Hopefully you here via the Debugging The Kernel page. If not, I'd highly recommend you start there, since it covers the basics of debugging the kernel.
This page describes the specifics of debugging a QNX Kernel that is running on a VMWare virtual machine. This page was originally written to be windows specific; where the steps between windows and linux differ, a section has been added for the linux instructions.
VMWare Workstation (the latest version is 6.0) is a useful tool in that it allows you to run one or more virtual x86 cpus on your development system. As far as the OS is concerned, it's an independant x86 machine, complete with BIOS, serial, disk and network devices.
Our standard qnxbasedma.build and bios.build will work with vmware.
Given that your kernel source environment is not on the vm, you need some way of getting the vm to boot your kernel. VMWare supports PXE netbooting, but probably the simplest method to load a fresh image is to generate a boot floppy image. You can do this with the mkifs filter, mkifsf_vmware.
To generate a floppy boot image with this filter, simple replace
[virtual=x86,bios +compress]
with
[virtual=x86,vmware +compress]
NB: If you see it complaining about not being able to find a watcom ld when you build your image, then you need to need to update to QNX 6.3.2 or later...
VMWare can redirect it's serial port to a named pipe. gdb doesn't support connected to windows named pipes (why not?!!) but there is a useful util called vmwaregateway.exe that is over on the L4Ka site. You can get it here:
http://l4ka.org/tools/vmwaregateway.php
Install it, register it as a services and start the service.
vmwaregateway.exe /rfollowed by
net start vmwaregateway
Go to the add hardware wizard, select a serial port and then redirect the output to the named pipe \\.\pipe\vmwaredebug
In the settings you should specify "This end is the client" "The other end is an application" and "Yield CPU on poll"
/tmp/vmwaredebug0 /tmp/vmwaredebug1In the settings for our second serial port, we need to specify that "This end is the server" and "The other end is an application", as well as selecting the option to "Yield CPU on poll"
Go the add hardware wizard, select floppy drive and specify the boot image you created in your build file
startup-bios -D 8250.2f8.38400 -K 8250.3f8.38400 -v gdb_kdebug -K -D1
Your debug port is now going out the second serial port, even though you probably don't have one setup. You need this though since otherwise when you 'ungrab' the input from vmware with ctrl-alt, you will abort the kernel debugger!
Start the vmware session. If you specified -K to gdb_kdebug then it should hang waiting for gdb to connect.
kdebug uses the gdb remote protocol. So load the procnto-instr.sym into gdb, and then connect to localhost port 567
(gdb) file procnto-instr.sym (gdb) target remote localhost:567You should now be live! Set a breakpoint and hit the slopes, dude!
socat -d -d /tmp/vmwaredebug1 TCP4-LISTEN:65333We use the second serial port because it is the one we are running the kernel debugger on. If you want to connect your first serial port, use /tmp/vmwaredebug0 instead of /tmp/vmwaredebug1, and use a different port for TCP4-LISTEN for each serial port you have connected. now we can connect to kdebug using the remote protocol:
(gdb) file procnto-instr.sym (gdb) target remote localhost:65333
... and now, just like windows, you're running live!
A couple of gotchas