They addresses are virtual, but the ppc kernel has 1-1 mappings setup,
so they mirror the physical.

When you get a dump, add a [+keeplinked] attribute to your procnto-400 - mkifs will then
leave a procnto-400.sym binary, which you can the load into ntoppc-gdb and examine the memory
addresses.

Colin

Robert D'Attilio wrote:
> Thanks guys.
> 
> Any idea if the memory addresses being referred to are physical
> addresses or virtual?
> 
> -----Original Message-----
> From: Joel Pilon [mailto:community-noreply@qnx.com] 
> Sent: Thursday, April 30, 2009 4:19 PM
> To: ostech-core_os
> Subject: Re: Looking for docs on decoding kernel panic dump
> 
> This is a good place to start:
> 
> http://www.qnx.com/developers/docs/6.4.0/neutrino/technotes/proc_dump.ht
> ml
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post28480
> 
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post28481
> 

-- 
cburgess@qnx.com

They are virtual addresses.

Addresses in the kernel space will match symbols in your procnto.sym
file (generated if you [+keeplinked] in your build file) so you can look
around with gdb.  Looks like there's lots of those in the information
you've got -- on PPC, kernel addresses are in the first 256M.

Addresses outside the kernel space are more difficult -- you need to
know what address space they belong to and use gdb on that program.
Probably not an issue here.

Addresses in kernel interrupt code are most difficult, as you don't have
any symbolic information to figure them out...

In this case, it looks like a procnto thread was running, servicing a
request made by proc/boot/ps (from the PID and ASPACE PID lines).  The
S/C/F codes indicate a segfault.  The state (d0 = now lock exit) are an
artifact of a fault in a procnto thread.

Aside from that I can't contribute much without digging into the .sym
file.

> -----Original Message-----
> From: Robert D'Attilio [mailto:community-noreply@qnx.com] 
> Sent: Thursday, April 30, 2009 4:27 PM
> To: ostech-core_os
> Subject: RE: Looking for docs on decoding kernel panic dump
> 
> Thanks guys.
> 
> Any idea if the memory addresses being referred to are 
> physical addresses or virtual?
> 
> -----Original Message-----
> From: Joel Pilon [mailto:community-noreply@qnx.com]
> Sent: Thursday, April 30, 2009 4:19 PM
> To: ostech-core_os
> Subject: Re: Looking for docs on decoding kernel panic dump
> 
> This is a good place to start:
> 
> http://www.qnx.com/developers/docs/6.4.0/neutrino/technotes/pr
> oc_dump.ht
> ml
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post28480
> 
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post28481
> 
> 

Robert D'Attilio wrote:
> Colin:
> 
> Many thanks for your analysis!
> 
> Some background: I am investigating what I think is a hardware problem
> (bad RAM?) on a new product in development. This particular card will
> periodically have different processes core with either a signal 4,
> illegal instruction error or a signal 11, segmentation fault along with
> a corrupted stack. I've also been able to capture some of these kernel
> panics now that I've re-enabled the serial port and disabled the
> watchdog. The kernel panics are not always in the same place either. So
> far the problem is isolated to this one card out of about 12 that have
> been undergoing extensive verification testing for a couple of months. 
> 
> I am hoping to use the kernel dumps to point to a region of RAM that
> should be probed with some sort of RAM test. So far, the basic data bus
> walking ones/zeros test and address lines tests haven't turned up
> anything.

Well the first thing I would check is the power.  I've seen plenty of boards
where the power was just barely in the spec'd margins, and fluctuations in the
supply could lead to memory corruption.  We spent a long time chasing one particular
corruption only to find out that it was as simple as that.  And yes, because
it was marginal it varied board by board.  It often would only fail under  heavy load
too (with peripherals sucking the power I guess).

> A couple of questions though about your analysis as working at this
> level (and with QNX) is still pretty new for me and interesting:
> 
> 1) How do you know that 0x7c005a14 is an invalid address?

The virtual address space is divided by kernel and user spaces.  The kernel lives (on PPC)
from 0-1G, usermode addresses are higher.  Hence a usermode address for a procnto variable
is not normal.

Note that it is normal for the kernel to access usermode vaddrs, but they must be verified
first, and in this case it was a kernel allocated variable.

> 2) The stack dump, is that the kernel's stack or the application's?

It's always the kernel stack.

> 3) In a subsequent posting, you did the following:
> 
> ntoppc-ld -T $QNX_TARGET/ppcbe/lib/nto.link -Ttext 0x46000 -o
> procnto-400.sym $QNX_TARGET/ppcbe/boot/sys/procnto-400
> 
> What exactly were you trying to do here? Was this how YOU were able to
> determine that the failure was in strcpy() - the above relinks the
> kernel to my relocation address and then you used the .sym to reverse
> engineer the instruction address?

The procnto-400 binary we ship is actually an object file.  When you run mkifs (try it with -vv) it
actually runs a linker to relocate the startup and the procnto to the appropriate addresses for your
image.

The [+keeplinked] attribute tells mkifs to leave the relocated symbol file around.  It's very useful for
this sort of debugging.

Once I had the relocated symbols I could simply load the .sym file into gdb and check the instruction where
the kernel faulted.

The register context (described in usr/include/ppc/context.h) gave me the registers.

> Sorry for all the questions...but I thought it was pretty cool that you
> could pull out that kind of info from what looks like nonsense :)

It does appear to be a bit of black magic from the outside, doesn't it?  And I assure you, it IS.  Now, I've
got to go drain the blood of a mouse.. :-)

Colin

> robert
> 
> 
> -----Original Message-----
> From: Colin Burgess [mailto:community-noreply@qnx.com] 
> Sent: Thursday, April 30, 2009 4:55 PM
> To: ostech-core_os
> Subject: Re: Looking for docs on decoding kernel panic dump
> 
> It appears to be crashing trying to strcpy a process' debug_name
> variable.
> 
> If you look at the instruction at 0x9840c, it's in strcpy, and it's
> doing
> 
> lbz r9,0(r4)
>...