Project Home
Project Home
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - io-pkt-v4 not responding: (12 Items)
   
io-pkt-v4 not responding  
A system in the field stopped responding. nicinfo would stay send blocked forever, same for ifconfig.

12297   1 roc/boot/io-pkt-v4  21r SIGWAITINFO      484K   19M  4096(516K)*
   12297   2 roc/boot/io-pkt-v4 100r MUTEX            484K   19M  8192(132K) 
   12297   3 roc/boot/io-pkt-v4 100r RECEIVE          484K   19M  4096(132K) 
   12297   4 roc/boot/io-pkt-v4  21r RECEIVE          484K   19M  4096(132K) 
   12297   5 roc/boot/io-pkt-v4  11r MUTEX            484K   19M  4096(132K) 
   12297   6 roc/boot/io-pkt-v4  21r CONDVAR          484K   19M  4096(132K) 
   12297   7 roc/boot/io-pkt-v4 100r MUTEX            484K   19M  4096(132K) 
   12297   8 roc/boot/io-pkt-v4  11r RECEIVE          484K   19M  4096(132K) 
            libc.so.3          @b0300000             468K   12K
            devnp-e1000.so     @b8200000             180K  4096
            lsm-qnet.so        @b822e000             172K   40K
            /dev/mem           @40100000 (fdd80000)        128K
            /dev/mem           @40120000 (fdde0000)        128K

A 3 seconde trace showed no interrupt at all.  Only thread is active.

Any pointers?
Re: io-pkt-v4 not responding  
On Thu, Oct 29, 2009 at 03:17:54PM -0400, Mario Charest wrote:
> A system in the field stopped responding. nicinfo would stay send blocked forever, same for ifconfig.
> 
> 12297   1 roc/boot/io-pkt-v4  21r SIGWAITINFO      484K   19M  4096(516K)*
>    12297   2 roc/boot/io-pkt-v4 100r MUTEX            484K   19M  8192(132K) 
>    12297   3 roc/boot/io-pkt-v4 100r RECEIVE          484K   19M  4096(132K) 
>    12297   4 roc/boot/io-pkt-v4  21r RECEIVE          484K   19M  4096(132K) 
>    12297   5 roc/boot/io-pkt-v4  11r MUTEX            484K   19M  4096(132K) 
>    12297   6 roc/boot/io-pkt-v4  21r CONDVAR          484K   19M  4096(132K) 
>    12297   7 roc/boot/io-pkt-v4 100r MUTEX            484K   19M  4096(132K) 
>    12297   8 roc/boot/io-pkt-v4  11r RECEIVE          484K   19M  4096(132K) 
>             libc.so.3          @b0300000             468K   12K
>             devnp-e1000.so     @b8200000             180K  4096
>             lsm-qnet.so        @b822e000             172K   40K
>             /dev/mem           @40100000 (fdd80000)        128K
>             /dev/mem           @40120000 (fdde0000)        128K
> 
> A 3 seconde trace showed no interrupt at all.  Only thread is active.
> 

What does a straight 'pidin' show?

-seanb
RE: io-pkt-v4 not responding  

> -----Original Message-----
> From: Sean Boudreau [mailto:community-noreply@qnx.com]
> Sent: Thursday, October 29, 2009 3:20 PM
> To: general-networking
> Subject: Re: io-pkt-v4 not responding
> 
> On Thu, Oct 29, 2009 at 03:17:54PM -0400, Mario Charest wrote:
> > A system in the field stopped responding. nicinfo would stay send
> blocked forever, same for ifconfig.
> >
> > 12297   1 roc/boot/io-pkt-v4  21r SIGWAITINFO      484K   19M
> 4096(516K)*
> >    12297   2 roc/boot/io-pkt-v4 100r MUTEX            484K   19M
> 8192(132K)
> >    12297   3 roc/boot/io-pkt-v4 100r RECEIVE          484K   19M
> 4096(132K)
> >    12297   4 roc/boot/io-pkt-v4  21r RECEIVE          484K   19M
> 4096(132K)
> >    12297   5 roc/boot/io-pkt-v4  11r MUTEX            484K   19M
> 4096(132K)
> >    12297   6 roc/boot/io-pkt-v4  21r CONDVAR          484K   19M
> 4096(132K)
> >    12297   7 roc/boot/io-pkt-v4 100r MUTEX            484K   19M
> 4096(132K)
> >    12297   8 roc/boot/io-pkt-v4  11r RECEIVE          484K   19M
> 4096(132K)
> >             libc.so.3          @b0300000             468K   12K
> >             devnp-e1000.so     @b8200000             180K  4096
> >             lsm-qnet.so        @b822e000             172K   40K
> >             /dev/mem           @40100000 (fdd80000)        128K
> >             /dev/mem           @40120000 (fdde0000)        128K
> >
> > A 3 seconde trace showed no interrupt at all.  Only thread is active.
> >
> 
> What does a straight 'pidin' show?

I`ve asked for that but the guy in the field didn`t do it, and he rebooted the machine.  I`m told it did froze twice 
today.   I'll make sure to get that info next time.

> 
> -seanb
> 
> 
> 
> _______________________________________________
> 
> General
> http://community.qnx.com/sf/go/post41011
> 
RE: io-pkt-v4 not responding  
I guess forcing a core dump is useless for you guys?

> -----Original Message-----
> From: Sean Boudreau [mailto:community-noreply@qnx.com]
> Sent: Thursday, October 29, 2009 3:20 PM
> To: general-networking
> Subject: Re: io-pkt-v4 not responding
> 
> On Thu, Oct 29, 2009 at 03:17:54PM -0400, Mario Charest wrote:
> > A system in the field stopped responding. nicinfo would stay send
> blocked forever, same for ifconfig.
> >
> > 12297   1 roc/boot/io-pkt-v4  21r SIGWAITINFO      484K   19M
> 4096(516K)*
> >    12297   2 roc/boot/io-pkt-v4 100r MUTEX            484K   19M
> 8192(132K)
> >    12297   3 roc/boot/io-pkt-v4 100r RECEIVE          484K   19M
> 4096(132K)
> >    12297   4 roc/boot/io-pkt-v4  21r RECEIVE          484K   19M
> 4096(132K)
> >    12297   5 roc/boot/io-pkt-v4  11r MUTEX            484K   19M
> 4096(132K)
> >    12297   6 roc/boot/io-pkt-v4  21r CONDVAR          484K   19M
> 4096(132K)
> >    12297   7 roc/boot/io-pkt-v4 100r MUTEX            484K   19M
> 4096(132K)
> >    12297   8 roc/boot/io-pkt-v4  11r RECEIVE          484K   19M
> 4096(132K)
> >             libc.so.3          @b0300000             468K   12K
> >             devnp-e1000.so     @b8200000             180K  4096
> >             lsm-qnet.so        @b822e000             172K   40K
> >             /dev/mem           @40100000 (fdd80000)        128K
> >             /dev/mem           @40120000 (fdde0000)        128K
> >
> > A 3 seconde trace showed no interrupt at all.  Only thread is active.
> >
> 
> What does a straight 'pidin' show?
> 
> -seanb
> 
> 
> 
> _______________________________________________
> 
> General
> http://community.qnx.com/sf/go/post41011
> 
Re: io-pkt-v4 not responding  
On Thu, Oct 29, 2009 at 03:26:10PM -0400, Mario Charest wrote:
> I guess forcing a core dump is useless for you guys?

Not necessarily.  You can tar it up along with the io-pkt-v4,
devnp-e1000.so, lsm-qnet.so and libc.

Regards,

-seanb
Re: io-pkt-v4 not responding  
> On Thu, Oct 29, 2009 at 03:26:10PM -0400, Mario Charest wrote:
> > I guess forcing a core dump is useless for you guys?
> 
> Not necessarily.  You can tar it up along with the io-pkt-v4,
> devnp-e1000.so, lsm-qnet.so and libc.

Here is it, the file contains also output of pidin (process-trenton). This is happening in-house now on a totally 
different hardware, crashes a few time a day.  This has start to happened after the machine was switched from 6.3.2 to 6
.4.1.


> 
> Regards,
> 
> -seanb


Attachment: Text network.7z 881.69 KB
Re: io-pkt-v4 not responding  
> > On Thu, Oct 29, 2009 at 03:26:10PM -0400, Mario Charest wrote:
> > > I guess forcing a core dump is useless for you guys?
> > 
> > Not necessarily.  You can tar it up along with the io-pkt-v4,
> > devnp-e1000.so, lsm-qnet.so and libc.
> 
> Here is it, the file contains also output of pidin (process-trenton). This is 
> happening in-house now on a totally different hardware, crashes a few time a 
> day.  This has start to happened after the machine was switched from 6.3.2 to 
> 6.4.1.
> 

Should I open a ticket?

> 
> > 
> > Regards,
> > 
> > -seanb
> 
> 


RE: io-pkt-v4 not responding  
Probably a good idea

--
aboyd
Re: io-pkt-v4 not responding  
On Mon, Nov 02, 2009 at 04:41:37PM -0500, Mario Charest wrote:
> > On Thu, Oct 29, 2009 at 03:26:10PM -0400, Mario Charest wrote:
> > > I guess forcing a core dump is useless for you guys?
> > 
> > Not necessarily.  You can tar it up along with the io-pkt-v4,
> > devnp-e1000.so, lsm-qnet.so and libc.
> 
> Here is it, the file contains also output of pidin (process-trenton). This is happening in-house now on a totally 
different hardware, crashes a few time a day.  This has start to happened after the machine was switched from 6.3.2 to 6
.4.1.

The .so files in the archive appear to be stripped.
The io-pkt-v4 is the 6.4.1 one so using the .so files
from 6.4.1 instead of the ones you sent I get this.
The only one I'm wondering about is the e1000 driver.
Is that the shipped 6.4.1 one?

The following doesn't look too out of the ordinary unless
the driver is spinning in its tx routine somehow...

-seanb

(gdb) info sharedlibrary
From        To          Syms Read   Shared Object Library
0xb0300000  0xb037924c  Yes         libc.so.3
0xb8200000  0xb822dcd8  Yes
/usr/qnx641/target/qnx6/x86/lib/dll/devnp-e1000.so
0xb822e000  0xb8236598  Yes
/usr/qnx641/target/qnx6/x86/lib/dll/devnp-shim.so
0xb8237000  0xb8242764  Yes
/usr/qnx641/target/qnx6/x86/lib/dll/devn-speedo.so
0xb8243000  0xb824dd74  Yes
/usr/qnx641/target/qnx6/x86/lib/dll/devn-pcnet.so
0xb824e000  0xb8282898  Yes
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
(gdb) bt
#0  0xb8226638 in i82544_start () at
/home/builder/hudson/641/daily/nto/usr/include/sys/pool.h:381
#1  0x0805fddc in ifq_enqueue_wtp (ifp=0x810b180, m=0x938d100,
wtp=0x80e4680)
    at /home/builder/hudson/641/svn/lib/io-pkt/sys/net/if.c:2450
#2  0x0806342f in ether_output (ifp0=0x810b180, m0=0x938d100,
dst=0x7fc6a80, rt0=0x0)
    at
/home/builder/hudson/641/svn/lib/io-pkt/sys/net/if_ethersubr.c:615
#3  0xb825ae63 in en_iopkt_tx_pkt () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#4  0xb826174a in l4_tx_pkt () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#5  0xb8261996 in l4_tx_timeout () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#6  0xb82627f2 in l4_tx_service () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#7  0xb826290d in l4_tx () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#8  0xb826f4f2 in tx_qos_pkt () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#9  0xb826c5d7 in qos_rx () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#10 0xb826099f in l4_rx () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#11 0xb825b62f in qnet_en_input () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#12 0x08062ef2 in ether_input (ifp=0x810b180, m=0x8674800)
    at
/home/builder/hudson/641/svn/lib/io-pkt/sys/net/if_ethersubr.c:1077
#13 0xb8222d4a in i82544_filter (i82544=0x810b180)
    at
/home/builder/hudson/641/svn/lib/io-pkt/sys/dev_qnx/e1000/i82544.c:391
#14 0x08077cb5 in process_interrupts_noswitch (wtp=0x80e4680)
    at /home/builder/hudson/641/svn/lib/io-pkt/sys/interrupt.c:385
#15 0x080913a1 in receive_loop_multi (arg=0x80c808c) at
/home/builder/hudson/641/svn/lib/io-pkt/sys/receive.c:546
#16 0x080881bf in thread_init (arg=0x80c808c) at
/home/builder/hudson/641/svn/lib/io-pkt/sys/nw_thread.c:765
#17 0xb031fa20 in ?? () from libc.so.3
(gdb)
RE: io-pkt-v4 not responding  
> -----Original Message-----
> From: Sean Boudreau [mailto:community-noreply@qnx.com]
> Sent: Monday, November 09, 2009 12:21 PM
> To: general-networking
> Subject: Re: io-pkt-v4 not responding
> 
> On Mon, Nov 02, 2009 at 04:41:37PM -0500, Mario Charest wrote:
> > > On Thu, Oct 29, 2009 at 03:26:10PM -0400, Mario Charest wrote:
> > > > I guess forcing a core dump is useless for you guys?
> > >
> > > Not necessarily.  You can tar it up along with the io-pkt-v4,
> > > devnp-e1000.so, lsm-qnet.so and libc.
> >
> > Here is it, the file contains also output of pidin (process-trenton).
> This is happening in-house now on a totally different hardware, crashes
> a few time a day.  This has start to happened after the machine was
> switched from 6.3.2 to 6.4.1.
> 
> The .so files in the archive appear to be stripped.
> The io-pkt-v4 is the 6.4.1 one so using the .so files
> from 6.4.1 instead of the ones you sent I get this.
> The only one I'm wondering about is the e1000 driver.
> Is that the shipped 6.4.1 one?

Yes, check it out with "use -i".

> 
> The following doesn't look too out of the ordinary unless
> the driver is spinning in its tx routine somehow...

Would that make io-pkt-v4 unresponsive to external command such as ifconfig and nicinfo?

I opened a ticket TicketID90401

> 
> -seanb
> 
> (gdb) info sharedlibrary
> From        To          Syms Read   Shared Object Library
> 0xb0300000  0xb037924c  Yes         libc.so.3
> 0xb8200000  0xb822dcd8  Yes
> /usr/qnx641/target/qnx6/x86/lib/dll/devnp-e1000.so
> 0xb822e000  0xb8236598  Yes
> /usr/qnx641/target/qnx6/x86/lib/dll/devnp-shim.so
> 0xb8237000  0xb8242764  Yes
> /usr/qnx641/target/qnx6/x86/lib/dll/devn-speedo.so
> 0xb8243000  0xb824dd74  Yes
> /usr/qnx641/target/qnx6/x86/lib/dll/devn-pcnet.so
> 0xb824e000  0xb8282898  Yes
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> (gdb) bt
> #0  0xb8226638 in i82544_start () at
> /home/builder/hudson/641/daily/nto/usr/include/sys/pool.h:381
> #1  0x0805fddc in ifq_enqueue_wtp (ifp=0x810b180, m=0x938d100,
> wtp=0x80e4680)
>     at /home/builder/hudson/641/svn/lib/io-pkt/sys/net/if.c:2450
> #2  0x0806342f in ether_output (ifp0=0x810b180, m0=0x938d100,
> dst=0x7fc6a80, rt0=0x0)
>     at
> /home/builder/hudson/641/svn/lib/io-pkt/sys/net/if_ethersubr.c:615
> #3  0xb825ae63 in en_iopkt_tx_pkt () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #4  0xb826174a in l4_tx_pkt () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #5  0xb8261996 in l4_tx_timeout () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #6  0xb82627f2 in l4_tx_service () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #7  0xb826290d in l4_tx () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #8  0xb826f4f2 in tx_qos_pkt () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #9  0xb826c5d7 in qos_rx () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #10 0xb826099f in l4_rx () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #11 0xb825b62f in qnet_en_input () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #12 0x08062ef2 in ether_input (ifp=0x810b180, m=0x8674800)
>     at
> /home/builder/hudson/641/svn/lib/io-pkt/sys/net/if_ethersubr.c:1077
> #13 0xb8222d4a in i82544_filter (i82544=0x810b180)
>     at
> /home/builder/hudson/641/svn/lib/io-pkt/sys/dev_qnx/e1000/i82544.c:391
> #14 0x08077cb5 in process_interrupts_noswitch (wtp=0x80e4680)
>     at /home/builder/hudson/641/svn/lib/io-pkt/sys/interrupt.c:385
> #15 0x080913a1 in receive_loop_multi (arg=0x80c808c) at
> /home/builder/hudson/641/svn/lib/io-pkt/sys/receive.c:546
> #16...
Re: io-pkt-v4 not responding  
On Mon, Nov 09, 2009 at 12:56:15PM -0500, Mario Charest wrote:
> 
> > 
> > The following doesn't look too out of the ordinary unless
> > the driver is spinning in its tx routine somehow...
> 
> Would that make io-pkt-v4 unresponsive to external command such as ifconfig and nicinfo?
> 
> I opened a ticket TicketID90401
> 

It has a mutex in its tx routine that other threads
are blocked on trying to acquire so yes.

-seanb
RE: io-pkt-v4 not responding  

> -----Original Message-----
> From: Sean Boudreau [mailto:community-noreply@qnx.com]
> Sent: Tuesday, November 10, 2009 10:43 AM
> To: general-networking
> Subject: Re: io-pkt-v4 not responding
> 
> On Mon, Nov 09, 2009 at 12:56:15PM -0500, Mario Charest wrote:
> >
> > >
> > > The following doesn't look too out of the ordinary unless
> > > the driver is spinning in its tx routine somehow...
> >
> > Would that make io-pkt-v4 unresponsive to external command such as
> ifconfig and nicinfo?
> >
> > I opened a ticket TicketID90401
> >
> 
> It has a mutex in its tx routine that other threads
> are blocked on trying to acquire so yes.

Ok so are you saying this is a hardware problem?  We are using the same image, but it shows on totally different 
hardware, so I doubt it's a hardware problem and 6.3.2 has been running fine.  Problem showed up the day we switch to 6.
4.1.
 
> 
> -seanb
> 
> 
> 
> _______________________________________________
> 
> General
> http://community.qnx.com/sf/go/post41718
>