Mario Charest
10/29/2009 3:17 PM
post41010
|
A system in the field stopped responding. nicinfo would stay send blocked forever, same for ifconfig.
12297 1 roc/boot/io-pkt-v4 21r SIGWAITINFO 484K 19M 4096(516K)*
12297 2 roc/boot/io-pkt-v4 100r MUTEX 484K 19M 8192(132K)
12297 3 roc/boot/io-pkt-v4 100r RECEIVE 484K 19M 4096(132K)
12297 4 roc/boot/io-pkt-v4 21r RECEIVE 484K 19M 4096(132K)
12297 5 roc/boot/io-pkt-v4 11r MUTEX 484K 19M 4096(132K)
12297 6 roc/boot/io-pkt-v4 21r CONDVAR 484K 19M 4096(132K)
12297 7 roc/boot/io-pkt-v4 100r MUTEX 484K 19M 4096(132K)
12297 8 roc/boot/io-pkt-v4 11r RECEIVE 484K 19M 4096(132K)
libc.so.3 @b0300000 468K 12K
devnp-e1000.so @b8200000 180K 4096
lsm-qnet.so @b822e000 172K 40K
/dev/mem @40100000 (fdd80000) 128K
/dev/mem @40120000 (fdde0000) 128K
A 3 seconde trace showed no interrupt at all. Only thread is active.
Any pointers?
|
|
|
Sean Boudreau(deleted)
|
Re: io-pkt-v4 not responding
|
Sean Boudreau(deleted)
10/29/2009 3:19 PM
post41011
|
Re: io-pkt-v4 not responding
On Thu, Oct 29, 2009 at 03:17:54PM -0400, Mario Charest wrote:
> A system in the field stopped responding. nicinfo would stay send blocked forever, same for ifconfig.
>
> 12297 1 roc/boot/io-pkt-v4 21r SIGWAITINFO 484K 19M 4096(516K)*
> 12297 2 roc/boot/io-pkt-v4 100r MUTEX 484K 19M 8192(132K)
> 12297 3 roc/boot/io-pkt-v4 100r RECEIVE 484K 19M 4096(132K)
> 12297 4 roc/boot/io-pkt-v4 21r RECEIVE 484K 19M 4096(132K)
> 12297 5 roc/boot/io-pkt-v4 11r MUTEX 484K 19M 4096(132K)
> 12297 6 roc/boot/io-pkt-v4 21r CONDVAR 484K 19M 4096(132K)
> 12297 7 roc/boot/io-pkt-v4 100r MUTEX 484K 19M 4096(132K)
> 12297 8 roc/boot/io-pkt-v4 11r RECEIVE 484K 19M 4096(132K)
> libc.so.3 @b0300000 468K 12K
> devnp-e1000.so @b8200000 180K 4096
> lsm-qnet.so @b822e000 172K 40K
> /dev/mem @40100000 (fdd80000) 128K
> /dev/mem @40120000 (fdde0000) 128K
>
> A 3 seconde trace showed no interrupt at all. Only thread is active.
>
What does a straight 'pidin' show?
-seanb
|
|
|
Mario Charest
|
RE: io-pkt-v4 not responding
|
Mario Charest
10/29/2009 3:25 PM
post41014
|
RE: io-pkt-v4 not responding
> -----Original Message-----
> From: Sean Boudreau [mailto:community-noreply@qnx.com]
> Sent: Thursday, October 29, 2009 3:20 PM
> To: general-networking
> Subject: Re: io-pkt-v4 not responding
>
> On Thu, Oct 29, 2009 at 03:17:54PM -0400, Mario Charest wrote:
> > A system in the field stopped responding. nicinfo would stay send
> blocked forever, same for ifconfig.
> >
> > 12297 1 roc/boot/io-pkt-v4 21r SIGWAITINFO 484K 19M
> 4096(516K)*
> > 12297 2 roc/boot/io-pkt-v4 100r MUTEX 484K 19M
> 8192(132K)
> > 12297 3 roc/boot/io-pkt-v4 100r RECEIVE 484K 19M
> 4096(132K)
> > 12297 4 roc/boot/io-pkt-v4 21r RECEIVE 484K 19M
> 4096(132K)
> > 12297 5 roc/boot/io-pkt-v4 11r MUTEX 484K 19M
> 4096(132K)
> > 12297 6 roc/boot/io-pkt-v4 21r CONDVAR 484K 19M
> 4096(132K)
> > 12297 7 roc/boot/io-pkt-v4 100r MUTEX 484K 19M
> 4096(132K)
> > 12297 8 roc/boot/io-pkt-v4 11r RECEIVE 484K 19M
> 4096(132K)
> > libc.so.3 @b0300000 468K 12K
> > devnp-e1000.so @b8200000 180K 4096
> > lsm-qnet.so @b822e000 172K 40K
> > /dev/mem @40100000 (fdd80000) 128K
> > /dev/mem @40120000 (fdde0000) 128K
> >
> > A 3 seconde trace showed no interrupt at all. Only thread is active.
> >
>
> What does a straight 'pidin' show?
I`ve asked for that but the guy in the field didn`t do it, and he rebooted the machine. I`m told it did froze twice
today. I'll make sure to get that info next time.
>
> -seanb
>
>
>
> _______________________________________________
>
> General
> http://community.qnx.com/sf/go/post41011
>
|
|
|
Mario Charest
|
RE: io-pkt-v4 not responding
|
Mario Charest
10/29/2009 3:26 PM
post41015
|
RE: io-pkt-v4 not responding
I guess forcing a core dump is useless for you guys?
> -----Original Message-----
> From: Sean Boudreau [mailto:community-noreply@qnx.com]
> Sent: Thursday, October 29, 2009 3:20 PM
> To: general-networking
> Subject: Re: io-pkt-v4 not responding
>
> On Thu, Oct 29, 2009 at 03:17:54PM -0400, Mario Charest wrote:
> > A system in the field stopped responding. nicinfo would stay send
> blocked forever, same for ifconfig.
> >
> > 12297 1 roc/boot/io-pkt-v4 21r SIGWAITINFO 484K 19M
> 4096(516K)*
> > 12297 2 roc/boot/io-pkt-v4 100r MUTEX 484K 19M
> 8192(132K)
> > 12297 3 roc/boot/io-pkt-v4 100r RECEIVE 484K 19M
> 4096(132K)
> > 12297 4 roc/boot/io-pkt-v4 21r RECEIVE 484K 19M
> 4096(132K)
> > 12297 5 roc/boot/io-pkt-v4 11r MUTEX 484K 19M
> 4096(132K)
> > 12297 6 roc/boot/io-pkt-v4 21r CONDVAR 484K 19M
> 4096(132K)
> > 12297 7 roc/boot/io-pkt-v4 100r MUTEX 484K 19M
> 4096(132K)
> > 12297 8 roc/boot/io-pkt-v4 11r RECEIVE 484K 19M
> 4096(132K)
> > libc.so.3 @b0300000 468K 12K
> > devnp-e1000.so @b8200000 180K 4096
> > lsm-qnet.so @b822e000 172K 40K
> > /dev/mem @40100000 (fdd80000) 128K
> > /dev/mem @40120000 (fdde0000) 128K
> >
> > A 3 seconde trace showed no interrupt at all. Only thread is active.
> >
>
> What does a straight 'pidin' show?
>
> -seanb
>
>
>
> _______________________________________________
>
> General
> http://community.qnx.com/sf/go/post41011
>
|
|
|
Sean Boudreau(deleted)
|
Re: io-pkt-v4 not responding
|
Sean Boudreau(deleted)
10/29/2009 3:27 PM
post41016
|
Re: io-pkt-v4 not responding
On Thu, Oct 29, 2009 at 03:26:10PM -0400, Mario Charest wrote:
> I guess forcing a core dump is useless for you guys?
Not necessarily. You can tar it up along with the io-pkt-v4,
devnp-e1000.so, lsm-qnet.so and libc.
Regards,
-seanb
|
|
|
Mario Charest
|
Re: io-pkt-v4 not responding
|
Mario Charest
11/02/2009 4:40 PM
post41187
|
Re: io-pkt-v4 not responding
> On Thu, Oct 29, 2009 at 03:26:10PM -0400, Mario Charest wrote:
> > I guess forcing a core dump is useless for you guys?
>
> Not necessarily. You can tar it up along with the io-pkt-v4,
> devnp-e1000.so, lsm-qnet.so and libc.
Here is it, the file contains also output of pidin (process-trenton). This is happening in-house now on a totally
different hardware, crashes a few time a day. This has start to happened after the machine was switched from 6.3.2 to 6
.4.1.
>
> Regards,
>
> -seanb
|
|
|
Mario Charest
|
Re: io-pkt-v4 not responding
|
Mario Charest
11/09/2009 10:44 AM
post41619
|
Re: io-pkt-v4 not responding
> > On Thu, Oct 29, 2009 at 03:26:10PM -0400, Mario Charest wrote:
> > > I guess forcing a core dump is useless for you guys?
> >
> > Not necessarily. You can tar it up along with the io-pkt-v4,
> > devnp-e1000.so, lsm-qnet.so and libc.
>
> Here is it, the file contains also output of pidin (process-trenton). This is
> happening in-house now on a totally different hardware, crashes a few time a
> day. This has start to happened after the machine was switched from 6.3.2 to
> 6.4.1.
>
Should I open a ticket?
>
> >
> > Regards,
> >
> > -seanb
>
>
|
|
|
Andrew Boyd(deleted)
|
RE: io-pkt-v4 not responding
|
Andrew Boyd(deleted)
11/09/2009 10:57 AM
post41622
|
RE: io-pkt-v4 not responding
Probably a good idea
--
aboyd
|
|
|
Sean Boudreau(deleted)
|
Re: io-pkt-v4 not responding
|
Sean Boudreau(deleted)
11/09/2009 12:20 PM
post41628
|
Re: io-pkt-v4 not responding
On Mon, Nov 02, 2009 at 04:41:37PM -0500, Mario Charest wrote:
> > On Thu, Oct 29, 2009 at 03:26:10PM -0400, Mario Charest wrote:
> > > I guess forcing a core dump is useless for you guys?
> >
> > Not necessarily. You can tar it up along with the io-pkt-v4,
> > devnp-e1000.so, lsm-qnet.so and libc.
>
> Here is it, the file contains also output of pidin (process-trenton). This is happening in-house now on a totally
different hardware, crashes a few time a day. This has start to happened after the machine was switched from 6.3.2 to 6
.4.1.
The .so files in the archive appear to be stripped.
The io-pkt-v4 is the 6.4.1 one so using the .so files
from 6.4.1 instead of the ones you sent I get this.
The only one I'm wondering about is the e1000 driver.
Is that the shipped 6.4.1 one?
The following doesn't look too out of the ordinary unless
the driver is spinning in its tx routine somehow...
-seanb
(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0xb0300000 0xb037924c Yes libc.so.3
0xb8200000 0xb822dcd8 Yes
/usr/qnx641/target/qnx6/x86/lib/dll/devnp-e1000.so
0xb822e000 0xb8236598 Yes
/usr/qnx641/target/qnx6/x86/lib/dll/devnp-shim.so
0xb8237000 0xb8242764 Yes
/usr/qnx641/target/qnx6/x86/lib/dll/devn-speedo.so
0xb8243000 0xb824dd74 Yes
/usr/qnx641/target/qnx6/x86/lib/dll/devn-pcnet.so
0xb824e000 0xb8282898 Yes
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
(gdb) bt
#0 0xb8226638 in i82544_start () at
/home/builder/hudson/641/daily/nto/usr/include/sys/pool.h:381
#1 0x0805fddc in ifq_enqueue_wtp (ifp=0x810b180, m=0x938d100,
wtp=0x80e4680)
at /home/builder/hudson/641/svn/lib/io-pkt/sys/net/if.c:2450
#2 0x0806342f in ether_output (ifp0=0x810b180, m0=0x938d100,
dst=0x7fc6a80, rt0=0x0)
at
/home/builder/hudson/641/svn/lib/io-pkt/sys/net/if_ethersubr.c:615
#3 0xb825ae63 in en_iopkt_tx_pkt () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#4 0xb826174a in l4_tx_pkt () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#5 0xb8261996 in l4_tx_timeout () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#6 0xb82627f2 in l4_tx_service () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#7 0xb826290d in l4_tx () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#8 0xb826f4f2 in tx_qos_pkt () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#9 0xb826c5d7 in qos_rx () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#10 0xb826099f in l4_rx () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#11 0xb825b62f in qnet_en_input () from
/usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
#12 0x08062ef2 in ether_input (ifp=0x810b180, m=0x8674800)
at
/home/builder/hudson/641/svn/lib/io-pkt/sys/net/if_ethersubr.c:1077
#13 0xb8222d4a in i82544_filter (i82544=0x810b180)
at
/home/builder/hudson/641/svn/lib/io-pkt/sys/dev_qnx/e1000/i82544.c:391
#14 0x08077cb5 in process_interrupts_noswitch (wtp=0x80e4680)
at /home/builder/hudson/641/svn/lib/io-pkt/sys/interrupt.c:385
#15 0x080913a1 in receive_loop_multi (arg=0x80c808c) at
/home/builder/hudson/641/svn/lib/io-pkt/sys/receive.c:546
#16 0x080881bf in thread_init (arg=0x80c808c) at
/home/builder/hudson/641/svn/lib/io-pkt/sys/nw_thread.c:765
#17 0xb031fa20 in ?? () from libc.so.3
(gdb)
|
|
|
Mario Charest
|
RE: io-pkt-v4 not responding
|
Mario Charest
11/09/2009 12:56 PM
post41631
|
RE: io-pkt-v4 not responding
> -----Original Message-----
> From: Sean Boudreau [mailto:community-noreply@qnx.com]
> Sent: Monday, November 09, 2009 12:21 PM
> To: general-networking
> Subject: Re: io-pkt-v4 not responding
>
> On Mon, Nov 02, 2009 at 04:41:37PM -0500, Mario Charest wrote:
> > > On Thu, Oct 29, 2009 at 03:26:10PM -0400, Mario Charest wrote:
> > > > I guess forcing a core dump is useless for you guys?
> > >
> > > Not necessarily. You can tar it up along with the io-pkt-v4,
> > > devnp-e1000.so, lsm-qnet.so and libc.
> >
> > Here is it, the file contains also output of pidin (process-trenton).
> This is happening in-house now on a totally different hardware, crashes
> a few time a day. This has start to happened after the machine was
> switched from 6.3.2 to 6.4.1.
>
> The .so files in the archive appear to be stripped.
> The io-pkt-v4 is the 6.4.1 one so using the .so files
> from 6.4.1 instead of the ones you sent I get this.
> The only one I'm wondering about is the e1000 driver.
> Is that the shipped 6.4.1 one?
Yes, check it out with "use -i".
>
> The following doesn't look too out of the ordinary unless
> the driver is spinning in its tx routine somehow...
Would that make io-pkt-v4 unresponsive to external command such as ifconfig and nicinfo?
I opened a ticket TicketID90401
>
> -seanb
>
> (gdb) info sharedlibrary
> From To Syms Read Shared Object Library
> 0xb0300000 0xb037924c Yes libc.so.3
> 0xb8200000 0xb822dcd8 Yes
> /usr/qnx641/target/qnx6/x86/lib/dll/devnp-e1000.so
> 0xb822e000 0xb8236598 Yes
> /usr/qnx641/target/qnx6/x86/lib/dll/devnp-shim.so
> 0xb8237000 0xb8242764 Yes
> /usr/qnx641/target/qnx6/x86/lib/dll/devn-speedo.so
> 0xb8243000 0xb824dd74 Yes
> /usr/qnx641/target/qnx6/x86/lib/dll/devn-pcnet.so
> 0xb824e000 0xb8282898 Yes
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> (gdb) bt
> #0 0xb8226638 in i82544_start () at
> /home/builder/hudson/641/daily/nto/usr/include/sys/pool.h:381
> #1 0x0805fddc in ifq_enqueue_wtp (ifp=0x810b180, m=0x938d100,
> wtp=0x80e4680)
> at /home/builder/hudson/641/svn/lib/io-pkt/sys/net/if.c:2450
> #2 0x0806342f in ether_output (ifp0=0x810b180, m0=0x938d100,
> dst=0x7fc6a80, rt0=0x0)
> at
> /home/builder/hudson/641/svn/lib/io-pkt/sys/net/if_ethersubr.c:615
> #3 0xb825ae63 in en_iopkt_tx_pkt () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #4 0xb826174a in l4_tx_pkt () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #5 0xb8261996 in l4_tx_timeout () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #6 0xb82627f2 in l4_tx_service () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #7 0xb826290d in l4_tx () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #8 0xb826f4f2 in tx_qos_pkt () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #9 0xb826c5d7 in qos_rx () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #10 0xb826099f in l4_rx () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #11 0xb825b62f in qnet_en_input () from
> /usr/qnx641/target/qnx6/x86/lib/dll/lsm-qnet.so
> #12 0x08062ef2 in ether_input (ifp=0x810b180, m=0x8674800)
> at
> /home/builder/hudson/641/svn/lib/io-pkt/sys/net/if_ethersubr.c:1077
> #13 0xb8222d4a in i82544_filter (i82544=0x810b180)
> at
> /home/builder/hudson/641/svn/lib/io-pkt/sys/dev_qnx/e1000/i82544.c:391
> #14 0x08077cb5 in process_interrupts_noswitch (wtp=0x80e4680)
> at /home/builder/hudson/641/svn/lib/io-pkt/sys/interrupt.c:385
> #15 0x080913a1 in receive_loop_multi (arg=0x80c808c) at
> /home/builder/hudson/641/svn/lib/io-pkt/sys/receive.c:546
> #16...
|
|
|
Sean Boudreau(deleted)
|
Re: io-pkt-v4 not responding
|
Sean Boudreau(deleted)
11/10/2009 10:43 AM
post41718
|
Re: io-pkt-v4 not responding
On Mon, Nov 09, 2009 at 12:56:15PM -0500, Mario Charest wrote:
>
> >
> > The following doesn't look too out of the ordinary unless
> > the driver is spinning in its tx routine somehow...
>
> Would that make io-pkt-v4 unresponsive to external command such as ifconfig and nicinfo?
>
> I opened a ticket TicketID90401
>
It has a mutex in its tx routine that other threads
are blocked on trying to acquire so yes.
-seanb
|
|
|
Mario Charest
|
RE: io-pkt-v4 not responding
|
Mario Charest
11/10/2009 10:46 AM
post41720
|
RE: io-pkt-v4 not responding
> -----Original Message-----
> From: Sean Boudreau [mailto:community-noreply@qnx.com]
> Sent: Tuesday, November 10, 2009 10:43 AM
> To: general-networking
> Subject: Re: io-pkt-v4 not responding
>
> On Mon, Nov 09, 2009 at 12:56:15PM -0500, Mario Charest wrote:
> >
> > >
> > > The following doesn't look too out of the ordinary unless
> > > the driver is spinning in its tx routine somehow...
> >
> > Would that make io-pkt-v4 unresponsive to external command such as
> ifconfig and nicinfo?
> >
> > I opened a ticket TicketID90401
> >
>
> It has a mutex in its tx routine that other threads
> are blocked on trying to acquire so yes.
Ok so are you saying this is a hardware problem? We are using the same image, but it shows on totally different
hardware, so I doubt it's a hardware problem and 6.3.2 has been running fine. Problem showed up the day we switch to 6.
4.1.
>
> -seanb
>
>
>
> _______________________________________________
>
> General
> http://community.qnx.com/sf/go/post41718
>
|
|
|
|