Project Home
Project Home
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - io-pkt crash on MPC8548 with two interfaces and prioritized RX path: (15 Items)
   
io-pkt crash on MPC8548 with two interfaces and prioritized RX path  
We use io-pkt (Milestone 3) with QNX 6.3.2 Kernel on a MPC8548 board with two network interfaces. We run one stack 
instance for both interfaces because later a layer 2 bridge is needed (not operational for this testcase). On tsec1 we 
want to receive and send UDP frames with low latency and do layer 2 bridging to and from tsec0. Especially the latency 
should not be influenced by "normal" traffic on tsec0. Because the stack is single threaded internally, we use the BPF 
(pcap library) to capture and send the UDP frames on tsec1. In our testcase a PC sends UDP frame and our MPC8548 board 
(tsec1) loops it back with the attached test program. The PC sends the next UDP frame (128 bytes UDP data) immediately 
after receipt of the response.

In this setup, io-pkt either terminates after a few frames with blown stack handling 0 or hangs with high CPU load (only
 restricted by the Adaptive Partitioning). Increasing the stacksize didn't help.

The stack is started with the following options:

io-pkt-v4-hc -t4 -dmpc85xx mac=xxx,pci=0,phy=0 -ptcpip

mount -Tio-pkt -o mac=yyy,pci=1,phy=1,rx_delay=0,rx_frame=0, priority=185 /lib/dll/devnp-mpc85xx.so

ifconfig tsec0 up
ifconfig tsec1 192.168.44.2 up

I have made the following observations:

1.) It works, if the -t4 option is omitted. But then the interrupt received thread is running on default priority -> no 
solution because of jitter

2.) It works, if only tsec1 is configured and used. It is not sufficient to have no network cable connected to a 
configured tsec0 -> no solution - both interfaces are needed

3.) It works, if the UDP frames are received and sent via the socket interface (#define VIA_SOCKET in the test program) 
and not over pcap_loop() and pcap_inject() -> no solution because of jitter when the single threaded stack is occupied 
by traffic from tsec0


Am I doing something wrong or is this a bug in io-pkt, the pcap library or the network driver ?

Thanks,
    Guenther

Attachment: Compressed file test_bpf.zip 215.18 KB
RE: io-pkt crash on MPC8548 with two interfaces and prioritized RX path  
Can you make a tar file with your:

 io-pkt binary
 io-pkt core files
 .so files (mpc85xx, libc, etc - see pidin mem for io-pkt)

after a crash?  You will need to run dumper on the 8548,
then restart the stack (without the funky options) to
ftp the core file off the 8548.

io-pkt should not crash, or run ready.

--
aboyd


-----Original Message-----
From: Guenther Weiss [mailto:community-noreply@qnx.com] 
Sent: Thursday, August 07, 2008 9:04 AM
To: technology-networking
Subject: io-pkt crash on MPC8548 with two interfaces and prioritized RX
path

We use io-pkt (Milestone 3) with QNX 6.3.2 Kernel on a MPC8548 board
with two network interfaces. We run one stack instance for both
interfaces because later a layer 2 bridge is needed (not operational for
this testcase). On tsec1 we want to receive and send UDP frames with low
latency and do layer 2 bridging to and from tsec0. Especially the
latency should not be influenced by "normal" traffic on tsec0. Because
the stack is single threaded internally, we use the BPF (pcap library)
to capture and send the UDP frames on tsec1. In our testcase a PC sends
UDP frame and our MPC8548 board (tsec1) loops it back with the attached
test program. The PC sends the next UDP frame (128 bytes UDP data)
immediately after receipt of the response.

In this setup, io-pkt either terminates after a few frames with blown
stack handling 0 or hangs with high CPU load (only restricted by the
Adaptive Partitioning). Increasing the stacksize didn't help.

The stack is started with the following options:

io-pkt-v4-hc -t4 -dmpc85xx mac=xxx,pci=0,phy=0 -ptcpip

mount -Tio-pkt -o mac=yyy,pci=1,phy=1,rx_delay=0,rx_frame=0,
priority=185 /lib/dll/devnp-mpc85xx.so

ifconfig tsec0 up
ifconfig tsec1 192.168.44.2 up

I have made the following observations:

1.) It works, if the -t4 option is omitted. But then the interrupt
received thread is running on default priority -> no solution because of
jitter

2.) It works, if only tsec1 is configured and used. It is not sufficient
to have no network cable connected to a configured tsec0 -> no solution
- both interfaces are needed

3.) It works, if the UDP frames are received and sent via the socket
interface (#define VIA_SOCKET in the test program) and not over
pcap_loop() and pcap_inject() -> no solution because of jitter when the
single threaded stack is occupied by traffic from tsec0


Am I doing something wrong or is this a bug in io-pkt, the pcap library
or the network driver ?

Thanks,
    Guenther



_______________________________________________
Technology
http://community.qnx.com/sf/go/post11522
RE: io-pkt crash on MPC8548 with two interfaces and prioritized RX path  
Hi Guenther:
	It certainly sounds like a bug of some sort.  The blown stack
may be why the pcap stuff isn't working.

Lets start with the easy one first.  Can you try using a "-p tcpip
rx_pulse_prio=185" instead of the per driver priority option?  I believe
that the per driver "priority" option should work, but this should be an
alternative which is applied to all driver threads.

	Robert.

-----Original Message-----
From: Guenther Weiss [mailto:community-noreply@qnx.com] 
Sent: Thursday, August 07, 2008 9:04 AM
To: technology-networking
Subject: io-pkt crash on MPC8548 with two interfaces and prioritized RX
path

We use io-pkt (Milestone 3) with QNX 6.3.2 Kernel on a MPC8548 board
with two network interfaces. We run one stack instance for both
interfaces because later a layer 2 bridge is needed (not operational for
this testcase). On tsec1 we want to receive and send UDP frames with low
latency and do layer 2 bridging to and from tsec0. Especially the
latency should not be influenced by "normal" traffic on tsec0. Because
the stack is single threaded internally, we use the BPF (pcap library)
to capture and send the UDP frames on tsec1. In our testcase a PC sends
UDP frame and our MPC8548 board (tsec1) loops it back with the attached
test program. The PC sends the next UDP frame (128 bytes UDP data)
immediately after receipt of the response.

In this setup, io-pkt either terminates after a few frames with blown
stack handling 0 or hangs with high CPU load (only restricted by the
Adaptive Partitioning). Increasing the stacksize didn't help.

The stack is started with the following options:

io-pkt-v4-hc -t4 -dmpc85xx mac=xxx,pci=0,phy=0 -ptcpip

mount -Tio-pkt -o mac=yyy,pci=1,phy=1,rx_delay=0,rx_frame=0,
priority=185 /lib/dll/devnp-mpc85xx.so

ifconfig tsec0 up
ifconfig tsec1 192.168.44.2 up

I have made the following observations:

1.) It works, if the -t4 option is omitted. But then the interrupt
received thread is running on default priority -> no solution because of
jitter

2.) It works, if only tsec1 is configured and used. It is not sufficient
to have no network cable connected to a configured tsec0 -> no solution
- both interfaces are needed

3.) It works, if the UDP frames are received and sent via the socket
interface (#define VIA_SOCKET in the test program) and not over
pcap_loop() and pcap_inject() -> no solution because of jitter when the
single threaded stack is occupied by traffic from tsec0


Am I doing something wrong or is this a bug in io-pkt, the pcap library
or the network driver ?

Thanks,
    Guenther



_______________________________________________
Technology
http://community.qnx.com/sf/go/post11522
Re: RE: io-pkt crash on MPC8548 with two interfaces and prioritized RX path  
Hi Robert,

I tried the following settings:

io-pkt-v4-hc -t4 -dmpc85xx mac=xxx,pci=0,phy=0 -ptcpip rx_pulse_prio=185

mount -Tio-pkt -o mac=yyy,pci=1,phy=1,rx_delay=0, rx_frame=0 /lib/dll/devnp-mpc85xx.so

This doesn't make any difference. Still got "blown stack handling 0".

      Guenther

Remark: Andrew Boyd has already received the requested files but cannot load it into the debugger. I have sent him today
 a core file that was written with a newer dumper version.
RE: RE: io-pkt crash on MPC8548 with two interfaces and prioritized RX path  
Hi Guenther:
	This option was to see if we could get the priority of the
receive thread set correctly (did it work?).   We still need to put some
effort into figuring out why the stack is getting blown...
   Robert.


-----Original Message-----
From: Guenther Weiss [mailto:community-noreply@qnx.com] 
Sent: Friday, August 08, 2008 1:43 AM
To: technology-networking
Subject: Re: RE: io-pkt crash on MPC8548 with two interfaces and
prioritized RX path

Hi Robert,

I tried the following settings:

io-pkt-v4-hc -t4 -dmpc85xx mac=xxx,pci=0,phy=0 -ptcpip rx_pulse_prio=185

mount -Tio-pkt -o mac=yyy,pci=1,phy=1,rx_delay=0, rx_frame=0
/lib/dll/devnp-mpc85xx.so

This doesn't make any difference. Still got "blown stack handling 0".

      Guenther

Remark: Andrew Boyd has already received the requested files but cannot
load it into the debugger. I have sent him today a core file that was
written with a newer dumper version.

_______________________________________________
Technology
http://community.qnx.com/sf/go/post11557
RE: RE: io-pkt crash on MPC8548 with two interfaces and prioritized RX path  
Rob:  Guenther, Alek and I are going back and 
forth a couple times a day, off-line on this.  

Guenter was running an older version of dumper, 
and truncating his dumps with the -s option 
(created PR for Colin re: -m ) but we are still 
having trouble getting sensible core dumps.  

This morning, Guenther was asked to specify only 
the -d and -m options to dumper, as suggested by 
Alek, to try to figure out his two different kinds
of io-pkt core dumps.

--
aboyd 
RE: io-pkt crash on MPC8548 with two interfaces and prioritized RX path  
Hi Guenther:
	Something else to check out with regards to the blown stack.
Are you using the most recent source for the mpc85xx driver?

If you look at the checkins for detect.c, you'll see Rev 445 which was a
change to specifically address a stack issue.

	Robert.

-----Original Message-----
From: Guenther Weiss [mailto:community-noreply@qnx.com] 
Sent: Thursday, August 07, 2008 9:04 AM
To: technology-networking
Subject: io-pkt crash on MPC8548 with two interfaces and prioritized RX
path

We use io-pkt (Milestone 3) with QNX 6.3.2 Kernel on a MPC8548 board
with two network interfaces. We run one stack instance for both
interfaces because later a layer 2 bridge is needed (not operational for
this testcase). On tsec1 we want to receive and send UDP frames with low
latency and do layer 2 bridging to and from tsec0. Especially the
latency should not be influenced by "normal" traffic on tsec0. Because
the stack is single threaded internally, we use the BPF (pcap library)
to capture and send the UDP frames on tsec1. In our testcase a PC sends
UDP frame and our MPC8548 board (tsec1) loops it back with the attached
test program. The PC sends the next UDP frame (128 bytes UDP data)
immediately after receipt of the response.

In this setup, io-pkt either terminates after a few frames with blown
stack handling 0 or hangs with high CPU load (only restricted by the
Adaptive Partitioning). Increasing the stacksize didn't help.

The stack is started with the following options:

io-pkt-v4-hc -t4 -dmpc85xx mac=xxx,pci=0,phy=0 -ptcpip

mount -Tio-pkt -o mac=yyy,pci=1,phy=1,rx_delay=0,rx_frame=0,
priority=185 /lib/dll/devnp-mpc85xx.so

ifconfig tsec0 up
ifconfig tsec1 192.168.44.2 up

I have made the following observations:

1.) It works, if the -t4 option is omitted. But then the interrupt
received thread is running on default priority -> no solution because of
jitter

2.) It works, if only tsec1 is configured and used. It is not sufficient
to have no network cable connected to a configured tsec0 -> no solution
- both interfaces are needed

3.) It works, if the UDP frames are received and sent via the socket
interface (#define VIA_SOCKET in the test program) and not over
pcap_loop() and pcap_inject() -> no solution because of jitter when the
single threaded stack is occupied by traffic from tsec0


Am I doing something wrong or is this a bug in io-pkt, the pcap library
or the network driver ?

Thanks,
    Guenther



_______________________________________________
Technology
http://community.qnx.com/sf/go/post11522
Re: RE: io-pkt crash on MPC8548 with two interfaces and prioritized RX path  
Hi Robert,

Christopher - currently on holiday just a stone's throw away from you - took a  copy of your trunk at 18th July 2008. 
This is later than revision 445. I also tried your Milestone 3 build. It shows the same behavior.

Guenther
Re: io-pkt crash on MPC8548 with two interfaces and prioritized RX path  

I wasn't able to reproduce the blown stack issue either.
At any rate please try the latest stuff.

Regards,

-seanb
Re: io-pkt crash on MPC8548 with two interfaces and prioritized RX path  
Hi Sean,

I have tried your trunk r478. With my test scenario (test_bpf) neither the "blown stack" problem nor the "loop" problem 
does occur any longer. Thanks for your work !!

Now I want to go to my actual setup. I have configured tsec0 to 192.168.40.16/24 and tsec1 to 192.168.40.32/24 (same 
subnet). Additionally, I need a layer 2 bridge between these interfaces.


ifconfig tsec0 inet up
ifconfig tsec1 inet up
ifconfig bridge0 create
brconfig bridge0 add tsec0 add tsec1 up
ifconfig tsec0 inet 192.168.40.16/24
ifconfig tsec1 inet 192.168.40.32/24

My PC is connected to tsec0 and another device with IP 192.168.40.31
is connected to tsec1.

When I open the pcap interface on tsec1 and set BIOCIMMEDIATE for the correponding file descriptor (line 306 in the 
attached file), the whole QNX freezes.

In the console, the following error message appears:

/sbin/bpf-ethident:pcap_loop: BIOCSETIF: bridge0: No such device or address

Am I doing something wrong ?

Thanks,
       Guenther


 

Attachment: Text udpSess.c 41.25 KB
Re: io-pkt crash on MPC8548 with two interfaces and prioritized RX path  
The same subnet sounds strange.  Have you created some sort of loop?
I did a test here through tsec* and it worked (I'm posting this
through it).  I have


    PC
 ----------------            ------------------------------------------
|10.42.101.236/20| <------> |tsec0 <no addr>     tsec1 10.42.101.237/20|----> internet
 ----------------            ------------------------------------------

Where I have a bridge containing tsec0, tsec1.  I don't need
the addr on tsec1 but it works when present.

io-pkt-v4-hc -t4 -d /proc/boot/devnp-mpc85xx.so verbose,mac=0000AB010203,pci=0,phy=0
mount -Tio-pkt -o verbose,mac=0000AB010204,pci=1,phy=1,rx_delay=0,rx_frame=0,priority=50 /proc/boot/devnp-mpc85xx.so
ifconfig tsec0 inet up
ifconfig tsec1 inet up
ifconfig bridge0 create
brconfig bridge0 maxaddr 2000
brconfig bridge0 add tsec0 add tsec1 up
ifconfig tsec1 inet 10.42.101.237/20

-seanb

On Fri, Aug 22, 2008 at 10:56:10AM -0400, Guenther Weiss wrote:
> Hi Sean,
> 
> I have tried your trunk r478. With my test scenario (test_bpf) neither the "blown stack" problem nor the "loop" 
problem does occur any longer. Thanks for your work !!
> 
> Now I want to go to my actual setup. I have configured tsec0 to 192.168.40.16/24 and tsec1 to 192.168.40.32/24 (same 
subnet). Additionally, I need a layer 2 bridge between these interfaces.
> 
> 
> ifconfig tsec0 inet up
> ifconfig tsec1 inet up
> ifconfig bridge0 create
> brconfig bridge0 add tsec0 add tsec1 up
> ifconfig tsec0 inet 192.168.40.16/24
> ifconfig tsec1 inet 192.168.40.32/24
> 
> My PC is connected to tsec0 and another device with IP 192.168.40.31
> is connected to tsec1.
> 
> When I open the pcap interface on tsec1 and set BIOCIMMEDIATE for the correponding file descriptor (line 306 in the 
attached file), the whole QNX freezes.
> 
> In the console, the following error message appears:
> 
> /sbin/bpf-ethident:pcap_loop: BIOCSETIF: bridge0: No such device or address
> 
> Am I doing something wrong ?
> 
> Thanks,
>        Guenther
> 
> 
>  
> 
> 
> 
> _______________________________________________
> Technology
> http://community.qnx.com/sf/go/post12253

Content-Description: Sourceforge Attachment
> #include <pthread.h>
> #include <string.h>
> #include <sys/types.h>
> #include <sys/socket.h>
> #include <sys/select.h>
> #include <arpa/inet.h>
> #include <errno.h>
> #include <unistd.h>
> #include <signal.h>
> #include <pcap.h>
> #include <net/if_ether.h>
> #include <netinet/udp.h>
> #include <netinet/ip.h>
> #include <netinet/ip_icmp.h>
> #include <net/if_arp.h>
> #include <net/ethertypes.h>
> #include <netdb.h>
> #include <net/if_dl.h>
> #include <sys/ioctl.h>
> #include <stdlib.h>
> 
> #include "etashead/etas_typ.h"
> #include "etasThreadPrio.h"
> #include "udpErr.h"
> #include "udpBuffer.h"
> #include "ethIpUdp.h"
> #include "libetas.h"
> 
> #include "udpSess.h"
> 
> 
> #define UDP_SERVER_SESSION_ATTRIBUTES_START_KEY    0x63528472
> #define UDP_SERVER_SESSION_ATTRIBUTES_END_KEY      0x78452374
> 
> #define NETWORK_DEVICE_NAME               "tsec1"
> /** Snap capture length for pcap library , choose 1500 bytes (full Ethernet frame) */
> #define PCAP_SNAPLEN            (1500u)
> 
> 
> #define FILTER_STRING_LEN        64
> 
> #define ETH_IP_UDP_HEADER_SIZE 42
> 
> #define PROTO_ICMP             1
> 
> #define PROTO_TYPE_ARP  0x0800
> 
> 
> #define CONNECT_TIMEOUT_MS                     5000
> #define ARP_REPLY_TIMEOUT_MS                    5000
> 
> 
> 
> typedef struct
> {
> ...
View Full Message
Re: io-pkt crash on MPC8548 with two interfaces and prioritized RX path  
> 
> 
> I wasn't able to reproduce the blown stack issue either.
> At any rate please try the latest stuff.
> 
> Regards,
> 
> -seanb


I introduced a nasty cut and paste issue in
rev 476.  Please update to rev 491 or greater.

-seanb
Guenther Weiss/DE/ETAS ist außer Haus.  
Ich werde ab  25.08.2008 nicht im Büro sein. Ich kehre zurück am
06.09.2008.

Ich werde Ihre Nachrichten nach meiner Rückkehr beantworten.

Antwort: Re: io-pkt crash on MPC8548 with two interfaces and prioritized RX path ("allow zip file")  
javax.mail.internet.MimeMultipart@ba856c
Attachment: Compressed file trace.zip 194.31 KB
Re: Antwort: Re: io-pkt crash on MPC8548 with two interfaces and prioritized RX path ("allow zip file")  
On Tue, Sep 09, 2008 at 03:47:34PM +0200, Guenther.Weiss@etas.com wrote:
> Hi Sean,
> 
> with the current 6.4 networking r498 (together with Neutrino 6.4 
> prerelease) our actual application is running in principal. But there is 
> unexpected high jitter on the UDP path via BPF and pcap library.
> 

<snip>

> 
> 
> 
> The network stack is started with the following options:
> 
> io-pkt-v4-hc -t4 -dmpc85xx mac=xxxx,pci=0 -ptcpip
> 
> mount -Tio-pkt -o mac=yyyy,pci=1,phy=1,rx_delay=0,rx_frame=0,priority=185 
> /lib/dll/devnp-mpc85xx.so
> 
>     /sbin/ifconfig tsec0 inet up
>     /sbin/ifconfig tsec1 inet up
>     /sbin/ifconfig bridge0 create
>     /sbin/brconfig bridge0 add tsec0 add tsec1 up
> 
> Several IP addresses are configured on tsec0. The pcap library is working 
> on tsec1 (low latency receiption and sending of UDP frames). 
> 
> 
> Process 581661 -> io-pkt-v4-hc
> Process 905246 Thread 16 ->  Priority 185 blocking in pcap_loop for 
> incomming UDP frames
> 
> When tsec1 rx interrupt (0x14)  occurs, the io-pkt-v4-hc / thread4 (prio 
> 185) is running immediately. But then this thread is receive blocked and 
> the io-pkt-v4-hc / thread2 is running on prio 7 processing other network 
> stuff - interrupted from other threads with higher priority. First after 
> approximately 340us after the RX interrupt, the Process 905246 thread 16 
> (prio 185) that blocks in pcap_loop is getting running.
> I would expect, that the tsec1 interrupt receive thread (4) puts the 
> incomming frame into the stack queue and then serves the pcap request 
> immediately or at least boosts the prioity of thread (2) to 185.
> 
> Is this correct ? As you know, we want to do a high priority / low latency 
> processing of UDP frames on tsec1 under use of the pcap library. 
> 

<snip>

> Do you have an explanation for this phenonema ? It occurs on the 6.4 as 
> well as on the 6.3.2 Networking / OS
> 

This is probably due to the switch from pure receive / layer 2
processing to uper layer stack processing.  Incoming packets
are queued for upper layer processing in a fifo manner and there's
no priority switch of the thread that already has, or may acquire
the stack context on this queue op.

> 
> Remark: On 6.3.2 OS, together with 6.4 networking, I have seen SIGSEGV 
> crashes of io-pkt in both setups after some runtime (e.g. 20 min). This 
> doesn't occur with 6.4 OS. Are there technical reasons not to use 6.4 
> Networking together with 6.3.2 OS ? 

Not that I'm aware of.

> 
> 
> I have attached a zip file with both kernel traces.
> 
> Guenther
> 
> 
> 
>  
> 
> 
> 
> Sean Boudreau <community-noreply@qnx.com> 
> 29.08.2008 17:27
> Bitte antworten an
> post12624@community.qnx.com
> 
> 
> An
> technology-networking <post12624@community.qnx.com>
> Kopie
> 
> Thema
> Re: io-pkt crash on MPC8548 with two interfaces and prioritized RX path
> 
> 
> 
> 
> 
> 
> > 
> > 
> > I wasn't able to reproduce the blown stack issue either.
> > At any rate please try the latest stuff.
> > 
> > Regards,
> > 
> > -seanb
> 
> 
> I introduced a nasty cut and paste issue in
> rev 476.  Please update to rev 491 or greater.
> 
> -seanb
> 
> _______________________________________________
> Technology
> http://community.qnx.com/sf/go/post12624
> 
> 
> 
> +----------------------------------------------------------------------+
> | ETAS Mail Security - http://intranet.etasgroup.com/encryption        |
> +----------------------------------------------------------------------+
> | - The message was...