Project Home
Project Home
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - Two instances of io-pkt interfering with each other: (14 Items)
   
Two instances of io-pkt interfering with each other  
First some background. We're running two instances of io-pkt in our system, one for 'normal' network traffic (i.e. TCP/
IP, UDP/IP, management, Qnet, Photon (that's another story), etc.) and a second for time critical traffic (data 
extracted/injected via BPF, no IP address configured, Qnet not started on this interface). These are x86 machines, one 
multicore and 2 or 3 single core. On the multicore machine, we're using the devnp-e1000 driver on both interfaces (NIC 
device ID 1096h); on the single core machines we're running the devnp-speedo driver on the 'normal' network (NIC device 
ID 1209h) and the devnp-e1000 driver on the time critical network (NIC device ID 1076h). Time critical system control 
traffic is sent using the second io-pkt instance every ms.

Using an internally developed traffic test program, we observe time critical data flowing from a single core machine to 
the multicore machine and then from the multicore machine back to the single core with a best case round trip time of ~
290 µsec. In some cases, however, this round trip time increases to ~450 µsec. You may be asking "what's 160 µsec 
between friends?" but we use this time critical channel for motor control data and when we use our full system (multiple
 single core machines sending data with our full motor control algorithms and other system functions), the problem 
magnifies to the point that we see traffic that doesn't make it back before the next cycle starts 1 ms later.

So, we took to looking at what was going on using the System Profiler (i.e. kernel trace) and observed that traffic on 
the time critical network (2nd io-pkt instance) slowed down when there was action on the management network (1st io-pkt 
instance). That was unexpected since the reason we started two instances of io-pkt was to avoid exactly that scenario, 
regular traffic negatively impacting time critical traffic. The second io-pkt instance is started with higher priority 
(see below). The kernel trace only gives us exposure to task activity, not NIC driver execution, so it's hard to say 
when network data actually "hits the wire". What could be going on in io-pkt that could be causing this type of adverse 
interaction? On the single core machine, it looks like the first io-pkt instance is running and delaying the driver from
 putting data on the wire. With two instances of io-pkt using devnp-e1000 on the multicore machine, could the shared 
driver object be 'binding' the two io-pkt instances and altering performance?

Thanks.

First io-pkt instance start-up (i.e. 'normal' network on multicore machine)
io-pkt-v4-hc -ptcpip
mount -Tio-pkt -opci=0,vid=0x8086,did=0x1096 /lib/dll/devnp-e1000.so

Second io-pkt instance start-up (i.e. time critical network on multicore machine)
io-pkt-v4-hc -i1 -ptcpip prefix=/alt
mount -Tio-pkt1 -opci=1,vid=0x8086,did=0x1096,priority=32,receive=512,transmit=4096 /lib/dll/devnp-e1000.so

Attachment: HTML sf-attachment-mime8601 5.22 KB
Re: Two instances of io-pkt interfering with each other  
Are you using this patch?

http://www.qnx.com/download/feature.html?programid=23666

Without it, the timer interrupt would be attached to both stacks - this may be somehow serializing the processing for 
the two stacks?
Re: Two instances of io-pkt interfering with each other  
Dennis Kellly wrote:
> Are you using this patch?
>
> http://www.qnx.com/download/feature.html?programid=23666

Did change SP1 anything with the e-1000 driver ?

I would bypassing io-pkt and devn-e1000  with a specialized resource 
manager w/o dependency to the QSS network implementation.
Regarding to the negative experiences with the PCI support of QNX 6.5 
it's recommended to go back to QNX 6.4.1.

Regards

--Armin

>
> Without it, the timer interrupt would be attached to both stacks - this may be somehow serializing the processing for 
the two stacks?
>
>
>
> _______________________________________________
>
> Networking Drivers
> http://community.qnx.com/sf/go/post98290
> To cancel your subscription to this discussion, please e-mail drivers-networking-unsubscribe@community.qnx.com
>

Re: Two instances of io-pkt interfering with each other  
>>>Did change SP1 anything with the e-1000 driver ?
Not that I am aware of.  The latest e1000 driver is on this page:
http://community.qnx.com/sf/wiki/do/viewPage/projects.bsp/wiki/ExperimentalDriversAndUtilities

Apparently the io-pkt patch I referenced is already rolled up in SP1.
Re: Two instances of io-pkt interfering with each other  
Ah, forgot to mention that. We are running 6.5.0 SP1. We can't move back to 6.4.1 because we have a new multicore 
machine that we're about to deploy and the NIC in that machine (Intel 82576, device id 10c9h) isn't supported in 6.4.1.

We're using the devnp-e1000 driver from SP1 (see below), I'll try the driver Dennis referenced to see if it makes any 
difference.

# use -i /lib/dll/devnp-e1000.so
NAME=devnp-e1000.so
DESCRIPTION=Driver for Intel 82544 Gigabit Ethernet controllers
DATE=2012/06/20-13:37:46-EDT
STATE=stable
HOST=gusbuild4
USER=builder
VERSION=6.5.0
TAGID=650SP1-166

Thanks.

Mark
Re: Two instances of io-pkt interfering with each other  
Mark-

You an look at at a system trace and make sure io-pkt has not InterruptAttach() to the system timer.   (Sounds like you 
already have this change.)  Hopefully, no other shared interrupts. 

I assume you are already raising the priority of your critical stack higher than the non-critical?
Re: Two instances of io-pkt interfering with each other  
When the timer interrupt fires, io-pkt is not scheduled. From the kernel trace:

Interrupt 0x0, Entry, who procnto-instr - CPU 1 idle interrupt 0x0 ip 0xf00879ae
Interrupt 0x0 Process 1, Handler Entry, pid 1 interrupt 0x0 ip 0xf005a430 area 0x0 process procnto-instr
Interrupt 0x0 Process 1, Handler Exit, interrupt 0x0 sigev_notify 0
Interrupt 0x0, Exit, interrupt 0x0 flags 0x81
<no Sigevent like other interrupts>

Yes, we use "priority=32" when mounting the e1000 driver on the second stack. This is reflected in the priority labels 
in the kernel trace (we ran wide mode to verify thread priorities).

Mark
Re: Two instances of io-pkt interfering with each other  
Mark Dowdy wrote:
> Ah, forgot to mention that. We are running 6.5.0 SP1. We can't move back to 6.4.1 because we have a new multicore 
machine that we're about to deploy and the NIC in that machine (Intel 82576, device id 10c9h) isn't supported in 6.4.1.

OK ... be careful if you use APIC/MSI.
An other critical point is the PCI interrupt if it is shared with io-usb 
...  the resource manager of io-usb seems to disable/block that shared 
interrupt for a longer time.

--Armin

>
> We're using the devnp-e1000 driver from SP1 (see below), I'll try the driver Dennis referenced to see if it makes any 
difference.
>
> # use -i /lib/dll/devnp-e1000.so
> NAME=devnp-e1000.so
> DESCRIPTION=Driver for Intel 82544 Gigabit Ethernet controllers
> DATE=2012/06/20-13:37:46-EDT
> STATE=stable
> HOST=gusbuild4
> USER=builder
> VERSION=6.5.0
> TAGID=650SP1-166
>
> Thanks.
>
> Mark
>
>
>
> _______________________________________________
>
> Networking Drivers
> http://community.qnx.com/sf/go/post98293
> To cancel your subscription to this discussion, please e-mail drivers-networking-unsubscribe@community.qnx.com
>

Re: Two instances of io-pkt interfering with each other  
Where do you get the information that io-usb disable interrupt for long time(how long), as far as I remember there was 
some host controller driver making lots delay, but that only on initalization. 
Sent from BlackBerry

----- Original Message -----
From: Armin Steinhoff [mailto:community-noreply@qnx.com]
Sent: Friday, December 21, 2012 05:03 PM
To: drivers-networking@community.qnx.com <drivers-networking@community.qnx.com>
Cc: Info System - IS Automated Notifications; Info System - IS Automated Notifications
Subject: Re: Two instances of io-pkt interfering with each other

Mark Dowdy wrote:
> Ah, forgot to mention that. We are running 6.5.0 SP1. We can't move back to 6.4.1 because we have a new multicore 
machine that we're about to deploy and the NIC in that machine (Intel 82576, device id 10c9h) isn't supported in 6.4.1.

OK ... be careful if you use APIC/MSI.
An other critical point is the PCI interrupt if it is shared with io-usb 
...  the resource manager of io-usb seems to disable/block that shared 
interrupt for a longer time.

--Armin

>
> We're using the devnp-e1000 driver from SP1 (see below), I'll try the driver Dennis referenced to see if it makes any 
difference.
>
> # use -i /lib/dll/devnp-e1000.so
> NAME=devnp-e1000.so
> DESCRIPTION=Driver for Intel 82544 Gigabit Ethernet controllers
> DATE=2012/06/20-13:37:46-EDT
> STATE=stable
> HOST=gusbuild4
> USER=builder
> VERSION=6.5.0
> TAGID=650SP1-166
>
> Thanks.
>
> Mark
>
>
>
> _______________________________________________
>
> Networking Drivers
> http://community.qnx.com/sf/go/post98293
> To cancel your subscription to this discussion, please e-mail drivers-networking-unsubscribe@community.qnx.com
>





_______________________________________________

Networking Drivers
http://community.qnx.com/sf/go/post98295
To cancel your subscription to this discussion, please e-mail drivers-networking-unsubscribe@community.qnx.com
Re: Two instances of io-pkt interfering with each other  
Lichun Zhu wrote:
> Where do you get the information that io-usb disable interrupt for long time(how long), as far as I remember there was
 some host controller driver making lots delay, but that only on initalization.

We have seen it in the context of an active io-usb ... there was the 
shared interrupt disabled for round about 8ms (visible in a kernel trace)
Also other (professional) customers did reporting it !

--Armin

> Sent from BlackBerry
>
> ----- Original Message -----
> From: Armin Steinhoff [mailto:community-noreply@qnx.com]
> Sent: Friday, December 21, 2012 05:03 PM
> To: drivers-networking@community.qnx.com <drivers-networking@community.qnx.com>
> Cc: Info System - IS Automated Notifications; Info System - IS Automated Notifications
> Subject: Re: Two instances of io-pkt interfering with each other
>
> Mark Dowdy wrote:
>> Ah, forgot to mention that. We are running 6.5.0 SP1. We can't move back to 6.4.1 because we have a new multicore 
machine that we're about to deploy and the NIC in that machine (Intel 82576, device id 10c9h) isn't supported in 6.4.1.
> OK ... be careful if you use APIC/MSI.
> An other critical point is the PCI interrupt if it is shared with io-usb
> ...  the resource manager of io-usb seems to disable/block that shared
> interrupt for a longer time.
>
> --Armin
>
>> We're using the devnp-e1000 driver from SP1 (see below), I'll try the driver Dennis referenced to see if it makes any
 difference.
>>
>> # use -i /lib/dll/devnp-e1000.so
>> NAME=devnp-e1000.so
>> DESCRIPTION=Driver for Intel 82544 Gigabit Ethernet controllers
>> DATE=2012/06/20-13:37:46-EDT
>> STATE=stable
>> HOST=gusbuild4
>> USER=builder
>> VERSION=6.5.0
>> TAGID=650SP1-166
>>
>> Thanks.
>>
>> Mark
>>
>>
>>
>> _______________________________________________
>>
>> Networking Drivers
>> http://community.qnx.com/sf/go/post98293
>> To cancel your subscription to this discussion, please e-mail drivers-networking-unsubscribe@community.qnx.com
>>
>
>
>
>
> _______________________________________________
>
> Networking Drivers
> http://community.qnx.com/sf/go/post98295
> To cancel your subscription to this discussion, please e-mail drivers-networking-unsubscribe@community.qnx.com
>
>
>
>
> _______________________________________________
>
> Networking Drivers
> http://community.qnx.com/sf/go/post98296
> To cancel your subscription to this discussion, please e-mail drivers-networking-unsubscribe@community.qnx.com
>

Re: Two instances of io-pkt interfering with each other  
There are some shared interrupts but not with io-usb.

# pci

PCI version    = 3.00

Class          = Mass Storage (IDE)
Vendor ID      = 8086h, Intel Corporation
Device ID      = 2680h,  631xESB/632xESB/3100 Chipset SATA IDE Controller
PCI index      = 0h
BAR - 0 [I/O]  = 0h enabled
BAR - 1 [I/O]  = 0h enabled
BAR - 2 [I/O]  = 0h enabled
BAR - 3 [I/O]  = 0h enabled
BAR - 4 [I/O]  = 1870h enabled
PCI Int Pin    = INT B
Interrupt line = no connection

Class          = Network (Ethernet)
Vendor ID      = 8086h, Intel Corporation
Device ID      = 1096h,  80003ES2LAN Gigabit Ethernet Controller (Copper)
PCI index      = 0h
BAR - 0 [Mem]  = d8020000h enabled
BAR - 1 [Mem]  = d8000000h enabled
BAR - 2 [I/O]  = 2000h enabled
PCI Expansion ROM = 0h disabled
PCI Int Pin    = INT A
Interrupt line = 11
CPU Interrupt  = bh

Class          = Network (Ethernet)
Vendor ID      = 8086h, Intel Corporation
Device ID      = 1096h,  80003ES2LAN Gigabit Ethernet Controller (Copper)
PCI index      = 1h
BAR - 0 [Mem]  = d8060000h enabled
BAR - 1 [Mem]  = d8040000h enabled
BAR - 2 [I/O]  = 2020h enabled
PCI Expansion ROM = 0h disabled
PCI Int Pin    = INT B
Interrupt line = 10
CPU Interrupt  = ah

Class          = Display (VGA)
Vendor ID      = 1002h, ATI Technologies Inc
Device ID      = 515eh,  ES1000
PCI index      = 0h
BAR - 0 [Mem]  = d0000000h enabled
BAR - 1 [I/O]  = 3000h enabled
BAR - 2 [Mem]  = d8200000h enabled
PCI Expansion ROM = 0h disabled
PCI Int Pin    = INT A
Interrupt line = 11
CPU Interrupt  = bh
RE: Two instances of io-pkt interfering with each other  
Are they real core or 1 core plus hyperthreading ?

> -----Message d'origine-----
> De : Mark Dowdy [mailto:community-noreply@qnx.com]
> Envoyé : 21 décembre 2012 15:17
> À : drivers-networking@community.qnx.com
> Objet : Two instances of io-pkt interfering with each other
> 
> First some background. We're running two instances of io-pkt in our system,
> one for 'normal' network traffic (i.e. TCP/IP, UDP/IP, management, Qnet,
> Photon (that's another story), etc.) and a second for time critical traffic (data
> extracted/injected via BPF, no IP address configured, Qnet not started on this
> interface). These are x86 machines, one multicore and 2 or 3 single core. On
> the multicore machine, we're using the devnp-e1000 driver on both interfaces
> (NIC device ID 1096h); on the single core machines we're running the devnp-
> speedo driver on the 'normal' network (NIC device ID 1209h) and the devnp-
> e1000 driver on the time critical network (NIC device ID 1076h). Time critical
> system control traffic is sent using the second io-pkt instance every ms.
> 
> Using an internally developed traffic test program, we observe time critical
> data flowing from a single core machine to the multicore machine and then
> from the multicore machine back to the single core with a best case round trip
> time of ~290 µsec. In some cases, however, this round trip time increases to
> ~450 µsec. You may be asking "what's 160 µsec between friends?" but we use
> this time critical channel for motor control data and when we use our full
> system (multiple single core machines sending data with our full motor control
> algorithms and other system functions), the problem magnifies to the point that
> we see traffic that doesn't make it back before the next cycle starts 1 ms later.
> 
> So, we took to looking at what was going on using the System Profiler (i.e.
> kernel trace) and observed that traffic on the time critical network (2nd io-pkt
> instance) slowed down when there was action on the management network (1st
> io-pkt instance). That was unexpected since the reason we started two
> instances of io-pkt was to avoid exactly that scenario, regular traffic negatively
> impacting time critical traffic. The second io-pkt instance is started with higher
> priority (see below). The kernel trace only gives us exposure to task activity, not
> NIC driver execution, so it's hard to say when network data actually "hits the
> wire". What could be going on in io-pkt that could be causing this type of
> adverse interaction? On the single core machine, it looks like the first io-pkt
> instance is running and delaying the driver from putting data on the wire. With
> two instances of io-pkt using devnp-e1000 on the multicore machine, could the
> shared driver object be 'binding' the two io-pkt instances and altering
> performance?
> 
> Thanks.
> 
> First io-pkt instance start-up (i.e. 'normal' network on multicore machine) io-
> pkt-v4-hc -ptcpip mount -Tio-pkt -opci=0,vid=0x8086,did=0x1096 /lib/dll/devnp-
> e1000.so
> 
> Second io-pkt instance start-up (i.e. time critical network on multicore
> machine) io-pkt-v4-hc -i1 -ptcpip prefix=/alt mount -Tio-pkt1 -
> opci=1,vid=0x8086,did=0x1096,priority=32,receive=512,transmit=4096
> /lib/dll/devnp-e1000.so
> 
> 
> 
> 
> 
> _______________________________________________
> 
> Networking Drivers
> http://community.qnx.com/sf/go/post98289
> To cancel your subscription to this discussion, please e-mail drivers-networking-
> unsubscribe@community.qnx.com
Re: RE: Two instances of io-pkt interfering with each other  
They're real cores, none of that hyperthreading nonsense that Intel tried to pass off as multicore.

# pidin info
CPU:X86 Release:6.5.0  FreeMem:838Mb/1023Mb BootTime:Dec 19 09:45:50 PST 2012
Processes: 51, Threads: 106
Processor1: 66222 Core 2 Extreme/Xeon Stepping 10 2835MHz FPU
Processor2: 66222 Core 2 Extreme/Xeon Stepping 10 2834MHz FPU
Processor3: 66222 Core 2 Extreme/Xeon Stepping 10 2835MHz FPU
Processor4: 66222 Core 2 Extreme/Xeon Stepping 10 2835MHz FPU
Re: Two instances of io-pkt interfering with each other  
Hi,

Did you find solution for this problem?  I have a similar setup like you except, I only have a single instance of io-pkt
.  And in the kernel trace I can clearly see the interrupt is coming, but the io-pkt is blocked for about 3 ms at times 
from the previous interrupt.