Mark Dowdy(deleted)
|
Two instances of io-pkt interfering with each other
|
Mark Dowdy(deleted)
12/21/2012 3:19 PM
post98289
|
Two instances of io-pkt interfering with each other
First some background. We're running two instances of io-pkt in our system, one for 'normal' network traffic (i.e. TCP/
IP, UDP/IP, management, Qnet, Photon (that's another story), etc.) and a second for time critical traffic (data
extracted/injected via BPF, no IP address configured, Qnet not started on this interface). These are x86 machines, one
multicore and 2 or 3 single core. On the multicore machine, we're using the devnp-e1000 driver on both interfaces (NIC
device ID 1096h); on the single core machines we're running the devnp-speedo driver on the 'normal' network (NIC device
ID 1209h) and the devnp-e1000 driver on the time critical network (NIC device ID 1076h). Time critical system control
traffic is sent using the second io-pkt instance every ms.
Using an internally developed traffic test program, we observe time critical data flowing from a single core machine to
the multicore machine and then from the multicore machine back to the single core with a best case round trip time of ~
290 µsec. In some cases, however, this round trip time increases to ~450 µsec. You may be asking "what's 160 µsec
between friends?" but we use this time critical channel for motor control data and when we use our full system (multiple
single core machines sending data with our full motor control algorithms and other system functions), the problem
magnifies to the point that we see traffic that doesn't make it back before the next cycle starts 1 ms later.
So, we took to looking at what was going on using the System Profiler (i.e. kernel trace) and observed that traffic on
the time critical network (2nd io-pkt instance) slowed down when there was action on the management network (1st io-pkt
instance). That was unexpected since the reason we started two instances of io-pkt was to avoid exactly that scenario,
regular traffic negatively impacting time critical traffic. The second io-pkt instance is started with higher priority
(see below). The kernel trace only gives us exposure to task activity, not NIC driver execution, so it's hard to say
when network data actually "hits the wire". What could be going on in io-pkt that could be causing this type of adverse
interaction? On the single core machine, it looks like the first io-pkt instance is running and delaying the driver from
putting data on the wire. With two instances of io-pkt using devnp-e1000 on the multicore machine, could the shared
driver object be 'binding' the two io-pkt instances and altering performance?
Thanks.
First io-pkt instance start-up (i.e. 'normal' network on multicore machine)
io-pkt-v4-hc -ptcpip
mount -Tio-pkt -opci=0,vid=0x8086,did=0x1096 /lib/dll/devnp-e1000.so
Second io-pkt instance start-up (i.e. time critical network on multicore machine)
io-pkt-v4-hc -i1 -ptcpip prefix=/alt
mount -Tio-pkt1 -opci=1,vid=0x8086,did=0x1096,priority=32,receive=512,transmit=4096 /lib/dll/devnp-e1000.so
|
|
|
Dennis Kellly
|
Re: Two instances of io-pkt interfering with each other
|
Dennis Kellly
12/21/2012 3:25 PM
post98290
|
Re: Two instances of io-pkt interfering with each other
Are you using this patch?
http://www.qnx.com/download/feature.html?programid=23666
Without it, the timer interrupt would be attached to both stacks - this may be somehow serializing the processing for
the two stacks?
|
|
|
Armin Steinhoff
|
Re: Two instances of io-pkt interfering with each other
|
Armin Steinhoff
12/21/2012 4:01 PM
post98291
|
Re: Two instances of io-pkt interfering with each other
Dennis Kellly wrote:
> Are you using this patch?
>
> http://www.qnx.com/download/feature.html?programid=23666
Did change SP1 anything with the e-1000 driver ?
I would bypassing io-pkt and devn-e1000 with a specialized resource
manager w/o dependency to the QSS network implementation.
Regarding to the negative experiences with the PCI support of QNX 6.5
it's recommended to go back to QNX 6.4.1.
Regards
--Armin
>
> Without it, the timer interrupt would be attached to both stacks - this may be somehow serializing the processing for
the two stacks?
>
>
>
> _______________________________________________
>
> Networking Drivers
> http://community.qnx.com/sf/go/post98290
> To cancel your subscription to this discussion, please e-mail drivers-networking-unsubscribe@community.qnx.com
>
|
|
|
Dennis Kellly
|
Re: Two instances of io-pkt interfering with each other
|
Dennis Kellly
12/21/2012 4:13 PM
post98292
|
Re: Two instances of io-pkt interfering with each other
>>>Did change SP1 anything with the e-1000 driver ?
Not that I am aware of. The latest e1000 driver is on this page:
http://community.qnx.com/sf/wiki/do/viewPage/projects.bsp/wiki/ExperimentalDriversAndUtilities
Apparently the io-pkt patch I referenced is already rolled up in SP1.
|
|
|
Mark Dowdy(deleted)
|
Re: Two instances of io-pkt interfering with each other
|
Mark Dowdy(deleted)
12/21/2012 4:55 PM
post98293
|
Re: Two instances of io-pkt interfering with each other
Ah, forgot to mention that. We are running 6.5.0 SP1. We can't move back to 6.4.1 because we have a new multicore
machine that we're about to deploy and the NIC in that machine (Intel 82576, device id 10c9h) isn't supported in 6.4.1.
We're using the devnp-e1000 driver from SP1 (see below), I'll try the driver Dennis referenced to see if it makes any
difference.
# use -i /lib/dll/devnp-e1000.so
NAME=devnp-e1000.so
DESCRIPTION=Driver for Intel 82544 Gigabit Ethernet controllers
DATE=2012/06/20-13:37:46-EDT
STATE=stable
HOST=gusbuild4
USER=builder
VERSION=6.5.0
TAGID=650SP1-166
Thanks.
Mark
|
|
|
Dennis Kellly
|
Re: Two instances of io-pkt interfering with each other
|
Dennis Kellly
12/21/2012 5:00 PM
post98294
|
Re: Two instances of io-pkt interfering with each other
Mark-
You an look at at a system trace and make sure io-pkt has not InterruptAttach() to the system timer. (Sounds like you
already have this change.) Hopefully, no other shared interrupts.
I assume you are already raising the priority of your critical stack higher than the non-critical?
|
|
|
Mark Dowdy(deleted)
|
Re: Two instances of io-pkt interfering with each other
|
Mark Dowdy(deleted)
12/21/2012 6:01 PM
post98297
|
Re: Two instances of io-pkt interfering with each other
When the timer interrupt fires, io-pkt is not scheduled. From the kernel trace:
Interrupt 0x0, Entry, who procnto-instr - CPU 1 idle interrupt 0x0 ip 0xf00879ae
Interrupt 0x0 Process 1, Handler Entry, pid 1 interrupt 0x0 ip 0xf005a430 area 0x0 process procnto-instr
Interrupt 0x0 Process 1, Handler Exit, interrupt 0x0 sigev_notify 0
Interrupt 0x0, Exit, interrupt 0x0 flags 0x81
<no Sigevent like other interrupts>
Yes, we use "priority=32" when mounting the e1000 driver on the second stack. This is reflected in the priority labels
in the kernel trace (we ran wide mode to verify thread priorities).
Mark
|
|
|
Armin Steinhoff
|
Re: Two instances of io-pkt interfering with each other
|
Armin Steinhoff
12/21/2012 5:04 PM
post98295
|
Re: Two instances of io-pkt interfering with each other
Mark Dowdy wrote:
> Ah, forgot to mention that. We are running 6.5.0 SP1. We can't move back to 6.4.1 because we have a new multicore
machine that we're about to deploy and the NIC in that machine (Intel 82576, device id 10c9h) isn't supported in 6.4.1.
OK ... be careful if you use APIC/MSI.
An other critical point is the PCI interrupt if it is shared with io-usb
... the resource manager of io-usb seems to disable/block that shared
interrupt for a longer time.
--Armin
>
> We're using the devnp-e1000 driver from SP1 (see below), I'll try the driver Dennis referenced to see if it makes any
difference.
>
> # use -i /lib/dll/devnp-e1000.so
> NAME=devnp-e1000.so
> DESCRIPTION=Driver for Intel 82544 Gigabit Ethernet controllers
> DATE=2012/06/20-13:37:46-EDT
> STATE=stable
> HOST=gusbuild4
> USER=builder
> VERSION=6.5.0
> TAGID=650SP1-166
>
> Thanks.
>
> Mark
>
>
>
> _______________________________________________
>
> Networking Drivers
> http://community.qnx.com/sf/go/post98293
> To cancel your subscription to this discussion, please e-mail drivers-networking-unsubscribe@community.qnx.com
>
|
|
|
Lichun Zhu
|
Re: Two instances of io-pkt interfering with each other
|
Lichun Zhu
12/21/2012 5:08 PM
post98296
|
Re: Two instances of io-pkt interfering with each other
Where do you get the information that io-usb disable interrupt for long time(how long), as far as I remember there was
some host controller driver making lots delay, but that only on initalization.
Sent from BlackBerry
----- Original Message -----
From: Armin Steinhoff [mailto:community-noreply@qnx.com]
Sent: Friday, December 21, 2012 05:03 PM
To: drivers-networking@community.qnx.com <drivers-networking@community.qnx.com>
Cc: Info System - IS Automated Notifications; Info System - IS Automated Notifications
Subject: Re: Two instances of io-pkt interfering with each other
Mark Dowdy wrote:
> Ah, forgot to mention that. We are running 6.5.0 SP1. We can't move back to 6.4.1 because we have a new multicore
machine that we're about to deploy and the NIC in that machine (Intel 82576, device id 10c9h) isn't supported in 6.4.1.
OK ... be careful if you use APIC/MSI.
An other critical point is the PCI interrupt if it is shared with io-usb
... the resource manager of io-usb seems to disable/block that shared
interrupt for a longer time.
--Armin
>
> We're using the devnp-e1000 driver from SP1 (see below), I'll try the driver Dennis referenced to see if it makes any
difference.
>
> # use -i /lib/dll/devnp-e1000.so
> NAME=devnp-e1000.so
> DESCRIPTION=Driver for Intel 82544 Gigabit Ethernet controllers
> DATE=2012/06/20-13:37:46-EDT
> STATE=stable
> HOST=gusbuild4
> USER=builder
> VERSION=6.5.0
> TAGID=650SP1-166
>
> Thanks.
>
> Mark
>
>
>
> _______________________________________________
>
> Networking Drivers
> http://community.qnx.com/sf/go/post98293
> To cancel your subscription to this discussion, please e-mail drivers-networking-unsubscribe@community.qnx.com
>
_______________________________________________
Networking Drivers
http://community.qnx.com/sf/go/post98295
To cancel your subscription to this discussion, please e-mail drivers-networking-unsubscribe@community.qnx.com
|
|
|
Armin Steinhoff
|
Re: Two instances of io-pkt interfering with each other
|
Armin Steinhoff
12/22/2012 5:11 AM
post98304
|
Re: Two instances of io-pkt interfering with each other
Lichun Zhu wrote:
> Where do you get the information that io-usb disable interrupt for long time(how long), as far as I remember there was
some host controller driver making lots delay, but that only on initalization.
We have seen it in the context of an active io-usb ... there was the
shared interrupt disabled for round about 8ms (visible in a kernel trace)
Also other (professional) customers did reporting it !
--Armin
> Sent from BlackBerry
>
> ----- Original Message -----
> From: Armin Steinhoff [mailto:community-noreply@qnx.com]
> Sent: Friday, December 21, 2012 05:03 PM
> To: drivers-networking@community.qnx.com <drivers-networking@community.qnx.com>
> Cc: Info System - IS Automated Notifications; Info System - IS Automated Notifications
> Subject: Re: Two instances of io-pkt interfering with each other
>
> Mark Dowdy wrote:
>> Ah, forgot to mention that. We are running 6.5.0 SP1. We can't move back to 6.4.1 because we have a new multicore
machine that we're about to deploy and the NIC in that machine (Intel 82576, device id 10c9h) isn't supported in 6.4.1.
> OK ... be careful if you use APIC/MSI.
> An other critical point is the PCI interrupt if it is shared with io-usb
> ... the resource manager of io-usb seems to disable/block that shared
> interrupt for a longer time.
>
> --Armin
>
>> We're using the devnp-e1000 driver from SP1 (see below), I'll try the driver Dennis referenced to see if it makes any
difference.
>>
>> # use -i /lib/dll/devnp-e1000.so
>> NAME=devnp-e1000.so
>> DESCRIPTION=Driver for Intel 82544 Gigabit Ethernet controllers
>> DATE=2012/06/20-13:37:46-EDT
>> STATE=stable
>> HOST=gusbuild4
>> USER=builder
>> VERSION=6.5.0
>> TAGID=650SP1-166
>>
>> Thanks.
>>
>> Mark
>>
>>
>>
>> _______________________________________________
>>
>> Networking Drivers
>> http://community.qnx.com/sf/go/post98293
>> To cancel your subscription to this discussion, please e-mail drivers-networking-unsubscribe@community.qnx.com
>>
>
>
>
>
> _______________________________________________
>
> Networking Drivers
> http://community.qnx.com/sf/go/post98295
> To cancel your subscription to this discussion, please e-mail drivers-networking-unsubscribe@community.qnx.com
>
>
>
>
> _______________________________________________
>
> Networking Drivers
> http://community.qnx.com/sf/go/post98296
> To cancel your subscription to this discussion, please e-mail drivers-networking-unsubscribe@community.qnx.com
>
|
|
|
Mark Dowdy(deleted)
|
Re: Two instances of io-pkt interfering with each other
|
Mark Dowdy(deleted)
12/21/2012 6:04 PM
post98298
|
Re: Two instances of io-pkt interfering with each other
There are some shared interrupts but not with io-usb.
# pci
PCI version = 3.00
Class = Mass Storage (IDE)
Vendor ID = 8086h, Intel Corporation
Device ID = 2680h, 631xESB/632xESB/3100 Chipset SATA IDE Controller
PCI index = 0h
BAR - 0 [I/O] = 0h enabled
BAR - 1 [I/O] = 0h enabled
BAR - 2 [I/O] = 0h enabled
BAR - 3 [I/O] = 0h enabled
BAR - 4 [I/O] = 1870h enabled
PCI Int Pin = INT B
Interrupt line = no connection
Class = Network (Ethernet)
Vendor ID = 8086h, Intel Corporation
Device ID = 1096h, 80003ES2LAN Gigabit Ethernet Controller (Copper)
PCI index = 0h
BAR - 0 [Mem] = d8020000h enabled
BAR - 1 [Mem] = d8000000h enabled
BAR - 2 [I/O] = 2000h enabled
PCI Expansion ROM = 0h disabled
PCI Int Pin = INT A
Interrupt line = 11
CPU Interrupt = bh
Class = Network (Ethernet)
Vendor ID = 8086h, Intel Corporation
Device ID = 1096h, 80003ES2LAN Gigabit Ethernet Controller (Copper)
PCI index = 1h
BAR - 0 [Mem] = d8060000h enabled
BAR - 1 [Mem] = d8040000h enabled
BAR - 2 [I/O] = 2020h enabled
PCI Expansion ROM = 0h disabled
PCI Int Pin = INT B
Interrupt line = 10
CPU Interrupt = ah
Class = Display (VGA)
Vendor ID = 1002h, ATI Technologies Inc
Device ID = 515eh, ES1000
PCI index = 0h
BAR - 0 [Mem] = d0000000h enabled
BAR - 1 [I/O] = 3000h enabled
BAR - 2 [Mem] = d8200000h enabled
PCI Expansion ROM = 0h disabled
PCI Int Pin = INT A
Interrupt line = 11
CPU Interrupt = bh
|
|
|
Mario Charest
|
RE: Two instances of io-pkt interfering with each other
|
Mario Charest
12/21/2012 6:29 PM
post98299
|
RE: Two instances of io-pkt interfering with each other
Are they real core or 1 core plus hyperthreading ?
> -----Message d'origine-----
> De : Mark Dowdy [mailto:community-noreply@qnx.com]
> Envoyé : 21 décembre 2012 15:17
> À : drivers-networking@community.qnx.com
> Objet : Two instances of io-pkt interfering with each other
>
> First some background. We're running two instances of io-pkt in our system,
> one for 'normal' network traffic (i.e. TCP/IP, UDP/IP, management, Qnet,
> Photon (that's another story), etc.) and a second for time critical traffic (data
> extracted/injected via BPF, no IP address configured, Qnet not started on this
> interface). These are x86 machines, one multicore and 2 or 3 single core. On
> the multicore machine, we're using the devnp-e1000 driver on both interfaces
> (NIC device ID 1096h); on the single core machines we're running the devnp-
> speedo driver on the 'normal' network (NIC device ID 1209h) and the devnp-
> e1000 driver on the time critical network (NIC device ID 1076h). Time critical
> system control traffic is sent using the second io-pkt instance every ms.
>
> Using an internally developed traffic test program, we observe time critical
> data flowing from a single core machine to the multicore machine and then
> from the multicore machine back to the single core with a best case round trip
> time of ~290 µsec. In some cases, however, this round trip time increases to
> ~450 µsec. You may be asking "what's 160 µsec between friends?" but we use
> this time critical channel for motor control data and when we use our full
> system (multiple single core machines sending data with our full motor control
> algorithms and other system functions), the problem magnifies to the point that
> we see traffic that doesn't make it back before the next cycle starts 1 ms later.
>
> So, we took to looking at what was going on using the System Profiler (i.e.
> kernel trace) and observed that traffic on the time critical network (2nd io-pkt
> instance) slowed down when there was action on the management network (1st
> io-pkt instance). That was unexpected since the reason we started two
> instances of io-pkt was to avoid exactly that scenario, regular traffic negatively
> impacting time critical traffic. The second io-pkt instance is started with higher
> priority (see below). The kernel trace only gives us exposure to task activity, not
> NIC driver execution, so it's hard to say when network data actually "hits the
> wire". What could be going on in io-pkt that could be causing this type of
> adverse interaction? On the single core machine, it looks like the first io-pkt
> instance is running and delaying the driver from putting data on the wire. With
> two instances of io-pkt using devnp-e1000 on the multicore machine, could the
> shared driver object be 'binding' the two io-pkt instances and altering
> performance?
>
> Thanks.
>
> First io-pkt instance start-up (i.e. 'normal' network on multicore machine) io-
> pkt-v4-hc -ptcpip mount -Tio-pkt -opci=0,vid=0x8086,did=0x1096 /lib/dll/devnp-
> e1000.so
>
> Second io-pkt instance start-up (i.e. time critical network on multicore
> machine) io-pkt-v4-hc -i1 -ptcpip prefix=/alt mount -Tio-pkt1 -
> opci=1,vid=0x8086,did=0x1096,priority=32,receive=512,transmit=4096
> /lib/dll/devnp-e1000.so
>
>
>
>
>
> _______________________________________________
>
> Networking Drivers
> http://community.qnx.com/sf/go/post98289
> To cancel your subscription to this discussion, please e-mail drivers-networking-
> unsubscribe@community.qnx.com
|
|
|
Mark Dowdy(deleted)
|
Re: RE: Two instances of io-pkt interfering with each other
|
Mark Dowdy(deleted)
12/21/2012 6:43 PM
post98300
|
Re: RE: Two instances of io-pkt interfering with each other
They're real cores, none of that hyperthreading nonsense that Intel tried to pass off as multicore.
# pidin info
CPU:X86 Release:6.5.0 FreeMem:838Mb/1023Mb BootTime:Dec 19 09:45:50 PST 2012
Processes: 51, Threads: 106
Processor1: 66222 Core 2 Extreme/Xeon Stepping 10 2835MHz FPU
Processor2: 66222 Core 2 Extreme/Xeon Stepping 10 2834MHz FPU
Processor3: 66222 Core 2 Extreme/Xeon Stepping 10 2835MHz FPU
Processor4: 66222 Core 2 Extreme/Xeon Stepping 10 2835MHz FPU
|
|
|
Suganthan Subramaniam
|
Re: Two instances of io-pkt interfering with each other
|
Suganthan Subramaniam
04/21/2013 7:17 AM
post100765
|
Re: Two instances of io-pkt interfering with each other
Hi,
Did you find solution for this problem? I have a similar setup like you except, I only have a single instance of io-pkt
. And in the kernel trace I can clearly see the interrupt is coming, but the io-pkt is blocked for about 3 ms at times
from the previous interrupt.
|
|
|
|