J Sinton(deleted)
|
Timing resolution for tcpdump / libpcap
|
J Sinton(deleted)
05/23/2014 4:11 PM
post110480
|
Timing resolution for tcpdump / libpcap
Hi,
I have an application that must send and receive various UDP packets (including unicast, broadcast, multicast - it's for
a test simulator) within fairly small time windows (500uS or less). I am currently desktop test hardware QNX localhost
6.5.0 2010/07/09-14:44:03EDT x86pc x86. The specific Ethernet adapter info is below.
By setting the tick resolution to a small value (around 10uS), and setting the NIC driver in enum to promiscuous mode,
and hand-tuning the delay in a timer calls, I can send UDP packets as I need using libpcap, which is what I've done on
other non-QNX platforms.
However, when I use libpcap to receive packets, it appears that packets are clumped together, and only delivered on
specific time intervals (about every 8.39mS on this system). I've tried using both pcap_loop and pcap_dispatch,
changing the priority of the receiving thread all the way to 63 (priority seems to make no difference, high or low), and
verifying using Wireshark from another platform that the incoming packets are at the expected times and are properly
separated in time. The load on the machine during testing is typically very low - hogs reports idle in very high 90s,
and the problem persists even when sending very few packets (e.g. one every 5mS). All time measurement in the
application code are using CLOCK_MONOTONIC. nanosleep is used on the sending side between outgoing messages, and the
receiver is message driven from pcap_loop or pcap_dispatch from a high-priority thread with nothing else (no GUI)
running on the box.
When I tried to investigate further on the machine with tcpdump, I discovered that the messages shown there seem to be
subject to the same clumping: Groups of messages all have the same timestamp, and groups of messages all have time gaps
of about 8.4mS.
Knowing little about QNX, it looks like there is timing granularity (<>10mS) that affects both libpcap and tcpdump, even
when the granularity at the user application level is successfully much less (<>10uS).
libpcap normally works wonderfully for what I need once the OS tick is OK (Linux and Windows), but I just need to
deliver the application and I'm not tied to it if this is a difficult-to-surmount libpcap issue.
Two questions:
1. Where could this timing granularity be coming from, and is there any way to configure or code around it?
2. Using some other approach, such as BPF (seems that may be old) or lsm-nraw, can anyone confirm that they are able to
receive Ethernet messages at the application level with timing granularity in the 200uS or better range? What general
hardware/OS/software approach did you use to achieve it?
Many thanks for any suggestions for solutions or further investigation. I'm stuck!
Best regards,
John
Ethernet driver / NIC info
io-pkt-v4-hc -d speedo promiscuous -p tcpip
INTEL 82558 Ethernet Controller
Physical Node ID ........................... 0019DB BB621E
Current Physical Node ID ................... 0019DB BB621E
Current Operation Rate ..................... 100.00 Mb/s full-duplex
Active Interface Type ...................... MII
Active PHY address ....................... 1
Maximum Transmittable data Unit ............ 1514
Maximum Receivable data Unit ............... 1514
Hardware Interrupt ......................... 0x5
I/O Aperture ............................... 0xef00 - 0xef3f
Memory Aperture ............................ 0xfdcff000 - 0xfdcfffff
ROM Aperture ............................... 0x80dbf6b07fa5a04
Promiscuous Mode ........................... On
Multicast Support .......................... Enabled
|
|
|
Gervais Mulongoy
|
Re: Timing resolution for tcpdump / libpcap
|
Gervais Mulongoy
05/26/2014 8:15 AM
post110486
|
Re: Timing resolution for tcpdump / libpcap
I will start by saying this is a known issue and there's already a
solution in the works.
1) Unfortunately io-pkt is deciding on the timing resolution of the
timestamps this affects pcap and tcpdump - there isn't anyway to
workaround it at this time
2) The timing resolution doesn't affect packet reception/transmission it
only affects the timestamp - you can still use BPF, for example, to
monitor the packets as they arrive and stamp them yourself.
The timing resolution is decided by io-pkt
On 14-05-23 04:11 PM, J Sinton wrote:
> Two questions:
>
> 1. Where could this timing granularity be coming from, and is there any way to configure or code around it?
>
> 2. Using some other approach, such as BPF (seems that may be old) or lsm-nraw, can anyone confirm that they are able
to receive Ethernet messages at the application level with timing granularity in the 200uS or better range? What
general hardware/OS/software approach did you use to achieve it?
|
|
|
J Sinton(deleted)
|
Re: Timing resolution for tcpdump / libpcap
|
J Sinton(deleted)
05/26/2014 11:04 AM
post110490
|
Re: Timing resolution for tcpdump / libpcap
Thank you for pointing in that direction.
It seems perhaps I was experiencing two problems with different causes.
I wrote a beautiful and inspiring post that had all the details, but it just vanished. I'm afraid the replacement will
have to be quite a bit shorter.
Following the above suggestion, I found a good post that explained exactly the resolution issue of tcpdump,
unfortunately searching for it again I can't find it a second time. But around 8 to 10mS is the limit, so trying to use
tcpdump to see what's happening at a finer resolution is not currently useful.
I should have noticed that the 'clumping' in tcpdump was different from the 'clumping' I saw (I had confirmed the
messages were arriving very close together - not just timestamped together - by using the equivalent of clockcycles()).
The tcpdump grouping was multiple messages receiving the same timestamp because of limited resolution, where the pcap
grouping is for another reason.
There is a parameter to pcap open live that gives the milliseconds to wait on receive. However, entering zero does NOT
cause it to return immediately, and it appears that QNX correctly implements the zero behaviour defined by BSD, which is
that on zero it will return after one buffer - how many messages that is depends on the size of the buffer you specify,
and how big incoming messages are. If all messages were more than 1/2 the size of the buffer, then it would work as
'immediate'.
But experimenting with the BPF alternative led me eventually to this post:
http://community.qnx.com/sf/discussion/do/listPosts/projects.networking/discussion.technology.topc3493
... which led to ...
http://hostap.epitest.fi/wpa_supplicant/
I added this ioctl after the pcap open:
unsigned int iOn = 1;
if( 0 > ioctl( pcap_fileno( g_psPlatformPcapDesc ), BIOCIMMEDIATE, &iOn ) )
{
fprintf(stderr, "Error: %.40s: %d: Cannot enable immediate mode on interface %s\n", __func__, __LINE__,
strerror(errno)); // TODO ideally add interface name here
return -1;
}
Timing of individual incoming messages was within 500uS of the expected value. I actually then had more problem with
sending, as for some reason I was getting up to 900uS of jitter (as measured by clock cycles around the sending
function). I'm guessing this is a result of needing to understand more now of priorities etc. on QNX, as I can get
normally better than that on Linux and Windows. I'd hope for about 10X better than that, maybe with using processor
affinity etc. (though on this QNX platform I only have two cores, so I was hoping QNX magic would allow me to avoid
dedicating a core to this).
In both cases, the timing limits above were exceeded about 30 times in the first 10 million 5mS message group tests, but
nearly 40 million 5mS frames since then, all send and receive messages have been under the above limits without
exception. More work to do to understand why, but the original issue is solved.
Summary:
1. It would be misleading to use tcpdump on 6.5 to investigate message resolution finer than 10mS,
2. To get immediate notification of received messages in libpcap, use a zero (some examples show -1) parameter for the
receive wait time, and use an ioctl with BIOCIMMEDIATE on the handle when it is opened.
Thanks!
|
|
|
Armin Steinhoff
|
Re: Timing resolution for tcpdump / libpcap
|
Armin Steinhoff
05/27/2014 5:37 AM
post110491
|
Re: Timing resolution for tcpdump / libpcap
J Sinton wrote:
> Thank you for pointing in that direction.
Just a proposal for an other direction ... a solution could be:
http://community.qnx.com/sf/discussion/do/listPosts/projects.networking/discussion.drivers.topc24497
"testpmd" is using pollmode drivers from Intel for Intel adapters. These
drivers are UIO based Linux drivers (user space drivers) ported to QNX6.x
Please have also a look to this;
http://community.qnx.com/sf/discussion/do/listPosts/projects.networking/discussion.drivers.topc24497?_pagenum=2
You will get a time resolution in the range of microseconds ...
--Armin
>
> It seems perhaps I was experiencing two problems with different causes.
>
> I wrote a beautiful and inspiring post that had all the details, but it just vanished. I'm afraid the replacement
will have to be quite a bit shorter.
>
> Following the above suggestion, I found a good post that explained exactly the resolution issue of tcpdump,
unfortunately searching for it again I can't find it a second time. But around 8 to 10mS is the limit, so trying to use
tcpdump to see what's happening at a finer resolution is not currently useful.
>
> I should have noticed that the 'clumping' in tcpdump was different from the 'clumping' I saw (I had confirmed the
messages were arriving very close together - not just timestamped together - by using the equivalent of clockcycles()).
The tcpdump grouping was multiple messages receiving the same timestamp because of limited resolution, where the pcap
grouping is for another reason.
>
> There is a parameter to pcap open live that gives the milliseconds to wait on receive. However, entering zero does
NOT cause it to return immediately, and it appears that QNX correctly implements the zero behaviour defined by BSD,
which is that on zero it will return after one buffer - how many messages that is depends on the size of the buffer you
specify, and how big incoming messages are. If all messages were more than 1/2 the size of the buffer, then it would
work as 'immediate'.
>
> But experimenting with the BPF alternative led me eventually to this post:
>
> http://community.qnx.com/sf/discussion/do/listPosts/projects.networking/discussion.technology.topc3493
>
> ... which led to ...
>
> http://hostap.epitest.fi/wpa_supplicant/
>
> I added this ioctl after the pcap open:
> unsigned int iOn = 1;
> if( 0 > ioctl( pcap_fileno( g_psPlatformPcapDesc ), BIOCIMMEDIATE, &iOn ) )
> {
> fprintf(stderr, "Error: %.40s: %d: Cannot enable immediate mode on interface %s\n", __func__, __LINE__,
strerror(errno)); // TODO ideally add interface name here
> return -1;
> }
>
> Timing of individual incoming messages was within 500uS of the expected value. I actually then had more problem with
sending, as for some reason I was getting up to 900uS of jitter (as measured by clock cycles around the sending
function). I'm guessing this is a result of needing to understand more now of priorities etc. on QNX, as I can get
normally better than that on Linux and Windows. I'd hope for about 10X better than that, maybe with using processor
affinity etc. (though on this QNX platform I only have two cores, so I was hoping QNX magic would allow me to avoid
dedicating a core to this).
>
> In both cases, the timing limits above were exceeded about 30 times in the first 10 million 5mS message group tests,
but nearly 40 million 5mS frames since then, all send and receive messages have been under the above limits without
exception. More work to do to understand why, but the original issue is solved.
>
> Summary:
>
> 1. It would be misleading to use tcpdump on 6.5 to investigate message resolution finer than 10mS,
>
> 2. To get immediate notification of received messages in libpcap, use a zero (some examples show -1) parameter for the
receive wait time, and use an...
View Full Message
|
|
|
|