Project Home
Project Home
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - Timing resolution for tcpdump / libpcap : (4 Items)
   
Timing resolution for tcpdump / libpcap  
Hi,

I have an application that must send and receive various UDP packets (including unicast, broadcast, multicast - it's for
 a test simulator) within fairly small time windows (500uS or less).  I am currently desktop test hardware QNX localhost
 6.5.0 2010/07/09-14:44:03EDT x86pc x86.  The specific Ethernet adapter info is below.

By setting the tick resolution to a small value (around 10uS), and setting the NIC driver in enum to promiscuous mode, 
and hand-tuning the delay in a timer calls, I can send UDP packets as I need using libpcap, which is what I've done on 
other non-QNX platforms.

However, when I use libpcap to receive packets, it appears that packets are clumped together, and only delivered on 
specific time intervals (about every 8.39mS on this system).  I've tried using both pcap_loop and pcap_dispatch, 
changing the priority of the receiving thread all the way to 63 (priority seems to make no difference, high or low), and
 verifying using Wireshark from another platform that the incoming packets are at the expected times and are properly 
separated in time.  The load on the machine during testing is typically very low - hogs reports idle in very high 90s, 
and the problem persists even when sending very few packets (e.g. one every 5mS).  All time measurement in the 
application code are using CLOCK_MONOTONIC.  nanosleep is used on the sending side between outgoing messages, and the 
receiver is message driven from pcap_loop or pcap_dispatch from a high-priority thread with nothing else (no GUI) 
running on the box.

When I tried to investigate further on the machine with tcpdump, I discovered that the messages shown there seem to be 
subject to the same clumping: Groups of messages all have the same timestamp, and groups of messages all have time gaps 
of about 8.4mS. 

Knowing little about QNX, it looks like there is timing granularity (<>10mS) that affects both libpcap and tcpdump, even
 when the granularity at the user application level is successfully much less (<>10uS).

libpcap normally works wonderfully for what I need once the OS tick is OK (Linux and Windows), but I just need to 
deliver the application and I'm not tied to it if this is a difficult-to-surmount libpcap issue.

Two questions:

1. Where could this timing granularity be coming from, and is there any way to configure or code around it?

2. Using some other approach, such as BPF (seems that may be old) or lsm-nraw, can anyone confirm that they are able to 
receive Ethernet messages at the application level with timing granularity in the 200uS or better range?  What general 
hardware/OS/software approach did you use to achieve it?

Many thanks for any suggestions for solutions or further investigation.  I'm stuck!

Best regards,

John

Ethernet driver / NIC info

io-pkt-v4-hc -d speedo promiscuous -p tcpip

 INTEL 82558 Ethernet Controller

  Physical Node ID ........................... 0019DB BB621E
  Current Physical Node ID ................... 0019DB BB621E
  Current Operation Rate ..................... 100.00 Mb/s full-duplex
  Active Interface Type ...................... MII
    Active PHY address ....................... 1
  Maximum Transmittable data Unit ............ 1514
  Maximum Receivable data Unit ............... 1514
  Hardware Interrupt ......................... 0x5
  I/O Aperture ............................... 0xef00 - 0xef3f
  Memory Aperture ............................ 0xfdcff000 - 0xfdcfffff
  ROM Aperture ............................... 0x80dbf6b07fa5a04
  Promiscuous Mode ........................... On
  Multicast Support .......................... Enabled
Re: Timing resolution for tcpdump / libpcap  
I will start by saying this is a known issue and there's already a 
solution in the works.

1) Unfortunately io-pkt is deciding on the timing resolution of the 
timestamps this affects pcap and tcpdump - there isn't anyway to 
workaround it at this time

2) The timing resolution doesn't affect packet reception/transmission it 
only affects the timestamp  - you can still use BPF, for example, to 
monitor the packets as they arrive and stamp them yourself.

The timing resolution is decided by io-pkt
On 14-05-23 04:11 PM, J Sinton wrote:
> Two questions:
>
> 1. Where could this timing granularity be coming from, and is there any way to configure or code around it?
>
> 2. Using some other approach, such as BPF (seems that may be old) or lsm-nraw, can anyone confirm that they are able 
to receive Ethernet messages at the application level with timing granularity in the 200uS or better range?  What 
general hardware/OS/software approach did you use to achieve it?

Re: Timing resolution for tcpdump / libpcap  
Thank you for pointing in that direction.  

It seems perhaps I was experiencing two problems with different causes.

I wrote a beautiful and inspiring post that had all the details, but it just vanished.  I'm afraid the replacement will 
have to be quite a bit shorter.

Following the above suggestion, I found a good post that explained exactly the resolution issue of tcpdump, 
unfortunately searching for it again I can't find it a second time.  But around 8 to 10mS is the limit, so trying to use
 tcpdump to see what's happening at a finer resolution is not currently useful.

I should have noticed that the 'clumping' in tcpdump was different from the 'clumping' I saw (I had confirmed the 
messages were arriving very close together - not just timestamped together - by using the equivalent of clockcycles()). 
 The tcpdump grouping was multiple messages receiving the same timestamp because of limited resolution, where the pcap 
grouping is for another reason.

There is a parameter to pcap open live that gives the milliseconds to wait on receive.  However, entering zero does NOT 
cause it to return immediately, and it appears that QNX correctly implements the zero behaviour defined by BSD, which is
 that on zero it will return after one buffer - how many messages that is depends on the size of the buffer you specify,
 and how big incoming messages are.  If all messages were more than 1/2 the size of the buffer, then it would work as 
'immediate'.

But experimenting with the BPF alternative led me eventually to this post:

http://community.qnx.com/sf/discussion/do/listPosts/projects.networking/discussion.technology.topc3493

... which led to ...

http://hostap.epitest.fi/wpa_supplicant/

I added this ioctl after the pcap open:
unsigned int iOn = 1;
  if( 0 > ioctl( pcap_fileno( g_psPlatformPcapDesc ), BIOCIMMEDIATE, &iOn ) )
  {
    fprintf(stderr, "Error: %.40s: %d: Cannot enable immediate mode on interface %s\n", __func__, __LINE__, 
strerror(errno));   // TODO ideally add interface name here
    return -1;
  }

Timing of individual incoming messages was within 500uS of the expected value.  I actually then had more problem with 
sending, as for some reason I was getting up to 900uS of jitter (as measured by clock cycles around the sending 
function).  I'm guessing this is a result of needing to understand more now of priorities etc. on QNX, as I can get 
normally better than that on Linux and Windows.  I'd hope for about 10X better than that, maybe with using processor 
affinity etc. (though on this QNX platform I only have two cores, so I was hoping QNX magic would allow me to avoid 
dedicating a core to this).

In both cases, the timing limits above were exceeded about 30 times in the first 10 million 5mS message group tests, but
 nearly 40 million 5mS frames since then, all send and receive messages have been under the above limits without 
exception.  More work to do to understand why, but the original issue is solved.

Summary:

1. It would be misleading to use tcpdump on 6.5 to investigate message resolution finer than 10mS, 

2. To get immediate notification of received messages in libpcap, use a zero (some examples show -1) parameter for the 
receive wait time, and use an ioctl with BIOCIMMEDIATE on the handle when it is opened.

Thanks!
Re: Timing resolution for tcpdump / libpcap  

J Sinton wrote:
> Thank you for pointing in that direction.  
Just a proposal for an other direction ... a solution could be:

http://community.qnx.com/sf/discussion/do/listPosts/projects.networking/discussion.drivers.topc24497

"testpmd" is using pollmode drivers from Intel for Intel adapters. These
drivers are UIO based Linux drivers (user space drivers) ported to QNX6.x

Please have also a look to this;
http://community.qnx.com/sf/discussion/do/listPosts/projects.networking/discussion.drivers.topc24497?_pagenum=2

You will get a time resolution in the range of microseconds ...

--Armin

>
> It seems perhaps I was experiencing two problems with different causes.
>
> I wrote a beautiful and inspiring post that had all the details, but it just vanished.  I'm afraid the replacement 
will have to be quite a bit shorter.
>
> Following the above suggestion, I found a good post that explained exactly the resolution issue of tcpdump, 
unfortunately searching for it again I can't find it a second time.  But around 8 to 10mS is the limit, so trying to use
 tcpdump to see what's happening at a finer resolution is not currently useful.
>
> I should have noticed that the 'clumping' in tcpdump was different from the 'clumping' I saw (I had confirmed the 
messages were arriving very close together - not just timestamped together - by using the equivalent of clockcycles()). 
 The tcpdump grouping was multiple messages receiving the same timestamp because of limited resolution, where the pcap 
grouping is for another reason.
>
> There is a parameter to pcap open live that gives the milliseconds to wait on receive.  However, entering zero does 
NOT cause it to return immediately, and it appears that QNX correctly implements the zero behaviour defined by BSD, 
which is that on zero it will return after one buffer - how many messages that is depends on the size of the buffer you 
specify, and how big incoming messages are.  If all messages were more than 1/2 the size of the buffer, then it would 
work as 'immediate'.
>
> But experimenting with the BPF alternative led me eventually to this post:
>
> http://community.qnx.com/sf/discussion/do/listPosts/projects.networking/discussion.technology.topc3493
>
> ... which led to ...
>
> http://hostap.epitest.fi/wpa_supplicant/
>
> I added this ioctl after the pcap open:
> unsigned int iOn = 1;
>   if( 0 > ioctl( pcap_fileno( g_psPlatformPcapDesc ), BIOCIMMEDIATE, &iOn ) )
>   {
>     fprintf(stderr, "Error: %.40s: %d: Cannot enable immediate mode on interface %s\n", __func__, __LINE__, 
strerror(errno));   // TODO ideally add interface name here
>     return -1;
>   }
>
> Timing of individual incoming messages was within 500uS of the expected value.  I actually then had more problem with 
sending, as for some reason I was getting up to 900uS of jitter (as measured by clock cycles around the sending 
function).  I'm guessing this is a result of needing to understand more now of priorities etc. on QNX, as I can get 
normally better than that on Linux and Windows.  I'd hope for about 10X better than that, maybe with using processor 
affinity etc. (though on this QNX platform I only have two cores, so I was hoping QNX magic would allow me to avoid 
dedicating a core to this).
>
> In both cases, the timing limits above were exceeded about 30 times in the first 10 million 5mS message group tests, 
but nearly 40 million 5mS frames since then, all send and receive messages have been under the above limits without 
exception.  More work to do to understand why, but the original issue is solved.
>
> Summary:
>
> 1. It would be misleading to use tcpdump on 6.5 to investigate message resolution finer than 10mS, 
>
> 2. To get immediate notification of received messages in libpcap, use a zero (some examples show -1) parameter for the
 receive wait time, and use an...
View Full Message