Add my vote to have an option to go polling only.  We have more cores then we can use.  Shared interrupt is adding a 
very significant overhead, we have no control over that for the moment.

> -----Original Message-----
> From: Lewis Donzis [mailto:community-noreply@qnx.com]
> Sent: Thursday, April 01, 2010 3:53 PM
> To: technology-networking
> Subject: Re: RE: High-Performance Network I/O Requirements
> 
> Andrew,
> 
> I was just about to post a related question, but this seems to be as
> good of a place as any.
> 
> We have also found that in our system, we can achieve better
> performance simply by dedicating a few CPU cores to polling for
> packets, i.e., running without interrupts at all.
> 
> As you mentioned, under heavy load, the driver essentially becomes
> polled anyway, however, there are a couple of areas where it's still
> not as good as a polled architecture.  In our world, real time response
> rate is critical, so using interrupt throttling is good for internal
> efficiency but not good for latency.  Then, the CPUs we're using are
> fast enough such that it takes a lot of network traffic to load them to
> the point that they will stay in the receive thread.  In most cases,
> there is just enough traffic to cause it to go in an out of the thread
> for nearly every packet.  The problem is that this incurs significant
> overhead in the kernel, which is always happening in CPU0, I believe,
> right?  So even though we have plenty of CPU resources available, the
> system is acting relatively sluggish.
> 
> Another problem has to do with shared interrupts.  We have many
> interfaces and not many interrupt levels available, to the point that
> when an interrupt occurs on a particular level, that may invoke four or
> five interrupt routines, and not just in the network code, the USB
> ports and so on are also getting hit with false interrupts.
> 
> Anyway, long story short, we have built an e1000 driver on io-net that
> can run either in the traditional interrupt mode or with a configurable
> number of dedicated polling threads, and the polled mode even with just
> two CPUs is already about 50% better than interrupt mode, plus the
> machine is completely responsive since we always have one or two CPUs
> doing almost nothing.
> 
> Which, finally, leads me to the real question/dilema: we can't figure
> out how to do this on io-pkt.  My knowledge of this is limited, but I
> get the impression that drivers aren't allowed to start threads, so how
> can we do something similar?
> 
> Thanks,
> lew
> 
> 
> 
> _______________________________________________
> 
> Technology
> http://community.qnx.com/sf/go/post51073
> 

> Which, finally, leads me to the real question/dilema: we can't figure out how 
> to do this on io-pkt.  My knowledge of this is limited, but I get the 
> impression that drivers aren't allowed to start threads, so how can we do 
> something similar?

In order to avoid context switches,  build your own lsm moduel wich handels all of your apllication specific tasks.  
Such a lsm modul is directly linked to the ethernet driver.
A good example of a lsm module is the lsm-raw module include in the network sources.

With 100000 packets per second you have to send every 10us one packet. IMHO that's not possible without a specialized 
ethernet hardware.  The best value we have measured was in the range of 15us.

Armin Steinhoff

http://www.steinhoff-automation.com

> it takes a lot of network traffic to load them 
> to the point that they will stay in the receive thread.  

I need to write a little wiki article on this ...

You need to change the return code of process_interrupt().
 
This is a driver function which is called by io-pkt
after an interrupt fires.  Normally all the servicing
is completed and the function returns, but by altering
the return code you can tell io-pkt that you have not
finished processing - even if you really have - and
io-pkt will call you again soon, with the interrupt
still masked!
 
You can create a little state machine with a static
variable so that your driver thread stays active longer 
after an interrupt, to straddle the interval between
packets in a burst.

The above is the elegant way of doing it.  You may not
prefer it - are you a manual transmission kind of guy?

If so, ensure that io-pkt has enough threads - one will
be created per cpu, you may wish to use the command line
options to create at *least* as many threads as interfaces - 
and in this case, you can "hog" the io-pkt thread by
not returning immediately from process_interrupt().
  
Intead what you might do, after all your processing is
complete, is loop perhaps calling SchedYield() and then 
reading your hardware register, looking for rxd packets.  

Seanb doesn't like this very much - he would prefer you return
from process_interrupt() asking to be called again - but
as long as you carefully configure your system with enough
threads for your number of cores and interfaces, spinning
in the process_interrupt() callback will work very well indeed.

If you've got cores to burn, skip the SchedYield() - that's
more for single-cpu configurations, to allow other threads at
the same priority - whom are ready to run - a chance at the cpu.

Hope this helps,

--
aboyd