On Sat, Dec 20, 2008 at 12:13:52PM -0500, Lewis Donzis wrote:
> We're trying to make sure we understand how threads are used in the io-pkt native driver model, especially as compared
 to io-net.
> 
> In io-net, we had no drivers that sent packets upstream in multiple threads from the same interface.  In other words, 
when an upstream filter received a packet, it would normally not be executing simultaneously with another thread 
processing a packet from the same interface.  Our use of multiple CPUs was essentially limited to one thread per 
physical network interface.
> 
> In io-pkt, there is a chunk of code devoted to dispatching patckets to multiple threads simultaneously and deciding 
which thread to use for an incoming packet, which looks advantageous, but may also create some challenges.

Multiple threads can be handling interrupts, but only one
thread will be servicing a particular interrupt (struct
_iopkt_inter) at a particular time.  From a driver's point
of view this should be similar to io-net.  Notice for
example that there's often no mutexing in the receive path
as opposed to the tx path which can be tickled by multiple
threads at once.

> 
> It also appears that things become essentially single-threaded once they get into the TCP/IP stack in that, if we 
understand the code correctly, there can be only one "stack thread" at a time.  If that's right, it would seem to 
eliminate the benefit of dispatching packets into multiple threads, but solves other problems.

This again is similar to io-net; the difference being that
in io-net there was a dedicated thread for "stack"
processing while in io-pkt a thread may switch from an
"interrupt processing" context to "stack" context.  In
both cases there is only ever one "stack" thread.

> 
> In our application, we like to see packets from a given source IP address in the same order they come into the machine
.  In contemplating methods of getting multiple CPUs active in io-net, we had considered allowing this, 

Allowing what exactly?

> but always dispatching packets from the same source IP to the same CPU.  In io-pkt, we don't have much control over 
how packets are dispatched to threads, and we would like to be able to take advantage of multiple CPUs, especially as we
 move toward supporting 10Gbps.  But at the same time, if locks have to be used heavily for protection, the benefit of 
multiple CPUs is reduced and in some cases the flow of execution becomes essentially single-threaded.

Currently it depends on packet flow.  Layer two processing
(eg bridging) as well as the ip forwarding "ipflow" layer is
more heavily optimised for multithreaded concurrency.
Things currently funnel into the stack thread for upper
layer processing but again this is similar to io-net. 

> 
> Getting to the point, my questions are whether these basic understandings are correct, whether there is any 
fundamental explanation of "how threads are used in io-pkt", and, more importantly, what can we expect in terms of 
threading model when processing packets from a pfil hook.

Again it depends on where the hook is located.  The ones
in ether_input() / ether_putput() can be tickled my
multiple threads at once but ones in ip_input() /
ip_output() (for example) are in the "stack" thread
context.

> 
> Our goal is to optimize performance to take advantage of multiple CPUs, but without having to lock so heavily that 
those benefits are lost.
> 
> Thanks,
> lew
> 
> _______________________________________________
> io-net migration
> http://community.qnx.com/sf/go/post19100
>