> A mutexing issue in the pool allocator.

That sounds promising and would explain a lot.

> > and where we could get
> > the fixed code to give it a try?
> 
> This I'm not sure of these days...

Well, we've compiled io-pkt from the released source, so if we knew what to change, we could modify the source and try 
it here.

Or maybe I should ask our FAE for an update?

What would be the best course of action?

Thanks,
lew

> A mutexing issue in the pool allocator.

Sean,

We're trying to come up with a test case that would let you see this more easily, and that is proving difficult.  One 
thing that helps a lot is changing the e1000 driver to do receive processing in its own separate thread, and using 
poke_stack_pkt_q() to get another thread to become the stack (more like the io-net model).  At that point, we can make 
it fail reasonably often by directing a decent load (about 500,000 packets/sec) at the machine and then running "tcpdump
 -c1" in a loop, to cause the interface to go in/out of promiscuous mode.  If we don't use poke_stack_pkt_q(), it fails 
so rarely as to be useless.  I was thinking that this might be similar to how the shim drivers operate, so we'll try 
replicating it with a released shim driver.

Note that the problem is a lot more serious for us than going into promiscuous mode -- we have occasional random 
failures, but they are very rare, so we're just searching for some way to cause it to happen more frequently.

In any event, after tracing through the driver, the mbuf library, and the pool allocator, it appears that something in 
the pool gets corrupted while it's unallocated.  99% of the time, everything works fine.  But every now and then, 
pcg_get() returns an object which points to 0x0f320700, which is not a valid pointer (which of course leads to a SEGV). 
 Sometimes, the pointer is null, and rarely it's -1, but about 90% of the time, it's the above magic value.  We can see 
that pcg_get() is returning a corrupted object, but we also instrumented pcg_put() and nothing ever intentionally frees 
a "bad" object.

Does any of this sound familiar?

Support sent us an io-pkt-v6 to try, and it made no difference, but I don't know how to verify whether it actually 
contains the fix you mentioned previously.

Thanks,
lew

Sean,

We thought we had this fixed, but the prior (e1000 related) problem was something entirely different -- the previous 
problem was corrupting mbufs by DMAing into them at bad times.

In any event, after almost a month of trying, we found a method of reproducing a highly intermittent problem in a matter
 of minutes.  Unfortunately, the lab setup is so complex that it would not be easy to replicate for you.  But the 
problem is likely due to passing an mbuf to another thread and having that thread free it, thus exercising the pool 
allocator because each thread's local mbuf cache is either empty or full all the time.

Long story short, there is a corruption occurring inside the pool allocator.  After several days of looking into it, 
there appear to be cases in the pool allocator where the PR_PROTECT flag is not set and therefore mutex locking is not 
occurring on certain pools.

Empirically, we found that we can eliminate the problem by locking a mutex on all pools, so now we're trying to find out
 exactly which one of the pools was causing the problem.

You mentioned that a mutexing problem had been found & fixed in the pool allocator, and we were theoretically given a 
copy of io-pkt that contained the fix, but we have no way to verify that the fix was contained within.

So my question is, can you tell us EXACTLY what was changed?  We'd like to make sure that we're in sync, both with 
whatever you fixed previously, as well as our findings in this particular case.

This is a very serious problem for us, we've even held up a major release due to a lurking known problem.  We now have 
light at the end of the tunnel, but would like to discuss further so we can make sure that our findings are consistent 
with your much deeper knowledge of this code, and that the fix is made appropriately so that it can be contributed back.


Thanks,
lew