On Thu, Nov 29, 2007 at 11:33:51AM -0500, Oleh Derevenko wrote:
> > The handling of the MsgSends are serialized in the stack.
> > This situatation arises when the send buffer fills up and
> > the send(), write(), sendto(), sendmsg() in the client has
> > to block.  When the send buffer drains we can continue
> > processing requests on this particular socket.  Since tcp
> > is a stream which request should be processed first?  Since
> > sockets have no concept like PIPE_BUF (that I could find)
> > it seems logical that the highest priority request should
> > win.
> 
> It may be logical from scheduler's point of view but it is completely
> senseless from point of view of socket's functionality. Client does not
> have any possibility of parsing data successfully after preemption like
> that and this makes socket inapplicable for multithreaded use. Even
> though there may not be a concept like PIPE_BUF for the socket its
> functionality should be "user friendly". Who wins from strict adherence
> to priority rules if the data is corrupted as a result? Can you show me
> at least one use case when there would be a benefit for the client from
> inserting unrelated data inside its data block?
> 
> > I'm pretty confident that the changes for this issue wouldn't
> > in themselves introduce an issue like this.
> 
> Well, I'll try to find out what is the state of threads when the
> connection is going down. However I'm running 6.3.0 SP3. I can't ruin my
> development/testing environment by upgrading it to 6.3.2. I can try
> September 6.3.2 release at one node at most. And I can't use the latest
> M2 build at all because kernel crashes after my application is started
> even though it is a pure user-mode application without any privileged
> access to ports or hardware.
> 

The fixes for this were local to npm-tcpip.so so you
shouldn't need to do anything except run the latest stack
and get rid of your mutex.

-seanb

On Thu, Nov 29, 2007 at 12:03:18PM -0500, Oleh Derevenko wrote:
> > The handling of the MsgSends are serialized in the stack.
> > This situatation arises when the send buffer fills up and
> > the send(), write(), sendto(), sendmsg() in the client has
> > to block.  When the send buffer drains we can continue
> > processing requests on this particular socket.  Since tcp
> > is a stream which request should be processed first?  Since
> > sockets have no concept like PIPE_BUF (that I could find)
> > it seems logical that the highest priority request should
> > win.
> 
> And also, please consider that the situation with send buffer overflow
> is just an implementation limitation. In theory, send buffer should be
> considered infinite and every new thread should just queue its data at
> the end. So, if the operation that should normally be atomic, is
> suspended because of physical limitations it should have priority for
> resume. All the rest threads which have not started their operations yet
> can be judged by their priority (you need not preserve thread acceptance
> order for data) but you must not discriminate a thread depending on its
> luck to arrive to empty buffer or to the buffer which is nearly full.

SO_SNDBUF is a documented, real limitation.  Currently the
requirements for not having data interleaved when multiple
threads write simultaneously on the same stream socket is to
run them at the same priority.  Note even this might not be
portable as it is ascribing undocumented characteristics to
stream sockets. I don't think this is a bug.  If you can
point out some spec, prior art or example (that doesn't work
by accident) that I may have missed I'll be happy to
reevaluate.

Regards,

-seanb

On Thu, Nov 29, 2007 at 12:12:14PM -0500, Oleh Derevenko wrote:
> > > > Your new issue sounds like you may be exhausting the number
> > > > of 'threads' in the stack.  Are thrads send blocked on
> > > > io-net in this situation?  Is there a sloginfo entry to this
> > > > effect (must be running the latest patch)?  You can increase
> > > > the number of stack 'threads' as follows:
> > > > 
> > > > # io-net -ptcpip threads_max=400
> > > 
> > > It was about 18 months ago and I did not check the state of threads.
> > > Documentation says there is 200 threads limit by default. My
> inspections
> > > of sender process showed 120-130 threads running. And also the last
> but
> > > not least, we had never seen that "denial of service" problem before
> > > patch even though we were running without send() serialized for
> quite a
> > > long time before we discovered it may intermix the data.
> > > 
> > > I can remove mutex and make some experiments in next few days to see
> > > what is the state of threads if you would like.
> > 
> > I'm pretty confident that the changes for this issue wouldn't
> > in themselves introduce an issue like this.
> 
> Well, if thread poll maximum is for whole system (and it is, since there
> is only one io-net process and I would not believe there could be a
> personal thread pool for every socket :) ), I can assume possibility to
> have more than 200 threads in several processes together. But 10 seconds
> is nearly the infinity! Several threads over 200 can't create a delay
> like that considering the fact that all of them send just one data
> envelope and block afterwards.
> 

The 'threads_max' argument to the stack controls how many
blocking operations the stack can service simultaneously.
They may be read(), write(), accept() ...

It's just an educated guess at this point.  Look for clients
SEND blocked on io-net and the aforementioned sloginfo
entry.

Thanks,

-seanb