|
Oleh Derevenko(deleted)
01/28/2008 2:51 PM
post4633
|
Hello
So, this is the continuation of my old topics.
Currently I have changed my program like described below.
1) A thread puts several data blocks (A, B, C, D) into queue.
2) One or more other threads lock the serialization mutex, extract current queue content (if any), send data with
sendmsg() (if any) and then unlock the serialization mutex.
So, since the data is pushed into queue by one thread the order of blocks in it is guarranteed to be "A, B, C, D". Since
the thread that does the transmission extracts the data and executed sendmsg() inside of a mutex lock, it is
guarranteed that data is delivered to the protocol stack in the same order it appears in queue. However today I have
cought an assertion check that the data has arrived to client in swapped order: "B, C, D, A".
So, it is evident that there were at least two sendmsg() calls, first with "A" and second with "B, C, D". The first call
exited and unlocked the mutex before second thread could lock the mutex and enter sendmsg(). Nevertheless the buffers
were swapped inside of io-net.
Can I have a confirmation this scenario is possible? And more important, can I have the confirmation that even though
the data of the sequential calls might be swapped, it can't be intermixed with each other, provided the access to
sendmsg() is serialized as described above?
|
|
|
|
|
|
Sean Boudreau(deleted)
01/28/2008 3:57 PM
post4636
|
On Mon, Jan 28, 2008 at 02:51:59PM -0500, Oleh Derevenko wrote:
> Hello
>
> So, this is the continuation of my old topics.
>
> Currently I have changed my program like described below.
>
> 1) A thread puts several data blocks (A, B, C, D) into queue.
> 2) One or more other threads lock the serialization mutex, extract current queue content (if any), send data with
sendmsg() (if any) and then unlock the serialization mutex.
>
> So, since the data is pushed into queue by one thread the order of blocks in it is guarranteed to be "A, B, C, D".
Since the thread that does the transmission extracts the data and executed sendmsg() inside of a mutex lock, it is
guarranteed that data is delivered to the protocol stack in the same order it appears in queue. However today I have
cought an assertion check that the data has arrived to client in swapped order: "B, C, D, A".
>
> So, it is evident that there were at least two sendmsg() calls, first with "A" and second with "B, C, D". The first
call exited and unlocked the mutex before second thread could lock the mutex and enter sendmsg(). Nevertheless the
buffers were swapped inside of io-net.
>
> Can I have a confirmation this scenario is possible? And more important, can I have the confirmation that even though
the data of the sequential calls might be swapped, it can't be intermixed with each other, provided the access to
sendmsg() is serialized as described above?
>
If this is in fact happening as described it's a bug. I've
never heard of anything like this. Is it possible to get a
test case?
-seanb
|
|
|
|
|
|
Oleh Derevenko(deleted)
01/29/2008 4:29 AM
post4639
|
> > Can I have a confirmation this scenario is possible? And more important, can
> I have the confirmation that even though the data of the sequential calls
> might be swapped, it can't be intermixed with each other, provided the access
> to sendmsg() is serialized as described above?
> >
>
> If this is in fact happening as described it's a bug. I've
> never heard of anything like this. Is it possible to get a
> test case?
OK. I can try to prepare one. However I'm not sure if I'll be able to reproduce it with a test application. I remember
how I tried to reproduce data intermixing and communication went fine from multiple parallel threads for several hours
while in real application it was easily reproducible on specific operation.
|
|
|
|
|
|
Sean Boudreau(deleted)
01/29/2008 8:10 AM
post4641
|
On Tue, Jan 29, 2008 at 04:29:25AM -0500, Oleh Derevenko wrote:
> > > Can I have a confirmation this scenario is possible? And more important, can
> > I have the confirmation that even though the data of the sequential calls
> > might be swapped, it can't be intermixed with each other, provided the access
> > to sendmsg() is serialized as described above?
> > >
> >
> > If this is in fact happening as described it's a bug. I've
> > never heard of anything like this. Is it possible to get a
> > test case?
>
> OK. I can try to prepare one. However I'm not sure if I'll be able to reproduce it with a test application. I remember
how I tried to reproduce data intermixing and communication went fine from multiple parallel threads for several hours
while in real application it was easily reproducible on specific operation.
>
I'd start by taking a close look at your application. What
you're describing here from the stack's point of view after
all the mutexing and queueing falls out seems to be:
for (;;)
send(socket, buf, sizeof(buf), 0);
I'm pretty sure the above doesn't have any re-arranging
issues.
Make sure you're reading() / writing() in a loop, checking
the return codes until the entire record is received / sent.
Make sure you're checking the return codes from
pthread_mutex_*(). If it's a static mutex, make sure it's
initialized with PTHREAD_MUTEX_INITIALIZER: one that isn't
will appear to work until there's actual contention.
-seanb
|
|
|
|
|
|
Oleh Derevenko(deleted)
01/29/2008 8:20 AM
post4642
|
> I'd start by taking a close look at your application. What
> you're describing here from the stack's point of view after
> all the mutexing and queueing falls out seems to be:
>
> for (;;)
> send(socket, buf, sizeof(buf), 0);
>
> I'm pretty sure the above doesn't have any re-arranging
> issues.
Yes, I know that it is just a sequential invocation of send(). But there is one peculiarity. Each time send() can be
invoked with different priority. And that creates the possibility for buffer swapping.
I'm pretty sure about my code. I'll try to create a test case.
|
|
|
|
|
|
Oleh Derevenko(deleted)
02/04/2008 3:00 PM
post4782
|
> Yes, I know that it is just a sequential invocation of send(). But there is
> one peculiarity. Each time send() can be invoked with different priority. And
> that creates the possibility for buffer swapping.
>
> I'm pretty sure about my code. I'll try to create a test case.
So I tried to make a test application but had no luck with reproducing the problem with it yet :(. I tried to make it
similar to real application but it is very difficult to recreate executing environment exactly.
Anyway, on sender side the code is quite clear and I do not see any possibility for data block swapping in my code. Also
on recepient (a windows app) each socket is processed by a dedicated thread and all the stream structure checks pass
with exception that sometimes the data blocks arrive in wrong order.
For now I have made a workaround in my code. I reassemble the blocks to restore correct order at client. Maybe later, if
I'll have some free time I'll try to dump data block structure at both sides and see if I will be able to provide an
example of data swapping.
|
|
|
|
|
|
Oleh Derevenko(deleted)
02/19/2008 8:07 AM
post5110
|
> > Yes, I know that it is just a sequential invocation of send(). But there is
>
> > one peculiarity. Each time send() can be invoked with different priority.
> And
> > that creates the possibility for buffer swapping.
> >
> > I'm pretty sure about my code. I'll try to create a test case.
>
> So I tried to make a test application but had no luck with reproducing the
> problem with it yet :(.
This was my bug. Sorry for false alert. I sent data in wrong order indeed.
|
|
|
|
|
|
