Rob Craig wrote:

> "something" is happening on the network that causes the 
> links to increase to max capacity

Network loop?  Broadcast storm?  I've caused a few of these
here in my time  :-)

Wireshark/ethereal might help.

--
aboyd

Thanx I will try it out.  I forgot to mention I am currently testing on
an 1.1 gHz box so definitely not fast by x86 standards.  The normal
system I work with is a core 2 duo 1.5 gHz.  I've seen the symptoms on
both machines.

I'll let you know what happens.

Thanx
Tim

-----Original Message-----
From: Robert Craig [mailto:rcraig@qnx.com] 
Sent: Wednesday, July 02, 2008 10:47 AM
To: technology-networking
Subject: RE: io-net comsumes all cpu time

Hi Tim:

	I can really only see two possibilities at this point.  1)
There's a bug in the driver 2) "something" is happening on the network
that causes the linksto increase to max capacity.  With 100Mbps links, I
wouldn't expect even two links to consume all CPU if you've got a fast
CPU.  I'd guess that what could be happening is that the one of receive
threads on the drivers is getting put into short term CPU starvation by
the other thread and "something bad" is happening as the driver deals
with the queuing of the packets.  

Dave Bott pointed out to me that some fixes have been checked into the
driver since it was last released that may be related.  I've attached a
private "experimental" x86 build from my sandbox (that I haven't tested
since I don't have the hardware...  notice all the provisos I put in
here :->) that would be worthwhile trying out.  

Let me know if it changes the behaviour at all...


   Robert.

-----Original Message-----
From: Tim Gessner [mailto:tim@deltacompsys.com]
Sent: Wednesday, July 02, 2008 12:48 PM
To: technology-networking
Subject: RE: io-net comsumes all cpu time

Io-net is started by diskboot (using the x86 runtime kit) though I see
the same results when I start it with io-net -drtl -ptcpip.  I have not
tried two instances of io-net yet.

I will have to get some new cards to try that.  I assume that you are
looking for different chip sets?

The links are currently both 100m though I have seen this with gigabit
connections on a different machine (which I don't have access to at the
moment).

I run hogs -n -%5 with pid 1 and io-net showing up until only io-net
appears.  My applications never show (that is never consume > 5%).  They
can do some communication using QNX messages MsgSend/MsgReceive but the
traffic is little if any most of the time.

There is no memory increase.

Tim

-----Original Message-----
From: Robert Craig [mailto:rcraig@qnx.com]
Sent: Wednesday, July 02, 2008 9:21 AM
To: technology-networking
Subject: RE: io-net comsumes all cpu time

Hi Tim:

	How is io-net started (command line options)?  Is it possible to
put in different network cards and see if the same sort of behaviour is
seen?  What speed of CPU are you using?  Are the links gigabit or 100
Mbps?  How does the CPU usage of your individual applications compare?
Are they interacting with each other in any way (e.g. through socket
calls)?  If you do a pidin -p io-net mem does the memory usage increase
with both apps running?

	(Lots of questions...)


	Robert.

-----Original Message-----
From: Tim Gessner [mailto:tim@deltacompsys.com]
Sent: Wednesday, July 02, 2008 12:15 PM
To: technology-networking
Subject: io-net comsumes all cpu time

I am running 6.3.2 on a non-smp x86 box with 2 network cards both
RealTek 8139 chips each on a different subnet.

I have two processes each with multiple threads.  Process 1 uses the
first network interface and process 2 uses the second network interface.
Both processes are reading and writing data over their respective
interfaces fairly heavily.  

Running just one process (either one) io-net never consumes even 5% of
cpu time.  Running both processes io-net consumes more and more over
about 15 minutes until the system quits responding.  

Any thoughts?  Is this a problem with io-net or with the driver or ???
Next steps to debug the...

I tried the new driver, but had the same problem.

Thanx
Tim

-----Original Message-----
From: Robert Craig [mailto:rcraig@qnx.com] 
Sent: Wednesday, July 02, 2008 10:47 AM
To: technology-networking
Subject: RE: io-net comsumes all cpu time

Hi Tim:

	I can really only see two possibilities at this point.  1)
There's a bug in the driver 2) "something" is happening on the network
that causes the linksto increase to max capacity.  With 100Mbps links, I
wouldn't expect even two links to consume all CPU if you've got a fast
CPU.  I'd guess that what could be happening is that the one of receive
threads on the drivers is getting put into short term CPU starvation by
the other thread and "something bad" is happening as the driver deals
with the queuing of the packets.  

Dave Bott pointed out to me that some fixes have been checked into the
driver since it was last released that may be related.  I've attached a
private "experimental" x86 build from my sandbox (that I haven't tested
since I don't have the hardware...  notice all the provisos I put in
here :->) that would be worthwhile trying out.  

Let me know if it changes the behaviour at all...


   Robert.

-----Original Message-----
From: Tim Gessner [mailto:tim@deltacompsys.com]
Sent: Wednesday, July 02, 2008 12:48 PM
To: technology-networking
Subject: RE: io-net comsumes all cpu time

Io-net is started by diskboot (using the x86 runtime kit) though I see
the same results when I start it with io-net -drtl -ptcpip.  I have not
tried two instances of io-net yet.

I will have to get some new cards to try that.  I assume that you are
looking for different chip sets?

The links are currently both 100m though I have seen this with gigabit
connections on a different machine (which I don't have access to at the
moment).

I run hogs -n -%5 with pid 1 and io-net showing up until only io-net
appears.  My applications never show (that is never consume > 5%).  They
can do some communication using QNX messages MsgSend/MsgReceive but the
traffic is little if any most of the time.

There is no memory increase.

Tim

-----Original Message-----
From: Robert Craig [mailto:rcraig@qnx.com]
Sent: Wednesday, July 02, 2008 9:21 AM
To: technology-networking
Subject: RE: io-net comsumes all cpu time

Hi Tim:

	How is io-net started (command line options)?  Is it possible to
put in different network cards and see if the same sort of behaviour is
seen?  What speed of CPU are you using?  Are the links gigabit or 100
Mbps?  How does the CPU usage of your individual applications compare?
Are they interacting with each other in any way (e.g. through socket
calls)?  If you do a pidin -p io-net mem does the memory usage increase
with both apps running?

	(Lots of questions...)


	Robert.

-----Original Message-----
From: Tim Gessner [mailto:tim@deltacompsys.com]
Sent: Wednesday, July 02, 2008 12:15 PM
To: technology-networking
Subject: io-net comsumes all cpu time

I am running 6.3.2 on a non-smp x86 box with 2 network cards both
RealTek 8139 chips each on a different subnet.

I have two processes each with multiple threads.  Process 1 uses the
first network interface and process 2 uses the second network interface.
Both processes are reading and writing data over their respective
interfaces fairly heavily.  

Running just one process (either one) io-net never consumes even 5% of
cpu time.  Running both processes io-net consumes more and more over
about 15 minutes until the system quits responding.  

Any thoughts?  Is this a problem with io-net or with the driver or ???
Next steps to debug the...

This comes from a bug in our libc implementation which will be released
with 6.4.0.  Not sure that I understand the question about the notice
(?)

This would be one of those areas where going through the standard
support channels to get a patched libc may be the way to go...

	Robert.

-----Original Message-----
From: Tim Gessner [mailto:tim@deltacompsys.com] 
Sent: Wednesday, July 02, 2008 5:42 PM
To: technology-networking
Subject: RE: io-net comsumes all cpu time

Yes, I do.  I assume you mean with a NULL timeval struct for the timeout
value.  I don't create a timeval with a 0 timeout, but I do call
select() with a NULL timeout param.

Where are you getting this notice?

Thanx
Tim

-----Original Message-----
From: Robert Craig [mailto:rcraig@qnx.com] 
Sent: Wednesday, July 02, 2008 2:39 PM
To: technology-networking
Subject: RE: io-net comsumes all cpu time

You know, Jeevan, that's a very good point.  This one keeps popping up
in reference to io-net CPU usage growing over time.   

Tim: You aren't calling select anywhere with a zero timeout anywhere in
your application are you?

Our PR for this says:
"select()/poll() with a timeout setting of 0 causes delays/ memory leak"


	Robert.

-----Original Message-----
From: Jeevan Mathew [mailto:jmathew@t-online.de] 
Sent: Wednesday, July 02, 2008 5:15 PM
To: technology-networking
Subject: Re: io-net comsumes all cpu time

> I am running 6.3.2 on a non-smp x86 box with 2 network cards both
RealTek 8139
>  chips each on a different subnet.
> 
> I have two processes each with multiple threads.  Process 1 uses the
first 
> network interface and process 2 uses the second network interface.
Both 
> processes are reading and writing data over their respective
interfaces fairly
>  heavily.  
> 
> Running just one process (either one) io-net never consumes even 5% of
cpu 
> time.  Running both processes io-net consumes more and more over about
15 
> minutes until the system quits responding.  
> 
> Any thoughts?  Is this a problem with io-net or with the driver or ???
Next 
> steps to debug the problem?
> 
> Thanx
> Tim

Hi Tim,

Let me bet that this won't occure if You use Trinity M7 respectively
Trinity M6.

Can You try this ?
I believe this is a libc bug regarding select().

M6 or M7 You can find here:

http://community.qnx.com/sf/sfmain/do/home

Ofcourse I might be completely wrong. :-)

Hope this helps.
Jeevan




_______________________________________________
Technology
http://community.qnx.com/sf/go/post9964


_______________________________________________
Technology
http://community.qnx.com/sf/go/post9965


_______________________________________________
Technology
http://community.qnx.com/sf/go/post9966

On Wed, Jul 02, 2008 at 05:42:27PM -0400, Tim Gessner wrote:
> Yes, I do.  I assume you mean with a NULL timeval struct for the timeout
> value.  I don't create a timeval with a 0 timeout, but I do call
> select() with a NULL timeout param.
> 
> Where are you getting this notice?
> 
> Thanx
> Tim

The bug in question was with a 0 timeout (a non blocking
select()) and IIRC the CPU got billed to the client not
io-pkt / io-net.

I think a test case may be required.

-seanb