Networking/Discussion/Technology/MsgSendPulse blocking/List of Posts
 
Forum Topic - MsgSendPulse blocking: (11 Items)
   
Update
Mario Charest
06/15/2009 1:24 PM
post31767
MsgSendPulse blocking   
On 6.3.2, trying to chase a weird delay in our software, we manage to get a capture of what is going on.

A process is doing a MsgSendPulse

- MsgSendPulse Enter 10.10.369
- Send Pulse 10.10.370
- Net Send 10.10.383

-- Here is idle is running on both cores

- Ready 11.886.439
- Running 11.886.602
- Ready 11.886.605
- Running 11.886.611
- MsgSendPulse Exit(0) 11.886.613

Notice how MsgSendPulse blocks for over 1.7 secondes.  Yet the doc says MsgSendPulse is non blocking even when going 
over a network. When this happens 3 other processes show the same behaviors, at the same time.  

I suspect we have a problem with the network driver / hardware. This is the first time we use the speedo driver. When 
changing the NIC to Iintel pro and e1000 the problem went away. 

My question is why is MsgSendPulse blocking?  I can understand that it woulc block until the pulse made it into qnet of 
the destination but that is not what the documentation says.
Andrew Boyd(deleted)
06/15/2009 1:49 PM
post31772
RE: MsgSendPulse blocking   
> I suspect we have a problem with the network 
> driver / hardware. This is the first time we 
> use the speedo driver. When changing the NIC 
> to Iintel pro and e1000 the problem went away. 

How old is your speedo driver?  I seem to recall
there was an unpleasant delay fixed in the devctl
which the shelf periodically called.

--
aboyd
Andrew Boyd(deleted)
06/15/2009 1:57 PM
post31774
Re: RE: MsgSendPulse blocking   
Whoops.  Not sure this has anything to do with
your problem, but it looks like I fixed that stats/delay
ONLY in the io-net driver, NOT in the io-pkt driver,
which clearly was branched off before then.

If you're running 6.4 or 6.4.1, try specifying the
io-net speedo driver instead of the io-pkt version - 
does that make any difference?
Mario Charest
06/15/2009 2:08 PM
post31778
Re: RE: MsgSendPulse blocking   
> Whoops.  Not sure this has anything to do with
> your problem, but it looks like I fixed that stats/delay
> ONLY in the io-net driver, NOT in the io-pkt driver,
> which clearly was branched off before then.
> 
> If you're running 6.4 or 6.4.1, try specifying the
> io-net speedo driver instead of the io-pkt version - 
> does that make any difference?


Over the week-end we try 6.4.1, out of desparation because we aren't 100% ready to deploy 6.4.1 in the field yet. It was
 worse, ifconfig would block forever (at least 30sec) when trying to configure the IP address. This was with io-pkt 
version.
Andrew Boyd(deleted)
06/15/2009 2:17 PM
post31780
RE: RE: MsgSendPulse blocking   
> we try 6.4.1 ... It was worse, ifconfig would block forever 
> (at least 30sec) when trying to configure the IP address. 

I think you probably mean DHCP, not ifconfig?  DHCP has had
problems with some drivers (not the speedo, that I can recall)
that report the link is up, but DHCP times out quickly - it's
first two attempts fail - and then it exponentially backs off
a loooooong time.  I think Hugh fixed something in the rtl8169
driver concerning this.

Anyways, like I said, try the io-net version of the speedo
driver by specifying the following to io-pkt:

  -d /lib/dll/devn-speedo.so

Any difference?

--
aboyd
Mario Charest
06/15/2009 2:55 PM
post31783
RE: RE: MsgSendPulse blocking   

> -----Original Message-----
> From: Andrew Boyd [mailto:community-noreply@qnx.com]
> Sent: June-15-09 2:18 PM
> To: technology-networking
> Subject: RE: RE: MsgSendPulse blocking
> 
> 
> > we try 6.4.1 ... It was worse, ifconfig would block forever
> > (at least 30sec) when trying to configure the IP address.
> 
> I think you probably mean DHCP, not ifconfig? 

No I meant ifconfig. This is a custom runtime image and IP addresses are static.

> DHCP has had
> problems with some drivers (not the speedo, that I can recall)
> that report the link is up, but DHCP times out quickly - it's
> first two attempts fail - and then it exponentially backs off
> a loooooong time.  I think Hugh fixed something in the rtl8169
> driver concerning this.
> 
> Anyways, like I said, try the io-net version of the speedo
> driver by specifying the following to io-pkt:
> 
>   -d /lib/dll/devn-speedo.so
> 
> Any difference?

The problem is happening in the field. I've send an updated image to our guy over there and will await feedback.
> 
> --
> aboyd
> 
> 
> _______________________________________________
> Technology
> http://community.qnx.com/sf/go/post31780
>
Mario Charest
06/15/2009 2:02 PM
post31775
Re: RE: MsgSendPulse blocking   
> 
> > I suspect we have a problem with the network 
> > driver / hardware. This is the first time we 
> > use the speedo driver. When changing the NIC 
> > to Iintel pro and e1000 the problem went away. 
> 
> How old is your speedo driver?  I seem to recall
> there was an unpleasant delay fixed in the devctl
> which the shelf periodically called.
> 
> --
> aboyd

NAME=devn-speedo.so
DESCRIPTION=Intel 82557/8/9 ethernet driver
DATE=2004/04/30-17:55:55-EDT
STATE=stable
HOST=hwbuild
USER=hwbuild
VERSION=6.3.0
TAGID=193

Over the week-end I looked on foundry27 bug couldn't find any updates.
Andrew Boyd(deleted)
06/15/2009 2:07 PM
post31777
RE: RE: MsgSendPulse blocking   
> Over the week-end I looked on foundry27 
> couldn't find any updates.

The fix to io-pkt speedo devctl.c speedo_get_stats()
is to replace the

  delay(10);

with

  for (I=0; I<1000000; I++) {
    if (ENDIAN_LE32(dstats->completion)) {
      break;		
    }
    SchedYield();  // testing shows we never get here - I is zero
  }

As I said, I'm not sure that's your problem with the speedo
driver, but it has caused problems in the past.

--
aboyd
Xiaodan Tang(deleted)
06/15/2009 2:12 PM
post31779
RE: MsgSendPulse blocking   

> -----Original Message-----
> From: Mario Charest [mailto:community-noreply@qnx.com]
> Sent: June 15, 2009 1:25 PM
> To: technology-networking
> Subject: MsgSendPulse blocking
> 
> My question is why is MsgSendPulse blocking?  I can understand that it
> woulc block until the pulse made it into qnet of the destination but
that
> is not what the documentation says.

You are correct.

The calling thread would put into "NET_SEND" block, the pulse delivered
to local QNET, then transmit to remote QNET. The Remote QNET will do a
MsgSendPulse() on behalf of the caller, and the return value will
transmit back, the calling thread will then unblocked.

The document is not matching in this case.

It is debatable that why sending pulse cross network need such a heavy
delay. But we have rare cases that calling thread is rely on the return
value of MsgSendPulse() to determine if the destination channel is exist
... :(

-xtang
Steve Reid
06/16/2009 10:45 AM
post31862
RE: MsgSendPulse blocking   
I've created a PR for the docs.


Steve Reid (stever@qnx.com)
Technical Editor
QNX Software Systems
Mario Charest
06/16/2009 10:58 AM
post31868
RE: MsgSendPulse blocking   

> 
> It is debatable that why sending pulse cross network need such a heavy
> delay. But we have rare cases that calling thread is rely on the return
> value of MsgSendPulse() to determine if the destination channel is
> exist

Maybe there should be some way to optionally make MsgSendPulse non blocking in all cases?

> ... :(
> 
> -xtang
> 
> _______________________________________________
> Technology
> http://community.qnx.com/sf/go/post31779
>

Return