Project Home
Project Home
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - How to find out if that's a HW fault?: Page 1 of 2 (43 Items)
   
How to find out if that's a HW fault?  
Hi

In one of customer installations on 6.3.0SP3 after some time EN1 interface drops offline (not actually offline, but 
loses all communications) while EN0 remains still operational. After rebooting everything works well again. Both 
interfaces are integrated on motherboard and are handled by a single io-net process. Is there a way to tell for sure 
what's happening there (if it's a hardware or software fault)?

The machine is kept running, so that I'll be able to request for additional info on Monday if necessary. Here's 
something to start with.

--------------------------
The symptoms are the same before: EN1 stops responding to both TCP/IP and QNET.  EN0 is still active and functioning.  I
 was able to grab the following info for you:

# nicinfo /dev/io-net/en1
INTEL 82544 Gigabit (Copper) Ethernet Controller
 
  Physical Node ID ........................... 003048 B037CB
  Current Physical Node ID ................... 003048 B037CB
  Current Operation Rate ..................... 100.00 Mb/s full-duplex
  Active Interface Type ...................... MII
    Active PHY address ....................... 0
  Maximum Transmittable data Unit ............ 1514
  Maximum Receivable data Unit ............... 0
  Hardware Interrupt ......................... 0xb
  Memory Aperture ............................ 0xd0200000 - 0xd021ffff
  Promiscuous Mode ........................... Off
  Multicast Support .......................... Enabled
 
  Packets Transmitted OK ..................... 16489506
  Bytes Transmitted OK ....................... 1550295660
  Broadcast Packets Transmitted OK ........... 27233
  Multicast Packets Transmitted OK ........... 0
  Memory Allocation Failures on Transmit ..... 0
 
  Packets Received OK ........................ 16603917
  Bytes Received OK .......................... 1545543297
  Broadcast Packets Received OK .............. 57710
  Multicast Packets Received OK .............. 0
  Memory Allocation Failures on Receive ...... 0
 
  Single Collisions on Transmit .............. 0
  Deferred Transmits ......................... 7
  Late Collision on Transmit errors .......... 0
  Transmits aborted (excessive collisions) ... 0
  No Carrier on Transmit ..................... 0
  Receive Alignment errors ................... 0
  Received packets with CRC errors ........... 0
  Packets Dropped on receive ................. 11905
#
 
# /sbin/ifconfig
lo0: flags=8009<UP,LOOPBACK,MULTICAST> mtu 33212
        capabilities=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
        enabled=0<>
        inet 127.0.0.1 netmask 0xff000000
en0: flags=8c43<UP,BROADCAST,RUNNING,OACTIVE,SIMPLEX,MULTICAST> mtu 1500
        capabilities=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
        enabled=0<>
        address: 00:30:48:b0:37:ca
        inet 192.168.1.70 netmask 0xffffff00 broadcast 192.168.1.255
en1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        capabilities=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
        enabled=0<>
        address: 00:30:48:b0:37:cb
        inet 192.168.10.44 netmask 0xffffff00 broadcast 192.168.10.255
#
 
# pci
 
PCI version    = 3.00
 
Class          = Display (VGA)
Vendor ID      = 8086h, Intel Corporation
Device ID      = 2972h,  Integrated Graphics Controller
PCI index      = 0h
PCI Mem Address = d0000000h enabled
PCI Mem Address = c0000000h enabled
PCI IO Address  = 30c0h enabled
PCI Int Pin    = INT A
Interrupt line = 10
CPU Interrupt  = ah
 
Class          = Mass Storage (IDE)
Vendor ID      = 8086h, Intel Corporation
Device ID      = 27c0h,  82801GB/GR/GH (ICH7 Family) Serial ATA Storage Controller IDE
PCI index      = 0h
PCI IO Address  = 0h enabled
PCI IO Address  = 0h enabled
PCI IO Address  = 0h enabled
PCI IO Address  = 0h enabled
PCI IO Address  = 30b0h enabled
PCI Mem Address = ffeffc00h enabled
PCI Int Pin    = INT B
Interrupt line = no connection
 
Class          = Network...
View Full Message
Re: How to find out if that's a HW fault?  
As an update I would like to say that this seems to be a driver fault. Today we tried it on backup hardware and EN1 
still dies.

Here is some io-net data as well.

# pidin -p 77835 mem
     pid tid name               prio STATE           code  data         stack
   77835   1 sbin/io-net         10o SIGWAITINFO       64K 5964K  8192(516K)*
   77835   2 sbin/io-net         10o RECEIVE           64K 5964K  4096(132K)
   77835   3 sbin/io-net         10o RECEIVE           64K 5964K   4096(68K)
   77835   5 sbin/io-net         10o RECEIVE           64K 5964K   4096(68K)
   77835   6 sbin/io-net         21o RECEIVE           64K 5964K  4096(132K)
   77835   7 sbin/io-net         21o RECEIVE           64K 5964K  4096(132K)
   77835   8 sbin/io-net         20o RECEIVE           64K 5964K  4096(132K)
   77835   9 sbin/io-net         20o RECEIVE           64K 5964K  4096(132K)
   77835  11 sbin/io-net         10o RECEIVE           64K 5964K   4096(68K)
   77835  12 sbin/io-net         10o RECEIVE           64K 5964K   4096(68K)
            ldqnx.so.2         @b0300000             348K   20K
            npm-tcpip.so       @b8200000             276K   32K
            devn-i82544.so     @b824d000              36K  4096
            npm-qnet.so        @b8257000             160K   36K
            /dev/mem           @40100000 (d0100000)        128K
            /dev/mem           @40120000 (d0200000)        128K
#

Io-net is started by hardware autodetect scripts and then qnet is mounted in with custom parameters. These parameters 
can vary depending on configuration settings available to user. The defaults (which I believe are used there) are

periodic_ticks=50,conn_est_timeout=10,conn_est_retries=1,conn_up_idle=10,conn_up_retries=10,tx_ticks=1,tx_retries=200,
do_crc=1,res_retries=0,probe_no_l4_time=500,slow_mode=12000,auto_add=1500

Also, here is binaries identification information

# ls -l /lib/dll/npm-tcpip.so
lrwxrwxrwx  1 root      root             15 Jan 27 01:16 /lib/dll/npm-tcpip.so -> npm-tcpip-v4.so
# ls -l /lib/dll/npm-tcpip-v4.so
-rwxrwxr-x  1 root      root         299788 Jun 05  2006 /lib/dll/npm-tcpip-v4.so
# cksum /lib/dll/npm-tcpip-v4.so
 945232789     299788 /lib/dll/npm-tcpip-v4.so
# ls -l /lib/dll/npm-qnet.so
lrwxrwxrwx  1 root      root             19 Jan 27 01:16 /lib/dll/npm-qnet.so -> npm-qnet-l4_lite.so
# ls -l /lib/dll/npm-qnet-l4_lite.so
-rwxrwxr-x  1 root      root         175620 Jun 21  2006 /lib/dll/npm-qnet-l4_lite.so
# cksum /lib/dll/npm-qnet-l4_lite.so
 283215240     175620 /lib/dll/npm-qnet-l4_lite.so
# ls -l /lib/dll/devn-i82544.so
-rwxrwxr-x  1 root      root          39420 Aug 30  2007 /lib/dll/devn-i82544.so
# cksum /lib/dll/devn-i82544.so
3741020423      39420 /lib/dll/devn-i82544.so
#

I would be grateful for any suggestions.


> In one of customer installations on 6.3.0SP3 after some time EN1 interface 
> drops offline (not actually offline, but loses all communications) while EN0 
> remains still operational. After rebooting everything works well again. Both 
> interfaces are integrated on motherboard and are handled by a single io-net 
> process. Is there a way to tell for sure what's happening there (if it's a 
> hardware or software fault)?
> 
> The machine is kept running, so that I'll be able to request for additional 
> info on Monday if necessary. Here's something to start with.
> 
> --------------------------
> The symptoms are the same before: EN1 stops responding to both TCP/IP and QNET
> .  EN0 is still active and functioning.  I was able to grab the following info
>  for you:
> 
> # nicinfo /dev/io-net/en1
> INTEL 82544 Gigabit (Copper) Ethernet Controller
>  
>   Physical Node ID ........................... 003048 B037CB
>   Current Physical Node ID ................... 003048 B037CB
>   Current Operation Rate ..................... 100.00 Mb/s...
View Full Message
RE: How to find out if that's a HW fault?  
After considerable effort on Hugh Brown's part, intel
discovered that some i82544 variants have a hardware
bug where they can lose interrupts during certain
race conditions.  Oops.

Later versions of the i82544 drivers (both io-net
and io-pkt variants) have fixes to work around this 
bug.

What is your DID?

--
aboyd
RE: How to find out if that's a HW fault?  

> -----Original Message-----
> From: Andrew Boyd [mailto:community-noreply@qnx.com]
> Sent: May-04-09 9:07 AM
> To: general-networking
> Subject: RE: How to find out if that's a HW fault?
> 
> 
> After considerable effort on Hugh Brown's part, intel
> discovered that some i82544 variants have a hardware
> bug where they can lose interrupts during certain
> race conditions.  Oops.
> 
> Later versions of the i82544 drivers (both io-net
> and io-pkt variants) have fixes to work around this
> bug.
> 


Please define later version.

> What is your DID?
> 
> --
> aboyd
> 
> 
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post28597
> 
RE: How to find out if that's a HW fault?  
> Please define later version.

For io-pkt, see r746 on 2009-01-16

--
aboyd
RE: How to find out if that's a HW fault?  
I was told a while ago that the e1000 driver would be available on 6.4.?  Now in M4 I noticed there is a devn-i82544.so 
and a devnp-i82544.so.  Why the two flavour? 

> -----Original Message-----
> From: Andrew Boyd [mailto:community-noreply@qnx.com]
> Sent: May-04-09 9:12 AM
> To: general-networking
> Subject: RE: How to find out if that's a HW fault?
> 
> 
> > Please define later version.
> 
> For io-pkt, see r746 on 2009-01-16
> 
> --
> aboyd
> 
> 
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post28600
> 
RE: How to find out if that's a HW fault?  
The 2 drivers are there for users still using the i82544 driver.
Eventually the i82544 driver will be dropped and the just the e1000
driver included.


-----Original Message-----
From: Mario Charest [mailto:community-noreply@qnx.com] 
Sent: Monday, May 04, 2009 9:17 AM
To: general-networking
Subject: RE: How to find out if that's a HW fault?

I was told a while ago that the e1000 driver would be available on 6.4.?
Now in M4 I noticed there is a devn-i82544.so and a devnp-i82544.so.
Why the two flavour? 

> -----Original Message-----
> From: Andrew Boyd [mailto:community-noreply@qnx.com]
> Sent: May-04-09 9:12 AM
> To: general-networking
> Subject: RE: How to find out if that's a HW fault?
> 
> 
> > Please define later version.
> 
> For io-pkt, see r746 on 2009-01-16
> 
> --
> aboyd
> 
> 
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post28600
> 


_______________________________________________
General
http://community.qnx.com/sf/go/post28601
RE: How to find out if that's a HW fault?  

> -----Original Message-----
> From: Hugh Brown [mailto:community-noreply@qnx.com]
> Sent: May-04-09 9:19 AM
> To: general-networking
> Subject: RE: How to find out if that's a HW fault?
> 
> The 2 drivers are there for users still using the i82544 driver.
> Eventually the i82544 driver will be dropped and the just the e1000
> driver included.
> 

Are you saying devnp-i82544 is e1000?  Because I don't see any e1000 driver in M4.

> 
> -----Original Message-----
> From: Mario Charest [mailto:community-noreply@qnx.com]
> Sent: Monday, May 04, 2009 9:17 AM
> To: general-networking
> Subject: RE: How to find out if that's a HW fault?
> 
> I was told a while ago that the e1000 driver would be available on
> 6.4.?
> Now in M4 I noticed there is a devn-i82544.so and a devnp-i82544.so.
> Why the two flavour?
> 
> > -----Original Message-----
> > From: Andrew Boyd [mailto:community-noreply@qnx.com]
> > Sent: May-04-09 9:12 AM
> > To: general-networking
> > Subject: RE: How to find out if that's a HW fault?
> >
> >
> > > Please define later version.
> >
> > For io-pkt, see r746 on 2009-01-16
> >
> > --
> > aboyd
> >
> >
> > _______________________________________________
> > General
> > http://community.qnx.com/sf/go/post28600
> >
> 
> 
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post28601
> 
> 
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post28604
> 
RE: How to find out if that's a HW fault?  
> Are you saying devnp-i82544 is e1000?  

No.  devnp-i82544.so and devnp-e1000.so are two
very different io-pkt drivers.

> I don't see any e1000 driver in M4.

Oops.  Hugh, is devnp-e1000.so shipping with 6.4.1?

--
aboyd
RE: How to find out if that's a HW fault?  
Yes, the devnp-e1000.so driver is being shipped with 6.4.1.


-----Original Message-----
From: Andrew Boyd [mailto:community-noreply@qnx.com] 
Sent: Monday, May 04, 2009 9:28 AM
To: general-networking
Subject: RE: How to find out if that's a HW fault?


> Are you saying devnp-i82544 is e1000?  

No.  devnp-i82544.so and devnp-e1000.so are two
very different io-pkt drivers.

> I don't see any e1000 driver in M4.

Oops.  Hugh, is devnp-e1000.so shipping with 6.4.1?

--
aboyd


_______________________________________________
General
http://community.qnx.com/sf/go/post28609
RE: How to find out if that's a HW fault?  
No, I don't think that the e1000 driver made M4. It is in M5.
devnp-e1000.so is a port of the devn-e1000.so driver to io-pkt.


-----Original Message-----
From: Mario Charest [mailto:community-noreply@qnx.com] 
Sent: Monday, May 04, 2009 9:23 AM
To: general-networking
Subject: RE: How to find out if that's a HW fault?



> -----Original Message-----
> From: Hugh Brown [mailto:community-noreply@qnx.com]
> Sent: May-04-09 9:19 AM
> To: general-networking
> Subject: RE: How to find out if that's a HW fault?
> 
> The 2 drivers are there for users still using the i82544 driver.
> Eventually the i82544 driver will be dropped and the just the e1000
> driver included.
> 

Are you saying devnp-i82544 is e1000?  Because I don't see any e1000
driver in M4.

> 
> -----Original Message-----
> From: Mario Charest [mailto:community-noreply@qnx.com]
> Sent: Monday, May 04, 2009 9:17 AM
> To: general-networking
> Subject: RE: How to find out if that's a HW fault?
> 
> I was told a while ago that the e1000 driver would be available on
> 6.4.?
> Now in M4 I noticed there is a devn-i82544.so and a devnp-i82544.so.
> Why the two flavour?
> 
> > -----Original Message-----
> > From: Andrew Boyd [mailto:community-noreply@qnx.com]
> > Sent: May-04-09 9:12 AM
> > To: general-networking
> > Subject: RE: How to find out if that's a HW fault?
> >
> >
> > > Please define later version.
> >
> > For io-pkt, see r746 on 2009-01-16
> >
> > --
> > aboyd
> >
> >
> > _______________________________________________
> > General
> > http://community.qnx.com/sf/go/post28600
> >
> 
> 
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post28601
> 
> 
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post28604
> 


_______________________________________________
General
http://community.qnx.com/sf/go/post28607
RE: How to find out if that's a HW fault?  
In the past, people have whined (with some merit) that
there weren't enough drivers for QNX.

At least with respect to intel i82544, this is no longer
the case.  Off the top of my head there chronologically are:

devn-i82544.so    (io-net and under shim for io-pkt)
devnp-i82544.so   (native io-pkt)
devnp-wm.so       (BSD io-pkt_
devn-e1000.so     (io-net port of intel source)
devnp-e1000.so    (native io-pkt port of intel source)

I probably have forgotten some.

Going forward, we hope that the e1000 driver
variants can provide the performance, reliability
and efficiency that people need.  They should work
with all hardware variants - which is a very big 
deal with i82544.

--
aboyd
RE: How to find out if that's a HW fault?  

> -----Original Message-----
> From: Andrew Boyd [mailto:community-noreply@qnx.com]
> Sent: May-04-09 9:21 AM
> To: general-networking
> Subject: RE: How to find out if that's a HW fault?
> 
> In the past, people have whined (with some merit) that
> there weren't enough drivers for QNX.
> 
> At least with respect to intel i82544, this is no longer
> the case.  Off the top of my head there chronologically are:
> 
> devn-i82544.so    (io-net and under shim for io-pkt)
> devnp-i82544.so   (native io-pkt)
> devnp-wm.so       (BSD io-pkt_
> devn-e1000.so     (io-net port of intel source)
> devnp-e1000.so    (native io-pkt port of intel source)
> 
> I probably have forgotten some.
> 
> Going forward, we hope 

I hope too because this is killing us  (over dramatic).

If only there was another 1G alternative.

> that the e1000 driver
> variants can provide the performance, reliability
> and efficiency that people need.  They should work
> with all hardware variants - which is a very big
> deal with i82544.


> 
> --
> aboyd
> 
> 
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post28605
> 
Re: RE: How to find out if that's a HW fault?  
> 
> After considerable effort on Hugh Brown's part, intel
> discovered that some i82544 variants have a hardware
> bug where they can lose interrupts during certain
> race conditions.  Oops.
> 
> Later versions of the i82544 drivers (both io-net
> and io-pkt variants) have fixes to work around this 
> bug.

Hm... Were there any updates to network drivers in 6.3.0 after SP3? I thought I had the latest ones available. I'll 
check downloads.

> What is your DID?

Class          = Network (Ethernet)
Vendor ID      = 8086h, Intel Corporation
Device ID      = 109ah,  82573L Gigabit Ethernet Controller



RE: RE: How to find out if that's a HW fault?  
The devn-e1000.so driver supersedes the i82544 driver. I have attached
an experimental copy. 


-----Original Message-----
From: Oleh Derevenko [mailto:community-noreply@qnx.com] 
Sent: Monday, May 04, 2009 9:19 AM
To: general-networking
Subject: Re: RE: How to find out if that's a HW fault?

> 
> After considerable effort on Hugh Brown's part, intel
> discovered that some i82544 variants have a hardware
> bug where they can lose interrupts during certain
> race conditions.  Oops.
> 
> Later versions of the i82544 drivers (both io-net
> and io-pkt variants) have fixes to work around this 
> bug.

Hm... Were there any updates to network drivers in 6.3.0 after SP3? I
thought I had the latest ones available. I'll check downloads.

> What is your DID?

Class          = Network (Ethernet)
Vendor ID      = 8086h, Intel Corporation
Device ID      = 109ah,  82573L Gigabit Ethernet Controller





_______________________________________________
General
http://community.qnx.com/sf/go/post28603

Attachment: Text devn-e1000.so 207.02 KB
Re: RE: RE: How to find out if that's a HW fault?  
Hi Hugh,

> The devn-e1000.so driver supersedes the i82544 driver. I have attached
> an experimental copy. 
> 

Sorry, but the driver did not solve the problem. :( Here is what our support person at client site reports.
------------
Installed the patched network drivers and restarted the server this morning:
 
# ls -la /etc/system/enum/devices/net && ls -la /lib/dll/devn-e1000.so
-rw-rw-r--  1 root      root          15941 May 04 07:01 /etc/system/enum/devices/net
-rwxrwxr-x  1 root      root         211989 May 04 06:58 /lib/dll/devn-e1000.so
#
 
About an hour later, however, en1 stopped working again.
 
When I connected to en0 and tried to telnet in, the telnet session ran really slow.  It wasn't slow to respond (as I 
would type characters I would see them echoed back quickly), but it was very slow to process (when I hit return at the 
end command it would take 1-2 minutes to act on the command).  I tried using a keyboard and mouse locally and found it 
responding in the same slow way.
 
I was wondering if it was a specific process using all available CPU time, but after spending 20 minutes figuring out 
the right ps command, the node process lag went away.  EN1 was still non-functional.

Below is some info I was able to grab off the node:
 
# nicinfo /dev/io-net/en1
INTEL PRO/1000 Gigabit (Copper) Ethernet Controller
 
  Physical Node ID ........................... 003048 B0CC4F
  Current Physical Node ID ................... 003048 B0CC4F
  Current Operation Rate ..................... 100.00 Mb/s full-duplex
  Active Interface Type ...................... MII
    Active PHY address ....................... 1
  Maximum Transmittable data Unit ............ 1514
  Maximum Receivable data Unit ............... 1514
  Hardware Interrupt ......................... 0xb
  Memory Aperture ............................ 0xd0200000 - 0xd021ffff
  Promiscuous Mode ........................... Off
  Multicast Support .......................... Enabled
 
  Packets Transmitted OK ..................... 700269
  Bytes Transmitted OK ....................... 80221003
  Broadcast Packets Transmitted OK ........... 3610
  Multicast Packets Transmitted OK ........... 0
  Memory Allocation Failures on Transmit ..... 0
 
  Packets Received OK ........................ 688414
  Bytes Received OK .......................... 74367203
  Broadcast Packets Received OK .............. 11976
  Multicast Packets Received OK .............. 0
  Memory Allocation Failures on Receive ...... 0
 
  Single Collisions on Transmit .............. 0
  Deferred Transmits ......................... 0
  Late Collision on Transmit errors .......... 0
  Transmits aborted (excessive collisions) ... 0
  No Carrier on Transmit ..................... 0
  Receive Alignment errors ................... 0
  Received packets with CRC errors ........... 0
  Packets Dropped on receive ................. 7481
# pidin -p 77835 mem
     pid tid name               prio STATE           code  data         stack
   77835   1 sbin/io-net         10o SIGWAITINFO       64K 5108K  8192(516K)*
   77835   2 sbin/io-net         10o RECEIVE           64K 5108K  4096(132K)
   77835   3 sbin/io-net         10o RECEIVE           64K 5108K   4096(68K)
   77835   4 sbin/io-net         10o RECEIVE           64K 5108K   4096(68K)
   77835   5 sbin/io-net         10o RECEIVE           64K 5108K   4096(68K)
   77835   6 sbin/io-net         21o RECEIVE           64K 5108K  4096(132K)
   77835   7 sbin/io-net         21o RECEIVE           64K 5108K  4096(132K)
   77835   8 sbin/io-net         20o RECEIVE           64K 5108K  4096(132K)
   77835   9 sbin/io-net         20o RECEIVE           64K 5108K  4096(132K)
   77835  10 sbin/io-net         10o RECEIVE           64K 5108K   4096(68K)
            ldqnx.so.2         @b0300000             348K   20K
            npm-tcpip.so       @b8200000             276K   32K
            devn-e1000.so     ...
View Full Message
RE: RE: RE: How to find out if that's a HW fault?  
Please can you supply me with the 'pci -v' output as well as how to
reproduce the problem.

Thanks, Hugh.

-----Original Message-----
From: Oleh Derevenko [mailto:community-noreply@qnx.com] 
Sent: Tuesday, May 05, 2009 6:56 AM
To: general-networking
Subject: Re: RE: RE: How to find out if that's a HW fault?

Hi Hugh,

> The devn-e1000.so driver supersedes the i82544 driver. I have attached
> an experimental copy. 
> 

Sorry, but the driver did not solve the problem. :( Here is what our
support person at client site reports.
------------
Installed the patched network drivers and restarted the server this
morning:
 
# ls -la /etc/system/enum/devices/net && ls -la /lib/dll/devn-e1000.so
-rw-rw-r--  1 root      root          15941 May 04 07:01
/etc/system/enum/devices/net
-rwxrwxr-x  1 root      root         211989 May 04 06:58
/lib/dll/devn-e1000.so
#
 
About an hour later, however, en1 stopped working again.
 
When I connected to en0 and tried to telnet in, the telnet session ran
really slow.  It wasn't slow to respond (as I would type characters I
would see them echoed back quickly), but it was very slow to process
(when I hit return at the end command it would take 1-2 minutes to act
on the command).  I tried using a keyboard and mouse locally and found
it responding in the same slow way.
 
I was wondering if it was a specific process using all available CPU
time, but after spending 20 minutes figuring out the right ps command,
the node process lag went away.  EN1 was still non-functional.

Below is some info I was able to grab off the node:
 
# nicinfo /dev/io-net/en1
INTEL PRO/1000 Gigabit (Copper) Ethernet Controller
 
  Physical Node ID ........................... 003048 B0CC4F
  Current Physical Node ID ................... 003048 B0CC4F
  Current Operation Rate ..................... 100.00 Mb/s full-duplex
  Active Interface Type ...................... MII
    Active PHY address ....................... 1
  Maximum Transmittable data Unit ............ 1514
  Maximum Receivable data Unit ............... 1514
  Hardware Interrupt ......................... 0xb
  Memory Aperture ............................ 0xd0200000 - 0xd021ffff
  Promiscuous Mode ........................... Off
  Multicast Support .......................... Enabled
 
  Packets Transmitted OK ..................... 700269
  Bytes Transmitted OK ....................... 80221003
  Broadcast Packets Transmitted OK ........... 3610
  Multicast Packets Transmitted OK ........... 0
  Memory Allocation Failures on Transmit ..... 0
 
  Packets Received OK ........................ 688414
  Bytes Received OK .......................... 74367203
  Broadcast Packets Received OK .............. 11976
  Multicast Packets Received OK .............. 0
  Memory Allocation Failures on Receive ...... 0
 
  Single Collisions on Transmit .............. 0
  Deferred Transmits ......................... 0
  Late Collision on Transmit errors .......... 0
  Transmits aborted (excessive collisions) ... 0
  No Carrier on Transmit ..................... 0
  Receive Alignment errors ................... 0
  Received packets with CRC errors ........... 0
  Packets Dropped on receive ................. 7481
# pidin -p 77835 mem
     pid tid name               prio STATE           code  data
stack
   77835   1 sbin/io-net         10o SIGWAITINFO       64K 5108K
8192(516K)*
   77835   2 sbin/io-net         10o RECEIVE           64K 5108K
4096(132K)
   77835   3 sbin/io-net         10o RECEIVE           64K 5108K
4096(68K)
   77835   4 sbin/io-net         10o RECEIVE           64K 5108K
4096(68K)
   77835   5 sbin/io-net         10o RECEIVE           64K 5108K
4096(68K)
   77835   6 sbin/io-net         21o RECEIVE           64K 5108K
4096(132K)
   77835   7 sbin/io-net         21o RECEIVE           64K 5108K
4096(132K)
   77835   8 sbin/io-net         20o RECEIVE           64K...
View Full Message
Re: RE: RE: RE: How to find out if that's a HW fault?  
> Please can you supply me with the 'pci -v' output as well as how to
> reproduce the problem.
> 
I do not have "pci -v" output yet, but this is what I've been told.

For 1 month the machine was running fine while en0 was used for network (QNet + TCP). We also used exactly the same 
computer configuration in previous installations. Then we had to add a driver that needs to communicate with hardware by
 UDP broadcasts and we had to make dedicated connection to it. However there were configuration limitations in the 
driver and we could not put it on en1. Instead we have moved main network to en1 and configured hardware device on en0.
First of all, there was a problem that when the hardware-dedicated ethernet (EN0) was not plugged in the network started
 to return "no buffer space available" on broadcast attempts. As soon as we have plugged in a cable and inserted it into
 an empty switch those errors were gone.
But also, since the time of employing en1, it started to go offline every few hours. Well, I suspect it was that way 
ever, we just never used and never checked if it was alive.
The last few days machine was running with our hardware driver disabled and EN0 unplugged (so that there was no traffic 
on it), but still there problem with EN1 dropping offline did not go away.  Even more, we have moved main network back 
to en0 and just connected a cable to en1 now and then to see if it still functioning, but the problem remained.
The new driver did not solve it and what's a little bit strange, even after restoring original /etc/system/enum/devices/
net back to have i82544 loaded for device again, the connection was slow after fault again, as if the server was 
overloaded. However this was just one test and I can't be sure that there was not some mistake and that the information 
about slowdown remaining after rollback to old driver is correct.
That's all I can tell about how the problem can be reproduced. There are no specific actions we could do to make it 
fault.
Re: RE: RE: RE: How to find out if that's a HW fault?  
Hi Hugh,

At last, they've sent me the 'pci -v' output. Here it is

=================================

PCI version    = 3.00

Class          = Bridge (Host/PCI)
Vendor ID      = 8086h, Intel Corporation 
Device ID      = 2970h,  Memory Controller Hub
PCI index      = 0h
Class Codes    = 060000h
Revision ID    = 2h
Bus number     = 0
Device number  = 0
Function num   = 0
Status Reg     = 2090h
Command Reg    = 106h
Header type    = 0h Single-function
BIST           = 0h Build-in-self-test not supported
Latency Timer  = 0h
Cache Line Size= 0h 
Subsystem Vendor ID = 15d9h
Subsystem ID        = b580h
Max Lat        = 0ns
Min Gnt        = 0ns
PCI Int Pin    = NC
Interrupt line = 0
CPU Interrupt  = 0h
Capabilities Pointer = e0h
Capability ID        = 9h - Vendor Specific
Capabilities         = 9109h - cb000368h

Class          = Display (VGA)
Vendor ID      = 8086h, Intel Corporation 
Device ID      = 2972h,  Integrated Graphics Controller
PCI index      = 0h
Class Codes    = 030000h
Revision ID    = 2h
Bus number     = 0
Device number  = 2
Function num   = 0
Status Reg     = 90h
Command Reg    = 7h
Header type    = 0h Single-function
BIST           = 0h Build-in-self-test not supported
Latency Timer  = 0h
Cache Line Size= 0h 
PCI Mem Address = d0000000h 32bit length 1048576 enabled
PCI Mem Address = c0000000h prefetchable 64bit length 268435456 enabled
PCI IO Address  = 30c0h length 8 enabled
Subsystem Vendor ID = 15d9h
Subsystem ID        = b580h
Max Lat        = 0ns
Min Gnt        = 0ns
PCI Int Pin    = INT A
Interrupt line = 10
CPU Interrupt  = ah
Capabilities Pointer = 90h
Capability ID        = 5h - Message Signaled Interrupts
Capabilities         = 0h - 0h
Capability ID        = 1h - Power Management
Capabilities         = 22h - 0h

Class          = Bridge (PCI/PCI)
Vendor ID      = 8086h, Intel Corporation 
Device ID      = 27d0h,  82801G (ICH7 Family) PCI Express Port 1
PCI index      = 0h
Class Codes    = 060400h
Revision ID    = 1h
Bus number     = 0
Device number  = 28
Function num   = 0
Status Reg     = 10h
Command Reg    = 4h
Header type    = 1h Multi-function
BIST           = 0h Build-in-self-test not supported
Latency Timer  = 0h
Cache Line Size= 8h un-cacheable
Primary Bus Number       = 0h
Secondary Bus Number     = 2h
Subordinate Bus Number   = 2h
Secondary Latency Timer  = 0h
I/O Base                 = f0h
I/O Limit                = 0h
Secondary Status         = 2000h
Memory Base              = fff0h
Memory Limit             = 0h
Prefetchable Memory Base = fff1h
Prefetchable Memory Limit= 1h
Prefetchable Base Upper 32 Bits  = 0h
Prefetchable Limit Upper 32 Bits = 0h
I/O Base Upper 16 Bits   = 0h
I/O Limit Upper 16 Bits  = 0h
Bridge Control           = 0h
PCI Int Pin              = INT A
Interrupt line           = 10
CPU Interrupt            = ah
Capabilities Pointer = 40h
Capability ID        = 10h - PCI Express
Capabilities         = 141h - fc0h
Capability ID        = 5h - Message Signaled Interrupts
Capabilities         = 0h - 0h
Capability ID        = dh - PCI Bridge Subsystem Vendor ID
Capabilities         = 0h - b58015d9h
Capability ID        = 1h - Power Management
Capabilities         = c802h - 0h

Class          = Bridge (PCI/PCI)
Vendor ID      = 8086h, Intel Corporation 
Device ID      = 27e0h,  82801GR/GH/GHM (ICH7 Family) PCI Express Port 5
PCI index      = 0h
Class Codes    = 060400h
Revision ID    = 1h
Bus number     = 0
Device number  = 28
Function num   = 4
Status Reg     = 10h
Command Reg    = 7h
Header type    = 1h Multi-function
BIST           = 0h Build-in-self-test not supported
Latency Timer  = 0h
Cache Line Size= 8h un-cacheable
Primary Bus Number       = 0h
Secondary Bus Number     = 6h
Subordinate Bus Number   = 6h
Secondary Latency Timer  = 0h
I/O Base                 = 40h
I/O Limit                = 40h
Secondary Status         = 2000h
Memory Base           ...
View Full Message
RE: RE: RE: RE: How to find out if that's a HW fault?  
Hi Oleh,

Unfortunately we don't have this device ID here. Is this Ethernet on the
motherboard or is it a plug-in adapter? If it is a plug-in adapter, I
need to know the make and model, and if it is on the motherboard, I need
to know the motherboard make and model.

I have a dual Intel gigabit adapter in one of my machines here and it
has been running for weeks without a problem. Is there heavy activity on
the customer's network when this problem occurs?

Thanks, Hugh.

-----Original Message-----
From: Oleh Derevenko [mailto:community-noreply@qnx.com] 
Sent: Tuesday, May 12, 2009 8:10 AM
To: general-networking
Subject: Re: RE: RE: RE: How to find out if that's a HW fault?

Hi Hugh,

At last, they've sent me the 'pci -v' output. Here it is

=================================

PCI version    = 3.00

Class          = Bridge (Host/PCI)
Vendor ID      = 8086h, Intel Corporation 
Device ID      = 2970h,  Memory Controller Hub
PCI index      = 0h
Class Codes    = 060000h
Revision ID    = 2h
Bus number     = 0
Device number  = 0
Function num   = 0
Status Reg     = 2090h
Command Reg    = 106h
Header type    = 0h Single-function
BIST           = 0h Build-in-self-test not supported
Latency Timer  = 0h
Cache Line Size= 0h 
Subsystem Vendor ID = 15d9h
Subsystem ID        = b580h
Max Lat        = 0ns
Min Gnt        = 0ns
PCI Int Pin    = NC
Interrupt line = 0
CPU Interrupt  = 0h
Capabilities Pointer = e0h
Capability ID        = 9h - Vendor Specific
Capabilities         = 9109h - cb000368h

Class          = Display (VGA)
Vendor ID      = 8086h, Intel Corporation 
Device ID      = 2972h,  Integrated Graphics Controller
PCI index      = 0h
Class Codes    = 030000h
Revision ID    = 2h
Bus number     = 0
Device number  = 2
Function num   = 0
Status Reg     = 90h
Command Reg    = 7h
Header type    = 0h Single-function
BIST           = 0h Build-in-self-test not supported
Latency Timer  = 0h
Cache Line Size= 0h 
PCI Mem Address = d0000000h 32bit length 1048576 enabled
PCI Mem Address = c0000000h prefetchable 64bit length 268435456 enabled
PCI IO Address  = 30c0h length 8 enabled
Subsystem Vendor ID = 15d9h
Subsystem ID        = b580h
Max Lat        = 0ns
Min Gnt        = 0ns
PCI Int Pin    = INT A
Interrupt line = 10
CPU Interrupt  = ah
Capabilities Pointer = 90h
Capability ID        = 5h - Message Signaled Interrupts
Capabilities         = 0h - 0h
Capability ID        = 1h - Power Management
Capabilities         = 22h - 0h

Class          = Bridge (PCI/PCI)
Vendor ID      = 8086h, Intel Corporation 
Device ID      = 27d0h,  82801G (ICH7 Family) PCI Express Port 1
PCI index      = 0h
Class Codes    = 060400h
Revision ID    = 1h
Bus number     = 0
Device number  = 28
Function num   = 0
Status Reg     = 10h
Command Reg    = 4h
Header type    = 1h Multi-function
BIST           = 0h Build-in-self-test not supported
Latency Timer  = 0h
Cache Line Size= 8h un-cacheable
Primary Bus Number       = 0h
Secondary Bus Number     = 2h
Subordinate Bus Number   = 2h
Secondary Latency Timer  = 0h
I/O Base                 = f0h
I/O Limit                = 0h
Secondary Status         = 2000h
Memory Base              = fff0h
Memory Limit             = 0h
Prefetchable Memory Base = fff1h
Prefetchable Memory Limit= 1h
Prefetchable Base Upper 32 Bits  = 0h
Prefetchable Limit Upper 32 Bits = 0h
I/O Base Upper 16 Bits   = 0h
I/O Limit Upper 16 Bits  = 0h
Bridge Control           = 0h
PCI Int Pin              = INT A
Interrupt line           = 10
CPU Interrupt            = ah
Capabilities Pointer = 40h
Capability ID        = 10h - PCI Express
Capabilities         = 141h - fc0h
Capability ID        = 5h - Message Signaled Interrupts
Capabilities         = 0h - 0h
Capability ID        = dh - PCI Bridge Subsystem Vendor ID
Capabilities         = 0h - b58015d9h
Capability ID        = 1h - Power Management
Capabilities         = c802h - 0h

Class        ...
View Full Message
Re: RE: RE: RE: RE: How to find out if that's a HW fault?  
> Unfortunately we don't have this device ID here. Is this Ethernet on the motherboard or is it a plug-in adapter? 

As far as I know it's Ethernet integrated on motherboard.

> If it is a plug-in adapter, I need to know the make and model, and if it is on the motherboard, I need to know the 
motherboard make and model.

This is what we use http://www.abmx.com/1u-short-depth-server

> 
> I have a dual Intel gigabit adapter in one of my machines here and it has been running for weeks without a problem. Is
 there heavy activity on the customer's network when this problem occurs?

Well, I do not have statistics from this particular installation, but I've been watching few previous places. Of course 
the network conditions are different everywhere but average network utilization was never above 5-10% there. So, 
generally the network should not be loaded very much even though there could be short bursts.
After all, as I already mentioned in previous posts, EN1 dies even if it is not plugged in at all. They just attach the 
cable after some time and find that EN1 is not operational any more.
RE: RE: RE: RE: RE: How to find out if that's a HW fault?  
I have managed to locate a machine with one 0x109a Intel device on it. I
will runs some tests and let you know. As a matter of interest, this
machine is from our test lab, and has just passed all tests!

Hugh.

-----Original Message-----
From: Oleh Derevenko [mailto:community-noreply@qnx.com] 
Sent: Tuesday, May 12, 2009 9:05 AM
To: general-networking
Subject: Re: RE: RE: RE: RE: How to find out if that's a HW fault?

> Unfortunately we don't have this device ID here. Is this Ethernet on
the motherboard or is it a plug-in adapter? 

As far as I know it's Ethernet integrated on motherboard.

> If it is a plug-in adapter, I need to know the make and model, and if
it is on the motherboard, I need to know the motherboard make and model.

This is what we use http://www.abmx.com/1u-short-depth-server

> 
> I have a dual Intel gigabit adapter in one of my machines here and it
has been running for weeks without a problem. Is there heavy activity on
the customer's network when this problem occurs?

Well, I do not have statistics from this particular installation, but
I've been watching few previous places. Of course the network conditions
are different everywhere but average network utilization was never above
5-10% there. So, generally the network should not be loaded very much
even though there could be short bursts.
After all, as I already mentioned in previous posts, EN1 dies even if it
is not plugged in at all. They just attach the cable after some time and
find that EN1 is not operational any more.

_______________________________________________
General
http://community.qnx.com/sf/go/post29246
RE: RE: RE: RE: RE: How to find out if that's a HW fault?  
Oleh,

Please can you find out what the customer means by "dropping offline"?
Does nicinfo still show activity on the link and is the link still up? I
have setup a machine on my desk and am running some tests, but would
like to know more about how to reproduce the problem.

Thanks, Hugh.

-----Original Message-----
From: Oleh Derevenko [mailto:community-noreply@qnx.com] 
Sent: Tuesday, May 12, 2009 9:05 AM
To: general-networking
Subject: Re: RE: RE: RE: RE: How to find out if that's a HW fault?

> Unfortunately we don't have this device ID here. Is this Ethernet on
the motherboard or is it a plug-in adapter? 

As far as I know it's Ethernet integrated on motherboard.

> If it is a plug-in adapter, I need to know the make and model, and if
it is on the motherboard, I need to know the motherboard make and model.

This is what we use http://www.abmx.com/1u-short-depth-server

> 
> I have a dual Intel gigabit adapter in one of my machines here and it
has been running for weeks without a problem. Is there heavy activity on
the customer's network when this problem occurs?

Well, I do not have statistics from this particular installation, but
I've been watching few previous places. Of course the network conditions
are different everywhere but average network utilization was never above
5-10% there. So, generally the network should not be loaded very much
even though there could be short bursts.
After all, as I already mentioned in previous posts, EN1 dies even if it
is not plugged in at all. They just attach the cable after some time and
find that EN1 is not operational any more.

_______________________________________________
General
http://community.qnx.com/sf/go/post29246
Re: RE: RE: RE: RE: RE: How to find out if that's a HW fault?  
> Please can you find out what the customer means by "dropping offline"?

It means "If trying 'ls /net' it returns the own node name only. An attempt to ping any other node on the network 
results in "Host 
is down" error." That is, the driver is running but all network communications are unavailable both over QNet and TCP/IP
.

> Does nicinfo still show activity on the link and is the link still up?
I've posted nicinfo output after the fault earlier in this thread for both i82544 and e1000.

Here it is again for i82544
===========
# nicinfo /dev/io-net/en1
INTEL 82544 Gigabit (Copper) Ethernet Controller
 
  Physical Node ID ........................... 003048 B037CB
  Current Physical Node ID ................... 003048 B037CB
  Current Operation Rate ..................... 100.00 Mb/s full-duplex
  Active Interface Type ...................... MII
    Active PHY address ....................... 0
  Maximum Transmittable data Unit ............ 1514
  Maximum Receivable data Unit ............... 0
  Hardware Interrupt ......................... 0xb
  Memory Aperture ............................ 0xd0200000 - 0xd021ffff
  Promiscuous Mode ........................... Off
  Multicast Support .......................... Enabled
 
  Packets Transmitted OK ..................... 16489506
  Bytes Transmitted OK ....................... 1550295660
  Broadcast Packets Transmitted OK ........... 27233
  Multicast Packets Transmitted OK ........... 0
  Memory Allocation Failures on Transmit ..... 0
 
  Packets Received OK ........................ 16603917
  Bytes Received OK .......................... 1545543297
  Broadcast Packets Received OK .............. 57710
  Multicast Packets Received OK .............. 0
  Memory Allocation Failures on Receive ...... 0
 
  Single Collisions on Transmit .............. 0
  Deferred Transmits ......................... 7
  Late Collision on Transmit errors .......... 0
  Transmits aborted (excessive collisions) ... 0
  No Carrier on Transmit ..................... 0
  Receive Alignment errors ................... 0
  Received packets with CRC errors ........... 0
  Packets Dropped on receive ................. 11905
#
 
# /sbin/ifconfig
lo0: flags=8009<UP,LOOPBACK,MULTICAST> mtu 33212
        capabilities=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
        enabled=0<>
        inet 127.0.0.1 netmask 0xff000000
en0: flags=8c43<UP,BROADCAST,RUNNING,OACTIVE,SIMPLEX,MULTICAST> mtu 1500
        capabilities=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
        enabled=0<>
        address: 00:30:48:b0:37:ca
        inet 192.168.1.70 netmask 0xffffff00 broadcast 192.168.1.255
en1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        capabilities=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
        enabled=0<>
        address: 00:30:48:b0:37:cb
        inet 192.168.10.44 netmask 0xffffff00 broadcast 192.168.10.255
#
===========
Here en1 is reported to be UP (<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>).

And here it is once again for e1000
===========
# nicinfo /dev/io-net/en1
INTEL PRO/1000 Gigabit (Copper) Ethernet Controller
 
  Physical Node ID ........................... 003048 B0CC4F
  Current Physical Node ID ................... 003048 B0CC4F
  Current Operation Rate ..................... 100.00 Mb/s full-duplex
  Active Interface Type ...................... MII
    Active PHY address ....................... 1
  Maximum Transmittable data Unit ............ 1514
  Maximum Receivable data Unit ............... 1514
  Hardware Interrupt ......................... 0xb
  Memory Aperture ............................ 0xd0200000 - 0xd021ffff
  Promiscuous Mode ........................... Off
  Multicast Support .......................... Enabled
 
  Packets Transmitted OK ..................... 700269
  Bytes Transmitted OK ....................... 80221003
  Broadcast Packets Transmitted OK ........... 3610
 ...
View Full Message
RE: RE: RE: RE: RE: RE: How to find out if that's a HW fault?  
If this could help, we have had strange behaviour (we didn't get to the bottom of it) when USB and Networking shared the
 same interrupt.

> -----Original Message-----
> From: Oleh Derevenko [mailto:community-noreply@qnx.com]
> Sent: May-12-09 11:45 AM
> To: general-networking
> Subject: Re: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
> 
> > Please can you find out what the customer means by "dropping
> offline"?
> 
> It means "If trying 'ls /net' it returns the own node name only. An
> attempt to ping any other node on the network results in "Host
> is down" error." That is, the driver is running but all network
> communications are unavailable both over QNet and TCP/IP.
> 
> > Does nicinfo still show activity on the link and is the link still
> up?
> I've posted nicinfo output after the fault earlier in this thread for
> both i82544 and e1000.
> 
> Here it is again for i82544
> ===========
> # nicinfo /dev/io-net/en1
> INTEL 82544 Gigabit (Copper) Ethernet Controller
> 
>   Physical Node ID ........................... 003048 B037CB
>   Current Physical Node ID ................... 003048 B037CB
>   Current Operation Rate ..................... 100.00 Mb/s full-duplex
>   Active Interface Type ...................... MII
>     Active PHY address ....................... 0
>   Maximum Transmittable data Unit ............ 1514
>   Maximum Receivable data Unit ............... 0
>   Hardware Interrupt ......................... 0xb
>   Memory Aperture ............................ 0xd0200000 - 0xd021ffff
>   Promiscuous Mode ........................... Off
>   Multicast Support .......................... Enabled
> 
>   Packets Transmitted OK ..................... 16489506
>   Bytes Transmitted OK ....................... 1550295660
>   Broadcast Packets Transmitted OK ........... 27233
>   Multicast Packets Transmitted OK ........... 0
>   Memory Allocation Failures on Transmit ..... 0
> 
>   Packets Received OK ........................ 16603917
>   Bytes Received OK .......................... 1545543297
>   Broadcast Packets Received OK .............. 57710
>   Multicast Packets Received OK .............. 0
>   Memory Allocation Failures on Receive ...... 0
> 
>   Single Collisions on Transmit .............. 0
>   Deferred Transmits ......................... 7
>   Late Collision on Transmit errors .......... 0
>   Transmits aborted (excessive collisions) ... 0
>   No Carrier on Transmit ..................... 0
>   Receive Alignment errors ................... 0
>   Received packets with CRC errors ........... 0
>   Packets Dropped on receive ................. 11905
> #
> 
> # /sbin/ifconfig
> lo0: flags=8009<UP,LOOPBACK,MULTICAST> mtu 33212
>         capabilities=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
>         enabled=0<>
>         inet 127.0.0.1 netmask 0xff000000
> en0: flags=8c43<UP,BROADCAST,RUNNING,OACTIVE,SIMPLEX,MULTICAST> mtu
> 1500
>         capabilities=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
>         enabled=0<>
>         address: 00:30:48:b0:37:ca
>         inet 192.168.1.70 netmask 0xffffff00 broadcast 192.168.1.255
> en1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>         capabilities=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
>         enabled=0<>
>         address: 00:30:48:b0:37:cb
>         inet 192.168.10.44 netmask 0xffffff00 broadcast 192.168.10.255
> #
> ===========
> Here en1 is reported to be UP
> (<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>).
> 
> And here it is once again for e1000
> ===========
> # nicinfo /dev/io-net/en1
> INTEL PRO/1000 Gigabit (Copper) Ethernet Controller
> 
>   Physical Node ID...
View Full Message