|
Mario Charest
|
6.5 M9 network stop working
|
|
Mario Charest
07/13/2010 1:58 PM
post59331
|
6.5 M9 network stop working
Driver is e1000, at some point in time networking stops working, TCP and QNET. The only think that I see which is weird
is nicinfo will report 1 Memory Allocation Failure on transmit.
In nicinfo number of TX packet isn`t increasing but RX is. If I rmdir /net I can see new names showing up so it`s
definitely receiving stuff.
Tx and Rx descriptors are set at 2048.
|
|
|
|
|
|
Vineet Garg
|
Re: 6.5 M9 network stop working
|
|
Vineet Garg
07/14/2010 1:45 AM
post59374
|
Re: 6.5 M9 network stop working
I dont know much about e1000 but network drivers normally create a buffer
pool from which buffer is picked for transmission. On successful transmit,
tx_done callback is called which should return this buffer to back to pool.
Similarly, if tx fails at driver, it should return the buffer to pool or
free it.
In your case, it seems that buffer used for transmission has not been
cleaned up or returned to pool properly. So, after sometime driver runs out
of buffers from transmission.
In simple words, it looks like a memory leak!!
HTH
VG
On Tue, Jul 13, 2010 at 11:28 PM, Mario Charest
<community-noreply@qnx.com>wrote:
> Driver is e1000, at some point in time networking stops working, TCP and
> QNET. The only think that I see which is weird is nicinfo will report 1
> Memory Allocation Failure on transmit.
>
> In nicinfo number of TX packet isn`t increasing but RX is. If I rmdir /net
> I can see new names showing up so it`s definitely receiving stuff.
>
> Tx and Rx descriptors are set at 2048.
>
>
>
> _______________________________________________
>
> General
> http://community.qnx.com/sf/go/post59331
>
>
|
|
|
|
|
|
Hugh Brown
|
Re: 6.5 M9 network stop working
|
|
Hugh Brown
07/14/2010 8:51 AM
post59394
|
Re: 6.5 M9 network stop working
Does this happen regularly? Does 'pidin mem' show the memory usage
increasing or does it remain static?
On 10-07-13 1:58 PM, "Mario Charest" <community-noreply@qnx.com> wrote:
> Driver is e1000, at some point in time networking stops working, TCP and QNET.
> The only think that I see which is weird is nicinfo will report 1 Memory
> Allocation Failure on transmit.
>
> In nicinfo number of TX packet isn`t increasing but RX is. If I rmdir /net I
> can see new names showing up so it`s definitely receiving stuff.
>
> Tx and Rx descriptors are set at 2048.
>
>
>
> _______________________________________________
>
> General
> http://community.qnx.com/sf/go/post59331
>
--
Hugh Brown (613) 591-0931 ext. 2209 (voice)
QNX Software Systems Ltd. (613) 591-3579 (fax)
175 Terence Matthews Cres. email: hsbrown@qnx.com
Kanata, Ontario, Canada.
K2M 1W8
|
|
|
|
|
|
Patrik Lahti
|
Re: 6.5 M9 network stop working
|
|
Patrik Lahti
07/14/2010 8:55 AM
post59395
|
Re: 6.5 M9 network stop working
mbuf memory usage can be monitored using 'netstat -m' as well.
/P
|
|
|
|
|
|
Mario Charest
|
RE: 6.5 M9 network stop working
|
|
Mario Charest
07/14/2010 9:21 AM
post59405
|
RE: 6.5 M9 network stop working
When our application starts doing some TCP stuff it happens within 5 minutes. The machine has 8 Gig when the problem
occurs there is still 7.7Gig left.
We start io-pkt-v4 like this:
io-pkt-v4 -t4 -de1000 transmit=2048,receive=2048 -pqnet
Network is 100Mbits, cpu dual core 1.8Gig. 6.5.0 is setup to use APIC.
Before starting the application io-pkt-v4 is using 5.3 Meg.
When application is running and network is ok netstat -m reports:
2105 mbufs in use
2094 mbufs allocated to data
11 mbufs allocated to packet header
0 calls to protocol drain runtimes
When crashed it says
4596 mbufs in use
3379 mbufs allocated for data
1167 mbuf allocated to packet header
0 calls to prodotoc drain runtimes
Io-pkt-v4 is using 7.1 Meg
Just in case it matters we are setting the kernel nmbclusters to 32000
-----Message d'origine-----
De : Hugh Brown [mailto:community-noreply@qnx.com]
Envoyé : 14 juillet 2010 08:51
À : general-networking
Objet : Re: 6.5 M9 network stop working
Does this happen regularly? Does 'pidin mem' show the memory usage increasing or does it remain static?
On 10-07-13 1:58 PM, "Mario Charest" <community-noreply@qnx.com> wrote:
> Driver is e1000, at some point in time networking stops working, TCP and QNET.
> The only think that I see which is weird is nicinfo will report 1
> Memory Allocation Failure on transmit.
>
> In nicinfo number of TX packet isn`t increasing but RX is. If I rmdir
> /net I can see new names showing up so it`s definitely receiving stuff.
>
> Tx and Rx descriptors are set at 2048.
>
>
>
> _______________________________________________
>
> General
> http://community.qnx.com/sf/go/post59331
>
--
Hugh Brown (613) 591-0931 ext. 2209 (voice)
QNX Software Systems Ltd. (613) 591-3579 (fax)
175 Terence Matthews Cres. email: hsbrown@qnx.com
Kanata, Ontario, Canada.
K2M 1W8
_______________________________________________
General
http://community.qnx.com/sf/go/post59394
|
|
|
|
|
|
Hugh Brown
|
Re: 6.5 M9 network stop working
|
|
Hugh Brown
07/14/2010 9:30 AM
post59408
|
Re: 6.5 M9 network stop working
If you slay pci-bios and restart it as 'pci-bios-v2 -M', then restart the
network and your application, does the problem still occur?
On 10-07-14 9:21 AM, "Mario Charest" <community-noreply@qnx.com> wrote:
>
> When our application starts doing some TCP stuff it happens within 5 minutes.
> The machine has 8 Gig when the problem occurs there is still 7.7Gig left.
>
> We start io-pkt-v4 like this:
>
> io-pkt-v4 -t4 -de1000 transmit=2048,receive=2048 -pqnet
>
> Network is 100Mbits, cpu dual core 1.8Gig. 6.5.0 is setup to use APIC.
>
> Before starting the application io-pkt-v4 is using 5.3 Meg.
>
> When application is running and network is ok netstat -m reports:
> 2105 mbufs in use
> 2094 mbufs allocated to data
> 11 mbufs allocated to packet header
> 0 calls to protocol drain runtimes
>
> When crashed it says
> 4596 mbufs in use
> 3379 mbufs allocated for data
> 1167 mbuf allocated to packet header
> 0 calls to prodotoc drain runtimes
>
> Io-pkt-v4 is using 7.1 Meg
>
> Just in case it matters we are setting the kernel nmbclusters to 32000
>
> -----Message d'origine-----
> De : Hugh Brown [mailto:community-noreply@qnx.com]
> Envoyé : 14 juillet 2010 08:51
> À : general-networking
> Objet : Re: 6.5 M9 network stop working
>
> Does this happen regularly? Does 'pidin mem' show the memory usage increasing
> or does it remain static?
>
>
> On 10-07-13 1:58 PM, "Mario Charest" <community-noreply@qnx.com> wrote:
>
>> Driver is e1000, at some point in time networking stops working, TCP and
>> QNET.
>> The only think that I see which is weird is nicinfo will report 1
>> Memory Allocation Failure on transmit.
>>
>> In nicinfo number of TX packet isn`t increasing but RX is. If I rmdir
>> /net I can see new names showing up so it`s definitely receiving stuff.
>>
>> Tx and Rx descriptors are set at 2048.
>>
>>
>>
>> _______________________________________________
>>
>> General
>> http://community.qnx.com/sf/go/post59331
>>
--
Hugh Brown (613) 591-0931 ext. 2209 (voice)
QNX Software Systems Ltd. (613) 591-3579 (fax)
175 Terence Matthews Cres. email: hsbrown@qnx.com
Kanata, Ontario, Canada.
K2M 1W8
|
|
|
|
|
|
Mario Charest
|
RE: 6.5 M9 network stop working
|
|
Mario Charest
07/14/2010 10:33 AM
post59421
|
RE: 6.5 M9 network stop working
No change in behaviour aside pidin irq showing interrupts were all assigned to interrupts below 0x17
-----Message d'origine-----
De : Hugh Brown [mailto:community-noreply@qnx.com]
Envoyé : 14 juillet 2010 09:30
À : general-networking
Objet : Re: 6.5 M9 network stop working
If you slay pci-bios and restart it as 'pci-bios-v2 -M', then restart the network and your application, does the problem
still occur?
On 10-07-14 9:21 AM, "Mario Charest" <community-noreply@qnx.com> wrote:
>
> When our application starts doing some TCP stuff it happens within 5 minutes.
> The machine has 8 Gig when the problem occurs there is still 7.7Gig left.
>
> We start io-pkt-v4 like this:
>
> io-pkt-v4 -t4 -de1000 transmit=2048,receive=2048 -pqnet
>
> Network is 100Mbits, cpu dual core 1.8Gig. 6.5.0 is setup to use APIC.
>
> Before starting the application io-pkt-v4 is using 5.3 Meg.
>
> When application is running and network is ok netstat -m reports:
> 2105 mbufs in use
> 2094 mbufs allocated to data
> 11 mbufs allocated to packet header
> 0 calls to protocol drain runtimes
>
> When crashed it says
> 4596 mbufs in use
> 3379 mbufs allocated for data
> 1167 mbuf allocated to packet header
> 0 calls to prodotoc drain runtimes
>
> Io-pkt-v4 is using 7.1 Meg
>
> Just in case it matters we are setting the kernel nmbclusters to 32000
>
> -----Message d'origine-----
> De : Hugh Brown [mailto:community-noreply@qnx.com]
> Envoyé : 14 juillet 2010 08:51
> À : general-networking
> Objet : Re: 6.5 M9 network stop working
>
> Does this happen regularly? Does 'pidin mem' show the memory usage
> increasing or does it remain static?
>
>
> On 10-07-13 1:58 PM, "Mario Charest" <community-noreply@qnx.com> wrote:
>
>> Driver is e1000, at some point in time networking stops working, TCP
>> and QNET.
>> The only think that I see which is weird is nicinfo will report 1
>> Memory Allocation Failure on transmit.
>>
>> In nicinfo number of TX packet isn`t increasing but RX is. If I
>> rmdir /net I can see new names showing up so it`s definitely receiving stuff.
>>
>> Tx and Rx descriptors are set at 2048.
>>
>>
>>
>> _______________________________________________
>>
>> General
>> http://community.qnx.com/sf/go/post59331
>>
--
Hugh Brown (613) 591-0931 ext. 2209 (voice)
QNX Software Systems Ltd. (613) 591-3579 (fax)
175 Terence Matthews Cres. email: hsbrown@qnx.com
Kanata, Ontario, Canada.
K2M 1W8
_______________________________________________
General
http://community.qnx.com/sf/go/post59408
|
|
|
|
|
|
Hugh Brown
|
Re: 6.5 M9 network stop working
|
|
Hugh Brown
07/14/2010 10:41 AM
post59424
|
Re: 6.5 M9 network stop working
Please post the output from 'pci -vv' as well as an explanation of what your
TCP application is doing.
On 10-07-14 10:33 AM, "Mario Charest" <community-noreply@qnx.com> wrote:
> No change in behaviour aside pidin irq showing interrupts were all assigned to
> interrupts below 0x17
>
> -----Message d'origine-----
> De : Hugh Brown [mailto:community-noreply@qnx.com]
> Envoyé : 14 juillet 2010 09:30
> À : general-networking
> Objet : Re: 6.5 M9 network stop working
>
> If you slay pci-bios and restart it as 'pci-bios-v2 -M', then restart the
> network and your application, does the problem still occur?
>
>
> On 10-07-14 9:21 AM, "Mario Charest" <community-noreply@qnx.com> wrote:
>
>>
>> When our application starts doing some TCP stuff it happens within 5 minutes.
>> The machine has 8 Gig when the problem occurs there is still 7.7Gig left.
>>
>> We start io-pkt-v4 like this:
>>
>> io-pkt-v4 -t4 -de1000 transmit=2048,receive=2048 -pqnet
>>
>> Network is 100Mbits, cpu dual core 1.8Gig. 6.5.0 is setup to use APIC.
>>
>> Before starting the application io-pkt-v4 is using 5.3 Meg.
>>
>> When application is running and network is ok netstat -m reports:
>> 2105 mbufs in use
>> 2094 mbufs allocated to data
>> 11 mbufs allocated to packet header
>> 0 calls to protocol drain runtimes
>>
>> When crashed it says
>> 4596 mbufs in use
>> 3379 mbufs allocated for data
>> 1167 mbuf allocated to packet header
>> 0 calls to prodotoc drain runtimes
>>
>> Io-pkt-v4 is using 7.1 Meg
>>
>> Just in case it matters we are setting the kernel nmbclusters to 32000
>>
>> -----Message d'origine-----
>> De : Hugh Brown [mailto:community-noreply@qnx.com]
>> Envoyé : 14 juillet 2010 08:51
>> À : general-networking
>> Objet : Re: 6.5 M9 network stop working
>>
>> Does this happen regularly? Does 'pidin mem' show the memory usage
>> increasing or does it remain static?
>>
>>
>> On 10-07-13 1:58 PM, "Mario Charest" <community-noreply@qnx.com> wrote:
>>
>>> Driver is e1000, at some point in time networking stops working, TCP
>>> and QNET.
>>> The only think that I see which is weird is nicinfo will report 1
>>> Memory Allocation Failure on transmit.
>>>
>>> In nicinfo number of TX packet isn`t increasing but RX is. If I
>>> rmdir /net I can see new names showing up so it`s definitely receiving
>>> stuff.
>>>
>>> Tx and Rx descriptors are set at 2048.
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> General
>>> http://community.qnx.com/sf/go/post59331
>>>
--
Hugh Brown (613) 591-0931 ext. 2209 (voice)
QNX Software Systems Ltd. (613) 591-3579 (fax)
175 Terence Matthews Cres. email: hsbrown@qnx.com
Kanata, Ontario, Canada.
K2M 1W8
|
|
|
|
|
|
Mario Charest
|
RE: 6.5 M9 network stop working
|
|
Mario Charest
07/14/2010 11:20 AM
post59435
|
RE: 6.5 M9 network stop working
The application is rather simple and is used on other system ( 6.4.1 although we have noted problem when there is a high
throughput ). However under 6.4.1 we have no clue to provide aside networking dies.
The application is multi-thread. The main process wait for a connections, upon detection a new connection a thread is
started to handle the connection. The pseudo-code for the thread look like this:
- Set receive timeout to 0
- Wait for one byte
- Set receive timeout to 5 seconds
- Until receive buffer is empty
- Call ioctl(FIONREAD, result) to get number of byte in receive buffer
- Call recv with number of byte to read
- The buffer should contain information about the number of bytes to expect (in the first 4 bytes)
- Until receive all data
Recv
- SendMsg data to QNX app
- Take the reply data and do a send() on the socket
- start from the top
This is program acts as a bridge between windows and the QNX applications.
-----Message d'origine-----
De : Hugh Brown [mailto:community-noreply@qnx.com]
Envoyé : 14 juillet 2010 10:41
À : general-networking
Objet : Re: 6.5 M9 network stop working
Please post the output from 'pci -vv' as well as an explanation of what your TCP application is doing.
On 10-07-14 10:33 AM, "Mario Charest" <community-noreply@qnx.com> wrote:
> No change in behaviour aside pidin irq showing interrupts were all
> assigned to interrupts below 0x17
>
> -----Message d'origine-----
> De : Hugh Brown [mailto:community-noreply@qnx.com]
> Envoyé : 14 juillet 2010 09:30
> À : general-networking
> Objet : Re: 6.5 M9 network stop working
>
> If you slay pci-bios and restart it as 'pci-bios-v2 -M', then restart
> the network and your application, does the problem still occur?
>
>
> On 10-07-14 9:21 AM, "Mario Charest" <community-noreply@qnx.com> wrote:
>
>>
>> When our application starts doing some TCP stuff it happens within 5 minutes.
>> The machine has 8 Gig when the problem occurs there is still 7.7Gig left.
>>
>> We start io-pkt-v4 like this:
>>
>> io-pkt-v4 -t4 -de1000 transmit=2048,receive=2048 -pqnet
>>
>> Network is 100Mbits, cpu dual core 1.8Gig. 6.5.0 is setup to use APIC.
>>
>> Before starting the application io-pkt-v4 is using 5.3 Meg.
>>
>> When application is running and network is ok netstat -m reports:
>> 2105 mbufs in use
>> 2094 mbufs allocated to data
>> 11 mbufs allocated to packet header
>> 0 calls to protocol drain runtimes
>>
>> When crashed it says
>> 4596 mbufs in use
>> 3379 mbufs allocated for data
>> 1167 mbuf allocated to packet header
>> 0 calls to prodotoc drain runtimes
>>
>> Io-pkt-v4 is using 7.1 Meg
>>
>> Just in case it matters we are setting the kernel nmbclusters to
>> 32000
>>
>> -----Message d'origine-----
>> De : Hugh Brown [mailto:community-noreply@qnx.com]
>> Envoyé : 14 juillet 2010 08:51
>> À : general-networking
>> Objet : Re: 6.5 M9 network stop working
>>
>> Does this happen regularly? Does 'pidin mem' show the memory usage
>> increasing or does it remain static?
>>
>>
>> On 10-07-13 1:58 PM, "Mario Charest" <community-noreply@qnx.com> wrote:
>>
>>> Driver is e1000, at some point in time networking stops working, TCP
>>> and QNET.
>>> The only think that I see which is weird is nicinfo will report 1
>>> Memory Allocation Failure on transmit.
>>>
>>> In nicinfo number of TX packet isn`t increasing but RX is. If I
>>> rmdir /net I can see new names showing up so it`s definitely
>>> receiving stuff.
>>>
>>> Tx and Rx descriptors are set at...
View Full Message
|
|
|
|
|
|
Mario Charest
|
RE: 6.5 M9 network stop working
|
|
Mario Charest
07/14/2010 11:22 AM
post59436
|
RE: 6.5 M9 network stop working
Woops forgot the pci -vv output
PCI version = 3.00
Class = Bridge (Host/PCI)
Vendor ID = 8086h, Intel Corporation
Device ID = 2990h, 82Q963/Q965 Memory Controller Hub
PCI index = 0h
Class Codes = 060000h
Revision ID = 2h
Bus number = 0
Device number = 0
Function num = 0
Status Reg = 2090h
Command Reg = 106h
I/O space access disabled
Memory space access enabled
Bus Master enabled
Special Cycle operations ignored
Memory Write and Invalidate disabled
Palette Snooping disabled
Parity Error Response disabled
Data/Address stepping disabled
SERR# driver enabled
Fast back-to-back transactions to different agents disabled
PCI INTx enabled
Header type = 0h Single-function
BIST = 0h Build-in-self-test not supported
Latency Timer = 0h
Cache Line Size= 0h
Subsystem Vendor ID = 17aah
Subsystem ID = 1011h
Max Lat = 0ns
Min Gnt = 0ns
PCI Int Pin = NC
Interrupt line = 0
CPU Interrupt = 0h
Capabilities Pointer = e0h
Capability ID = 9h - Vendor Specific
Capabilities = 5109h - 4800036eh
Device Dependent Registers:
0x040: 0190 f1fe 0000 0000 0040 d1fe 0000 0000
0x050: 0000 3000 0900 0000 0000 0000 0000 0000
0x060: 0100 00e0 0000 0000 0180 d1fe 0000 0000
0x070: 0000 0000 0000 0000 0000 0000 0000 0000
...
0x090: 1011 1100 0011 1100 8000 8c00 001a 3900
0x0a0: 8000 4023 0000 80bf 0000 0000 0000 70bf
0x0b0: 00c0 0000 0000 0000 0000 0000 0000 0000
0x0c0: 0000 0000 0000 0000 0000 0000 0000 0000
0x0d0: 0000 0000 0000 0000 4900 0000 0005 0213
0x0e0: 0900 0951 6e03 0048 060c 0000 0000 0000
0x0f0: 0200 0000 0000 0000 900f 0400 0000 0000
Class = Display (VGA)
Vendor ID = 8086h, Intel Corporation
Device ID = 2992h, 82Q963/Q965 Integrated Graphics Controller
PCI index = 0h
Class Codes = 030000h
Revision ID = 2h
Bus number = 0
Device number = 2
Function num = 0
Status Reg = 90h
Command Reg = 7h
I/O space access enabled
Memory space access enabled
Bus Master enabled
Special Cycle operations ignored
Memory Write and Invalidate disabled
Palette Snooping disabled
Parity Error Response disabled
Data/Address stepping disabled
SERR# driver disabled
Fast back-to-back transactions to different agents disabled
PCI INTx enabled
Header type = 0h Single-function
BIST = 0h Build-in-self-test not supported
Latency Timer = 0h
Cache Line Size= 0h
BAR - 0 [Mem] = d0000000h 32bit length 1048576 enabled
BAR - 2 [Mem] = c0000000h prefetchable 64bit length 268435456 enabled
BAR - 4 [I/O] = 3400h length 8 enabled
Subsystem Vendor ID = 17aah
Subsystem ID = 1011h
Max Lat = 0ns
Min Gnt = 0ns
PCI Int Pin = INT A
Interrupt line = 11
CPU Interrupt = bh
Capabilities Pointer = 90h
Capability ID = 5h - Message Signaled Interrupts
Capabilities = 0h - 0h
Capability ID = 1h - Power Management
Capabilities = 22h - 0h
Device Dependent Registers:
0x040: 0000 0000 e000 0000 0900 0951 6e03 0048
0x050: 060c 3000 0900 0000 0000 0000 0000 80bf
0x060: 0000 0200 0000 0000 0000 0000 0000 0000
0x070: 0000 0000 0000 0000 0000 0000 0000 0000
...
0x090: 05d0 0000 0000 0000 0000 0000 0000 0000
0x0a0: 1111 0000 0000 0000 0000 0000 0000 0000
0x0b0: 0000 0000 0000 0000 0000 0000 0000 0000
0x0c0: 0000 0000 0f0f 071e 8000 1008 bc01 0000
0x0d0: 0100 2200 0000 0000 0000 0000 0001 0200
0x0e0: 0000 0000 0000 0000 0000 0000 0000 0000
0x0f0: 3400 0300 0000 0000 900f 0400 fcb6 6cbf
Class = Serial Bus (Universal Serial Bus)
Vendor ID = 8086h, Intel Corporation
Device ID = 2834h, 82801H (ICH8 Family) USB UHCI Controller #4
PCI index ...
View Full Message
|
|
|
|
|
|
