|
Mario Charest
07/23/2009 3:52 PM
post34456
|
Is there a known memory leak in io-net of 6.3.2. It's consuming 1 meg per 10 hours till the system runs out.
I have look for leaking connections and other resources used by the our software but it all looked peachy to me.
|
|
|
|
|
|
Yao Zhao(deleted)
07/23/2009 3:58 PM
post34459
|
yes, we recently found one in tcpip stack.
but it depends on your case. Are you using select/poll with multiple fds?
I will recommend you use IDE's malloc information to monitor malloc/free, if never free then it is leak and what the
size of memory is leaking. then we can decide it is mbuf or others.
|
|
|
|
|
|
Sean Boudreau(deleted)
07/23/2009 4:00 PM
post34460
|
On Thu, Jul 23, 2009 at 03:58:18PM -0400, Yao Zhao wrote:
> yes, we recently found one in tcpip stack.
> but it depends on your case. Are you using select/poll with multiple fds?
multiple sockets. This isn't in io-pkt if that helps...
-seanb
|
|
|
|
|
|
Mario Charest
07/23/2009 4:01 PM
post34461
|
-----Original Message-----
From: Yao Zhao [mailto:community-noreply@qnx.com]
Sent: Thursday, July 23, 2009 3:58 PM
To: technology-networking
Subject: Re: io-net memory leak
yes, we recently found one in tcpip stack.
but it depends on your case. Are you using select/poll with multiple fds?
Yes. Does it also exists in 6.4.1
I will recommend you use IDE's malloc information to monitor malloc/free, if never free then it is leak and what the
size of memory is leaking. then we can decide it is mbuf or others.
It's at a customer site, and access is limited.
_______________________________________________
Technology
http://community.qnx.com/sf/go/post34459
|
|
|
|
|
|
Yao Zhao(deleted)
|
Re: RE: io-net memory leak
|
|
Yao Zhao(deleted)
07/23/2009 4:27 PM
post34463
|
Re: RE: io-net memory leak
Yes, I should say multiple sockets as Sean said.
If you can't use IDE then you can write an app to read io-net process' libc's malloc statistics to know, they are global
array so it is easy to know the address of these then you can read it to know what size of memory is leaking.
if the size is 32-64 then probably it is if not then it would be something else.
there is no this problem in io-pkt's stack.
|
|
|
|
|
|
Mario Charest
|
Re: RE: io-net memory leak
|
|
Mario Charest
07/27/2009 10:38 AM
post34632
|
Re: RE: io-net memory leak
> Yes, I should say multiple sockets as Sean said.
> If you can't use IDE then you can write an app to read io-net process' libc's
> malloc statistics to know, they are global array so it is easy to know the
> address of these then you can read it to know what size of memory is leaking.
> if the size is 32-64 then probably it is if not then it would be something
> else.
Must be something else because it's 128-256 and there is 349483 of it and it's growing by 1 every 5 seconds ( refresh
rate of IDE was set a 1 sec. I'll look through system to see what exactly is being performed every 5 seconds.
>
> there is no this problem in io-pkt's stack.
|
|
|
|
|
|
Yao Zhao(deleted)
|
Re: RE: io-net memory leak
|
|
Yao Zhao(deleted)
07/27/2009 11:51 AM
post34645
|
Re: RE: io-net memory leak
> > Yes, I should say multiple sockets as Sean said.
> > If you can't use IDE then you can write an app to read io-net process'
> libc's
> > malloc statistics to know, they are global array so it is easy to know the
> > address of these then you can read it to know what size of memory is leaking
> .
> > if the size is 32-64 then probably it is if not then it would be something
> > else.
>
> Must be something else because it's 128-256 and there is 349483 of it and it's
> growing by 1 every 5 seconds ( refresh rate of IDE was set a 1 sec. I'll
> look through system to see what exactly is being performed every 5 seconds.
>
> >
> > there is no this problem in io-pkt's stack.
>
>
That problem we found is when you select/poll multiple sockets, if you close them and then select/poll multiple other
sockets and repeat these steps, then you will see it is leaking, if you didn't close them then you won't notice they are
leaking although it is.
128-256: it is not mbuf either, I remember mbuf is 256. 128-256 should contain some overhead by malloc. try netstat -m
to show mbuf statistics and maybe netstat -an to show socket queues.
another suggestion is: because you are using IDE maybe you can use gdb too(if you have 2 ports then run 2nd io-net to
use gdb), attach to that io-net, break malloc with conditional break if size > 123 (to be safe make it smaller as
malloc's management overhead), if you can use size then use register to set condition. Or easily set mmap as breakpoint:
gdb 2>&1 | tee log
handle SIGUSR2 pass nostop noprint
break mmap
command breakpoint number
silent
bt
continue
end
this will log all mmap's backtrace and it should log malloc(12x) because it is too frequent. then you should be able to
know where is leaking or not.
another try: kill all of io-net's client to see whether it will free memory. if used bind with unix socket you should
unlink or it will leak too. You can use gdb to check this too as the list is a global variable.
|
|
|
|
|
|
Mario Charest
|
Re: RE: io-net memory leak
|
|
Mario Charest
07/27/2009 3:31 PM
post34665
|
Re: RE: io-net memory leak
> > > Yes, I should say multiple sockets as Sean said.
> > > If you can't use IDE then you can write an app to read io-net process'
> > libc's
> > > malloc statistics to know, they are global array so it is easy to know the
>
> > > address of these then you can read it to know what size of memory is
> leaking
> > .
> > > if the size is 32-64 then probably it is if not then it would be something
>
> > > else.
> >
> > Must be something else because it's 128-256 and there is 349483 of it and
> it's
> > growing by 1 every 5 seconds ( refresh rate of IDE was set a 1 sec. I'll
> > look through system to see what exactly is being performed every 5 seconds.
>
> >
> > >
> > > there is no this problem in io-pkt's stack.
> >
> >
>
>
> That problem we found is when you select/poll multiple sockets, if you close
> them and then select/poll multiple other sockets and repeat these steps, then
> you will see it is leaking, if you didn't close them then you won't notice
> they are leaking although it is.
>
> 128-256: it is not mbuf either, I remember mbuf is 256. 128-256 should contain
> some overhead by malloc. try netstat -m to show mbuf statistics and maybe
> netstat -an to show socket queues.
>
Not 100% sure yet but I think it's cause be an experimental version of devn-e1000.
|
|
|
|
|
|
Yao Zhao(deleted)
|
Re: RE: io-net memory leak
|
|
Yao Zhao(deleted)
08/09/2009 10:07 PM
post35522
|
Re: RE: io-net memory leak
|
|
|
|
|
|
Mario Charest
|
RE: RE: io-net memory leak
|
|
Mario Charest
08/09/2009 11:05 PM
post35523
|
RE: RE: io-net memory leak
Thanks I was just to post on this issue which the latest devn-e1000 has not solved. This week-end I killed all the
program that I could. Only keeping the one that I needed to keep the machine remotely accessible. None of our
softwares were running and memory was still leaking.
I`ll keep you posted
-----Original Message-----
From: Yao Zhao [mailto:community-noreply@qnx.com]
Sent: August 9, 2009 10:07 PM
To: technology-networking
Subject: Re: RE: io-net memory leak
This weekend I wrote the command line utility to monitor a process' heap usage.
code is attached.
svn can be checked out from here:
http://community.qnx.com/integration/viewvc/viewvc.cgi/trunk/utils/h/hmon/monitor.c?revision=42&root=qfuse&system=
exsy1001&view=markup
_______________________________________________
Technology
http://community.qnx.com/sf/go/post35522
|
|
|
|
|
|
Yao Zhao(deleted)
|
Re: RE: RE: io-net memory leak
|
|
Yao Zhao(deleted)
08/10/2009 8:38 AM
post35538
|
Re: RE: RE: io-net memory leak
> Thanks I was just to post on this issue which the latest devn-e1000 has not
> solved. This week-end I killed all the program that I could. Only keeping the
> one that I needed to keep the machine remotely accessible. None of our
> softwares were running and memory was still leaking.
>
> I`ll keep you posted
>
> -----Original Message-----
> From: Yao Zhao [mailto:community-noreply@qnx.com]
> Sent: August 9, 2009 10:07 PM
> To: technology-networking
> Subject: Re: RE: io-net memory leak
>
> This weekend I wrote the command line utility to monitor a process' heap usage
> .
> code is attached.
> svn can be checked out from here:
> http://community.qnx.com/integration/viewvc/viewvc.cgi/trunk/utils/h/hmon/
> monitor.c?revision=42&root=qfuse&system=exsy1001&view=markup
>
>
>
> _______________________________________________
>
> Technology
> http://community.qnx.com/sf/go/post35522
I thought it was solved:(
io-net and io-pkt are special as the thread stacks may increase too although there is a hard limit.
|
|
|
|
|
|
Mario Charest
|
RE: RE: RE: io-net memory leak
|
|
Mario Charest
08/10/2009 9:55 AM
post35561
|
RE: RE: RE: io-net memory leak
> I thought it was solved:(
> io-net and io-pkt are special as the thread stacks may increase too
> although there is a hard limit.
Nah it's the heap, not the stacks.
|
|
|
|
|
|
Mario Charest
|
RE: RE: RE: io-net memory leak
|
|
Mario Charest
08/11/2009 1:10 PM
post35731
|
RE: RE: RE: io-net memory leak
How much ram does a tx and rx descriptor use? I assume it's close to the packet size?
> -----Original Message-----
> From: Yao Zhao [mailto:community-noreply@qnx.com]
> Sent: Monday, August 10, 2009 8:39 AM
> To: technology-networking
> Subject: Re: RE: RE: io-net memory leak
>
> > Thanks I was just to post on this issue which the latest devn-e1000
> has not
> > solved. This week-end I killed all the program that I could. Only
> keeping the
> > one that I needed to keep the machine remotely accessible. None of
> our
> > softwares were running and memory was still leaking.
> >
> > I`ll keep you posted
> >
> > -----Original Message-----
> > From: Yao Zhao [mailto:community-noreply@qnx.com]
> > Sent: August 9, 2009 10:07 PM
> > To: technology-networking
> > Subject: Re: RE: io-net memory leak
> >
> > This weekend I wrote the command line utility to monitor a process'
> heap usage
> > .
> > code is attached.
> > svn can be checked out from here:
> >
> http://community.qnx.com/integration/viewvc/viewvc.cgi/trunk/utils/h/hm
> on/
> > monitor.c?revision=42&root=qfuse&system=exsy1001&view=markup
> >
> >
> >
> > _______________________________________________
> >
> > Technology
> > http://community.qnx.com/sf/go/post35522
>
> I thought it was solved:(
> io-net and io-pkt are special as the thread stacks may increase too
> although there is a hard limit.
>
>
>
> _______________________________________________
>
> Technology
> http://community.qnx.com/sf/go/post35538
>
|
|
|
|
|
|
Andrew Boyd(deleted)
|
RE: RE: RE: io-net memory leak
|
|
Andrew Boyd(deleted)
08/11/2009 1:13 PM
post35732
|
RE: RE: RE: io-net memory leak
> How much ram does a tx and rx descriptor use?
It depends upon each driver, but generally, a
tx descriptor only costs you 16 or 32 bytes,
but a rx descriptor costs you 16 or 32 bytes
PLUS 2048 bytes (half a page) with io-net or
io-pkt.
Generally, tx descriptors are cheap and well
worth the very few bytes they cost you. Rx
descriptors - because they are assigned an
empty packet buffer - chew up the memory.
But, it's probably better to use a megabyte
or two, than to lose packets.
--
aboyd
|
|
|
|
|
|
Mario Charest
|
RE: RE: RE: io-net memory leak
|
|
Mario Charest
08/11/2009 1:31 PM
post35734
|
RE: RE: RE: io-net memory leak
Hum, tx=4096 and rx at 406 is a little over 8Meg though. Yet a few seconds after booting io-net uses over 68Meg of
memory ???? Something doesn't add up.
io-net is passed 4 drivers to start (ie1000, tigon, pcnet, speedo) 3 of them have the rx=4096 and tx=4096. Out of the 4
drivers only 1 stay in memory because there is only one model of NIC in the machine. That could explain the 68Meg, but
not the memory leak. I'll run some test.
Just did the test and the same image that is a the customer site but running on VMWare here in the office ( so pcnet
instead of e1000) used only 1.5Meg at startup.
I'll keep digging.
> -----Original Message-----
> From: Andrew Boyd [mailto:community-noreply@qnx.com]
> Sent: Tuesday, August 11, 2009 1:13 PM
> To: technology-networking
> Subject: RE: RE: RE: io-net memory leak
>
>
>
> > How much ram does a tx and rx descriptor use?
>
> It depends upon each driver, but generally, a
> tx descriptor only costs you 16 or 32 bytes,
> but a rx descriptor costs you 16 or 32 bytes
> PLUS 2048 bytes (half a page) with io-net or
> io-pkt.
>
> Generally, tx descriptors are cheap and well
> worth the very few bytes they cost you. Rx
> descriptors - because they are assigned an
> empty packet buffer - chew up the memory.
>
> But, it's probably better to use a megabyte
> or two, than to lose packets.
>
> --
> aboyd
>
>
>
>
>
> _______________________________________________
>
> Technology
> http://community.qnx.com/sf/go/post35732
>
|
|
|
|
|
|
Mario Charest
|
RE: RE: io-net memory leak
|
|
Mario Charest
08/10/2009 9:36 AM
post35550
|
RE: RE: io-net memory leak
> -----Original Message-----
> From: Yao Zhao [mailto:community-noreply@qnx.com]
> Sent: Sunday, August 09, 2009 10:07 PM
> To: technology-networking
> Subject: Re: RE: io-net memory leak
>
> This weekend I wrote the command line utility to monitor a process'
> heap usage.
> code is attached.
> svn can be checked out from here:
> http://community.qnx.com/integration/viewvc/viewvc.cgi/trunk/utils/h/hm
> on/monitor.c?revision=42&root=qfuse&system=exsy1001&view=markup
I get the feeling, mere mortals don't have access to this stuff?
>
>
>
> _______________________________________________
>
> Technology
> http://community.qnx.com/sf/go/post35522
|
|
|
|
|
|
Vineet Garg
|
Re: RE: io-net memory leak
|
|
Vineet Garg
08/31/2010 11:27 PM
post65363
|
Re: RE: io-net memory leak
Hi
This is an old thread, but i'll like to know if this problem was resolved.
I am facing a memory leak where "pidin mem | grep io-net " shows continuous increase in memory utilization over a period
of time. We are running qnx 6.3.0 and io-net.
Any idea if this could be linked to an already known issue on io-net?
Looking for help badly on this.
Regards
Vineet
|
|
|
|
|
|
Andrew Boyd(deleted)
|
RE: RE: io-net memory leak
|
|
Andrew Boyd(deleted)
08/31/2010 11:58 PM
post65365
|
RE: RE: io-net memory leak
99% of the time, this is caused by the driver
leaking packets in a race condition somewhere.
What driver are you using?
--
aboyd
________________________________
From: Vineet Garg [mailto:community-noreply@qnx.com]
Sent: Tue 8/31/2010 8:27 PM
To: technology-networking
Subject: Re: RE: io-net memory leak
Hi
This is an old thread, but i'll like to know if this problem was resolved.
I am facing a memory leak where "pidin mem | grep io-net " shows continuous increase in memory utilization over a period
of time. We are running qnx 6.3.0 and io-net.
Any idea if this could be linked to an already known issue on io-net?
Looking for help badly on this.
Regards
Vineet
_______________________________________________
Technology
http://community.qnx.com/sf/go/post65363
|
|
|
|
|
|
Sean Boudreau(deleted)
|
Re: RE: io-net memory leak
|
|
Sean Boudreau(deleted)
09/01/2010 2:10 PM
post65529
|
Re: RE: io-net memory leak
On Tue, Aug 31, 2010 at 11:27:14PM -0400, Vineet Garg wrote:
> Hi
>
> This is an old thread, but i'll like to know if this problem was resolved.
>
> I am facing a memory leak where "pidin mem | grep io-net " shows continuous increase in memory utilization over a
period of time. We are running qnx 6.3.0 and io-net.
>
> Any idea if this could be linked to an already known issue on io-net?
>
> Looking for help badly on this.
See the top of this thread. One leak was plugged
in select() / poll() handling on multiple sockets.
Regards,
-seanb
|
|
|
|
|
|
Vineet Garg
|
Re: RE: io-net memory leak
|
|
Vineet Garg
09/28/2010 2:00 PM
post68856
|
Re: RE: io-net memory leak
Thanks Andrew & Sean for replying.
I have been a little late in this update but I eventually found a leak in ethernet driver (proprietary driver feeding
stack from a resource manager) in case packets coming from stack had more than 8 buffers in the chain.
That brings me to question about why stack uses different number of buffer for packets of different sizes. Is that
because io-net has a buffer pool of small size and it uses a number of them to create complete packet?
|
|
|
|
|
|
Vineet Garg
|
Re: RE: io-net memory leak
|
|
Vineet Garg
09/28/2010 2:03 PM
post68857
|
Re: RE: io-net memory leak
Hi Sean
Fixed the leak that I was facing, however for future dev can you please let me know the way to find out if the select()/
poll() memory leak fix in io-net is present in OS version that I am using.
Regards
VG
> On Tue, Aug 31, 2010 at 11:27:14PM -0400, Vineet Garg wrote:
> > Hi
> >
> > This is an old thread, but i'll like to know if this problem was resolved.
> >
> > I am facing a memory leak where "pidin mem | grep io-net " shows continuous
> increase in memory utilization over a period of time. We are running qnx 6.3.0
> and io-net.
> >
> > Any idea if this could be linked to an already known issue on io-net?
> >
> > Looking for help badly on this.
>
> See the top of this thread. One leak was plugged
> in select() / poll() handling on multiple sockets.
>
> Regards,
>
> -seanb
|
|
|
|
|
|
