Forum Topic - Tracking dynamic memory allocation for a given process:
   
Tracking dynamic memory allocation for a given process  
Hello,

I'm looking for a way to verify that a given process no longer allocates memory after having completed its 
initialization phase.

So I need a piece of C code to read the current memory allocated by a given process.
What would be the best option to implement this?
I see that "/proc/%d/vmstat" could be parsed, but it seems to me that it provides number of mapped pages, while I would 
need the exact allocated memory size.
Any idea?

Thanks.
Re: Tracking dynamic memory allocation for a given process  
From the system's point of view. the number of pages used by the process is the amount of memory allocated (there are 
subtleties, like accounting for shared mappings). It's not clear what metric you are looking for.

--Elad
Re: Tracking dynamic memory allocation for a given process  
> Hello,
> 
> I'm looking for a way to verify that a given process no longer allocates 
> memory after having completed its initialization phase.

What problem are you trying to solve?

IMO, the only way you can be even moderately sure that a given program won't allocate any more memory after a given 
point is to analyse the code ... just observing its allocation pattern from "outside" says nothing about future 
behaviour.

Having said that, in addition to what Elad wrote, you can use /proc/PID/pmap to find out what process with pid PID has 
mapped *at that instant*, and even do diffs between consecutive runs, but as I said above, that's no guarantee.
You can read that data like any text file, but since it's a pseudo-file generated by the system on the fly, you need to 
close and reopen it for consecutive runs.

HTH
Michael
Re: Tracking dynamic memory allocation for a given process  
Which version of QNX, 7.1 or 8.0?
Re: Tracking dynamic memory allocation for a given process  
and are you aiming to verify from within the process or by looking externally at a process?
Re: Tracking dynamic memory allocation for a given process  
Hello,

OK, sorry. Let me try to clarify my question:

First, I'm using QNX 8.0.
For a certification process, I need to prove that a given program won't allocate any more memory after a given point: So
, no more malloc() or equivalent after a given point (End of initialization phase).
Analyzing the code by code review could be an option, but I wanted to automate this par inspecting this program from an 
external C application.

I wrote a C routine which parses the  "as_stats.rss" field from "/proc/%d/vmstat" file.
This returns me the numbers of pages mapped by the process, and I could multiply rss * _SC_PAGESIZE to obtain the size 
of the memory mapped by the process.
But what I would really need is the size of the memory currently used by the process.

For example, if my process allocates some memory after the end of its initialization phase, the OS can potentially use 
the remaining unused memory in mapped pages, without having to map a new memory page.
And, in this specific case, parsing "as_stats.rss" field will not indicate that memory has been allocated after the init
 phase end.

Hoping this is clearer now.

Thanks.
Regards
Jean-Michel
Re: Tracking dynamic memory allocation for a given process  
> Hello,
> 
> OK, sorry. Let me try to clarify my question:
> 
> First, I'm using QNX 8.0.
> For a certification process, I need to prove that a given program won't 
> allocate any more memory after a given point: So, no more malloc() or 
> equivalent after a given point (End of initialization phase).
> Analyzing the code by code review could be an option, but I wanted to automate
>  this par inspecting this program from an external C application.

As I wrote, I don't believe that can work.

> I wrote a C routine which parses the  "as_stats.rss" field from "/proc/%d/
> vmstat" file.
> This returns me the numbers of pages mapped by the process, and I could 
> multiply rss * _SC_PAGESIZE to obtain the size of the memory mapped by the 
> process.
> But what I would really need is the size of the memory currently used by the 
> process.

There is no distinction from the POV of the OS between "mapped" and "in use"  (with the exception of lazy mapping, which
 I think you're not talking about, and which we currently don't support in QNX 8) by a process. Also, mappings happen on
 per-page basis, you can't go smaller than that.
 
> For example, if my process allocates some memory after the end of its 
> initialization phase, the OS can potentially use the remaining unused memory 
> in mapped pages, without having to map a new memory page.

This doesn't make much sense to me, sorry. Perhaps other can follow better.

> And, in this specific case, parsing "as_stats.rss" field will not indicate 
> that memory has been allocated after the init phase end.
> 
> Hoping this is clearer now.

erm... not really (to me at least)

Michael

> 
> Thanks.
> Regards
> Jean-Michel


Re: Tracking dynamic memory allocation for a given process  
> For example, if my process allocates some memory after the end of its 
> initialization phase, the OS can potentially use the remaining unused memory 
> in mapped pages, without having to map a new memory page.
> And, in this specific case, parsing "as_stats.rss" field will not indicate 
> that memory has been allocated after the init phase end.

If you want to say that a malloc() or free() call will not necessarily change the number of mapped pages, you are right.


What about a preload library with stubs for malloc() and free().
You could count them and check then number by dumping them in a signal handler.
Maybe you can also reset the counters in a signal handler after the init phase and check if new calls are made later.

Not sure however, if this is an acceptable approach for your certification process.
Re: Tracking dynamic memory allocation for a given process  
Yes, we thought about using stubs to "disable" malloc operations after the end of our application initialization, but we
 're using other libraries which may also allocate memory...

Thanks anyway.
Regards.
Jean-Michel
Re: Tracking dynamic memory allocation for a given process  
stubs in your own code, or stubs in a preload library that is loaded by LD_PRELOAD?
Even calls from other libraries will use the stubs in a preload library.

https://www.qnx.com/developers/docs/8.0/com.qnx.doc.neutrino.sys_arch/topic/dll_SYMBOLNAME.html
Re: Tracking dynamic memory allocation for a given process  
You would need own implementations of all heap-related functions (malloc, realloc, free, strdup, ...) in a dedicated 
shared object, that you then LD_PRELOAD. This should capture all heap operations, even from external libs - as long as 
they are not statically linked against libc.

This is how the memory analysis tools work.

Besides, even the standard heap allocator does have some instrumentation, maybe this is already sufficient, it must be 
possible to read that from inside via mallopt(). (is that still available in QNX 8?)

Regards,
Albrecht
Re: Tracking dynamic memory allocation for a given process  
The problem with only looking at heap functions is that this actually misses the case of dynamic memory allocations that
 don't use the C library heap. Any code can invoke mmap() and get more memory to manage outside the primary heap.
It is not clear what safety requirement the original question pertains to. Is it really trying to lock the memory usage 
of the process, or is it trying to avoid standard library calls to the heap?

--Elad