QNX Operating System/Discussion/OSTech/Specifying contiguous memory for static data/List of Posts
 
Forum Topic - Specifying contiguous memory for static data: (8 Items)
   
Update
Jim Mikola
07/29/2008 11:50 AM
post11200
Specifying contiguous memory for static data   
I'm developing a application that requires contiguous memory for statically initialized byte arrays.  The arrays are 
declared similar to:

const unsigned char bit_data[BIT_DATA_SIZE] __attribute__((aligned(64))) = 
{
    ##, ##, ##, ….
};

I obtain a physical memory address for this data to use with the embedded hardware using the mem_offset() function.  The
 returned physical address is correct but the data is not contiguous in physical memory.

Allocating a contiguous block using mmap() and copying the data doubles up the RAM utilization which is not desired 
(since i cannot "free" the original static array).  The static arrays are auto-generated and contain uncompressed image 
data.  The size of the static array can be very large (over a megabyte).

Is there any way to force the static data to be contiguous in the application?

Thanks!

Jim
Mario Charest
07/29/2008 4:24 PM
post11205
Re: Specifying contiguous memory for static data   
> I'm developing a application that requires contiguous memory for statically 
> initialized byte arrays.  The arrays are declared similar to:
> 
> const unsigned char bit_data[BIT_DATA_SIZE] __attribute__((aligned(64))) = 
> {
>     ##, ##, ##, ….
> };
> 
> I obtain a physical memory address for this data to use with the embedded 
> hardware using the mem_offset() function.  The returned physical address is 
> correct but the data is not contiguous in physical memory.
> 
> Allocating a contiguous block using mmap() and copying the data doubles up the
>  RAM utilization which is not desired (since i cannot "free" the original 
> static array).  The static arrays are auto-generated and contain uncompressed 
> image data.  The size of the static array can be very large (over a megabyte).
> 
> 
> Is there any way to force the static data to be contiguous in the application?
> 

I'm not from QNX. I don't think that's possible. Couldn't you load the data from an external file.


> 
> Thanks!
> 
> Jim
Colin Burgess(deleted)
07/29/2008 4:42 PM
post11206
Re: Specifying contiguous memory for static data   
Really the only way you could guarantee that your initialized data is contiguous is if you
are running from within an image filesystem.

The gotcha is that you can't UNMOUNT an image filesystem, so you would be stuck with one
version of your application per boot.

BTW why would you use uncompressed data - wouldn't it be better to decompress them?
And FWIW you can unmap your initialized data, although you have to be careful since it's
done on a per page basis - however with careful alignment and padding it would be doable.

Mario Charest wrote:
>> I'm developing a application that requires contiguous memory for statically 
>> initialized byte arrays.  The arrays are declared similar to:
>>
>> const unsigned char bit_data[BIT_DATA_SIZE] __attribute__((aligned(64))) = 
>> {
>>     ##, ##, ##, ….
>> };
>>
>> I obtain a physical memory address for this data to use with the embedded 
>> hardware using the mem_offset() function.  The returned physical address is 
>> correct but the data is not contiguous in physical memory.
>>
>> Allocating a contiguous block using mmap() and copying the data doubles up the
>>  RAM utilization which is not desired (since i cannot "free" the original 
>> static array).  The static arrays are auto-generated and contain uncompressed 
>> image data.  The size of the static array can be very large (over a megabyte).
>>
>>
>> Is there any way to force the static data to be contiguous in the application?
>>
> 
> I'm not from QNX. I don't think that's possible. Couldn't you load the data from an external file.
> 
> 
>> Thanks!
>>
>> Jim
> 
> 
> 
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post11205

-- 
cburgess@qnx.com
Jim Mikola
07/29/2008 6:03 PM
post11210
Re: Specifying contiguous memory for static data   
The uncompressed images are to speed up initialization time.  It also reduces RAM consumption since QNX copies the 
application into RAM before execution.  If the images were stored in the application in a compressed format then they 
would be copied into RAM with the application and then uncompressed, resulting in more RAM consumption.

An application with uncompressed images would benefit from the compression of the OS image.  Putting the application 
into a file system would remove that benefit.  Decompressing the OS image as a whole is faster than decompressing the 
images individually.

I suspect the GNU compilers used by QNX Momentics do no fragment the array.  I'm guessing QNX does that when the 
application is loaded into RAM and the virtual memory addresses are created from the physical memory locations.  I'm not
 sure how fragmenting a static array provides any benefit.

Is it possible to create a section for the images in a linker script and somehow define the section as "non-fragmented"?


It would be unfortunate if I cannot find a solution for keeping the arrays contiguous.
Colin Burgess(deleted)
07/29/2008 6:51 PM
post11212
RE: Specifying contiguous memory for static data   
If the application was in the OS image then it is already continguous - UNLESS you
have selected code=copy
 
Or if the array is not in the readonly data section - which it should be since the data
is marked const.
 
Colin

________________________________

From: Jim Mikola [mailto:community-noreply@qnx.com]
Sent: Tue 29/07/2008 6:03 PM
To: ostech-core_os
Subject: Re: Specifying contiguous memory for static data



The uncompressed images are to speed up initialization time.  It also reduces RAM consumption since QNX copies the 
application into RAM before execution.  If the images were stored in the application in a compressed format then they 
would be copied into RAM with the application and then uncompressed, resulting in more RAM consumption.

An application with uncompressed images would benefit from the compression of the OS image.  Putting the application 
into a file system would remove that benefit.  Decompressing the OS image as a whole is faster than decompressing the 
images individually.

I suspect the GNU compilers used by QNX Momentics do no fragment the array.  I'm guessing QNX does that when the 
application is loaded into RAM and the virtual memory addresses are created from the physical memory locations.  I'm not
 sure how fragmenting a static array provides any benefit.

Is it possible to create a section for the images in a linker script and somehow define the section as "non-fragmented"?


It would be unfortunate if I cannot find a solution for keeping the arrays contiguous.


_______________________________________________
OSTech
http://community.qnx.com/sf/go/post11210
Jim Mikola
07/30/2008 3:45 PM
post11261
Re: RE: Specifying contiguous memory for static data   
Awesome that worked.  It's a slow process downloading a new OS image via IPL each time to test but at least it works :).
Colin Burgess(deleted)
07/30/2008 3:55 PM
post11262
Re: Specifying contiguous memory for static data   
Presumeably you can use the bad old way of copying into allocated memory for debugging at least? :-)

mmap( 0, size, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PHYS|MAP_PRIVATE, NOFD, 0 );

will give you contig memory (if any is to be found)

Jim Mikola wrote:
> Awesome that worked.  It's a slow process downloading a new OS image via IPL each time to test but at least it works :
).
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post11261
> 

-- 
cburgess@qnx.com
Yao Zhao(deleted)
08/06/2008 9:37 AM
post11480
Re: Specifying contiguous memory for static data   
How about specify it into a section you named which align to 4k? because ldd will use mmap to map it then I assume you 
can unmap it too(but not quite sure), before you unmap it then copy to contiguous memory. If it is in same segment with 
other .data then you can do partial unmap (seems that QNX support partitial unmap now).
Even it doesn't support I think you can add a seperate segment and only this section in this segment, ldd will 
seperately map it then you can unmap it.

Return