Project Home
Project Home
Documents
Documents
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - Ooops! Few problems with ARM memory management.: (8 Items)
   
Ooops! Few problems with ARM memory management.  
Hi!

I do some experiments with organization of shmem in ARM (xScale pxa255) in QNX 6.3.2.

My current task is to allocate in process as much memory as it's possible.
As soon as application address space in ARM limited by 32M and maximal malloc()able block is about 15Mb, I'm trying my 
best with shared memory.
So reading http://www.qnx.com/developers/docs/6.3.2/neutrino/prog/arm_memory.html
I found that my problem can be solved by using global shmem. :-)

So, I wrote an small test application (see attach).
I have problem with understanding the behavior of QNX under my tests...

My system says:
# pidin info
CPU:ARM Release:6.3.0  FreeMem:46Mb/64Mb BootTime:Jan 01 00:00:00 UTC 1970
Processor1: 1761946886 pxa250 399MHz

1. Lats say I want to allocate 33 Mb of shmem...

When I set SHMEM_SIZE = (33 * 1024 * 1024) my program prints following:
SHMEM allocation test.
shm_open() -> fd = 3 OK
ftruncate() -> r = 0 OK
shm_ctl() -> r = -1 Function not implemented
mmap() -> shmem = 0xFFFFFFFF Not enough memory
Code address:  &main() = 0x00100940
Data address:  &fd     = 0x00101DC4
Stack address: &shmem  = 0x000FFCE0
Shmem address: shmem   = 0xFFFFFFFF
Memsetting... <SIGSEGV>

Shmem block is created but not mapped in address space!
# ls -l /dev/shmem
total 67584
nrwxrwxrwx  3 root      0          34603008 Mar 18 15:53 big_memory_block

Why if it's possible to map (according to docs) more than 32 Mb of physical RAM, it's impossible to map such amount of 
shared RAM?

2. When I set SHMEM_SIZE to 18 or 16 Mb system can't map it globally too!

3. When I set SHMEM_SIZE in 5 Mb memory is mapped successfully, but I see different behavior of C library when I change 
an order of ftruncate() and shm_ctl().
You can see 2 outputs here below.

3a. Case shm_open()->ftruncate()->shm_ctl()->mmap().
SHMEM allocation test.
shm_open() -> fd = 3 OK
ftruncate() -> r = 0 OK
shm_ctl() -> r = -1 Function not implemented
mmap() -> shmem = 0x01800000 OK
Code address:  &main() = 0x00100940
Data address:  &fd     = 0x00101DC4
Stack address: &shmem  = 0x000FFCE0
Shmem address: shmem   = 0x01800000
Memsetting... OK

Result is that memory is mapped locally (0x01800000).

3b. Case shm_open()->shm_ctl()->ftruncate()->mmap().
SHMEM allocation test.
shm_open() -> fd = 3 OK
shm_ctl() -> r = 0 OK
ftruncate() -> r = -1 Invalid argument
mmap() -> shmem = 0x82000000 OK
Code address:  &main() = 0x00100940
Data address:  &fd     = 0x00101DC4
Stack address: &shmem  = 0x000FFCE0
Shmem address: shmem   = 0x82000000
Memsetting... OK

Result is that memory is mapped globally (0x82000000).

Here I have some problem with usage of ftruncate() and shm_ctl() together.

And main question is: want I'm doing wrong when I trying to map big peace of RAM in global address space zone?

Thanks!

PS: Document (see link above) says that it's possible to create up to 63 processes in ARM based system. But this doc 
didn't say that it's absolutely impossible to change this number to 95 or so... What about new option for procnto what 
change this parameter at startup?
For our project it's really critical limitation...

--
AG
Attachment: Compressed file shmem-test.zip 24.69 KB
Re: Ooops! Few problems with ARM memory management.  
You can't mix ftruncate() and shm_ctl():
- ftruncate() is used to set the size of "ordinary" shared memory objects
- shm_ctl() is used to set the size of "special" shared memory objects

shm_ctl() is a one-time operation to set up a special shared memory object:
- if its size has already been set by ftruncate(), shm_ctl() will fail
- if it has already been initialised by a previous shm_ctl() call, a 2nd
   shm_ctl() will fail

Your code should work if you remove the ftruncate() call.

Also, be aware that by default these global mappings are protected so that
only those processes that mmap() the object can access the global mapping.
This requires updating L1 page table entries on context switches to change
the access protections. You can avoid this cost by using one of these flags
along with the SHMCTL_GLOBAL:
- SHMCTL_PRIV: mapping is visible to all processes, but only processes with
   I/O privilege (ie. used ThreadCtl) can access it; other processes will
   get a sigsegv on an attempted access
- SHMCTL_LOWERPROT: mapping is visible and accessible to all processes.

If you don't use either of these flags, ou may need to create and mmap() the
global object _before_ you call ThreadCtl() to work around a problem where
the memmgr doesn't record the protection status for the mmap() because the
ThreadCtl() causes all mappings to be locked - this invokes a slightly
different code path which is missing the code to indicate that the global
mapping needs to be manipulated on context switches...

Comments on your observations are below:

Andrew Golovnia wrote:
> Hi!
> 
> I do some experiments with organization of shmem in ARM (xScale pxa255) 
> in QNX 6.3.2.
> 
> My current task is to allocate in process as much memory as it's possible.
> As soon as application address space in ARM limited by 32M and maximal 
> malloc()able block is about 15Mb, I'm trying my best with shared memory.
> 
> So reading 
> http://www.qnx.com/developers/docs/6.3.2/neutrino/prog/arm_memory.html
> I found that my problem can be solved by using global shmem. :-)
> 
> So, I wrote an small test application (see attach).
> I have problem with understanding the behavior of QNX under my tests...
> 
> My system says:
> # pidin info
> CPU:ARM Release:6.3.0  FreeMem:46Mb/64Mb BootTime:Jan 01 00:00:00 UTC 1970
> Processor1: 1761946886 pxa250 399MHz
> 
> 1. Lats say I want to allocate 33 Mb of shmem...
> 
> When I set SHMEM_SIZE = (33 * 1024 * 1024) my program prints following:
> SHMEM allocation test.
> shm_open() -> fd = 3 OK
> ftruncate() -> r = 0 OK
> shm_ctl() -> r = -1 Function not implemented

Your ftruncate() created an ordinary 33MB object so the shm_ctl() fails.

> mmap() -> shmem = 0xFFFFFFFF Not enough memory

The mmap() fails because an ordinary shared memory object is mapped into
the process address space, but it cannot fit.

> Code address:  &main() = 0x00100940
> Data address:  &fd     = 0x00101DC4
> Stack address: &shmem  = 0x000FFCE0
> Shmem address: shmem   = 0xFFFFFFFF
> Memsetting... <SIGSEGV>
> 
> Shmem block is created but not mapped in address space!
> # ls -l /dev/shmem
> total 67584
> nrwxrwxrwx  3 root      0          34603008 Mar 18 15:53 big_memory_block
> 
> Why if it's possible to map (according to docs) more than 32 Mb of 
> physical RAM, it's impossible to map such amount of shared RAM?

You can only map more than 32MB into the "global address space".
Your ftruncate() call created an ordinary shared memory object that will
be mapped into the process address space, NOT the global address space.

> 2. When I set SHMEM_SIZE to 18 or 16 Mb system can't map it globally too!
> 
> 3. When I set SHMEM_SIZE in 5 Mb memory is mapped successfully, but I 
> see different behavior of C library when I change an...
View Full Message
Re: Ooops! Few problems with ARM memory management.  
Thanks for your fast reply!

Lat's say I want to have this memory only for me as a workaround for 32Mb limitation of ARM. Let's make it "privat". :-)


I made following changes in code:
...
#define SHMEM_SIZE      (33 * 1024 * 1024)   // 33Mb
...
   fd = shm_open(SHMEM_NAME, O_RDWR | O_CREAT /*| O_EXCL*/, 0777);
   printf("shm_open() -> fd = %d %s\n", fd, (fd == -1)? strerror(errno): "OK");

// I remove ftruncate() here

   r = shm_ctl(fd, SHMCTL_ANON | SHMCTL_GLOBAL | SHMCTL_PRIV, 0, SHMEM_SIZE);
   printf("shm_ctl() -> r = %d %s\n", r, (r == -1)? strerror(errno): "OK");

   shmem = mmap(0, SHMEM_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
   printf("mmap() -> shmem = 0x%08X %s\n", (unsigned int)shmem, (shmem == MAP_FAILED)? strerror(errno): "OK");
...

But it cannot map 33 Mb block because of shm_ctl() fails:
SHMEM allocation test.
shm_open() -> fd = 3 OK
shm_ctl() -> r = -1 Function not implemented
mmap() -> shmem = 0xFFFFFFFF Invalid argument
...

Then I set:
...
#define SHMEM_SIZE      (5 * 1024 * 1024)   // 5Mb
...
And now it works fine:
SHMEM allocation test.
shm_open() -> fd = 3 OK
shm_ctl() -> r = 0 OK
mmap() -> shmem = 0x82000000 OK
...

Here is some restriction by size in shm_ctl(). Or I didn't get something? :-/

--
AG

> You can't mix ftruncate() and shm_ctl():
> - ftruncate() is used to set the size of "ordinary" shared memory objects
> - shm_ctl() is used to set the size of "special" shared memory objects
> 
> shm_ctl() is a one-time operation to set up a special shared memory object:
> - if its size has already been set by ftruncate(), shm_ctl() will fail
> - if it has already been initialised by a previous shm_ctl() call, a 2nd
>    shm_ctl() will fail
> 
> Your code should work if you remove the ftruncate() call.
> 
> Also, be aware that by default these global mappings are protected so that
> only those processes that mmap() the object can access the global mapping.
> This requires updating L1 page table entries on context switches to change
> the access protections. You can avoid this cost by using one of these flags
> along with the SHMCTL_GLOBAL:
> - SHMCTL_PRIV: mapping is visible to all processes, but only processes with
>    I/O privilege (ie. used ThreadCtl) can access it; other processes will
>    get a sigsegv on an attempted access
> - SHMCTL_LOWERPROT: mapping is visible and accessible to all processes.
> 
> If you don't use either of these flags, ou may need to create and mmap() the
> global object _before_ you call ThreadCtl() to work around a problem where
> the memmgr doesn't record the protection status for the mmap() because the
> ThreadCtl() causes all mappings to be locked - this invokes a slightly
> different code path which is missing the code to indicate that the global
> mapping needs to be manipulated on context switches...
> 
> Comments on your observations are below:
> 
> Andrew Golovnia wrote:
> > Hi!
> > 
> > I do some experiments with organization of shmem in ARM (xScale pxa255) 
> > in QNX 6.3.2.
> > 
> > My current task is to allocate in process as much memory as it's possible.
> > As soon as application address space in ARM limited by 32M and maximal 
> > malloc()able block is about 15Mb, I'm trying my best with shared memory.
> > 
> > So reading 
> > http://www.qnx.com/developers/docs/6.3.2/neutrino/prog/arm_memory.html
> > I found that my problem can be solved by using global shmem. :-)
> > 
> > So, I wrote an small test application (see attach).
> > I have problem with understanding the behavior of QNX under my tests...
> > 
> > My system says:
> > # pidin info
> > CPU:ARM Release:6.3.0  FreeMem:46Mb/64Mb BootTime:Jan 01 00:00:00 UTC 1970
> >...
View Full Message
Re: Ooops! Few problems with ARM memory management.  
That shm_ctl/mmap should work (assuming you have > 33MB of free memory).
Are you sure you've unlinked the object so that the shm_ctl() operates on
a newly created shared memory object?

The following simple test case worked for me:

#include <sys/neutrino.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <errno.h>

#define SHMEM_SIZE      (33*1024*1024)

int
main()
{
         int     fd;
         int     r;
         void    *p;

         fd = shm_open("/foo", O_RDWR|O_CREAT, 0777);
         printf("shm_open: %d\n", fd);

         r = shm_ctl(fd, SHMCTL_ANON|SHMCTL_GLOBAL|SHMCTL_PRIV, 0,
SHMEM_SIZE);
         printf("shm_ctl: %d %d\n", r, errno);

         p = mmap(0, SHMEM_SIZE, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
         printf("mmap: %x %d\n", p, errno);

         shm_unlink("/foo");
}

Andrew Golovnia wrote:
> Thanks for your fast reply!
> 
> Lat's say I want to have this memory only for me as a workaround for 
> 32Mb limitation of ARM. Let's make it "privat". :-)
> 
> I made following changes in code:
> ...
> #define SHMEM_SIZE      (33 * 1024 * 1024)   // 33Mb
> ...
>    fd = shm_open(SHMEM_NAME, O_RDWR | O_CREAT /*| O_EXCL*/, 0777);
>    printf("shm_open() -> fd = %d %s\n", fd, (fd == -1)? strerror(errno): 
> "OK");
> 
> // I remove ftruncate() here
> 
>    r = shm_ctl(fd, SHMCTL_ANON | SHMCTL_GLOBAL | SHMCTL_PRIV, 0, 
> SHMEM_SIZE);
>    printf("shm_ctl() -> r = %d %s\n", r, (r == -1)? strerror(errno):
"OK");
> 
>    shmem = mmap(0, SHMEM_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
>    printf("mmap() -> shmem = 0x%08X %s\n", (unsigned int)shmem, (shmem 
> == MAP_FAILED)? strerror(errno): "OK");
> ...
> 
> But it cannot map 33 Mb block because of shm_ctl() fails:
> SHMEM allocation test.
> shm_open() -> fd = 3 OK
> shm_ctl() -> r = -1 Function not implemented
> mmap() -> shmem = 0xFFFFFFFF Invalid argument
> ...
> 
> Then I set:
> ...
> #define SHMEM_SIZE      (5 * 1024 * 1024)   // 5Mb
> ...
> And now it works fine:
> SHMEM allocation test.
> shm_open() -> fd = 3 OK
> shm_ctl() -> r = 0 OK
> mmap() -> shmem = 0x82000000 OK
> ...
> 
> Here is some restriction by size in shm_ctl(). Or I didn't get 
> something? :-/
> 
> -- 
> AG
> 
>  > You can't mix ftruncate() and shm_ctl():
>  > - ftruncate() is used to set the size of "ordinary" shared memory 
> objects
>  > - shm_ctl() is used to set the size of "special" shared memory objects
>  >
>  > shm_ctl() is a one-time operation to set up a special shared memory 
> object:
>  > - if its size has already been set by ftruncate(), shm_ctl() will fail
>  > - if it has already been initialised by a previous shm_ctl() call, a
2nd
>  >    shm_ctl() will fail
>  >
>  > Your code should work if you remove the ftruncate() call.
>  >
>  > Also, be aware that by default these global mappings are protected so 
> that
>  > only those processes that mmap() the object can access the global 
> mapping.
>  > This requires updating L1 page table entries on context switches to 
> change
>  > the access protections. You can avoid this cost by using one of these 
> flags
>  > along with the SHMCTL_GLOBAL:
>  > - SHMCTL_PRIV: mapping is visible to all processes, but only 
> processes with
>  >    I/O privilege (ie. used ThreadCtl) can access it; other processes 
> will
>  >    get a sigsegv on an attempted access
>  > - SHMCTL_LOWERPROT: mapping is visible and accessible to all processes.
>  >
>  > If you don't use either of these flags, ou may need to create and 
> mmap() the
>  > global object _before_ you call...
View Full Message
Re: Ooops! Few problems with ARM memory management.  
All shmem objects are destroyed.

# ls -l /dev/shmem
total 0
# pidin info
CPU:ARM Release:6.3.0  FreeMem:46Mb/64Mb BootTime:Jan 01 00:00:00 UTC 1970
Processor1: 1761946886 pxa250 399MHz

And I have this output:
shm_open: 3
shm_ctl: -1 89
mmap: ffffffff 22

I run this code over qconn and from telnet. Both show the same output.
Hmm... :-/

--
AG

> That shm_ctl/mmap should work (assuming you have > 33MB of free memory).
> Are you sure you've unlinked the object so that the shm_ctl() operates on
> a newly created shared memory object?
> 
> The following simple test case worked for me:
> 
> #include <sys/neutrino.h>
> #include <sys/mman.h>
> #include <fcntl.h>
> #include <errno.h>
> 
> #define SHMEM_SIZE      (33*1024*1024)
> 
> int
> main()
> {
>          int     fd;
>          int     r;
>          void    *p;
> 
>          fd = shm_open("/foo", O_RDWR|O_CREAT, 0777);
>          printf("shm_open: %d\n", fd);
> 
>          r = shm_ctl(fd, SHMCTL_ANON|SHMCTL_GLOBAL|SHMCTL_PRIV, 0,
> SHMEM_SIZE);
>          printf("shm_ctl: %d %d\n", r, errno);
> 
>          p = mmap(0, SHMEM_SIZE, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
>          printf("mmap: %x %d\n", p, errno);
> 
>          shm_unlink("/foo");
> }
> 
> Andrew Golovnia wrote:
<CUT>
Re: Ooops! Few problems with ARM memory management.  
oops! I think I have a wrong procnto running in ARM...
It says it's 6.3.0 instead of 6.3.2....
I'll try tomorrow with new procnto.

Thanks!

--
AG
Re: Ooops! Few problems with ARM memory management.  
I just tried with a 630 procnto:

# uname -a
QNX prejacinto-1 6.3.0 2005/08/09-14:59:23EDT
Texas_Instruments_DAVINCI_DM6446 a
rmle
# pidin in
CPU:ARM Release:6.3.0  FreeMem:124Mb/128Mb BootTime:Dec 31 19:00:00 est 1969
Processor1: 1090949733 arm926 270MHz
# shm
shm_open: 3
shm_ctl: 0 0
mmap: 80000000 0
#

Can you post your source?

	Sunil.

Andrew Golovnia wrote:
> oops! I think I have a wrong procnto running in ARM...
> It says it's 6.3.0 instead of 6.3.2....
> I'll try tomorrow with new procnto.
> 
> Thanks!
> 
> -- 
> AG
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post5911
> 
Re: Ooops! Few problems with ARM memory management.  
Hi!

I updated my procnto to 6.3.2 and your and my code work fine now.

# pidin info
CPU:ARM Release:6.3.2  FreeMem:46Mb/64Mb BootTime:Jan 01 00:00:00 UTC 1970
Processor1: 1761946886 pxa250 399MHz
# /H*/shmem-sunil
shm_open: 3
shm_ctl: 0 0
mmap: 82000000 0
# /H*/shmem-test
SHMEM allocation test.
shm_open() -> fd = 3 OK
shm_ctl() -> r = 0 OK
mmap() -> shmem = 0x82000000 OK
Code address:  &main() = 0x001008EC
Data address:  &fd     = 0x00101D14
Stack address: &shmem  = 0x000FFCE0
Shmem address: shmem   = 0x82000000
Memsetting... OK

Yesterday it was procnto from QNX 6.3.0 SP2 and today it's from QNX 6.3.2.
It was all about procnto.. :-/

Thanks for your help!

--
AG

> I just tried with a 630 procnto:
> 
> # uname -a
> QNX prejacinto-1 6.3.0 2005/08/09-14:59:23EDT
> Texas_Instruments_DAVINCI_DM6446 a
> rmle
> # pidin in
> CPU:ARM Release:6.3.0  FreeMem:124Mb/128Mb BootTime:Dec 31 19:00:00 est 1969
> Processor1: 1090949733 arm926 270MHz
> # shm
> shm_open: 3
> shm_ctl: 0 0
> mmap: 80000000 0
> #
> 
> Can you post your source?
> 
> 	Sunil.
> 
> Andrew Golovnia wrote:
> > oops! I think I have a wrong procnto running in ARM...
> > It says it's 6.3.0 instead of 6.3.2....
> > I'll try tomorrow with new procnto.
> > 
> > Thanks!
> > 
> > -- 
> > AG
> > 
> > _______________________________________________
> > OSTech
> > http://community.qnx.com/sf/go/post5911
> >