Project Home
Project Home
Documents
Documents
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - >3Gb userspace VM on x86 QNX: (5 Items)
   
>3Gb userspace VM on x86 QNX  
Hello,

I've an application which requires >3Gb of VM space and I'd like to try and compress down the 1Gb of VM which the QNX 
kernel reserves such that my application will run.  Reducing its VM requirements in this instance is not an option, so 
I'd really like a solution to this if possible.

I can see from system/memmgr/x86/cpu_mm_internal.h that the CPU_USER_VADDR_END address defaults to 0xbfffffffu.  I've 
experimented with moving this up further and had some success, along with success in moving the L2MAP_BASE, PROCMEM_BASE
 and also success in compressing some of these regions down in size (MAP_SIZE to 64Mb from 128Mb, and L2MAP_SIZE (though
 this #define doesn't exist in its own right) down to 4Mb and both work fine for me). 

What I'm really stuck on is moving the 1-TO-1 Virtual->Physical region.  I've found the code in startup-bios which 
matches that in x86/cpu_mm_internal.h, specifically:

hardware/startup/lib/x86/x86_mmu_4m.c

#define RAM_VADDR               0xe0000000
#define RAM_MAP_SIZE          0x10000000

and system/memmgr/x86/cpu_mm_internal.h 

#define CPU_SYSTEM_PADDR_START  0
#define CPU_SYSTEM_PADDR_END    ((256*1024*1024)-1)
...
#define CPU_1TO1_VADDR_BIAS             0xe0000000u

I've succeeded in booting a kernel which has this region shrunk from 256Mb to 128Mb, but what I cannot get working is a 
world in which this region is moved to any address other than 0xe0000000. Even if I move it off by a single page, QNX 
hangs during bootup.

I've littered the code with kprintf's and tracked the hang down to a nano_alloc() call, specifically this section of 
code:

                                        if((size -= new_size)) {
                                                add = PTRADD(new, new_size);
                                                add->size = size;
                                        }

I have printouts each side of the add->size write, but I never see any printouts following this write.  &(add->size) in this 
case is 0xe80000c4, which is curiously inside the last page in the 1-TO-1 region in my world (where I've moved the 1-TO-
1 region to 0xe0001000).  At first I thought this was the issue, but looking at a stock world which hasn't been moved 
off by a page, the address of add->size is also a page earlier, at 0xe7fff0c4 (again, the last page of the 1-TO-1 
region).

I've grepped around for other references to this address, in case I'd missed anywhere it was also hard coded, but I've 
not found any.

Does anyone understand why I'm having difficulty moving this region or can you suggest how best to attack this problem 
going forward?


Re: >3Gb userspace VM on x86 QNX  
I'm not absolutely positive why it's screwing up, but I'm pretty sure
it's because the code that sets up the 1-to-1 region is designed to
have it done with big pages (2/4MB). With you moving it by 4K, you can't
use the big page mappings and I'm guessing that's messing something
up (e.g. not allocating all the level 2 (page table) entries needed).

The 1-to-1 region isn't required on X86's (we don't have it on chips
that don't support the Page Size Extension), so my suggestion would be
to just delete it entirely if you're trying to shrink the system vaddr
space. The easiest way of doing that it just to make sure that the
X86_CPU_PSE bit doesn't get turned on by the init_one_cpuinfo() function
in startup. 

	Brian

On Wed, Jan 02, 2008 at 12:22:37PM -0500, James Mulcahy wrote:
> 
> Hello,
> 
> I've an application which requires >3Gb of VM space and I'd like to try
and compress down the 1Gb of VM which the QNX kernel reserves such that my
application will run.  Reducing its VM requirements in this instance is not
an option, so I'd really like a solution to this if possible.
> 
> I can see from system/memmgr/x86/cpu_mm_internal.h that the
CPU_USER_VADDR_END address defaults to 0xbfffffffu.  I've experimented with
moving this up further and had some success, along with success in moving
the L2MAP_BASE, PROCMEM_BASE and also success in compressing some of these
regions down in size (MAP_SIZE to 64Mb from 128Mb, and L2MAP_SIZE (though
this #define doesn't exist in its own right) down to 4Mb and both work fine
for me). 
> 
> What I'm really stuck on is moving the 1-TO-1 Virtual->Physical region.
I've found the code in startup-bios which matches that in
x86/cpu_mm_internal.h, specifically:
> 
> hardware/startup/lib/x86/x86_mmu_4m.c
> 
> #define RAM_VADDR               0xe0000000
> #define RAM_MAP_SIZE          0x10000000
> 
> and system/memmgr/x86/cpu_mm_internal.h 
> 
> #define CPU_SYSTEM_PADDR_START  0
> #define CPU_SYSTEM_PADDR_END    ((256*1024*1024)-1)
> ...
> #define CPU_1TO1_VADDR_BIAS             0xe0000000u
> 
> I've succeeded in booting a kernel which has this region shrunk from 256Mb
to 128Mb, but what I cannot get working is a world in which this region is
moved to any address other than 0xe0000000. Even if I move it off by a
single page, QNX hangs during bootup.
> 
> I've littered the code with kprintf's and tracked the hang down to a
nano_alloc() call, specifically this section of code:
> 
>                                         if((size -= new_size)) {
>                                                 add = PTRADD(new,
new_size);
>                                                 add->size = size;
>                                         }
> 
> I have printouts each side of the add->size write, but I never see any
printouts following this write.  &(add->size) in this case is 0xe80000c4,
which is curiously inside the last page in the 1-TO-1 region in my world
(where I've moved the 1-TO-1 region to 0xe0001000).  At first I thought this
was the issue, but looking at a stock world which hasn't been moved off by a
page, the address of add->size is also a page earlier, at 0xe7fff0c4 (again,
the last page of the 1-TO-1 region).
> 
> I've grepped around for other references to this address, in case I'd
missed anywhere it was also hard coded, but I've not found any.
> 
> Does anyone understand why I'm having difficulty moving this region or can
you suggest how best to attack this problem going forward?
> 
> 
> 
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post4021

-- 
Brian Stecher (bstecher@qnx.com)        QNX Software Systems
phone: +1 (613) 591-0931 (voice)        175 Terence Matthews Cr.
       +1 (613) 591-3579 (fax)          Kanata, Ontario,...
Re: >3Gb userspace VM on x86 QNX  
> I'm not absolutely positive why it's screwing up, but I'm pretty sure
> it's because the code that sets up the 1-to-1 region is designed to
> have it done with big pages (2/4MB). With you moving it by 4K, you can't
> use the big page mappings and I'm guessing that's messing something
> up (e.g. not allocating all the level 2 (page table) entries needed).
> 
> The 1-to-1 region isn't required on X86's (we don't have it on chips
> that don't support the Page Size Extension), so my suggestion would be
> to just delete it entirely if you're trying to shrink the system vaddr
> space. The easiest way of doing that it just to make sure that the
> X86_CPU_PSE bit doesn't get turned on by the init_one_cpuinfo() function
> in startup. 
>

Hi Brian,

Thanks for your response -- I've had a shot at this but it hasn't gone too smoothly.

I disabled the PSE flag in init_one_cpuinfo() as suggested, but as soon as I move the 1TO1 BIAS I hit a crash() in 
mm_pte.c.

It crashes at the very first statement:

if((manip->data.start & (__PAGESIZE-1)) != 0) crash();

data.start = 0xd0400288
__PAGE_SIZE = 0x1000

I've traced this back one level to pte_unmap(), but there are many calls to there, so I'm not sure where it originates. 
 I tried simply page aligning the address in pte_unmap() before calling pte_manip() but this just results in the 
walk_asinfo("imagefs", map_ifs, &data) call in bootimage_start failing to fill in the data.boot value, which results in 
a crash there instead.

I should add that in this instance I've moved the 1TO1 off by 4Mb instead of the 4K as per last time, just to be sure 
that it wasn't that causing me issues.

It appears there's a bit more reliance on the 1TO1 region than just the PSE stuff by the sounds of it.  I notice various
 _early memory allocation calls (in memmgr/pc.a) use this region too) -- is it perhaps this which is relating to my 
troubles, or are all these allocations thrown away when we move into a latter stage of kernel initialisation?

Any tips would be greatly appreciated!

 
Re: >3Gb userspace VM on x86 QNX  
Ignore my previous message for the time being -- whilst the PSE flag isn't enabled in init_one_cpuinfo() -- it does 
appear to still be enabled in the syspage's cpu flags entry.  

Forcing it off early in the procnto initialisation has got me a little further for now.
Re: >3Gb userspace VM on x86 QNX  
On Fri, Jan 04, 2008 at 08:24:03AM -0500, James Mulcahy wrote:
> Ignore my previous message for the time being -- whilst the PSE flag isn't
enabled in init_one_cpuinfo() -- it does appear to still be enabled in the
syspage's cpu flags entry.  

I'm surprised by that - the line in init_one_cpuinfo() that checks
X86_FEATURE_PSE and sets X86_CPU_PSE is the only one that turns that
flag on. Commenting it out should be good enough to disable X86_CPU_PSE
and make us run without the 1-to-1 region - same as we normally would
on a 386.

-- 
Brian Stecher (bstecher@qnx.com)        QNX Software Systems
phone: +1 (613) 591-0931 (voice)        175 Terence Matthews Cr.
       +1 (613) 591-3579 (fax)          Kanata, Ontario, Canada K2M 1W8