Project Home
Project Home
Documents
Documents
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - 4Gig: (20 Items)
   
4Gig  
There is a discussion over openqnx.com about going over the 4Gig limit and its implication on driver using DMA.

My take is that the OS should not provide contiguous memory above 4Gig unless specifically requested/allowed by the 
driver.  I base my statement on being able to provide backward compatibility with existing user driver.

It's being proposed that to provide backward compatibility that memory partitioning be used, but my counter argument is 
that this is a 6.4 feature. 
Plus it might might be an expensive add-on. Lastly it seems to me like it's the kind of thing that would is easy to 
configure improperly.  

How is that managed for 6.3. ( I believe 4Gig support was broken in 6.3.2 right?)

What do the Gurus have to say about this?

4Gig will probably be the "norm" in 1 year from now and it should be as easy as sticking ram in the machine.




Re: 4Gig  
That brings the question how can I force contiguous memory to be allocated above 4G, below 4G or anywhere?

- Mario
Re: 4Gig  
Hmmm, that would be POSIX typed memory, I think.

Mario Charest wrote:
> That brings the question how can I force contiguous memory to be 
> allocated above 4G, below 4G or anywhere?
> 
> - Mario
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post5541
> 

-- 
cburgess@qnx.com
Re: 4Gig  
> Hmmm, that would be POSIX typed memory, I think.

Are you talking about posix_typed_mem_open().  

So in a nut shell if I want the memory bloc to be below 4G I use mmap, if 'd want it to be above 4G i would use posix_..
..  If I don't care, I'm force to make a choice?

That doesn't sound right.

> 
> Mario Charest wrote:
> > That brings the question how can I force contiguous memory to be 
> > allocated above 4G, below 4G or anywhere?
> > 
> > - Mario
> > 
> > _______________________________________________
> > OSTech
> > http://community.qnx.com/sf/go/post5541
> > 
> 
> -- 
> cburgess@qnx.com


Re: 4Gig  
The future proof (and POSIX) way of doing ANY restriction of ram allocation
is the posix typed memory interface.  The old MAP_* flags are kinda hokey,
and had lots of 'hacks' to make things like this work.

You are supposed to setup the relevant regions in startup.  Now I've mentioned
to people before we need to have these defined BEFORE the 6.4 release...

Time for a PR, methinks...

Mario Charest wrote:
>  > Hmmm, that would be POSIX typed memory, I think.
> 
> Are you talking about posix_typed_mem_open(). 
> 
> So in a nut shell if I want the memory bloc to be below 4G I use mmap, 
> if 'd want it to be above 4G i would use posix_....  If I don't care, 
> I'm force to make a choice?
> 
> That doesn't sound right.
> 
>  >
>  > Mario Charest wrote:
>  > > That brings the question how can I force contiguous memory to be
>  > > allocated above 4G, below 4G or anywhere?
>  > >
>  > > - Mario
>  > >
>  > > _______________________________________________
>  > > OSTech
>  > > http://community.qnx.com/sf/go/post5541
>  > >
>  >
>  > --
>  > cburgess@qnx.com
> 
> 
> 
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post5560
> 

-- 
cburgess@qnx.com
Re: 4Gig  
> The future proof (and POSIX) way of doing ANY restriction of ram allocation
> is the posix typed memory interface.  The old MAP_* flags are kinda hokey,
> and had lots of 'hacks' to make things like this work.
> 
> You are supposed to setup the relevant regions in startup.  Now I've mentioned
> 
> to people before we need to have these defined BEFORE the 6.4 release...
> 

Are ya saying that if I add memory to a machine i need to twiddle with the startup stuff?

About 6.4 don't know if it's close to release or not but it's been a while since there hasn't been a Milestone release (
 last december )

> Time for a PR, methinks...
> 
> Mario Charest wrote:
> >  > Hmmm, that would be POSIX typed memory, I think.
> > 
> > Are you talking about posix_typed_mem_open(). 
> > 
> > So in a nut shell if I want the memory bloc to be below 4G I use mmap, 
> > if 'd want it to be above 4G i would use posix_....  If I don't care, 
> > I'm force to make a choice?
> > 
> > That doesn't sound right.
> > 
> >  >
> >  > Mario Charest wrote:
> >  > > That brings the question how can I force contiguous memory to be
> >  > > allocated above 4G, below 4G or anywhere?
> >  > >
> >  > > - Mario
> >  > >
> >  > > _______________________________________________
> >  > > OSTech
> >  > > http://community.qnx.com/sf/go/post5541
> >  > >
> >  >
> >  > --
> >  > cburgess@qnx.com
> > 
> > 
> > 
> > 
> > _______________________________________________
> > OSTech
> > http://community.qnx.com/sf/go/post5560
> > 
> 
> -- 
> cburgess@qnx.com


Re: 4Gig  
Mario Charest wrote:
>  > The future proof (and POSIX) way of doing ANY restriction of ram 
> allocation
>  > is the posix typed memory interface.  The old MAP_* flags are kinda 
> hokey,
>  > and had lots of 'hacks' to make things like this work.
>  >
>  > You are supposed to setup the relevant regions in startup.  Now I've 
> mentioned
>  >
>  > to people before we need to have these defined BEFORE the 6.4 release...
>  >
> 
> Are ya saying that if I add memory to a machine i need to twiddle with 
> the startup stuff?

No, I'm saying WE should have a default set of 'types' suitable for x86 systems,
and all our BSPs really.

None of our drivers use typed memory (to my knowledge yet) because they were written
before the availability of that interface, however they should be being migrated
over AFAIK.

Colin

> About 6.4 don't know if it's close to release or not but it's been a 
> while since there hasn't been a Milestone release ( last december )
> 
>  > Time for a PR, methinks...
>  >
>  > Mario Charest wrote:
>  > >  > Hmmm, that would be POSIX typed memory, I think.
>  > >
>  > > Are you talking about posix_typed_mem_open().
>  > >
>  > > So in a nut shell if I want the memory bloc to be below 4G I use mmap,
>  > > if 'd want it to be above 4G i would use posix_....  If I don't care,
>  > > I'm force to make a choice?
>  > >
>  > > That doesn't sound right.
>  > >
>  > >  >
>  > >  > Mario Charest wrote:
>  > >  > > That brings the question how can I force contiguous memory to be
>  > >  > > allocated above 4G, below 4G or anywhere?
>  > >  > >
>  > >  > > - Mario
>  > >  > >
>  > >  > > _______________________________________________
>  > >  > > OSTech
>  > >  > > http://community.qnx.com/sf/go/post5541
>  > >  > >
>  > >  >
>  > >  > --
>  > >  > cburgess@qnx.com
>  > >
>  > >
>  > >
>  > >
>  > > _______________________________________________
>  > > OSTech
>  > > http://community.qnx.com/sf/go/post5560
>  > >
>  >
>  > --
>  > cburgess@qnx.com
> 
> 
> 
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post5564
> 

-- 
cburgess@qnx.com
Re: 4Gig  
> Mario Charest wrote:
> >  > The future proof (and POSIX) way of doing ANY restriction of ram 
> > allocation
> >  > is the posix typed memory interface.  The old MAP_* flags are kinda 
> > hokey,
> >  > and had lots of 'hacks' to make things like this work.
> >  >
> >  > You are supposed to setup the relevant regions in startup.  Now I've 
> > mentioned
> >  >
> >  > to people before we need to have these defined BEFORE the 6.4 release...
> >  >

Are these "regions" related to the new memory partition scheme?
Re: 4Gig  
On Wed, Mar 05, 2008 at 11:16:54PM -0500, Mario Charest wrote:
> > Mario Charest wrote:
> > >  > The future proof (and POSIX) way of doing ANY restriction of ram 
> > > allocation
> > >  > is the posix typed memory interface.  The old MAP_* flags are kinda

> > > hokey,
> > >  > and had lots of 'hacks' to make things like this work.
> > >  >
> > >  > You are supposed to setup the relevant regions in startup.  Now
I've 
> > > mentioned
> > >  >
> > >  > to people before we need to have these defined BEFORE the 6.4
release...
> > >  >
> 
> Are these "regions" related to the new memory partition scheme?

The partitioning code can impose a limit on the amount of memory a
process can allocate from one of these typed memory names, but the use
of typed memory names to require allocation from a certain physical address
range is not dependent on the partitioning code being present in the system.


-- 
Brian Stecher (bstecher@qnx.com)        QNX Software Systems
phone: +1 (613) 591-0931 (voice)        175 Terence Matthews Cr.
       +1 (613) 591-3579 (fax)          Kanata, Ontario, Canada K2M 1W8
Re: 4Gig  
> 
> No, I'm saying WE should have a default set of 'types' suitable for x86 
> systems,
> and all our BSPs really.
> 
> None of our drivers use typed memory (to my knowledge yet) because they were 
> written
> before the availability of that interface, however they should be being 
> migrated
> over AFAIK.
> 
> Colin

Was that done for the M6 drop or should I not waste my time and test this on >4Gig machine?
Re: 4Gig  
It hasn't been done yet, no...

Mario Charest wrote:
> 
>  >
>  > No, I'm saying WE should have a default set of 'types' suitable for x86
>  > systems,
>  > and all our BSPs really.
>  >
>  > None of our drivers use typed memory (to my knowledge yet) because 
> they were
>  > written
>  > before the availability of that interface, however they should be being
>  > migrated
>  > over AFAIK.
>  >
>  > Colin
> 
> Was that done for the M6 drop or should I not waste my time and test 
> this on >4Gig machine?
> 
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post6687
> 

-- 
cburgess@qnx.com
Re: 4Gig  
> It hasn't been done yet, no...

What if I make a custom boot image and in that boot image I would start process as early as possible.  This program 
would allocated memory in chunks of 64K and it would keep allocating them until their physical memory address is below 
4G.  The idea is that all memory above 4G would be consumed while drivers are starting.  Then once the startup is 
completed the program would get killed, making that memory available.

Of course there are a lot of caveats but does the general idea make sense?


> 
> Mario Charest wrote:
> > 
> >  >
> >  > No, I'm saying WE should have a default set of 'types' suitable for x86
> >  > systems,
> >  > and all our BSPs really.
> >  >
> >  > None of our drivers use typed memory (to my knowledge yet) because 
> > they were
> >  > written
> >  > before the availability of that interface, however they should be being
> >  > migrated
> >  > over AFAIK.
> >  >
> >  > Colin
> > 
> > Was that done for the M6 drop or should I not waste my time and test 
> > this on >4Gig machine?
> > 
> > 
> > _______________________________________________
> > OSTech
> > http://community.qnx.com/sf/go/post6687
> > 
> 
> -- 
> cburgess@qnx.com


Re: 4Gig  
On Tue, Mar 04, 2008 at 05:38:55PM -0500, Mario Charest wrote:
> There is a discussion over openqnx.com about going over the 4Gig limit and
its implication on driver using DMA.
> 
> My take is that the OS should not provide contiguous memory above 4Gig
unless specifically requested/allowed by the driver.  I base my statement on
being able to provide backward compatibility with existing user driver.
> 
> It's being proposed that to provide backward compatibility that memory
partitioning be used, but my counter argument is that this is a 6.4 feature.

> Plus it might might be an expensive add-on. Lastly it seems to me like
it's the kind of thing that would is easy to configure improperly.  
> 
> How is that managed for 6.3. ( I believe 4Gig support was broken in 6.3.2
right?)

For 6.3.0, the allocation of contiguous physical memory was changed so
that the allocator tried to return memory that was below 4G and only went
above the line if there was nothing left below it. That particular hack
was accidently left out of the memmgr rewrite of 6.3.2, but has made
a re-appearance in the source (search for "restrict_user_4G_first" in
services/system/memmgr/mm_memobj.c). It's not sufficient, of course, since
there's no guarantee that the memory won't come from above 4G and there
are drivers that don't allocate their device buffer memory contiguously.

The solution that our drivers will be using in 6.4 is to switch to
the posix typed memory API.

-- 
Brian Stecher (bstecher@qnx.com)        QNX Software Systems
phone: +1 (613) 591-0931 (voice)        175 Terence Matthews Cr.
       +1 (613) 591-3579 (fax)          Kanata, Ontario, Canada K2M 1W8
Re: 4Gig  
>
> The solution that our drivers will be using in 6.4 is to switch to
> the posix typed memory API.

And old 3rd party drivers using mmap(MAP_ANON...) will be guarantied to be below 4G?

> 
> -- 
> Brian Stecher (bstecher@qnx.com)        QNX Software Systems
> phone: +1 (613) 591-0931 (voice)        175 Terence Matthews Cr.
>        +1 (613) 591-3579 (fax)          Kanata, Ontario, Canada K2M 1W8


Re: 4Gig  
On Wed, Mar 05, 2008 at 11:13:10PM -0500, Mario Charest wrote:
> >
> > The solution that our drivers will be using in 6.4 is to switch to
> > the posix typed memory API.
> 
> And old 3rd party drivers using mmap(MAP_ANON...) will be guarantied to be
below 4G?

No, but the contiguous allocation requests will continue to be resolved from
below 4G preferentially. 

The OS doesn't know if the the request is really for a device buffer or not,

nor does it know if the physical address is going to be passed to a piece of

hardware that requires a 32 bit paddr, so it can't refuse to satisfy the 
allocation if there's memory to be had above the 4G line.

-- 
Brian Stecher (bstecher@qnx.com)        QNX Software Systems
phone: +1 (613) 591-0931 (voice)        175 Terence Matthews Cr.
       +1 (613) 591-3579 (fax)          Kanata, Ontario, Canada K2M 1W8
Re: 4Gig  
> On Wed, Mar 05, 2008 at 11:13:10PM -0500, Mario Charest wrote:
> > >
> > > The solution that our drivers will be using in 6.4 is to switch to
> > > the posix typed memory API.
> > 
> > And old 3rd party drivers using mmap(MAP_ANON...) will be guarantied to be
> below 4G?
> 
> No, but the contiguous allocation requests will continue to be resolved from
> below 4G preferentially. 
> 
> The OS doesn't know if the the request is really for a device buffer or not,
> 
> nor does it know if the physical address is going to be passed to a piece of
> 
> hardware that requires a 32 bit paddr, so it can't refuse to satisfy the 
> allocation if there's memory to be had above the 4G line.
> 

So what you are saying is that 6.4 will break old driver if use with more then 4G.  

I though the OS could make the assumption that if mmap is used with MAP_PHYS and MAP_ANON that it could only then 
allocate it below 4G.

> -- 
> Brian Stecher (bstecher@qnx.com)        QNX Software Systems
> phone: +1 (613) 591-0931 (voice)        175 Terence Matthews Cr.
>        +1 (613) 591-3579 (fax)          Kanata, Ontario, Canada K2M 1W8


Re: 4Gig  
On Thu, Mar 06, 2008 at 09:42:37AM -0500, Mario Charest wrote:
> So what you are saying is that 6.4 will break old driver if use with more
then 4G.  

No, it _may_ break. If the driver is requesting contiguous memory it will
likely 
work - 'normal' user allocations are preferentially done from above 4G to
leave 
the memory below for allocations that need it. That means that there will 
normally be memory free below the line.

There's another out - implicit in the memmgr code is ability to have
an individual restriction list for each memory object (a restriction list is

what a typed memory name turns into - a linked list of paddr start/end
ranges 
that the physical allocator is allowed to use for a particular allocation).
It's always been in our plan (but not yet implemented) to have an API that
let you start a process and specify a typed memory name to use for its
non-typed memory mmap()'s (what's happening right now is that all objects
are using the typed memory name "sysram" for their allocations). If the API
was implmented, you could theoretically do something along the line of:

	on -M"sysram_below_4G" old_driver

and all the mmap()'s that the process does would come from below the 4G
line.
A little bit of overkill since not all the memory the driver uses is
required
to be below the line, but better than nothing.  Unfortunately, we're not 
likely to have the time to do that before the 6.4 release.	

> I though the OS could make the assumption that if mmap is used with
MAP_PHYS 
> and MAP_ANON that it could only then allocate it below 4G.

As I said previously, we can't make that assumption. The memory manager
code has no knowledge of if the memory is even going to be handed off to
a device or what the restrictions on the device memory usage are.

-- 
Brian Stecher (bstecher@qnx.com)        QNX Software Systems
phone: +1 (613) 591-0931 (voice)        175 Terence Matthews Cr.
       +1 (613) 591-3579 (fax)          Kanata, Ontario, Canada K2M 1W8
Re: 4Gig  
> On Thu, Mar 06, 2008 at 09:42:37AM -0500, Mario Charest wrote:
> > So what you are saying is that 6.4 will break old driver if use with more
> then 4G.  
> 
> No, it _may_ break. 

Well we can't really deliver a product to customer and tell them: it will most likely work but may not work some times..
.


Re: 4Gig  
I've posted diff's to OSREV forum for anyone who wishes to comment on the changes.

http://community.qnx.com/sf/go/projects.core_os/discussion.osrev.topc2940