> I've been working with a client who has some 16Gig 6.6 machines and they want 

6.5 or 6.6? (confusion between Subject and text here)

> to use all of the memory and be able to allocate the maximum amount for the 
> address space.  Somewhere I read that this is 3.5Gig.
> 
> We are using -x on startup-* and we see the memory.
> We start test programs with "on -P".  This works.  We can allocate all the 
> memory, just not more than 2Gig per process.
> 
> We've try to use setrlimit(RLIMIT_AS, ...) and it doesn't work.  We read after
>  writing the limit and it doesn't change.

can you describe in detail how you're testing this? From what you write, it seems you're assuming that this will change 
the size of the process's address space right away ... is that the case?

> Mysteriously "ulimit -a" show that memory is unlimited.
> 
> We've tried allocating using malloc() which will only handle 2Gig at a time 
> and mmap() with the same results.

The available user address space on a 32bit system on qnx depends on the architecture. 
see http://www.qnx.com/developers/docs/6.6.0_anm11_wf10/#com.qnx.doc.neutrino.user_guide/topic/limits_PlatformSpecific.
html

so having 2/3.5g available user address space doesn't mean you are able to do a contiguous virtual address space 
allocation of that size. 
 There is space also used for the stack, heap, executable, libs, etc... 

you can use something like pidin mem to see the memory layout in the process.
http://www.qnx.com/developers/docs/7.0.0/#com.qnx.doc.neutrino.utilities/topic/p/pidin.html