Project Home
Project Home		 Documents
Documents		 Wiki
Wiki		 Discussion Forums
Discussions		 Project Information
Project Info
QNX Operating System/Discussion/OSTech/Shared memory coherence between processes/List of Posts
Forum Topic - Shared memory coherence between processes: (9 Items)
   
Update
Todd Peterson(deleted)
01/10/2010 12:05 PM
post44858
Shared memory coherence between processes   
I have written an A/D driver that stores samples in a circular buffer in shared memory. Upon a certain number of samples
, it pulses another process, which stores the last n samples to a file. Works fine for a while, then the file shows 
zeros for a gap. When I shutdown the processes and examine the shared memory, the zeros are not there, leading me to 
believe the file writer process is reading stale contents from the shared memory. 

How can I make the driver process force coherence before it pulses the other application?
John Garvey
01/11/2010 2:20 PM
post44917
Re: Shared memory coherence between processes   
> I have written an A/D driver that stores samples in a circular buffer in 
> shared memory. Upon a certain number of samples, it pulses another process, 
> which stores the last n samples to a file. Works fine for a while, then the 
> file shows zeros for a gap. When I shutdown the processes and examine the 
> shared memory, the zeros are not there, leading me to believe the file writer 
> process is reading stale contents from the shared memory. 
> 
> How can I make the driver process force coherence before it pulses the other 
> application?

mmap()ing your shared buffer PROT_NOCACHE is one approach, but may degrade performance accessing the buffer, especially 
since you are batching samples.

Look at the msync() function to synchronise to physical memory.  You want MS_SYNC at the writer side prior to pulsing; 
you may also need to MS_INVALIDATE from the reader process after receiving the pulse (if volatile pointers are not 
enough hint and no CPU bus-snooping).

There is also the cache API (libcache, refer cache_init() in the docs) which can provide more efficient memory operation
 than a msync syscall (if your h/w is supported by a library variant), with CACHE_FLUSH and CACHE_INVAL taking the place
 of the aforementioned msync flags, respectively.
Colin Burgess(deleted)
01/11/2010 2:38 PM
post44919
Re: Shared memory coherence between processes   
You don't mention which platform this is, nor how you are setting up 
your shared mappings?  If you are simply
using MAP_PHYS them yes you could get cache aliasing.

A better way to handle this is to make a shared object in /dev/shmem - 
that way the kernel will ensure that
processes mapping the object will have the same cache colour in the 
virtual mapping.

If the mapping has to overlay a particular physical address, see shmctl( 
SHMCTL_PHYS... for how to make a shared
object of that type.

On 10-01-10 12:05 PM, Todd Peterson wrote:
> I have written an A/D driver that stores samples in a circular buffer in shared memory. Upon a certain number of 
samples, it pulses another process, which stores the last n samples to a file. Works fine for a while, then the file 
shows zeros for a gap. When I shutdown the processes and examine the shared memory, the zeros are not there, leading me 
to believe the file writer process is reading stale contents from the shared memory.
>
> How can I make the driver process force coherence before it pulses the other application?
>
>
>
> _______________________________________________
>
> OSTech
> http://community.qnx.com/sf/go/post44858
>
>    

-- 
cburgess@qnx.com
Todd Peterson(deleted)
01/11/2010 7:11 PM
post44973
Re: Shared memory coherence between processes   
 The platform is ARM 926EJS-based. Here is what I use to create and map shmem on driver process:

    shmem_fd = shm_open( HS_ADC_SHMEM_NAME, O_RDWR | O_CREAT, S_IRWXU | S_IRWXG | S_IRWXO);

    ftruncate( shmem_fd, amount );

    const char *adc_buf = (const char *) mmap( NULL, amount, PROT_READ|PROT_WRITE|PROT_NOCACHE, MAP_SHARED, shmem_fd, 0 
);


On the client side:

  shmem_hdl = shm_open("/adc_hs", O_RDWR, 755);

  addr = (uint8_t*) mmap( NULL, amount, PROT_READ | PROT_WRITE | PROT_NOCACHE, MAP_SHARED, shmem_hdl, 0 );

I added the PROT_NOCACHE in both mmap calls since when I first posted and the problem seems to be fixed, but more 
testing is required.

Thanks

Todd

> You don't mention which platform this is, nor how you are setting up 
> your shared mappings?  If you are simply
> using MAP_PHYS them yes you could get cache aliasing.
> 
> A better way to handle this is to make a shared object in /dev/shmem - 
> that way the kernel will ensure that
> processes mapping the object will have the same cache colour in the 
> virtual mapping.
> 
> If the mapping has to overlay a particular physical address, see shmctl( 
> SHMCTL_PHYS... for how to make a shared
> object of that type.
> 
> On 10-01-10 12:05 PM, Todd Peterson wrote:
> > I have written an A/D driver that stores samples in a circular buffer in 
> shared memory. Upon a certain number of samples, it pulses another process, 
> which stores the last n samples to a file. Works fine for a while, then the 
> file shows zeros for a gap. When I shutdown the processes and examine the 
> shared memory, the zeros are not there, leading me to believe the file writer 
> process is reading stale contents from the shared memory.
> >
> > How can I make the driver process force coherence before it pulses the other
>  application?
> >
> >
> >
> > _______________________________________________
> >
> > OSTech
> > http://community.qnx.com/sf/go/post44858
> >
> >    
> 
> -- 
> cburgess@qnx.com
Colin Burgess(deleted)
01/12/2010 12:14 AM
post44986
Re: Shared memory coherence between processes   
How is the driver process writing to the shared memory?  Is the 
peripheral writing to it or
the driver reading from the peripheral and then writing to the shared 
object?

Also, what version of the OS is this for?

Colin

On 10-01-11 7:11 PM, Todd Peterson wrote:
>   The platform is ARM 926EJS-based. Here is what I use to create and map shmem on driver process:
>
>      shmem_fd = shm_open( HS_ADC_SHMEM_NAME, O_RDWR | O_CREAT, S_IRWXU | S_IRWXG | S_IRWXO);
>
>      ftruncate( shmem_fd, amount );
>
>      const char *adc_buf = (const char *) mmap( NULL, amount, PROT_READ|PROT_WRITE|PROT_NOCACHE, MAP_SHARED, shmem_fd,
 0 );
>
>
> On the client side:
>
>    shmem_hdl = shm_open("/adc_hs", O_RDWR, 755);
>
>    addr = (uint8_t*) mmap( NULL, amount, PROT_READ | PROT_WRITE | PROT_NOCACHE, MAP_SHARED, shmem_hdl, 0 );
>
> I added the PROT_NOCACHE in both mmap calls since when I first posted and the problem seems to be fixed, but more 
testing is required.
>
> Thanks
>
> Todd
>
>    
>> You don't mention which platform this is, nor how you are setting up
>> your shared mappings?  If you are simply
>> using MAP_PHYS them yes you could get cache aliasing.
>>
>> A better way to handle this is to make a shared object in /dev/shmem -
>> that way the kernel will ensure that
>> processes mapping the object will have the same cache colour in the
>> virtual mapping.
>>
>> If the mapping has to overlay a particular physical address, see shmctl(
>> SHMCTL_PHYS... for how to make a shared
>> object of that type.
>>
>> On 10-01-10 12:05 PM, Todd Peterson wrote:
>>      
>>> I have written an A/D driver that stores samples in a circular buffer in
>>>        
>> shared memory. Upon a certain number of samples, it pulses another process,
>> which stores the last n samples to a file. Works fine for a while, then the
>> file shows zeros for a gap. When I shutdown the processes and examine the
>> shared memory, the zeros are not there, leading me to believe the file writer
>> process is reading stale contents from the shared memory.
>>      
>>> How can I make the driver process force coherence before it pulses the other
>>>        
>>   application?
>>      
>>>
>>>
>>> _______________________________________________
>>>
>>> OSTech
>>> http://community.qnx.com/sf/go/post44858
>>>
>>>
>>>        
>> -- 
>> cburgess@qnx.com
>>      
>
>
>
>
>
> _______________________________________________
>
> OSTech
> http://community.qnx.com/sf/go/post44973
>
>    

-- 
cburgess@qnx.com
Todd Peterson(deleted)
01/12/2010 3:06 PM
post45094
Re: Shared memory coherence between processes   
The driver is writing directly to shared memory (after reading the peripheral over SPI).
We're using production 6.4.1.

> How is the driver process writing to the shared memory?  Is the 
> peripheral writing to it or
> the driver reading from the peripheral and then writing to the shared 
> object?
> 
> Also, what version of the OS is this for?
> 
> Colin
>
Sunil Kittur(deleted)
01/12/2010 4:09 PM
post45098
Re: Shared memory coherence between processes   
The ARM926 has a virtually indexed cache so the
memmgr is supposed to force MAP_SHARED
mappings to be uncached to avoid cache aliases.
I don't understand why explicitly adding NOCACHE
appears to fix the problem...

    Sunil.

>  The platform is ARM 926EJS-based. Here is what I use to create and map shmem 
> on driver process:
> 
>     shmem_fd = shm_open( HS_ADC_SHMEM_NAME, O_RDWR | O_CREAT, S_IRWXU | 
> S_IRWXG | S_IRWXO);
> 
>     ftruncate( shmem_fd, amount );
> 
>     const char *adc_buf = (const char *) mmap( NULL, amount, PROT_READ|
> PROT_WRITE|PROT_NOCACHE, MAP_SHARED, shmem_fd, 0 );
> 
> 
> On the client side:
> 
>   shmem_hdl = shm_open("/adc_hs", O_RDWR, 755);
> 
>   addr = (uint8_t*) mmap( NULL, amount, PROT_READ | PROT_WRITE | PROT_NOCACHE,
>  MAP_SHARED, shmem_hdl, 0 );
> 
> I added the PROT_NOCACHE in both mmap calls since when I first posted and the 
> problem seems to be fixed, but more testing is required.
> 
> Thanks
> 
> Todd
> 
> > You don't mention which platform this is, nor how you are setting up 
> > your shared mappings?  If you are simply
> > using MAP_PHYS them yes you could get cache aliasing.
> > 
> > A better way to handle this is to make a shared object in /dev/shmem - 
> > that way the kernel will ensure that
> > processes mapping the object will have the same cache colour in the 
> > virtual mapping.
> > 
> > If the mapping has to overlay a particular physical address, see shmctl( 
> > SHMCTL_PHYS... for how to make a shared
> > object of that type.
> > 
> > On 10-01-10 12:05 PM, Todd Peterson wrote:
> > > I have written an A/D driver that stores samples in a circular buffer in 
> > shared memory. Upon a certain number of samples, it pulses another process, 
> 
> > which stores the last n samples to a file. Works fine for a while, then the 
> 
> > file shows zeros for a gap. When I shutdown the processes and examine the 
> > shared memory, the zeros are not there, leading me to believe the file 
> writer 
> > process is reading stale contents from the shared memory.
> > >
> > > How can I make the driver process force coherence before it pulses the 
> other
> >  application?
> > >
> > >
> > >
> > > _______________________________________________
> > >
> > > OSTech
> > > http://community.qnx.com/sf/go/post44858
> > >
> > >    
> > 
> > -- 
> > cburgess@qnx.com
> 
>
Todd Peterson(deleted)
01/12/2010 11:49 PM
post45113
Re: Shared memory coherence between processes   
do you say this because of the following quote from http://www.qnx.com/developers/docs/6.4.1/neutrino/prog/arm_memory.
html?

If under ARM, shared memory is *always* mapped uncached, then I fixed the problem in some other manner (or it will rear 
its ugly head again during testing).

quote from ref above:
"Because the MMU transparently remaps each process's virtual address, shared memory objects must be mapped uncached, 
since they're always mapped at different virtual addresses.

Strictly speaking, this is required only if at least one writable mapping exists, but the current VM implementation 
doesn't track this, and *unconditionally makes all mappings uncached*.

The consequence of this is that performance of memory accesses to shared memory object mappings will be bound by the 
uncached memory performance of the system. "

> The ARM926 has a virtually indexed cache so the
> memmgr is supposed to force MAP_SHARED
> mappings to be uncached to avoid cache aliases.
> I don't understand why explicitly adding NOCACHE
> appears to fix the problem...
> 
>     Sunil.
> 
> >  The platform is ARM 926EJS-based. Here is what I use to create and map 
> shmem 
> > on driver process:
> > 
> >     shmem_fd = shm_open( HS_ADC_SHMEM_NAME, O_RDWR | O_CREAT, S_IRWXU | 
> > S_IRWXG | S_IRWXO);
> > 
> >     ftruncate( shmem_fd, amount );
> > 
> >     const char *adc_buf = (const char *) mmap( NULL, amount, PROT_READ|
> > PROT_WRITE|PROT_NOCACHE, MAP_SHARED, shmem_fd, 0 );
> > 
> > 
> > On the client side:
> > 
> >   shmem_hdl = shm_open("/adc_hs", O_RDWR, 755);
> > 
> >   addr = (uint8_t*) mmap( NULL, amount, PROT_READ | PROT_WRITE | 
> PROT_NOCACHE,
> >  MAP_SHARED, shmem_hdl, 0 );
> > 
> > I added the PROT_NOCACHE in both mmap calls since when I first posted and 
> the 
> > problem seems to be fixed, but more testing is required.
> > 
> > Thanks
> > 
> > Todd
> > 
> > > You don't mention which platform this is, nor how you are setting up 
> > > your shared mappings?  If you are simply
> > > using MAP_PHYS them yes you could get cache aliasing.
> > > 
> > > A better way to handle this is to make a shared object in /dev/shmem - 
> > > that way the kernel will ensure that
> > > processes mapping the object will have the same cache colour in the 
> > > virtual mapping.
> > > 
> > > If the mapping has to overlay a particular physical address, see shmctl( 
> > > SHMCTL_PHYS... for how to make a shared
> > > object of that type.
> > > 
> > > On 10-01-10 12:05 PM, Todd Peterson wrote:
> > > > I have written an A/D driver that stores samples in a circular buffer in
>  
> > > shared memory. Upon a certain number of samples, it pulses another process
> , 
> > 
> > > which stores the last n samples to a file. Works fine for a while, then 
> the 
> > 
> > > file shows zeros for a gap. When I shutdown the processes and examine the 
> 
> > > shared memory, the zeros are not there, leading me to believe the file 
> > writer 
> > > process is reading stale contents from the shared memory.
> > > >
> > > > How can I make the driver process force coherence before it pulses the 
> > other
> > >  application?
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > >
> > > > OSTech
> > > > http://community.qnx.com/sf/go/post44858
> > > >
> > > >    
> > > 
> > > -- 
> > >...
Sunil Kittur(deleted)
01/13/2010 4:31 AM
post45123
Re: Shared memory coherence between processes   
Yes - that's exactly what should happen.

The only shared mappings that are not forced to be uncached are:

- executable/shared library text sections.
   These are read-only so even though aliases may exist they are coherent.

- SHMCTL_GLOBAL mappings.
   These have a global virtual address (outside the FCSE remapped ranges)
   so no aliases can exist.

	Sunil.

Todd Peterson wrote:
> do you say this because of the following quote from http://www.qnx.com/developers/docs/6.4.1/neutrino/prog/arm_memory.
html?
> 
> If under ARM, shared memory is *always* mapped uncached, then I fixed the problem in some other manner (or it will 
rear its ugly head again during testing).
> 
> quote from ref above:
> "Because the MMU transparently remaps each process's virtual address, shared memory objects must be mapped uncached, 
since they're always mapped at different virtual addresses.
> 
> Strictly speaking, this is required only if at least one writable mapping exists, but the current VM implementation 
doesn't track this, and *unconditionally makes all mappings uncached*.
> 
> The consequence of this is that performance of memory accesses to shared memory object mappings will be bound by the 
uncached memory performance of the system. "
> 
> 
>>The ARM926 has a virtually indexed cache so the
>>memmgr is supposed to force MAP_SHARED
>>mappings to be uncached to avoid cache aliases.
>>I don't understand why explicitly adding NOCACHE
>>appears to fix the problem...
>>
>>    Sunil.

Return


  A subsidiary of BlackBerry             All content ©2004-2013, QNX Software Systems