Project Home
Project Home
Documents
Documents
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - shm_ctl() fails when setting up a shared memory physical DMA buffer for a PCI card: (5 Items)
   
shm_ctl() fails when setting up a shared memory physical DMA buffer for a PCI card  
I am trying to set up a shared memory buffer for DMA, so I can DMA directly in/out from one process and memcpy in/out 
from other processes but shm_ctl() fails. I have been "doing the dance" with shm_open(), shm_ctl(), mmap() and 
mem_offset64() but not managed to combine these in a way that enables the call to shm_ctl to succeed with the physical 
flag set. There are other threads  on this topic and I have followed their suggestions but they don't work for me. (for 
example: http://www.openqnx.com/newsgroups/viewtopic.php?f=13&t=12398) The "tested example" that thread works for me but
 it specifies the phyiscal address directly and does not obtain it from an mmap allocation followed by mem_offset64. As 
soon as I pass the mmap allocation physical address to shm_ctl(), it fails with "error: Function not implemented".
 I'm testing on a 4 core x86 PC.


An extract from my code is:

/* Allocate contiguous memory for DMA buffer */
/* byte size is page aligned */
uint32_t *  ptr = mmap(NULL, bytesize,   PROT_READ | PROT_WRITE | PROT_NOCACHE,   MAP_PHYS | MAP_ANON,  NOFD,  0);

/* Get the physical addresss, note that mem_offset fails so I use mem_offset64 */
mem_offset64(ptr, NOFD, bytesize, &physical_addr_64, &contig_len

/* Create a shared memory object */
int fd = shm_open("/turnip", O_RDWR | O_CREAT, 0644);

/* Overlay the shared memory object to the physical memory address */
if (shm_ctl(fd, SHMCTL_GLOBAL|SHMCTL_PHYS, physical_addr_64, bytesize) == -1)
{
   perror("shm_open");
   exit(EXIT_FAILURE);
}

If I replace physical_addr_64 with 0xa0000, shm_ctl() works but this seems too risky because I am allocating a large 
amount for memory for driver DMA buffers so I'm not happy just picking any physical address for a driver buffer.

If I just use mmap and memoffset_64 to have a DMA buffer in process memory space this works fine and if I create a 
shared memory objects, this also works fine. I have read conflicting accounts of using ltruncate to allocate the 
physical buffer but I couldn't get that to work either and Ithink I'm going round in circles now!

Any help or advice would be greatly appreciated. Is it possible to allocate a DMA buffer that can be shared between 
processes?

RE: shm_ctl() fails when setting up a shared memory physical DMA buffer for a PCI card  
Hi Iain,

I can reproduce the problem - if the shared memory object already exists and has physical memory assigned to it. 

When I run your code for the first time, it works perfectly. From then on, it will fail on shm_ctl. This kind of makes 
sense, since you're trying to associate physical memory with a shared memory object which already has physical memory 
associated with it. It's mostly the error code that is misleading.

Cheers,
Thomas

-----Original Message-----
From: Iain Martin [mailto:community-noreply@qnx.com] 
Sent: Dienstag, 13. März 2012 13:54
To: ostech-core_os
Subject: shm_ctl() fails when setting up a shared memory physical DMA buffer for a PCI card

I am trying to set up a shared memory buffer for DMA, so I can DMA directly in/out from one process and memcpy in/out 
from other processes but shm_ctl() fails. I have been "doing the dance" with shm_open(), shm_ctl(), mmap() and 
mem_offset64() but not managed to combine these in a way that enables the call to shm_ctl to succeed with the physical 
flag set. There are other threads  on this topic and I have followed their suggestions but they don't work for me. (for 
example: http://www.openqnx.com/newsgroups/viewtopic.php?f=13&t=12398) The "tested example" that thread works for me but
 it specifies the phyiscal address directly and does not obtain it from an mmap allocation followed by mem_offset64. As 
soon as I pass the mmap allocation physical address to shm_ctl(), it fails with "error: Function not implemented".
 I'm testing on a 4 core x86 PC.


An extract from my code is:

/* Allocate contiguous memory for DMA buffer */
/* byte size is page aligned */
uint32_t *  ptr = mmap(NULL, bytesize,   PROT_READ | PROT_WRITE | PROT_NOCACHE,   MAP_PHYS | MAP_ANON,  NOFD,  0);

/* Get the physical addresss, note that mem_offset fails so I use mem_offset64 */ mem_offset64(ptr, NOFD, bytesize, &
physical_addr_64, &contig_len

/* Create a shared memory object */
int fd = shm_open("/turnip", O_RDWR | O_CREAT, 0644);

/* Overlay the shared memory object to the physical memory address */ if (shm_ctl(fd, SHMCTL_GLOBAL|SHMCTL_PHYS, 
physical_addr_64, bytesize) == -1) {
   perror("shm_open");
   exit(EXIT_FAILURE);
}

If I replace physical_addr_64 with 0xa0000, shm_ctl() works but this seems too risky because I am allocating a large 
amount for memory for driver DMA buffers so I'm not happy just picking any physical address for a driver buffer.

If I just use mmap and memoffset_64 to have a DMA buffer in process memory space this works fine and if I create a 
shared memory objects, this also works fine. I have read conflicting accounts of using ltruncate to allocate the 
physical buffer but I couldn't get that to work either and Ithink I'm going round in circles now!

Any help or advice would be greatly appreciated. Is it possible to allocate a DMA buffer that can be shared between 
processes?





_______________________________________________

OSTech
http://community.qnx.com/sf/go/post92072
Re: RE: shm_ctl() fails when setting up a shared memory physical DMA buffer for a PCI card  
Thanks for your quick reply. Unfortunately, it also fails for me the first time I run it, when there is no physical 
memory associated with the shared memory object.  I retested with a new memory object to confirm this.

As it works for you the first time, that suggests to me that the approach I'm taking is sensible? 

I neglected to mention that I am using QNX 6.5.0.
RE: RE: shm_ctl() fails when setting up a shared memory physical DMA buffer for a PCI card  
Hi Iain,

that's a bit weird; I tested on a quad-core SMP x86 machine with QNX 6.5.0 as well...

Here's a question for you: Do you actually need that DMA memory to be in a shared memory object? If you only needed 
access to the memory in the application that allocates the memory, you could use only mmap() and mem_offset().

Cheers,
Thomas

-----Original Message-----
From: Iain Martin [mailto:community-noreply@qnx.com] 
Sent: Dienstag, 13. März 2012 14:50
To: ostech-core_os
Subject: Re: RE: shm_ctl() fails when setting up a shared memory physical DMA buffer for a PCI card

Thanks for your quick reply. Unfortunately, it also fails for me the first time I run it, when there is no physical 
memory associated with the shared memory object.  I retested with a new memory object to confirm this.

As it works for you the first time, that suggests to me that the approach I'm taking is sensible? 

I neglected to mention that I am using QNX 6.5.0.



_______________________________________________

OSTech
http://community.qnx.com/sf/go/post92078
Re: RE: RE: shm_ctl() fails when setting up a shared memory physical DMA buffer for a PCI card  
It's a long story so I'll be as brief as possible. I'm writing a QNX driver for our PCI and PCIe boards which have 3 
data links. I do have a working driver using mmap and memoffset64 but this limits access to the driver buffers to the 
driver process.

I originally designesd our QNX driver as a full resource manager driver but I found that there was a problem with 
multiple reads and writes (i.e. serviing 3 duplex data links concurrently). I documented this issue at: http://community
.qnx.com/sf/discussion/do/listPosts/projects.community/discussion.community.topc21108

and never found a solution. We needed to get driver released so I kept the resource manager but bypassed read and write,
  instead using a seperate thread (and shared events) for each readlink and write link to copy data to/from the dma 
buffers to/from shared memory buffers accessed by a higher level API running in a user process. This works reliably but 
is too processor intensive. So, to improve this, the obvious solution is to use a single buffer to DMA in and out of 
that our driver interface API (running in a user process) can memcpy in/out of.