kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives?

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
I have got a high speed multi-channel data acquisition PCIE device that  
would benefit from allocating huge buffers for the dma controller. I have  
got a system with 48Gb of ram and I would like to lock down something  
like 20Gb dedicated to the dma controllers on this high-speed adc device.

running 64 bit kernel 3.8.0
    kmalloc is limited to 4Mb  
    bigphysarea patch hangs my system if I allocate anything above 1Gb
    could use mem28%G to lop off upper memory but it could get messy
    accessing that memory in user space as well as kernel

What would you suggest?

Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives?
On Sun, 20 Oct 2013 05:41:32 +0000, jackbenimble wrote:

Quoted text here. Click to load it
memory
well as

Presumably you're running Linux.  I'm spinning off into a dark and  
mysterious land for me (meaning: I have no clue what I'm saying).

In a smaller embedded system, where you have to tell the OS (or what  
passes for one) where the memory is, the solution to this would be to  
simply lie to the OS, and then let your drivers play as they will.

Is there a way that you can configure/patch/whatever the kernel to ignore  
a big chunk of memory, and then write your drivers to talk to that  
chunk?  This would probably be easiest if you always had the same amount  
of memory available, which would leave you hardware-bound -- but it  
sounds like you're doing something special enough that you may be so  
anyway.

--  
Tim Wescott
Control system and signal processing consulting
We've slightly trimmed the long signature. Click to see the full one.
Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives?
Quoted text here. Click to load it

Yup

Cant do that under Linux and still expect other apps in the system to
just play well in the sandbox. Memory allocation/use isnt the sort of
thing I want to play with - dma controllers have a way of causing problems
if left to run amok with memory the OS has other ideas about.
  
Quoted text here. Click to load it

Yeah bottom and top memory allocation. But I have to make sure the memory
can be accessed in both kernel and userspace (build pages with a virtual
address for the physical block the dma has control of)

Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives?
On 10/19/2013 10:41 PM, jackbenimble wrote:
Quoted text here. Click to load it

Can you allocate (early in boot) several < 1GB buffers *expecting*
them to be contiguously located (you could examine the sources
to *know* how consecutive allocation requests will be serviced).
Then, treat them as a single contiguous object?  Once they're
*yours*, use them as *you* see fit!

(eventually releasing them in the same piecemeal fashion -- if need
be)

[Any limitations on what the DMAC can access?]

Of course, you could also dig through the sources to see *why*
the limit exists on kmalloc -- possibly implementing a similar
function with a different entry point -- or the problem "patch".
Thats the whole idea behind *having* the sources (not just,  "Hey,
I didn't have to pay any license fees for this stuff!  Yippee!")

Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives?
Quoted text here. Click to load it

with bootmem_alloc? perhaps. will need to study this more.


Quoted text here. Click to load it

No limitation. its got 512 descriptors each house -64 bit addresses and  
byte transfer counts

Quoted text here. Click to load it

You are right of course. Completely. I was hoping for a reason NOT to have
to go down this road. I set it aside as a LAST alternative. When all other
options have been exhausted ... seems I am nearing that point ...  

Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives?
On Sun, 20 Oct 2013 22:08:42 GMT, jackbenimble

Quoted text here. Click to load it
limit
have
other

As I understand it, bigphysarea uses alloc_bootmem_low_pages to get
the physical memory.  AFAIK, kmalloc is used only to grab little bits
of kmem for administration, so the size limit on kmalloc shouldn't
matter.

Are you sure the call to bigphysarea_setup succeeds?
Are you sure the call to bigphysarea_alloc(_pages) succeeds?  

Maybe you said this somewhere, but does the hang occur when trying to
allocate the buffer, or later trying to use the buffer?

I've only ever toyed with bigphysarea, but I *think* I recall that it
didn't actually set permissions on the buffer pages - you still had to
map the buffer into your address space just as if you'd grabbed a
bootmem block directly.

George

Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives?


Quoted text here. Click to load it

DMA controllers typically work with physical addresses, while the
program access uses virtual addresses mapped to physical memory
addresses by the MMU.

So you need some mechanism to allocate  some physical memory, for the
huge buffer, so that the OS can't directly access it (forging the top
physical address or possibly some RAMdisk system).

To access the data transferred to that memory by the DMA, you need to
map some MMU registers into that area (or at least to a part of it at
a time), so that the user/kernel mode code can access a section of
that huge buffer.

If the RAMdisk way works, then using the normal memory mapping files
should suffice.


Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives?
jackbenimble wrote:
Quoted text here. Click to load it

Hm. One possible approach ( I have not tried it; this is where I'd
start):

Use custom linker scripts to declare a peer memory section to .bss,
.data and the like. You'll have to declare the buffer in 'C' with
overrides for the section name to match what your linker script uses.

https://answers.launchpad.net/gcc-arm-embedded/+question/205467

seems to be a start to that. no idea what the constraints are;
you may be back to 2GB sections or something.

--
Les Cargill





Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives?
Quoted text here. Click to load it

Unfortunately this buffer would work for USER land programs but not for
a dma controller. memory buffers/addresses passed to a dma controller must
have certain properties:

physically contiguous and page locked (non swappable) being the most  
important.

Would your suggestion above satisfy that? In my experience memory  
allocated by the os for processes are virtually contiguous and rarely
physically so (malloc vs kmalloc).


Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives?
jackbenimble wrote:
Quoted text here. Click to load it

I presume you're in a kernel module, so in that case, yes it would work
exactly as you'd expect - if it works at all.

Quoted text here. Click to load it


I would think it could be made to do so, but I haven't gone through the
exersize. I know it'd be contiguous, and since it would be allocated  
only to this device driver ( through a mechanism to be named later?
I am unsure if insmod takes care of that for you, but I woould think it  
would - it's the moral equivalent of "ld" ...  ).

SFAIK ), locking just accepts an address range:
int mlock(const void *addr, size_t len); I beleive you must be  
privileged to call mlock(), and I'm not sure what addr means in kernel  
space.

I am also unsure how to interface linker scripts and such through the  
device driver build process, which may or may not use the usual
tools in the same way.

Quoted text here. Click to load it

Right. A peer section to .bss would be all one big hunk.

--
Les Cargill


Re: kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives?
On Sunday 20 October 2013 13:48, in comp.os.linux.embedded, "Les Cargill"

Quoted text here. Click to load it
[snip]

Another approach may be to
- use the mem= and memmap= kernel parameters to reserve a block of memory
for your PCIE device, and then
- hard code the reserved address (or possibly derive it from the kernel)
into the driver, so as to be able to initialize the dma controllers to the
proper addresses.

Since the kernel would be explicitly forbidden from using the memmap='ed
memory, it would be free for the OP's use. Since he would have manually
selected the address and size of the block, he can be assured of it's
placement and extent, and can code such in his drivers.


--  
Lew Pitcher
"In Skills, We Trust"
We've slightly trimmed the long signature. Click to see the full one.

Site Timeline