kmalloc()ing HUGE buffers in excess of 1Gb - bigphysarea alternatives?

I have got a high speed multi-channel data acquisition PCIE device that would benefit from allocating huge buffers for the dma controller. I have got a system with 48Gb of ram and I would like to lock down something like 20Gb dedicated to the dma controllers on this high-speed adc device.

running 64 bit kernel 3.8.0 kmalloc is limited to 4Mb bigphysarea patch hangs my system if I allocate anything above 1Gb could use mem=28G to lop off upper memory but it could get messy accessing that memory in user space as well as kernel

What would you suggest?

Reply to
jackbenimble
Loading thread data ...

memory

well as

Presumably you're running Linux. I'm spinning off into a dark and mysterious land for me (meaning: I have no clue what I'm saying).

In a smaller embedded system, where you have to tell the OS (or what passes for one) where the memory is, the solution to this would be to simply lie to the OS, and then let your drivers play as they will.

Is there a way that you can configure/patch/whatever the kernel to ignore a big chunk of memory, and then write your drivers to talk to that chunk? This would probably be easiest if you always had the same amount of memory available, which would leave you hardware-bound -- but it sounds like you're doing something special enough that you may be so anyway.

--
Tim Wescott 
Control system and signal processing consulting 
 Click to see the full signature
Reply to
Tim Wescott

Can you allocate (early in boot) several < 1GB buffers *expecting* them to be contiguously located (you could examine the sources to *know* how consecutive allocation requests will be serviced). Then, treat them as a single contiguous object? Once they're

*yours*, use them as *you* see fit!

(eventually releasing them in the same piecemeal fashion -- if need be)

[Any limitations on what the DMAC can access?]

Of course, you could also dig through the sources to see *why* the limit exists on kmalloc -- possibly implementing a similar function with a different entry point -- or the problem "patch". Thats the whole idea behind *having* the sources (not just, "Hey, I didn't have to pay any license fees for this stuff! Yippee!")

Reply to
Don Y

Hm. One possible approach ( I have not tried it; this is where I'd start):

Use custom linker scripts to declare a peer memory section to .bss, .data and the like. You'll have to declare the buffer in 'C' with overrides for the section name to match what your linker script uses.

formatting link

seems to be a start to that. no idea what the constraints are; you may be back to 2GB sections or something.

--
Les Cargill
Reply to
Les Cargill

Unfortunately this buffer would work for USER land programs but not for a dma controller. memory buffers/addresses passed to a dma controller must have certain properties:

physically contiguous and page locked (non swappable) being the most important.

Would your suggestion above satisfy that? In my experience memory allocated by the os for processes are virtually contiguous and rarely physically so (malloc vs kmalloc).

Reply to
jackbenimble

[snip]

Another approach may be to

- use the mem= and memmap= kernel parameters to reserve a block of memory for your PCIE device, and then

- hard code the reserved address (or possibly derive it from the kernel) into the driver, so as to be able to initialize the dma controllers to the proper addresses.

Since the kernel would be explicitly forbidden from using the memmap='ed memory, it would be free for the OP's use. Since he would have manually selected the address and size of the block, he can be assured of it's placement and extent, and can code such in his drivers.

--
Lew Pitcher 
"In Skills, We Trust" 
 Click to see the full signature
Reply to
Lew Pitcher

with bootmem_alloc? perhaps. will need to study this more.

No limitation. its got 512 descriptors each house -64 bit addresses and byte transfer counts

You are right of course. Completely. I was hoping for a reason NOT to have to go down this road. I set it aside as a LAST alternative. When all other options have been exhausted ... seems I am nearing that point ...

Reply to
jackbenimble

Yup

Cant do that under Linux and still expect other apps in the system to just play well in the sandbox. Memory allocation/use isnt the sort of thing I want to play with - dma controllers have a way of causing problems if left to run amok with memory the OS has other ideas about.

Yeah bottom and top memory allocation. But I have to make sure the memory can be accessed in both kernel and userspace (build pages with a virtual address for the physical block the dma has control of)

Reply to
jackbenimble

I presume you're in a kernel module, so in that case, yes it would work exactly as you'd expect - if it works at all.

I would think it could be made to do so, but I haven't gone through the exersize. I know it'd be contiguous, and since it would be allocated only to this device driver ( through a mechanism to be named later? I am unsure if insmod takes care of that for you, but I woould think it would - it's the moral equivalent of "ld" ... ).

SFAIK ), locking just accepts an address range: int mlock(const void *addr, size_t len); I beleive you must be privileged to call mlock(), and I'm not sure what addr means in kernel space.

I am also unsure how to interface linker scripts and such through the device driver build process, which may or may not use the usual tools in the same way.

Right. A peer section to .bss would be all one big hunk.

--
Les Cargill
Reply to
Les Cargill

DMA controllers typically work with physical addresses, while the program access uses virtual addresses mapped to physical memory addresses by the MMU.

So you need some mechanism to allocate some physical memory, for the huge buffer, so that the OS can't directly access it (forging the top physical address or possibly some RAMdisk system).

To access the data transferred to that memory by the DMA, you need to map some MMU registers into that area (or at least to a part of it at a time), so that the user/kernel mode code can access a section of that huge buffer.

If the RAMdisk way works, then using the normal memory mapping files should suffice.

Reply to
upsidedown

limit

have

other

As I understand it, bigphysarea uses alloc_bootmem_low_pages to get the physical memory. AFAIK, kmalloc is used only to grab little bits of kmem for administration, so the size limit on kmalloc shouldn't matter.

Are you sure the call to bigphysarea_setup succeeds? Are you sure the call to bigphysarea_alloc(_pages) succeeds?

Maybe you said this somewhere, but does the hang occur when trying to allocate the buffer, or later trying to use the buffer?

I've only ever toyed with bigphysarea, but I *think* I recall that it didn't actually set permissions on the buffer pages - you still had to map the buffer into your address space just as if you'd grabbed a bootmem block directly.

George

Reply to
George Neuner

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.