problem allocating large amount of memory

Mark Hahn hahn at physics.mcmaster.ca
Thu Dec 4 20:06:59 EST 2003


> I've tried your code and, yes, I am able to allocate up to 3G of memory
> in 124K chunks.

I probably should have commented on the code a bit more.  it demonstrates
three separate things: that for <128K allocations, libc uses the heap
first, then when that fills (hits the mmap arena) it switches to allocating
in the mmap arena.  if allocations are 128K or more, it *starts* in the 
mmap arena (since mmap has advantages when doing large allocations - munmap).  
finally, if you statically link and avoid the use of stdio,
you can make one giant allocation from the end of text up to stack.

you can't make that one giant allocation with malloc, though, simply
because glibc has this big-alloc-via-mmap policy.  I dimly recall that 
you can change this behavior at runtime.

> Unfortunately this doesn't not help me because the
> memory needed is allocated for a large software package, written in
> Fortran, that makes heavy use of all kinds of libraries (libc among
> others) over which I have no control. 

I'd suggest moving TASK_UNMAPPED_BASE down, and possibly going to a 
3.5 or 4GB userspace.  I think I also mentioned there's a patch to make
the mmap arena grow down - start it below your max stack extent, and 
let it grow towards the heap.

> Also, if I change your code to try to allocate the available memory in
> one chunk I am obviously in the same situation as before. If I
> understand you correctly, this is because small chunks of memory are
> allocated with sbrk, large ones with mmap.

right, though that's a purely user-space choice, nothing to do with the OS.

> I notice from the output of
> your program that the allocated memory is also not in a contiguous
> block. 

the demo program operates in three modes, one of which is a single chunk,
the other is a contiguous series of small chunks, and the other is two 
series of chunks.

> This must be because Redhat's prelinking of glibc to a fixed
> address in memory as noted by David Lombard. 

as I mentioned, this is irrelevant if you link statically.

> What I dont understand at all then is why your second code example
> (mmap) is able to return
>  2861 MB : 157286400
> or even more memory upon changing size to 4.e9. Isn't this supposently
> simply overwriting the area where glibc is in?

if you link my demo statically, there *is* no mmaped glibc chopping
up the address space.

> Will that prevent me from using stdio. 

stdio (last time I checked) used mmap even when statically linked - 
a single page, presumably a conversion buffer.  you'd have to check the 
source to see whether that can be changed.  I presume it's trying to 
initialize the buffer before the malloc heap is set up, or something 
like that.

> There is no problem linking
> statically for me. I am doing that for other reasons anyway. 

remember, no one says you have to use glibc...

regards, mark hahn.

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



More information about the Beowulf mailing list