problem allocating large amount of memory

Lombard, David N david.n.lombard at
Thu Dec 4 10:39:24 EST 2003

From: Mark Hahn; Sent: Wednesday, December 03, 2003 9:50 PM
> From: Roland Krause; Sent: Wednesday, December 03, 2003 6:25 PM
> > Here is what I _am_ able to do. Using a little test program that I
> > written I can pretty much get 3 GB of memory allocated in chunks.
> > largest chunk is 2,143 GBytes, then one of 0.939 GBytes size and
> > finally some smaller chunks of 10MBytes. So the total amount of

The 2.143 GB chunk is above TASK_UNMAPPED_BASE and the 0.939 chunk is

> yes.  unless you are quite careful, your address space looks like
> 0-128M		zero page
> 128M + small	program text
> 		sbrk heap (grows up)
> 1GB		mmap arena (grows up)
> 3GB - small	stack base (grows down)
> 3GB-4GB		kernel direct-mapped area
> your ~1GB is allocated in the sbrk heap (above text, below 1GB).
> the ~2GB is allocated in the mmap arena (glibc puts large allocations
> there, if possible, since you can munmap arbitrary pages, but heaps
> only rarely shrink).


> interestingly, you can avoid the mmap arena entirely if you try
> linking,
> avoid even static stdio).  that leaves nearly 3 GB available for the
> or stack.

Interesting, never tried static linking.  While I worked with an app
that needed dynamic linking, this is an experiment I will certainly try.

> also interesting is that you can use mmap with MAP_FIXED to avoid the
> default
> mmap-arena at 1GB.  the following code demonstrates all of these.  the
> last time
> I tried, you could also move around the default mmap base
> and could squeeze the 3G barier, too (TASK_SIZE).  I've seen patches
> make
> TASK_UNMAPPED_BASE a /proc setting, and to make the mmap arena grow
> (which lets you start it at a little under 3G, leaving a few hundred
> for stack).

Prior to RH 7.3, you could use one of the extant TASK_UNMAPPED_BASE
patches to address this problem.  I always used the patch to move
TASK_UNMAPPED_BASE UP, so that the brk() area (the 0.939 chunk above)
could get larger.  I could reliably get this up to about 2.2 GB or so
(on a per-process basis). The original requestor would want to move
TASK_UNMAPPED_BASE DOWN, so that the first big malloc() could be larger.

Starting at RH Linux 7.3, Red Hat prelinked glibc to the fixed value of
TASK_UNMAPPED_BASE so that moving TASK_UNMAPPED_BASE around only caused
heartache and despair, a.k.a., you app crashed and burned as you
trampled over glibc.

I have rebuilt a few pairs of RH kernels and glibc's to add the kernel
patch and not prelink glibc, thereby restoring the wonders of the
per-process TASK_UNMAPPED_BASE patch.  But, this must be done to both
the kernel and glibc.

So, the biggest issue in an unpatched RH world is not the user app, but

> finally, there is a patch which does away with the kernel's 1G chunk
> entirely
> (leaving 4G:4G, but necessitating some nastiness on context switches)

This is something I want to look at, to quantify how bad it actually is.

David N. Lombard

My comments represent my opinions, not those of Intel.
Beowulf mailing list, Beowulf at
To change your subscription (digest mode or unsubscribe) visit

More information about the Beowulf mailing list