problem allocating large amount of memory

[Lombard, David N]

From: Mark Hahn; Sent: Wednesday, December 03, 2003 9:50 PM
> 
> From: Roland Krause; Sent: Wednesday, December 03, 2003 6:25 PM
> > Here is what I _am_ able to do. Using a little test program that I
have
> > written I can pretty much get 3 GB of memory allocated in chunks.
The
> > largest chunk is 2,143 GBytes, then one of 0.939 GBytes size and
> > finally some smaller chunks of 10MBytes. So the total amount of
memory

The 2.143 GB chunk is above TASK_UNMAPPED_BASE and the 0.939 chunk is
below TASK_UNMAPPED_BASE.

> yes.  unless you are quite careful, your address space looks like
this:
> 
> 0-128M		zero page
> 128M + small	program text
> 		sbrk heap (grows up)
> 1GB		mmap arena (grows up)
> 3GB - small	stack base (grows down)
> 3GB-4GB		kernel direct-mapped area
> 
> your ~1GB is allocated in the sbrk heap (above text, below 1GB).
> the ~2GB is allocated in the mmap arena (glibc puts large allocations
> there, if possible, since you can munmap arbitrary pages, but heaps
can
> only rarely shrink).

Right.

> interestingly, you can avoid the mmap arena entirely if you try
(static
> linking,
> avoid even static stdio).  that leaves nearly 3 GB available for the
heap
> or stack.

Interesting, never tried static linking.  While I worked with an app
that needed dynamic linking, this is an experiment I will certainly try.

> also interesting is that you can use mmap with MAP_FIXED to avoid the
> default
> mmap-arena at 1GB.  the following code demonstrates all of these.  the
> last time
> I tried, you could also move around the default mmap base
> (TASK_UNMAPPED_BASE,
> and could squeeze the 3G barier, too (TASK_SIZE).  I've seen patches
to
> make
> TASK_UNMAPPED_BASE a /proc setting, and to make the mmap arena grow
down
> (which lets you start it at a little under 3G, leaving a few hundred
MB
> for stack).

Prior to RH 7.3, you could use one of the extant TASK_UNMAPPED_BASE
patches to address this problem.  I always used the patch to move
TASK_UNMAPPED_BASE UP, so that the brk() area (the 0.939 chunk above)
could get larger.  I could reliably get this up to about 2.2 GB or so
(on a per-process basis). The original requestor would want to move
TASK_UNMAPPED_BASE DOWN, so that the first big malloc() could be larger.


Starting at RH Linux 7.3, Red Hat prelinked glibc to the fixed value of
TASK_UNMAPPED_BASE so that moving TASK_UNMAPPED_BASE around only caused
heartache and despair, a.k.a., you app crashed and burned as you
trampled over glibc.

I have rebuilt a few pairs of RH kernels and glibc's to add the kernel
patch and not prelink glibc, thereby restoring the wonders of the
per-process TASK_UNMAPPED_BASE patch.  But, this must be done to both
the kernel and glibc.

So, the biggest issue in an unpatched RH world is not the user app, but
glibc.

> finally, there is a patch which does away with the kernel's 1G chunk
> entirely
> (leaving 4G:4G, but necessitating some nastiness on context switches)

This is something I want to look at, to quantify how bad it actually is.

-- 
David N. Lombard

My comments represent my opinions, not those of Intel.
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf