problem allocating large amount of memory

Mark Hahn hahn at physics.mcmaster.ca
Thu Dec 4 00:49:50 EST 2003


> Here is what I _am_ able to do. Using a little test program that I have
> written I can pretty much get 3 GB of memory allocated in chunks. The
> largest chunk is 2,143 GBytes, then one of 0.939 GBytes size and
> finally some smaller chunks of 10MBytes. So the total amount of memory

yes.  unless you are quite careful, your address space looks like this:

0-128M		zero page
128M + small	program text
		sbrk heap (grows up)
1GB		mmap arena (grows up)
3GB - small	stack base (grows down)
3GB-4GB		kernel direct-mapped area

your ~1GB is allocated in the sbrk heap (above text, below 1GB).
the ~2GB is allocated in the mmap arena (glibc puts large allocations
there, if possible, since you can munmap arbitrary pages, but heaps can 
only rarely shrink).

interestingly, you can avoid the mmap arena entirely if you try (static linking,
avoid even static stdio).  that leaves nearly 3 GB available for the heap or stack.  
also interesting is that you can use mmap with MAP_FIXED to avoid the default 
mmap-arena at 1GB.  the following code demonstrates all of these.  the last time
I tried, you could also move around the default mmap base (TASK_UNMAPPED_BASE,
and could squeeze the 3G barier, too (TASK_SIZE).  I've seen patches to make 
TASK_UNMAPPED_BASE a /proc setting, and to make the mmap arena grow down
(which lets you start it at a little under 3G, leaving a few hundred MB for stack).
finally, there is a patch which does away with the kernel's 1G chunk entirely
(leaving 4G:4G, but necessitating some nastiness on context switches)


#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>

void print(char *message) {
    unsigned l = strlen(message);
    write(1,message,l);
}
void printuint(unsigned u) {
    char buf[20];
    char *p = buf + sizeof(buf) - 1;
    *p-- = 0;
    do {
        *p-- = "0123456789"[u % 10];
        u /= 10;
    } while (u);
    print(p+1);
}

int main() {
#if 1
//    unsigned chunk = 128*1024;                                                
    unsigned chunk = 124*1024;
    unsigned total = 0;
    void *p;

    while (p = malloc(chunk)) {
        total += chunk;
        printuint(total);
        print("MB\t: ");
        printuint((unsigned)p);
        print("\n");
    }
#else
    unsigned offset = 150*1024*1024;
    unsigned size = (unsigned) 3e9;
    void *p = mmap((void*) offset,
                   size,
                   PROT_READ|PROT_WRITE,
                   MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS,
                   0,0);
    printuint(size >> 20);
    print(" MB\t: ");
    printuint((unsigned) p);
    print("\n");
#endif
    return 0;
}

> Also has someone experience with the various kernel patches for large
> memory out there (im's 4G/4G or IBM's 3.5G/0.5G hack)? 

there's nothing IBM-specific about 3.5/.5, that's for sure.

as it happens, I'm going to be doing some measurements of performance soon.

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



More information about the Beowulf mailing list