Allocating Memory in a 2.4 kernel
On December third 2003, Roland Krause asked a fairly common question on the Beowulf mailing list. Roland asked why he couldn't allocate more than 2GB of memory in a continuous chunk using the malloc function. He was running a code on a dual Athlon system with 4 GB of memory. The system is running Red Hat 9 with a 2.4.20 kernel. He built the kernel with CONFIG_HIGHMEM and CONFIG_HIGHMEM4G turned on so the kernel can address all 4 GB of memory. He could allocate a total amount of memory close to the 3GB limit. However, he could only allocated about 2GB in a continuous chunk and wanted to get closer to 3 GB in a single chunk. The unpatched Linux kernel compiled with the typical 4 GB options (as mentioned previously), splits memory into two parts. The first 1 GB belongs to the kernel and the remaining 3GB belongs to user applications. This is typically called the 3/1 VM (Virtual Memory) split.Mark Hahn quickly replied and gave us all a lesson in how memory works on 32-bit Linux. He replied that the address space should look something like the following:
0-128MB zero page
128MB + small program text
sbrk heap (grows up)
1GB mmap areana (grows up)
3GB - small stack base (grows down)
3GB-4GB kernel direct-mapped area
Mark explained that the 2 GB continuous chunk was allocated in
the mmap arena and its associated 2 GB limit. The other
1 GB was allocated in the sbrk heap. He also explained that
by statically linking your code, you could bypass the mmap
area entirely and see almost 3 GB for the heap or stack. He also
posted a very useful C code to explain and demonstrate all of
these concepts.
Mark also mentioned that it is possible to move these memory limits
around by modifying your kernel. He said that you could move the
default C
Finally, Mark commented that is a patch available that eliminates the kernel's 1 GB area entirely. This is the 4G:4G patch from the well-known kernel coder, Al Viro. The patch allows a full separate 4GB VM for the kernel and separate, full (and per-process) 4GB VM's for user applications. However, the patch can come at a price because it can impact kernel performance.
Roland Krause replied back to the list that he modified his kernel
changing the value of the C
This short discussion provides some valuable insight into how memory is allocated. Even though this isn't necessarily cluster specific, it does help you tune the nodes in your cluster for your application. It also points out how convenient it is to have the source code for the kernel. It allows you to adjust certain kernel parameters to match your needs. If anyone has any comments on kernel 2.6 memory management, please add a comment below. A good summary of Linux Memory can be found here.
| Sidebar One: Links Mentioned in Column |
This article was originally published in ClusterWorld Magazine. It has been updated and formated for the web. If you want to read more about HPC clusters and Linux you may wish to visit Linux Magazine.
Jeff Layton has been a cluster enthusiast since 1997 and spends far too much time reading mailing lists. He occasionally finds time to perform experiments on clusters in his basement. He also has a Ph. D. in Aeronautical and Astronautical Engineering and he's not afraid to use it.
