[Beowulf] newb question - help with NUMA + mpich2 + GPUs

Christopher Samuel samuel at unimelb.edu.au
Mon Feb 20 22:31:12 EST 2012

Hash: SHA1

On 21/02/12 09:52, Micha wrote:

> Things are actually more complex as this is a mixed GPU/GPU (CUDA) 
> system so I'm also looking for how to effectively transfer data
> between GPUs siting on different PCIe slots and find the affinity
> between GPUs and CPUs. Also at what stage is the support for using
> MPI to copy between GPUs?

The hwloc library from the Open-MPI folks will probably help with some
of it:


It can show you which cores are near which PCI devices for instance
and lstopo is a fantastic tool for getting a quick overview of a node.

I *believe* that code is in the 1.5 series but it'd be well worth
asking the question on the open-mpi lists to get a definitive answer
from someone who knows what they're talking about. :-)

There was also a discussion on the Open-MPI devel list recently about
why MVAPICH2 appears to do better than it with GPUs (for the moment),
the summary is here:


Hope this helps!
- -- 
    Christopher Samuel - Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545

Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

More information about the Beowulf mailing list