Which MPI implementation for MPI-2? ...

Greg Lindahl lindahl at keyresearch.com
Thu Apr 3 14:56:08 EST 2003

On Thu, Apr 03, 2003 at 08:51:50AM -0600, Richard Walsh wrote:

>  I see ... then I should also anticipate some loss of performance 
>  (higher latency) when using one-sided MPI communications compared 
>  to SHMEM. Or perhaps this is one-time overhead paid at registration
>  only?

Registration is a one-time overhead. However, the exact semantics of
MPI-2 are annoying and may end up introducing some significant
overhead for tiny messages. A machine which only allows a limited
amout of memory to be registered might have significant overhead all
the time. The T3E didn't have that problem because it had a direct
mapping from virtual to physical addresses, so the communications
system didn't need to know what the TLB mappings looked like.

For modern interconnects like Myrinet, there's enough SRAM on the card
to map the entire process: 3 bytes per page times (4 GB/4k per page) = 3
megabytes, so the larger memory version of the card suffices for a
32-bit system. The current GM only supports put, not get.

I have no idea how much memory SCI or Quadrics could map. You may be
able to hack Linux such that it always handed out groups of pages;
this would waste some memory, but could reduce the memory needed to
hold the full set of mappings by a factor of 4 for 16k groups, 16 for
64k groups, etc. It's a shame that x86 doesn't have support for
slightly larger pages; the Opteron has the same problem.

>  As far as I know, SHMEM requires a known address either explicitly 
>  passed (asymmetric location) between partners or a implicitly determined 
>  from the symmetry relationships of the images communicating (static 
>  or common).  As you say, this is "free" for COMMON/STATIC data.
>  Perhaps we are actually agreeing ... explicitly exchange addresses
>  of malloc-ed locations in different binaries would be fine.

We are agreeing. It comes down to "does this object have the same
address in both processes?"

Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list