[Beowulf] Infiniband modular switches

Joe Landman landman at scalableinformatics.com
Sun Jun 15 12:36:15 EDT 2008

On a different note ...

Gilad Shainer wrote:

> routing capabilities. And if you are bandwidth bounded, using IB QDR
> will help. You will be able to drive more than 3GB/s from each server. 

This is theoretical max, correct?  What sort of real observed DMA 
bandwidth are people seeing with the QDR adapters?  That is, how quickly 
can I copy 32MB from main memory to the PCIe (even gen2) based card?  Or 
how quickly (e.g. what is the real observed data rate) for copying 
streaming data, say from a nice NFS over RDMA session, to the card?

Please correct me if I am wrong, but I suspect that this will be one of 
rate limiting factors at least in the near term.

Not that this is a bad thing, 3GB/s is quite good, but if the practical 
best case memory->PCIe bandwidth only let us get 2GB/s, or even 1.5 
GB/s, this is important to know.

If you or someone out there with QDR could measure this, I think a few 
of us (at least on the storage side) would like to know this.  If 
someone wants to loan us cards and cables (cough cough), we would be 
happy to do this measurement between pairs of JackRabbits.


Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

More information about the Beowulf mailing list