[Beowulf] How Would You Test Infiniband in New Cluster?

Prentice Bisbal prentice at ias.edu
Wed Nov 18 00:11:41 EST 2009


Jon Forrest wrote:
> Let's say you have a brand new cluster with
> brand new Infiniband hardware, and that
> you've installed OFED 1.4 and the
> appropriate drivers for your IB
> HCAs (i.e. you see ib0 devices
> on the frontend and all compute nodes).
> The cluster appears to be working
> fine but you're not sure about IB.
>
> How would you test your IB network
> to make sure all is well?
>
> Cordially,
I would start with the basic IB diagnostic utilities. On RHEL-based
systems, they are in the infiniband-diags rpm. I have limited experience
of them myself, but you can check the man pages. They may not give you
performance metrics, but can definitely help you determine if everything
is connected and working properly.

Here's a list of the commands available from this package in my RHEL 5.3
rebuild:

$ rpm -ql infiniband-diags | grep bin
/usr/sbin/check_lft_balance.pl
/usr/sbin/dump_lfts.sh
/usr/sbin/dump_mfts.sh
/usr/sbin/ibaddr
/usr/sbin/ibcheckerrors
/usr/sbin/ibcheckerrs
/usr/sbin/ibchecknet
/usr/sbin/ibchecknode
/usr/sbin/ibcheckport
/usr/sbin/ibcheckportstate
/usr/sbin/ibcheckportwidth
/usr/sbin/ibcheckstate
/usr/sbin/ibcheckwidth
/usr/sbin/ibclearcounters
/usr/sbin/ibclearerrors
/usr/sbin/ibdatacounters
/usr/sbin/ibdatacounts
/usr/sbin/ibdiscover.pl
/usr/sbin/ibfindnodesusing.pl
/usr/sbin/ibhosts
/usr/sbin/ibidsverify.pl
/usr/sbin/iblinkinfo.pl
/usr/sbin/ibnetdiscover
/usr/sbin/ibnodes
/usr/sbin/ibping
/usr/sbin/ibportstate
/usr/sbin/ibprintca.pl
/usr/sbin/ibprintrt.pl
/usr/sbin/ibprintswitch.pl
/usr/sbin/ibqueryerrors.pl
/usr/sbin/ibroute
/usr/sbin/ibrouters
/usr/sbin/ibstat
/usr/sbin/ibstatus
/usr/sbin/ibswitches
/usr/sbin/ibswportwatch.pl
/usr/sbin/ibsysstat
/usr/sbin/ibtracert
/usr/sbin/perfquery
/usr/sbin/saquery
/usr/sbin/set_nodedesc.sh
/usr/sbin/sminfo
/usr/sbin/smpdump
/usr/sbin/smpquery
/usr/sbin/vendstat


_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



More information about the Beowulf mailing list