[Beowulf] IB troubles - mca_mpool_openib_register

Greg Lindahl greg.lindahl at qlogic.com
Thu Jun 22 13:13:33 EDT 2006


> The problem is that when I run with nodes=64:ppn=2 and only use -np 64 
> with my openmpi (v 1.0.2 gcc compiled), it still runs fine, but when I 
> run with -np 65 I get megabytes of error messages and the job never 
> completes.  The errors all look like this:
> 
> mca_mpool_openib_register: ibv_reg_mr(0x2a96641000,1060864)
> failed with error: Cannot allocate memory

.. which means you need to raise that memlock limit higher than your
current 8192, which is only 8 megabytes. Most MPIs on IB lock lots of
memory, more if you use more processes, and more for apps that send
large messages.

-- greg

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



More information about the Beowulf mailing list