MPI & linux compilers

Xavier Andrade xavier at zeth.ciencias.uchile.cl
Sun Aug 3 21:59:56 EDT 2003


On Sun, 3 Aug 2003, Levente Horvath wrote:

> To whom it may concern,
>
> We have 12 PCs set up for parallel computation. All are running linux
> (Redhat 7.3) and MPI.
> We would like to compute eigenvalues and eigenvectors for large matrices.
>
> We have managed to do up to 10000x10000 matrix no problem. Our program uses
> Scalapack and Blacs
> routines. These routines require two matrix to be declared. On single
> precision two 10000x10000
> matrix occupies 800Mb of memory which is already exceeds the 512Mb local
> memory of
> each computer in our cluster. This memory were equally distributed over the
> 12 computers
> upon computation. So, we think that in theory we shouldn't have any problem
> going
> to large matrices; as our distributed memory is quite large 12*512Mb.
>
> Now, if we try to run a larger size then the compiler mpif77 returns
> a "large matrix" error. We have traced the compiler and found that mpif77 is
> a script
> that calls up f77 and mpi libraries. Upon replacing the f77 with g77-3, we
> found that
> there is no problem with the compilation up to a size of 15000x15000, then
> the
> compiler crashes. After tracing the compilation procedure, we found that
> the linker "as" cannot link some of the .o and .s files in our /tmp
> directory.
>
> So, we used C rather than fortran. Statically, we cannot declare more than
> a 1500x1500 matrix (that put in to a hello world program for MPI). We
> thought
> it might be the problem with the static allocation of memory. So, we tried
> to allocate this space dynamically without any success....
>
> Our questions are: Are we doing something wrong here. Or are the compilers
> gcc and g77-3
> responsible for such an array limit. Or are we missing the ways to allocate
> memory for large matrices....
>
> This is not the end of our story. We tried "ifc" IBM fortran 90 compiler.
> Unfortunately, we
> cannot link mpi libraries against this "ifc" compiler. It just doesn't see
> them. We have
> tried to compile ifc with the full path names of libraries using either
> static and dynamics libraries.
> In either case we had no success...
>

Running "mpif77 -showme" will show you the line that mpif77 actually
calls for compiling, if you want to change the compiler that mpif77 calls
set the enviroment variable LAMHF77 (i.e. with `export LAMHF77=ifc`
mpif77 will compile using ifc instead of f77).

Xavier

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



More information about the Beowulf mailing list