MPI & linux compilers

[Levente Horvath]

To whom it may concern,

We have 12 PCs set up for parallel computation. All are running linux 
(Redhat 7.3) and MPI.
We would like to compute eigenvalues and eigenvectors for large matrices.

We have managed to do up to 10000x10000 matrix no problem. Our program uses 
Scalapack and Blacs
routines. These routines require two matrix to be declared. On single 
precision two 10000x10000
matrix occupies 800Mb of memory which is already exceeds the 512Mb local 
memory of
each computer in our cluster. This memory were equally distributed over the 
12 computers
upon computation. So, we think that in theory we shouldn't have any problem 
going
to large matrices; as our distributed memory is quite large 12*512Mb.

Now, if we try to run a larger size then the compiler mpif77 returns
a "large matrix" error. We have traced the compiler and found that mpif77 is 
a script
that calls up f77 and mpi libraries. Upon replacing the f77 with g77-3, we 
found that
there is no problem with the compilation up to a size of 15000x15000, then 
the
compiler crashes. After tracing the compilation procedure, we found that
the linker "as" cannot link some of the .o and .s files in our /tmp 
directory.

So, we used C rather than fortran. Statically, we cannot declare more than
a 1500x1500 matrix (that put in to a hello world program for MPI). We 
thought
it might be the problem with the static allocation of memory. So, we tried
to allocate this space dynamically without any success....

Our questions are: Are we doing something wrong here. Or are the compilers 
gcc and g77-3
responsible for such an array limit. Or are we missing the ways to allocate
memory for large matrices....

This is not the end of our story. We tried "ifc" IBM fortran 90 compiler. 
Unfortunately, we
cannot link mpi libraries against this "ifc" compiler. It just doesn't see 
them. We have
tried to compile ifc with the full path names of libraries using either 
static and dynamics libraries.
In either case we had no success...

We would appreciate all of your comments and suggestions.
Thank you in advance....

_________________________________________________________________
ninemsn Extra Storage comes with McAfee Virus Scanning - to keep your 
Hotmail account and PC safe. Click here  http://join.msn.com/

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf