Article Index

Software continues to be one of the largest challenges to the parallel computing market. When considering parallel and multi-core computing, questions about software are most important. To help set expectations and ensure a successful project, Interactive Supercomputing has prepared some important questions worth asking about parallel software (and some answers!).


A growing number of problems demand parallel computing capabilities these days, and the processing power of high performance technical computing servers have kept pace with this demand. Yet a hurdle to unlocking the true potential of parallel computing remains: affordable, easy-to-use parallel programming tools. The "software gap" – the gap between hardware capabilities and actual benefits we can extract through programming – is growing wider. Many applications are available for parallel computers, yet the custom development required by these tools is exceedingly complex, takes months or years to develop, and runs in batch mode over hours or days.

 As engineers and scientists explore parallel programming approaches, whether they be first timers or experienced users, an increasing array of choices is available to them. What is better for your next project - a parallel extension to high-level desktop tools? Programming toolkits and APIs for C and Fortran developers? An interactive parallel computing platform?

No one-size-fits-all solution exits. The optimal choice is driven by several factors, including ease of use, scalability, platform scope, and vendors’ market focus. The weight of each factor depends on your unique project, situation and environment. To help you sort through these factors to determine the best parallel tool for your unique needs, the following are key questions to ask your potential vendor, and why the questions matter.

Can I use my familiar desktop tools to develop the parallel application?

Familiar desktop tools such as Python, MATLAB® and Mathematica have largely supplanted C and Fortran in developing custom models, algorithms, and simulations, due to their ease of use, high-level constructs, and interactive and graphical environment. If you have such a tool of choice, then the ability to use it to develop new parallel applications would reduce the learning curve substantially, and may dramatically accelerate the development of custom parallel applications. Furthermore, the more the parallelization of your code is transparent and automatic, the closer would be "time to partial satisfaction" – the ability to run your model in parallel, testing and scaling it with real data.

Does the solution require users to substantially rewrite their code?

Related to the previous question, this question probes how different is the serial code from the parallel code. The best parallel code is a serial code that executes in parallel. There are two ways to reuse code: by executing in a global array syntax, or through task parallelism. A system that allows for the reuse of code with both paradigms is far more powerful than one with one or the other separately.

Do I have to worry about how many processors I have access to explicitly?

A measure of parallel abstraction is that a program should execute independent of the number of processors it has. Performance might vary, but the correctness of the program should not. Does the user or the system explicitly worry about the number of available processors?

Do I have to worry about data distribution explicitly?

Another measure of parallel abstraction is whether the user must know how data is distributed in order to execute. Again,, performance might vary, but the correctness of the program should not. Suppose, for example, you are inverting 1000 medium sized matrices, but each one is distributed. Does the user or the system worry about the data? Or suppose you divide a problem into pieces, but it doesn’t divide up evenly. Do you worry about those annoying remainders, and those blocks that cross processor boundaries, or does the system take care of this automatically?

Does the solution hide the complexities of message passing programming?

Message passing, especially the MPI standard, is a low level method for programming parallel computers. It has traditionally given expert users access to the performance they seek, though often this performance does not come easily. Although experimental hybrids are popping up now where higher level languages are being equipped with message passing, such solutions still require the users to be versed in message-passing programming.

What is the SLOC (software lines of code) expansion factor when going from serial to parallel?

Every user dreams that his or her serial program will just work in parallel. How close is that to reality? One measure is to count the lines of code in the serial prototype as compared to the working parallel implementation. The less the original code needs to be modified, the better.

Does it scale to large memory and processor sizes?

Today’s mainstream 8- and 16-processor clusters easily offer an order of magnitude in performance over a desktop PC. For some applications, this is more than ample. But many of today’s toughest computational problems are getting larger and more complex every year. A 10 MB data file generated by airborne radar today may swell to a Terabyte-sized data set generated by an array of satellites.

Does the solution support both embarrassingly parallel and data parallel algorithms?

Coarse-grained parallelism (sometimes called "embarrassingly-parallel" or "task parallelism") is a powerful method to carry out many independent calculations in parallel, such as Monte Carlo simulations, or "un-rolling" serial FOR loops. Fine-grained parallelism (sometimes called "data parallel" or "global array syntax") is used for high-level matrix and vector operations on huge data sets. It turns out that most modern production-level parallel applications require both. Some of the computations may be on data sets that fill the machine’s huge memories and require global operations such as sum of squares, or an average, or Fourier transforms or a linear system solution. However, the global operation may also interoperate with a FOR loop over smaller pieces of data, and the support for and interoperability between the coarse- and fine-grained parallelism may be critical. Taking that example, parallelizing the Monte Carlo may be a large piece of a computation, but a global analysis of the statistics that have emerged may also be equally important.

You have no rights to post comments


Login And Newsletter

Create an account to access exclusive content, comment on articles, and receive our newsletters.


This work is licensed under CC BY-NC-SA 4.0

©2005-2023 Copyright Seagrove LLC, Some rights reserved. Except where otherwise noted, this site is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International. The Cluster Monkey Logo and Monkey Character are Trademarks of Seagrove LLC.