Software continues to be one of the largest challenges to the parallel computing market. When considering parallel and multi-core computing, questions about software are most important. To help set expectations and ensure a successful project, Interactive Supercomputing has prepared some important questions worth asking about parallel software (and some answers!).
ï»¿ As engineers and scientists explore parallel programming approaches, whether they be first timers or experienced users, an increasing array of choices is available to them. What is better for your next project - a parallel extension to high-level desktop tools? Programming toolkits and APIs for C and Fortran developers? An interactive parallel computing platform?
No one-size-fits-all solution exits. The optimal choice is driven by several factors, including ease of use, scalability, platform scope, and vendorsâ market focus. The weight of each factor depends on your unique project, situation and environment. To help you sort through these factors to determine the best parallel tool for your unique needs, the following are key questions to ask your potential vendor, and why the questions matter.
Familiar desktop tools such as Python, MATLABÂ® and Mathematica have largely supplanted C and Fortran in developing custom models, algorithms, and simulations, due to their ease of use, high-level constructs, and interactive and graphical environment. If you have such a tool of choice, then the ability to use it to develop new parallel applications would reduce the learning curve substantially, and may dramatically accelerate the development of custom parallel applications. Furthermore, the more the parallelization of your code is transparent and automatic, the closer would be "time to partial satisfaction" â the ability to run your model in parallel, testing and scaling it with real data.
A measure of parallel abstraction is that a program should execute independent of the number of processors it has. Performance might vary, but the correctness of the program should not. Does the user or the system explicitly worry about the number of available processors?
Another measure of parallel abstraction is whether the user must know how data is distributed in order to execute. Again,, performance might vary, but the correctness of the program should not. Suppose, for example, you are inverting 1000 medium sized matrices, but each one is distributed. Does the user or the system worry about the data? Or suppose you divide a problem into pieces, but it doesnât divide up evenly. Do you worry about those annoying remainders, and those blocks that cross processor boundaries, or does the system take care of this automatically?
Message passing, especially the MPI standard, is a low level method for programming parallel computers. It has traditionally given expert users access to the performance they seek, though often this performance does not come easily. Although experimental hybrids are popping up now where higher level languages are being equipped with message passing, such solutions still require the users to be versed in message-passing programming.
Every user dreams that his or her serial program will just work in parallel. How close is that to reality? One measure is to count the lines of code in the serial prototype as compared to the working parallel implementation. The less the original code needs to be modified, the better.
Todayâs mainstream 8- and 16-processor clusters easily offer an order of magnitude in performance over a desktop PC. For some applications, this is more than ample. But many of todayâs toughest computational problems are getting larger and more complex every year. A 10 MB data file generated by airborne radar today may swell to a Terabyte-sized data set generated by an array of satellites.
Coarse-grained parallelism (sometimes called "embarrassingly-parallel" or "task parallelism") is a powerful method to carry out many independent calculations in parallel, such as Monte Carlo simulations, or "un-rolling" serial FOR loops. Fine-grained parallelism (sometimes called "data parallel" or "global array syntax") is used for high-level matrix and vector operations on huge data sets. It turns out that most modern production-level parallel applications require both. Some of the computations may be on data sets that fill the machineâs huge memories and require global operations such as sum of squares, or an average, or Fourier transforms or a linear system solution. However, the global operation may also interoperate with a FOR loop over smaller pieces of data, and the support for and interoperability between the coarse- and fine-grained parallelism may be critical. Taking that example, parallelizing the Monte Carlo may be a large piece of a computation, but a global analysis of the statistics that have emerged may also be equally important.
Many parallel numerical libraries, perhaps SCALAPACK being the most famous, are open source and familiar to some users. Nonetheless, these libraries are newer, more complex, and less well tested than their older counterparts. These libraries often go under the heading of research projects. Ask your provider if they just plugged in the library or if they have made proprietary tests and improvements to the basic software. Many vendors just take the libraries hoping that they are correct, leaving the user in a "buyer beware" situation.
Depending on the scope and importance of the new parallel programming project, a vendorâs commitment may be relevant. For example, if a small team is purchasing an 8-processor cluster for offloading occasional computations, the vendorâs focus on parallel computing may not be a critical issue. But for many enterprise environments, where parallel codes are in service for years (and sometimes decades), a vendorâs focus is critical. One relevant metric is the fraction of revenue that comes from serving the high-performance parallel computing segment, because this can serve as a leading indicator of how much time and money companies invest to understand and invest in solving a marketâs problems.
You need to understand the productâs maturity. A reasonable measure is the number of major releases the product has gone through (remember Windows 2.0, 3.0, 3.1? Widespread acceptance of Windows did not really start until 3.1). This scope would also extend to the research and community developments for the various open-source or academic projects that were ultimately taken commercial.
The right HPC software will not just get the job done in a current project, but also serve as the foundation for future work. How rich and broad such a platform needs to be is situation-specific, and would depend on questions like: What suite of desktop tools does the team want to use in the development of custom parallel applications? How important is the productâs extensibility, through an API or SDK, to plug in future off-the-shelf or custom codes? How important is it that the platform support both interactivity during the application development, refinement, and discovery process; as well as large batch runs?
Naturally, this may not be important in every situation. But when solving the largest, most complex â and often most important problems â vendor innovation matters. In those cases, the problem with choosing a follower is not just in the timing of a new product or feature availability, but that the company processes, culture, and approach may hinder it from delivering necessary breakthroughs. Put another way, "if they are not the lead dog, their view never changes" â and this has potentially important implications for customers.
About Interactive Supercomputing
Interactive Supercomputing (ISC) launched in 2004 to commercialize Star-P, an interactive parallel computing platform. Star-P enables automatic parallelization and interactive execution of existing desktop simulation applications on high-performance computers. Based in Waltham, Mass., the privately held company markets Star-P for a range of security, intelligence, manufacturing, energy, biomedical, financial, and scientific research applications. More information about ISC is available at www.interactivesupercomputing.com