Opinions are like clusters, everybody should have one. Well we have clusters and opinions in this category. We also welcome your feedback. Registered users can comment on articles including opinions.

You can't always get what you want. But that doesn't stop me from asking. Besides, Buddha's got my back. Here's my HPC wish list.

[Note: A recent article by Al Geist, How To Kill A Supercomputer: Dirty Power, Cosmic Rays, and Bad Solder reminds us that statistics at scale represents a big challenge for HPC. The following updated article was originally published in Linux Magazine in May 2006 and offers some further thoughts on this important issue. ]

Twenty five years ago I wrote a short article in a now defunct parallel computing magazine (Parallelogram) entitled "How Will You Program 1000 Processors?" Back then, it was a good question that had no easy answer. Today, it's still a good question with no easy answer, except now it seems a bit more urgent as we step into the "multi-core" era. Indeed, when I originally wrote the article, using 1,000 processors was a far off, but real possibility. Today, 1,000 processors are a reality for many practitioners of high-performance computing (HPC). And as dual-cores (and now 18-core processors) hit the server room, effectively doubling processor counts, many more people will be joining the 1,000P club very soon.

So let's get adventurous and ask, "How will you program 10,000 processors?" As I realized twenty five years ago, such a question may never really have a complete answer. In the history of computers, no one has ever answered such a question to my liking — even when considering ten processors. Of course, there are plenty of methods and ideas like threads, messages, barrier synchronization, and so on, but when I have to think more about the computer than about my problem, something is wrong.

From the cult of personality department

Image Ever since Watson became the Jeopardy Champ, I have been following his (its) developments. IBM and others seem to think deep learning AI's are the future. Recently, I happened upon this web page for IBM Watson Personality Insights where I can "Gain insight into how and why people think, act, and feel the way they do. This service applies linguistic analytics and personality theory to infer attributes from a person's unstructured text. The web page invites you to submit writing samples that are analyzed by Watson using some kind of linguistic API. Such a page is ripe for testing.

Back in 2009, I was frustrated. Worshiping the Top500 list was all the rage. I just did not understand what all the fuss was about. I certainly appreciated the goal of the Top500 benchmark and the valuable historical data it has collected over the years. However, using it as a metric to measure real-world HPC performance was in my mind a "high tech pissing contest." In my opinion, things have gotten a little better, but not much. The focus on one data point was great for marketing types, but scientists and engineers know better.

Hadoop has been growing clusters in datacenters at a rapid pace. Is Hadoop the new corporate HPC?

Apache Hadoop has been generating a lot of headlines lately. For those that are not aware, Hadoop is an open source project that provides a distributed file system and MapReduce framework for massive amounts of data. The primary hardware used for Hadoop is clusters of commodity servers. File sizes can easily be in the petabyte range and can easily use hundreds or thousands of compute servers.

Open the #pod bay doors, @HAL

Despite over promising the likes of HAL 9000 the Artificial Intelligence (AI) community has been making steady progress. Indeed, the famous Watson Jeopardy Experiment was a great demonstration of the coming era of "smart systems." Other examples are Apple's Siri, and smart search engines (including Google, which seems to be getting smarter about its search results each year.)

All of these efforts have several things in common; AI based software, piles of data, and racks of commodity hardware. Popular conversations include terms like business intelligence, knowledge discovery, Big Data, Hadoop, and other new buzz words. Is this yet another fad being oversold by the marketing types or is this a game changing set of technologies that will shape how we interact with almost everything we touch?


Login And Newsletter

Create an account to access exclusive content, comment on articles, and receive our newsletters.


Share The Bananas

Creative Commons License
©2005-2016 Copyright Seagrove LLC, Some rights reserved. Except where otherwise noted, this site is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License. The Cluster Monkey Logo and Monkey Character are Trademarks of Seagrove LLC.