Can You Learn Hadoop and Spark in One Day?

Details

Written by Number Six

Published: 30 March 2016

Hits: 2503

From the Princess Bride guide to Hadoop

Apache Hadoop and Spark have received a large amount of attention in recent years. Understanding what Hadoop and Spark bring to the Big Data "revolution" can be difficult because the ecosystem of tools and applications is quite vast, changing, and somewhat convoluted. If one applies Betteridge's Law to the above headline, the answer would certainly be "No" and the mystique of Hadoop, Spark, and Big Data may continue to persist for some time.

Because Cluster Monkey does not like hearing the word "No," we decided to interview our own editor, Douglas Eadline, who has been crushing the notion that Hadoop is difficult and complex by presenting a one day workshop called Apache Hadoop with Spark in One Day. As background, Eadline has, in addition to writing numerous articles and presentations, authored several books and videos on Hadoop and Big Data, his most recent Hadoop 2 Quick-Start Guide is the genesis for the "Hadoop in One Day" concept that continues to intrigue users, administrators, and managers from all sectors of the market.

By way of full disclosure, when not writing or consulting, Eadline shares his time as Editor of Cluster Monkey and assisting Basement Supercomputing with desk-side High Performance Computing (HPC) and Hadoop computing designs.

No GPU, No Problem: Five Open Source Machine Learning Tools

Details

Written by Douglas Eadline

Published: 22 January 2016

Hits: 6936

From the this article was written by a puny human department

There is a notion floating about that suggests machine learning with deep learning is a GPU focused application. While GPUs excel at deep learning, they are not exclusively required to teach your servers some smarts. There are many free and open tools for machine learning that use good old fashion CPUs (some can also use GPUs). Before we point out some of the tools, however, a few background comments about machine learning are in order.

Machine learning can take many forms and cover many methods one of which is deep learning. Back in the 1980's machine learning was called Artificial Intelligence (AI). As with many technologies, AI was oversold and lost credibility because promised breakthroughs never materialized. Beyond the hype, there has been steady progress with machine learning. In commercial contexts, machine learning methods may be referred to as data science (statistics), predictive analytics, or predictive modeling.

You (Still) Can't Always Get What You Want

Details

Written by Douglas Eadline

Published: 25 February 2016

Hits: 1532

You can't always get what you want. But that doesn't stop me from asking. Besides, Buddha's got my back. Here's my HPC wish list.

[Note: A recent article by Al Geist, How To Kill A Supercomputer: Dirty Power, Cosmic Rays, and Bad Solder reminds us that statistics at scale represents a big challenge for HPC. The following updated article was originally published in Linux Magazine in May 2006 and offers some further thoughts on this important issue. ]

Twenty five years ago I wrote a short article in a now defunct parallel computing magazine (Parallelogram) entitled "How Will You Program 1000 Processors?" Back then, it was a good question that had no easy answer. Today, it's still a good question with no easy answer, except now it seems a bit more urgent as we step into the "multi-core" era. Indeed, when I originally wrote the article, using 1,000 processors was a far off, but real possibility. Today, 1,000 processors are a reality for many practitioners of high-performance computing (HPC). And as dual-cores (and now 18-core processors) hit the server room, effectively doubling processor counts, many more people will be joining the 1,000P club very soon.

So let's get adventurous and ask, "How will you program 10,000 processors?" As I realized twenty five years ago, such a question may never really have a complete answer. In the history of computers, no one has ever answered such a question to my liking — even when considering ten processors. Of course, there are plenty of methods and ideas like threads, messages, barrier synchronization, and so on, but when I have to think more about the computer than about my problem, something is wrong.