Hadoop Is Not a Toaster

Details

Written by Douglas Eadline

Published: 18 September 2019

Hits: 3655

From the "Here comes the cluestick" department

Apache Hadoop has been in the press lately. Some of the content has not been positive and, often times, reflects a misunderstanding of how Hadoop relates to data processing. Indeed, we seem to be in the Trough of Disillusionment in the Technology Hype Cycle. In my opinion, many of these recent "insights" seem to come from the belief that Hadoop is some kind of toaster. For the record, Hadoop can make great toast and as it traveled up the hype curve, market exuberance thought good tasting toast could do anything. Turns out, these days, people want something to go along with their toast. What happened to the great toast?

Nothing happened to the toast. It turns out that Hadoop may have started out as a toaster, but now it is quite a bit more. Hadoop has evolved into a full kitchen. To understand modern Hadoop technology, one must understand that just like kitchen that is designed to prepare food for consumption, Hadoop is designed as a platform to prepare data for analysis and insights.

Close to the Edge: If I Don't Use ECC, Have I Made an Error?

Details

Written by Douglas Eadline

Published: 24 June 2019

Hits: 3250

A continuing installment of our (Close to the) Edge Computing series.

The main memory in computing devices is an ephemeral repository of information. Processing units (CPUs) may analyze and change the information, but ultimately the results of everything eventually end up in the main memory store. Of course, the information may move to nonvolatile memory or disk storage, which provides a more permanent resting place.

Main memory integrity is important. If errors occur in main memory, anything from nothing to a full crash of the entire computer is possible. In order to prevent and possibly correct memory errors, Error-Correcting Code memory (ECC) memory has been developed and deployed in systems where data errors may result in harmful results, like in real-time financial systems, for instance. The goal is that the data written to memory should be the same when it is read back in the future.

New Documents Available to Site Users

Details

Written by Administrator

Published: 22 August 2019

Hits: 4273

From the secret double bonus department

A somewhat un-publicized feature of Cluster Monkey are the PDF documents that registered users can download. These documents include white papers on Limulus Personal computing, a brief book called HPC for Dummies and the PowerPoint slides describing the 3D printing at SC19. To download any of the files you need to register with your email address. When logged in, a Downloads item will appear in the blue Main Menu area on the top left of the page. There you will find the materials.

We ask for your email because we are working on starting two low frequency newsletters covering HPC and Hadoop/Spark. We explain how we treat your information in our Privacy Policy. Also, as a registered user you can leave comments on articles. Cluster Monkey is a community based web-site.

One of our newest documents is a white paper comparing an on-prem Limulus personal appliance to Amazon Cloud instances. The economics are quite interesting. You can also find the slides from our 3D printing demonstration at SC19 and a 2018 white paper on building a Modern Scalable Analytics Classroom--on-prem/no data center needed.

Spring 2019 HPC Market Update

Details

Written by Number Six

Published: 21 June 2019

Hits: 1270

As part of this years ISC-19 a new Top500 list has been posted. There was very little churn at the top of the list, however the entire list is now Peta-flop scale systems. In addition, the number 136 system was an AWS cloud based cluster. There has been some buzz as to whether a cloud based system violates the Top500 rule of not allowing "stunt" clusters to be ranked on the list. The intention is that the cluster should be permanent and used for production work after the list publication.

It was also another good year for HPC according to our friends at Intersect360:

This was a solid year for the HPC market, although there was some variability between categories. The total HPC market revenue was $36.1 billion in 2018, growing 3.2% over 2017. Public cloud consumption for HPC had a second strong year, with 16.0% growth. Spending on HPC servers grew 9.1%, driven in large part by end users shifting to more computationally rich environments, fueled greatly by new requirements in machine learning. HPC services declined, due to a number of market trends including: competition with the cloud and bundling of services with server sales.

More information can be found here. Full reports can be purchased by contacting sales(you know what foes here) intersect360.com.