New cluster benchmark proposal (Re: top500 list)

Jakob Oestergaard jakob at unthought.net
Tue Nov 18 02:12:46 EST 2003


On Mon, Nov 17, 2003 at 01:36:28PM -0800, Bill Broadley wrote:
> 
> After all this discussion of the top 500 list, it got me thinking about a
> "better" benchmark.  Where "better" means more useful to evaluating my
> idea of cluster goodness.  

There are lies, damn lies, and statistics...

Your points about a more appropriate benchmark are valid - but we must
realize that there is not such thing as "the one true benchmark".

Some clusters are tailored for one specific workload - one app. that has
been written for the cluster, as the cluster was built for the app. In
those situations, you can run that app on the cluster and get your "true
performance" metric.

For most of the top machines, I'd be rather surprised if there hadn't
been a pretty clear idea about what the machines would be running, prior
to purchase.

A general list such as Top500 needs one benchmark which will arguably be
both unfair and even irrelevant for a large number of the systems on the
list.  (example: if all I do is factor large numbers, I don't care what
the Linpack performance of my machine is - I may well have a system that
does factoring 10 times faster than the Earth Simulator, while my system
cannot even make the Top500).

All in all - for a list as Top500, having *one* *simple* benchmark that
is *well known*, is really the true value of the list.

Having a "fairer" benchmark with more numbers (one number is as you
argue and as per my previous example, irrelevant in many if not most
cases), would in my oppinion not be a gain for the usefulness of the
list as such.

It's not what the Top500 is for. The Top500 is for "who's got big iron
that can do Linpack really fast".  Chances are such big iron will
perform other tasks really fast as well, but we don't know, and if the
Top500 could tell us, the list would be so massively complicated that we
couldn't use it for anything at all in the first place anyway.

I think that having one poor (but well known and simple) metric is the
better solution.


-- 
................................................................
:   jakob at unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



More information about the Beowulf mailing list