[Beowulf] ...Re: Benchmark results

Robert G. Brown rgb at phy.duke.edu
Thu Jan 8 09:10:24 EST 2004

On Wed, 7 Jan 2004, Jim Lux wrote:

> >Here is my challenge to the Beowulf and MPI communities: Achieve 
> >microsecond-level *global* synchronization and tracking of all system 
> >clocks within a cluster, for application use via portable calls 
> >gettimeofday() and/or MPI_Wtime().
> Is added hardware on each node legal along with distributing some periodic 
> time mark (like a 1 pulse per second from a GPS receiver)?

IIRC, we discussed this on list within the last couple of years.  There
were only two approaches that seemed worth considering -- a network
based schema that used ntp or an ntp-like daemon to gradually refine the
system local clock to be accurate within a microsecond across a network.
This isn't horribly easy on a network with a latency on the order of
hundreds of microseconds, but it is doable -- it just takes a lot of
packets and complete control of the network.  This synchronization won't
be accurate to absolute time unless the master clock is, but it would be

It seems reasonable to do this as a grant funded project.  I've got a
couple or three student hackers handy that might tackle it if I could
get it funded (with a very small grant).  The nice thing about this
approach is that it could be a fully GPL, fully portable solution that
could be built right into e.g. a bproc cluster.  Note also that all that
is really required in terms of counters is to be able to match the
distributed/refined time base to a particular value of the CPU cycle
counter.  Then a trivial revision of the non-portable CPU-based
nanoscale gettimeofday I posted (or an architecture-aware replacement of
the archaic gettimeofday library functions) would make this effectively

The second approach is hardware, which seems perfectly "legal" but which
requires an additional investment and which pushes one a bit away from
the "commodity" ideal and into the "hardware hacker" ideal which is not
everybody's cup of tea.  GPS is (as was discussed) a good way to get a
very accurate time base, with the caveat that GPS signals are very weak
(my handheld GPS won't work indoors, or even inside a car, let alone in
a basement server room underneat a few hundred tons of steel and

GPS is getting to be VERY cheap -- almost all cell phones now apparently
have built in GPS, and for $45 or so I bought a GPS-sync'd clock from
Wal-Mart (which won't work inside the house because GPS signals are so
weak, sigh).  GPS thus seems like something that would actually be
fairly tricky to get working inside a lot of server room environments
unless an external antenna were used and piped down to the device, which
would add to the expense and hassle and be moderately inefficient on a
per-system basis.

I have my own reasons for liking the design you indicate below, and
would like it very much if two PCs separated by tens of kilometers and
not even ON a common network could synchronize an internal clock to less
than a microsecond, better yet to less than 100 nanoseconds.  For this
to happen, the whole thing REALLY needs to move onto the motherboard and
be tightly integrated with the system design, although a PCI card or USB
device both are alternatives that would likely "work", especially if one
wrote kernel drivers for them that could do the entire job of reading
and setting the internal cycle-counter-to-absolute-time variable as an
atomic operation AND actually correct the result for the time it took to
do the set.  If one could build a hardware device that kept
nanosecond-accurate time, it should be possible to write a gettimeofday
or inline assembler fragment that is accurate to within a few
nanoseconds (the minimal overhead associated with reading the counter
and saving the result as a timestamp for later conversion).

Perhaps Intel and/or AMD would consider (or are considering) adding a
GPS receiver to their future mobo designs.  Add a single port on the
back for an external antenna, or figure out how to use some of the
considerable quantity of metal attached to the system as a default
antenna.  Sell amplified GPS antennae as an inexpensive add-on.  Or
ditto on a PCI card, but again one has to find a manufacturer who will
make and sell it for $25, or maybe $35, or it won't be worth it to most
people and the price then has to rise.

Personally I can think of all sorts of benefits to having each and every
PC clock in the universe set and locked to real, absolute time within a
few nanoseconds, and similarly located in space to within a few meters
(those same nanoseconds times c).  The little radiotelescope project
I've got a student (still, sigh) working on being one of them, but far
from the only one.  I'd really like to be able to sit at my PC and have
it go "happy new year" at PRECISELY the new year, for example.

While we are at it sailing around in the blue sky, can we also put an
entropy generator on the add-on card?  Something that can generate
random bits (in parallel) at oh, 32 GHz, buffer them internally, and
deliver the entire buffer to memory via DMA on a single interrupt?  My
systems get SO tired of generating pseudorands...:-)


Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu

Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list