Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results

Jim Lux james.p.lux at jpl.nasa.gov
Thu Jan 8 11:29:46 EST 2004

----- Original Message -----
Subject: Re: [Beowulf] ...Re: Benchmark results

> On Wed, 7 Jan 2004, Jim Lux wrote:
> > >Here is my challenge to the Beowulf and MPI communities: Achieve
> > >microsecond-level *global* synchronization and tracking of all system
> > >clocks within a cluster, for application use via portable calls
> > >gettimeofday() and/or MPI_Wtime().
> >
> > Is added hardware on each node legal along with distributing some
> > time mark (like a 1 pulse per second from a GPS receiver)?
> IIRC, we discussed this on list within the last couple of years.  There
> were only two approaches that seemed worth considering -- a network
> based schema that used ntp or an ntp-like daemon to gradually refine the
> system local clock to be accurate within a microsecond across a network.
> This isn't horribly easy on a network with a latency on the order of
> hundreds of microseconds, but it is doable -- it just takes a lot of
> packets and complete control of the network.  This synchronization won't
> be accurate to absolute time unless the master clock is, but it would be
> synchronization.
> It seems reasonable to do this as a grant funded project.  I've got a
> couple or three student hackers handy that might tackle it if I could
> get it funded (with a very small grant).  The nice thing about this
> approach is that it could be a fully GPL, fully portable solution that
> could be built right into e.g. a bproc cluster.  Note also that all that
> is really required in terms of counters is to be able to match the
> distributed/refined time base to a particular value of the CPU cycle
> counter.  Then a trivial revision of the non-portable CPU-based
> nanoscale gettimeofday I posted (or an architecture-aware replacement of
> the archaic gettimeofday library functions) would make this effectively
> transparent.

But, does the CPU cycle counter run at constant speed?  Or does it ratchet
up and down with power management, for instance.  This was a curse for us in
the DSP world, where we did alter CPU clock rate(as in stopping the
processor)  to reduce power during "idle time", which is why we went to
using a counter that happened to be in a peripheral chip.

> The second approach is hardware, which seems perfectly "legal" but which
> requires an additional investment and which pushes one a bit away from
> the "commodity" ideal and into the "hardware hacker" ideal which is not
> everybody's cup of tea.  GPS is (as was discussed) a good way to get a
> very accurate time base, with the caveat that GPS signals are very weak
> (my handheld GPS won't work indoors, or even inside a car, let alone in
> a basement server room underneat a few hundred tons of steel and
> concrete).

I think that one can probably do it with no mods to the motherboard or
outside hardware. For instance, the NTP servers can bring in the 1pps on one
of the serial port lines. And, "legacy free" notwithstanding, most
motherboards still have a parallel printer port.  Perhaps something along
the lines of PAPERS?

> GPS is getting to be VERY cheap -- almost all cell phones now apparently
> have built in GPS,

In the GPS mfrs dreams maybe!... Soon.  Motorola just released their latest
widget, a GPS receiver that's about 10x15 mm, and can do what's called
"assisted" fixes, where the cell site does a lot of the work in terms of
keeping track of satellite ephemerides, and so forth. (Greatly reducing the
processing, and power, in the phone).
It's all part of the E911 mandate which requires cellphone providers to give
locations within a few meters when someone makes a 911 call on their
cellphone.  (Such position information is also quite commercially valuable..
they could send you customized SMS text messages: "Looking for Starbucks? 2
blocks ahead on the left?")

 and for $45 or so I bought a GPS-sync'd clock from
> Wal-Mart (which won't work inside the house because GPS signals are so
> weak, sigh).  GPS thus seems like something that would actually be
> fairly tricky to get working inside a lot of server room environments
> unless an external antenna were used and piped down to the device, which
> would add to the expense and hassle and be moderately inefficient on a
> per-system basis.

That clock is probably not GPS, but uses the 60 kHz WWVB broadcast (at least
most of the widely advertised "atomic clocks" do that...)

You don't need to have GPS on each computer (in fact, that would be an
egregious waste of money), just the sync signal piped to them all.  A single
wire, pulsing at 1pps, is sufficient.  Put the GPS somewhere convenient.
You can send the 1pps down cat5 wire, or on a fiber optic link, or almost

Even better (if you're into this kind of thing), is a GPS timing receiver
like the surplus Z3801, which use GPS to "discipline" a very high quality
quartz oscillator, which then generates a very clean and precise 10MHz, as
well as the 1 pps.  You can also generate precision IRIG (or AES/EBU or
SMPTE) time code, and distribute that.

> I have my own reasons for liking the design you indicate below, and
> would like it very much if two PCs separated by tens of kilometers and
> not even ON a common network could synchronize an internal clock to less
> than a microsecond, better yet to less than 100 nanoseconds.

Such things are available off the shelf now, just not as a commodity. I
would think, though, that a Z3801 type receiver ($250 surplus) hooked to
each PC can do it to microsecond precision.

Companies like Symmetricom/True Time/Datum/Odetics (they're all merging and
reforming...) sell a variety of PCI, PCMCIA, and ethernet boxes that do all
these sorts of things, with external trigger inputs and so forth.

  For this
> to happen, the whole thing REALLY needs to move onto the motherboard and
> be tightly integrated with the system design, although a PCI card or USB
> device both are alternatives that would likely "work", especially if one
> wrote kernel drivers for them that could do the entire job of reading
> and setting the internal cycle-counter-to-absolute-time variable as an
> atomic operation AND actually correct the result for the time it took to
> do the set.  If one could build a hardware device that kept
> nanosecond-accurate time, it should be possible to write a gettimeofday
> or inline assembler fragment that is accurate to within a few
> nanoseconds (the minimal overhead associated with reading the counter
> and saving the result as a timestamp for later conversion).

This might already exist on the motherboard.. one just needs to find
something that latches the counter deterministically.  I'd look at the game
port interface.

> Perhaps Intel and/or AMD would consider (or are considering) adding a
> GPS receiver to their future mobo designs.  Add a single port on the
> back for an external antenna, or figure out how to use some of the
> considerable quantity of metal attached to the system as a default
> antenna.  Sell amplified GPS antennae as an inexpensive add-on.

Nahh.. have the user buy a GPS receiver pod that is connected via a
power/data cable and keep all the RF stuff at the antenna.  Like the DeLorme
EarthMate or TripMate widgets.  The key is getting the 1pps tick into the
box in a "nice" fashion.

> Personally I can think of all sorts of benefits to having each and every
> PC clock in the universe set and locked to real, absolute time within a
> few nanoseconds, and similarly located in space to within a few meters
> (those same nanoseconds times c).  The little radiotelescope project
> I've got a student (still, sigh) working on being one of them, but far
> from the only one.  I'd really like to be able to sit at my PC and have
> it go "happy new year" at PRECISELY the new year, for example.

This is what LOFAR  and the SKA are all about... I think the Allen Telescope
Array up in Hat Creek is also using GPS disciplined clocks to sync

> While we are at it sailing around in the blue sky, can we also put an
> entropy generator on the add-on card?  Something that can generate
> random bits (in parallel) at oh, 32 GHz, buffer them internally, and
> deliver the entire buffer to memory via DMA on a single interrupt?  My
> systems get SO tired of generating pseudorands...:-)

How about a free running counter, latched by the output from a geiger
counter instead of the 1pps..
(Tough to get numbers at 32 Gnumbers/second though.. but rates in the 10s of
kHz would be easy to get with a few microcuries of Am241 and a suitable
detector tube/amplifier. )
Or, wide band noise fed to a video A/D and digitized. (you'd have some
spectral whitening issues and A/D nonlinearities to deal with)

Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list