Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results

Robert G. Brown rgb at phy.duke.edu
Thu Jan 8 14:05:57 EST 2004

On Thu, 8 Jan 2004, Jim Lux wrote:

> But, does the CPU cycle counter run at constant speed?  Or does it ratchet
> up and down with power management, for instance.  This was a curse for us in
> the DSP world, where we did alter CPU clock rate(as in stopping the
> processor)  to reduce power during "idle time", which is why we went to
> using a counter that happened to be in a peripheral chip.

One would have to check the Intel or AMD specs to be sure, but I recall
that it is a hardware counter of the stabilized baseline clock crystal
for the system and is hence very, very stable and quite accurate.
However, I admit that I'm not certain -- clock drift in gettimeofday on
some systems USED to be a really annoying problem and a load dependent
one at that.  Also, I have no idea what happens to it on a CPU that does
cycle throttling when it gets hot (e.g. P4).

The problem with putting it in a peripheral chip is that it inevitably
increases the overhead associated with reading it, probably to hundreds
of nanoseconds or worse.  Probably adequate to achieve usec resolution,
but not adequate to get nsec resolution.

> > The second approach is hardware, which seems perfectly "legal" but which
> > requires an additional investment and which pushes one a bit away from
> > the "commodity" ideal and into the "hardware hacker" ideal which is not
> > everybody's cup of tea.  GPS is (as was discussed) a good way to get a
> > very accurate time base, with the caveat that GPS signals are very weak
> > (my handheld GPS won't work indoors, or even inside a car, let alone in
> > a basement server room underneat a few hundred tons of steel and
> > concrete).
> I think that one can probably do it with no mods to the motherboard or
> outside hardware. For instance, the NTP servers can bring in the 1pps on one
> of the serial port lines. And, "legacy free" notwithstanding, most
> motherboards still have a parallel printer port.  Perhaps something along
> the lines of PAPERS?

Again, one has to navigate the device driver for the peripheral and
likely hack the kernel or suffer random delays.  If you're going this
route one might as well just use the network and work harder, as either
way you'll need the same techniques to overcome the various sources of
random latency and refine an average to the desired resolution.

> > GPS is getting to be VERY cheap -- almost all cell phones now apparently
> > have built in GPS,
> In the GPS mfrs dreams maybe!... Soon.  Motorola just released their latest
> widget, a GPS receiver that's about 10x15 mm, and can do what's called
> "assisted" fixes, where the cell site does a lot of the work in terms of
> keeping track of satellite ephemerides, and so forth. (Greatly reducing the
> processing, and power, in the phone).
> It's all part of the E911 mandate which requires cellphone providers to give
> locations within a few meters when someone makes a 911 call on their
> cellphone.  (Such position information is also quite commercially valuable..
> they could send you customized SMS text messages: "Looking for Starbucks? 2
> blocks ahead on the left?")

I was pretty surprised, but then my Christmas present in 2002 was a
cell-phone sized GPS that cost about $100, complete with nifty screen,
built in maps, leedle controls -- and it is still mostly battery, as I
take it the GPS signals from the various satellites are quite weak and
require a lot of amplification.

>  and for $45 or so I bought a GPS-sync'd clock from
> > Wal-Mart (which won't work inside the house because GPS signals are so
> > weak, sigh).  GPS thus seems like something that would actually be
> > fairly tricky to get working inside a lot of server room environments
> > unless an external antenna were used and piped down to the device, which
> > would add to the expense and hassle and be moderately inefficient on a
> > per-system basis.
> That clock is probably not GPS, but uses the 60 kHz WWVB broadcast (at least
> most of the widely advertised "atomic clocks" do that...)

Hmmm, checking it out on the web it does indeed.  I'm probably getting
my wife's new cell phone and the various new PDA's scrambled with the
thermometer/clock (Acurite Wireless Weather and Atomic Clock Station,
$22.44 at Wal-Mart:-).  Silly me -- I assume(d) that it just meant that
somebody in China had cloned a single-chip GPS receiver and was selling
it for $0.50 to the clock's maker.  I didn't realize that the cell phone
devices were also indirectly being run through the cell towers instead
of directly from satellites, although it makes sense since cellular
signals work indoors but GPS's don't.

At $22.44 I'm tempted to buy one and hack it up and see if one can leech
the clock register contents out in a machine readable form.  Might not
get microsecond resolution that way, but it would be good enough for an
NTP server for my house and beat the hell out of the $400 cards sold
for that purpose over the counter...;-).  I've been TEMPTED to see if my
$100 GPS will do the same thing -- somewhere in there it must be reading
the time base(s) to do the location computation.

> You don't need to have GPS on each computer (in fact, that would be an
> egregious waste of money), just the sync signal piped to them all.  A single

Egregious at current prices, yes.

> wire, pulsing at 1pps, is sufficient.  Put the GPS somewhere convenient.
> You can send the 1pps down cat5 wire, or on a fiber optic link, or almost
> anything.
> Even better (if you're into this kind of thing), is a GPS timing receiver
> like the surplus Z3801, which use GPS to "discipline" a very high quality
> quartz oscillator, which then generates a very clean and precise 10MHz, as
> well as the 1 pps.  You can also generate precision IRIG (or AES/EBU or
> SMPTE) time code, and distribute that.

And read it with a $400 card?

Alas, I'm not into this kind of thing at that level.  What I REALLY want
is an over the counter device that sells for $22.95, attaches to a USB
port or a PCI slot, works in a server room deep in the bowels of the
world, and magically keeps the best clock available to linux on a given
architecture accurate to a few nsec.  Not too much, really.

Couple of years, mass market GPS, prices drop, or somebody will figure
out a way to use cell phone towers to achieve the same thing.

> > I have my own reasons for liking the design you indicate below, and
> > would like it very much if two PCs separated by tens of kilometers and
> > not even ON a common network could synchronize an internal clock to less
> > than a microsecond, better yet to less than 100 nanoseconds.
> Such things are available off the shelf now, just not as a commodity. I
> would think, though, that a Z3801 type receiver ($250 surplus) hooked to
> each PC can do it to microsecond precision.
> Companies like Symmetricom/True Time/Datum/Odetics (they're all merging and
> reforming...) sell a variety of PCI, PCMCIA, and ethernet boxes that do all
> these sorts of things, with external trigger inputs and so forth.

Ya, for prices that seem to exceed $500 a pop by a goodly amount.  Even
a card to read a time reference signal from a provided standard
interface is $400 from e.g. masterclock, even a WWVB card is $400 from
e.g. beagle.  This is nuts (especially for the WWVB card).  If Wal-Mart
can sell a WWVB clock thermometer with remote wireless thermometer
included for less than $25 full retail, and a PCI card base costs a few
dollars, I can't believe that one couldn't build and sell a WWVB PCI
time base card (and hell, throw temperature in too with a remote sensor
probe that plugs into the back or onto the card internally) for
considerably less than $100.

I suppose there is a limited market but that in part is CREATED by the
high prices.  I might buy a card for my home network for $100.  I might
buy a card for every system in my home network for $25.  Prices of $1000
(ballpark of a lot of the GPS-based offerings) might be ok for a
departmental LAN or an institution, but they are not by any means
"consumer" level offerings.

Odd at a time that radio clocks are over the counter available and even
becoming "common" at less than $50 full retail.

> > to happen, the whole thing REALLY needs to move onto the motherboard and
> > be tightly integrated with the system design, although a PCI card or USB
> > device both are alternatives that would likely "work", especially if one
> > wrote kernel drivers for them that could do the entire job of reading
> > and setting the internal cycle-counter-to-absolute-time variable as an
> > atomic operation AND actually correct the result for the time it took to
> > do the set.  If one could build a hardware device that kept
> > nanosecond-accurate time, it should be possible to write a gettimeofday
> > or inline assembler fragment that is accurate to within a few
> > nanoseconds (the minimal overhead associated with reading the counter
> > and saving the result as a timestamp for later conversion).
> This might already exist on the motherboard.. one just needs to find
> something that latches the counter deterministically.  I'd look at the game
> port interface.

Wouldn't one need something that could manage pulses with
sub-microsecond rise times and trigger reliably?  I'd expect a game
interface to be way too slow.  I'd even worry about RS232.  As I said, a
lot of the $300-500 cards sold appear to be nothing but an interface
that can reliably manage a time pulse accurate to within a usec or so.

> > Personally I can think of all sorts of benefits to having each and every
> > PC clock in the universe set and locked to real, absolute time within a
> > few nanoseconds, and similarly located in space to within a few meters
> > (those same nanoseconds times c).  The little radiotelescope project
> > I've got a student (still, sigh) working on being one of them, but far
> > from the only one.  I'd really like to be able to sit at my PC and have
> > it go "happy new year" at PRECISELY the new year, for example.
> This is what LOFAR  and the SKA are all about... I think the Allen Telescope
> Array up in Hat Creek is also using GPS disciplined clocks to sync
> everything.

nsec syncronization on a global scale has lots of benefits, some of
which could enable projects that would dwarf SETI at home.  The heck with
turning every cellular tower in the country into a giant radiotelescope
array -- turn every suitably equipped PC in the country into a receiver
in a really enormous radiotelescope array.  Now there is a project for a

Frankly, it would probably work better for SETI as well.

Maybe even tabletop relativity experiments as well.

> > While we are at it sailing around in the blue sky, can we also put an
> > entropy generator on the add-on card?  Something that can generate
> > random bits (in parallel) at oh, 32 GHz, buffer them internally, and
> > deliver the entire buffer to memory via DMA on a single interrupt?  My
> > systems get SO tired of generating pseudorands...:-)
> How about a free running counter, latched by the output from a geiger
> counter instead of the 1pps..
> (Tough to get numbers at 32 Gnumbers/second though.. but rates in the 10s of
> kHz would be easy to get with a few microcuries of Am241 and a suitable
> detector tube/amplifier. )
> Or, wide band noise fed to a video A/D and digitized. (you'd have some
> spectral whitening issues and A/D nonlinearities to deal with)

Oh, there are people looking and and even implementing a lot of these
sorts of things.  Unfortunately, a lot of "random" physical sources turn
out not be random at all, just to have a relatively short
autocorrelation function and to have distributions that are not uniform
as well. A lot of the noise-based generators fall into this category,
but quantum generators can as well (one has to look at second and higher
order correlation functions, for example -- even resonance fluorescence,
a textbook example of a "random quantum process" turns out to be photon
antibunched, and there is also the venerable work of Hanbury Brown and
Twiss.  It turns out not to be horribly easy to build a hardware
generator that passes diehard or the NIST suite, let alone one that
works fast.

Lots of effort is actually expended here in corporate america as random
numbers are a key element of cryptography, and some devices are on the
market.  They're just expensive and slow and in some cases not terribly
high quality -- "unpredictable" but not necessarily "random".
Unpredictable can be made random, sort of, sometimes -- but only via
transformations that slow the generator down.  Right now I don't know of
any process or device that can deliver uniform deviates as fast or
faster (or of any better quality) than the faster/better pseudorandom
generators, although there may be some under development or out there as
the world keeps changing.


Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu

Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list