Time and synchronization, was Re: [Beowulf] ...Re: Benchmark results

Jim Lux James.P.Lux at jpl.nasa.gov
Thu Jan 8 14:38:03 EST 2004

At 02:05 PM 1/8/2004 -0500, Robert G. Brown wrote:
>On Thu, 8 Jan 2004, Jim Lux wrote:
> >
> > I think that one can probably do it with no mods to the motherboard or
> > outside hardware. For instance, the NTP servers can bring in the 1pps 
> on one
> > of the serial port lines. And, "legacy free" notwithstanding, most
> > motherboards still have a parallel printer port.  Perhaps something along
> > the lines of PAPERS?
>Again, one has to navigate the device driver for the peripheral and
>likely hack the kernel or suffer random delays.  If you're going this
>route one might as well just use the network and work harder, as either
>way you'll need the same techniques to overcome the various sources of
>random latency and refine an average to the desired resolution.

hacking the kernel is not necessarily evil. I suppose it's a tradeoff: hack 
the kernel and use commodity hardware; use stock kernel and use hacked 

> > > GPS is getting to be VERY cheap -- almost all cell phones now apparently
> > > have built in GPS,
> >
> > In the GPS mfrs dreams maybe!... Soon.  Motorola just released their latest
> > widget, a GPS receiver that's about 10x15 mm, and can do what's called

>I was pretty surprised, but then my Christmas present in 2002 was a
>cell-phone sized GPS that cost about $100, complete with nifty screen,
>built in maps, leedle controls -- and it is still mostly battery, as I
>take it the GPS signals from the various satellites are quite weak and
>require a lot of amplification.

yep.. as a standalone device, there are lots of $50 (retail) GPS units 
available in one form or another. The basic module, with antenna, runs 
about $30-$50 in single quantities.

But, there aren't a whole lot of phones with GPS in them yet.  No real 
customer demand yet (although, I can certainly see a "On-Star" type service 
being attractive to people.. I'm lost, how do I get to the freeway)

> >
> > That clock is probably not GPS, but uses the 60 kHz WWVB broadcast (at 
> least
> > most of the widely advertised "atomic clocks" do that...)
>Hmmm, checking it out on the web it does indeed.  I'm probably getting
>my wife's new cell phone and the various new PDA's scrambled with the
>thermometer/clock (Acurite Wireless Weather and Atomic Clock Station,
>$22.44 at Wal-Mart:-).  Silly me -- I assume(d) that it just meant that
>somebody in China had cloned a single-chip GPS receiver and was selling
>it for $0.50 to the clock's maker.  I didn't realize that the cell phone
>devices were also indirectly being run through the cell towers instead
>of directly from satellites, although it makes sense since cellular
>signals work indoors but GPS's don't.

Sort of a combination of the two.. Rather than have to do the entire fix in 
the phone, they can just receive the GPS signals and make some raw 
measurements of code and carrier phase, then send it to the cell site, 
where it can get turned into position.

>At $22.44 I'm tempted to buy one and hack it up and see if one can leech
>the clock register contents out in a machine readable form.  Might not
>get microsecond resolution that way, but it would be good enough for an
>NTP server for my house and beat the hell out of the $400 cards sold
>for that purpose over the counter...;-).  I've been TEMPTED to see if my
>$100 GPS will do the same thing -- somewhere in there it must be reading
>the time base(s) to do the location computation.

Check out the documentation for the various Time Servers software.. Almost 
all GPS receivers have a 1pps output, and many synchronize the NMEA serial 
output string to the 1pps (that is, you can use the start bit of the first 
character in the sentence for sync).

The highly integrated GPS chips do NOT make it easy to read the internal 
registers, for a variety of reasons. http://www.synergy-gps.com/ is a big 
dealer of Motorola Oncore modules and has a lot of documentation.  Googling 
for the ST Microdevices ST20GP6 will find a data sheet for a typical GPS 
receiver/processor chip. (several years old design, though.. )

Search for "Totally Accurate Clock" and you'll find stuff about using WWVB 
and/or GPS for this sort of thing.

> > You don't need to have GPS on each computer (in fact, that would be an
> > egregious waste of money), just the sync signal piped to them all.  A 
> single
>Egregious at current prices, yes.
> > wire, pulsing at 1pps, is sufficient.  Put the GPS somewhere convenient.
> > You can send the 1pps down cat5 wire, or on a fiber optic link, or almost
> > anything.
> >
> > Even better (if you're into this kind of thing), is a GPS timing receiver
> > like the surplus Z3801, which use GPS to "discipline" a very high quality
> > quartz oscillator, which then generates a very clean and precise 10MHz, as
> > well as the 1 pps.  You can also generate precision IRIG (or AES/EBU or
> > SMPTE) time code, and distribute that.
>And read it with a $400 card?

Read it with software and a single bit input.  If you can latch that 
register on the edges, you're all set, because the time code itself is 
quite slow, and because time marches forward in a fairly regular fashion, 
you can take a lot of time decoding the time code sequence (you could do 1 
bit per second, and after a few minutes, have the complete sequence locked up).

You can certainly decode most audio timecode signals (i.e. IRIG) (but not 
synchronize to microseconds) with a sound card.

>Alas, I'm not into this kind of thing at that level.  What I REALLY want
>is an over the counter device that sells for $22.95, attaches to a USB
>port or a PCI slot, works in a server room deep in the bowels of the
>world, and magically keeps the best clock available to linux on a given
>architecture accurate to a few nsec.  Not too much, really.
>Couple of years, mass market GPS, prices drop, or somebody will figure
>out a way to use cell phone towers to achieve the same thing.

You could do it today for <$100, and a free serial port, and enough wire to 
get the GPS receiver to a view of the sky.

> > Companies like Symmetricom/True Time/Datum/Odetics (they're all merging and
> > reforming...) sell a variety of PCI, PCMCIA, and ethernet boxes that do all
> > these sorts of things, with external trigger inputs and so forth.
>Ya, for prices that seem to exceed $500 a pop by a goodly amount.  Even
>a card to read a time reference signal from a provided standard
>interface is $400 from e.g. masterclock, even a WWVB card is $400 from
>e.g. beagle.  This is nuts (especially for the WWVB card).  If Wal-Mart
>can sell a WWVB clock thermometer with remote wireless thermometer
>included for less than $25 full retail, and a PCI card base costs a few
>dollars, I can't believe that one couldn't build and sell a WWVB PCI
>time base card (and hell, throw temperature in too with a remote sensor
>probe that plugs into the back or onto the card internally) for
>considerably less than $100.

You could, if you could convince Wal-Mart that they could sell millions of 

>Odd at a time that radio clocks are over the counter available and even
>becoming "common" at less than $50 full retail.

Wireless LAN interfaces used to be $200, and are now <$30... it might happen.

> >
> > This might already exist on the motherboard.. one just needs to find
> > something that latches the counter deterministically.  I'd look at the game
> > port interface.
>Wouldn't one need something that could manage pulses with
>sub-microsecond rise times and trigger reliably?  I'd expect a game
>interface to be way too slow.

The original game interface latched a counter driven off the processor 
clock. Yes, overkill to measure a potentiometer position to 1 part in 100, 
but that's the hardware available.  The question would be how slavishly 
have the chipset designers held to the original design concept.

>  I'd even worry about RS232.  As I said, a
>lot of the $300-500 cards sold appear to be nothing but an interface
>that can reliably manage a time pulse accurate to within a usec or so.

For RS232, it's the interrupt latency on the RS232 status line that 
determines the performance, which can be quite impressive. Essentially a 
kernel hack, but someone's already done it for the NTB clock drivers. Sub 
microsecond is fairly easy.  All the ISR has to do is latch the processor 
time counter and set a flag, so it can run unmasked, etc.

Likewise the IRQ7 line coming in from the printer port can be used. 
Although it's low priority, you don't need perfect accuracy.. you just 
throw out the interrupts that are obviously late, having been overridden by 
a higher priority interrupt.

For those of you working with Win NT, take a look at AboutTime, which has a 
kernel driver doing just this sort of thing.

I don't know how deterministic the interrupt latency is on the modern x-ium 
and x-lon processors though.  It kind of depends on if the designer wanted 
to flush the pipeline before servicing the interrupt, or if they stall 
it.  It might be a bit tricky to even find out from the x-ium documentation 
exactly what it does, since for most users, the few nanoseconds difference 
doesn't make any difference.

There's probably also some latching pipeline logic running at fairly low 
clock rates between the IRQ line and the actually processor. Theinterrupt 
controller could introduce some random latency depending on the relative 
phases of the clock latching the interrupt request and the processor seeing 
the IRQ asserted on the chip.

James Lux, P.E.
Spacecraft Telecommunications Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
tel: (818)354-2075
fax: (818)393-6875

Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list