[Beowulf] ...Re: Benchmark results

Jim Lux James.P.Lux at jpl.nasa.gov
Thu Jan 8 19:14:38 EST 2004

At 02:43 PM 1/8/2004 -0700, Josip Loncaric wrote:
>Jim Lux wrote:
>>At 11:13 AM 1/8/2004 -0700, Josip Loncaric wrote:
>>>"On Efficiently Implementing Global Time for Performance Evaluation on 
>>>Multiprocessor Systems," Eric Maillet and Cecile Tron, Journal of 
>>>Parallel and Distributed Computing, No. 28, pp. 84-93 (1995).
>>What was the measurement/integration interval for their frequency 
>>measurements?  Robert (and I, and others) are interested in stability 
>>over fairly short time scales (milliseconds?).
>Alas, their sampling interval was about 6 seconds.  They had 500 message 
>exchanges in 50 minutes.
>However, drift due to independent random events should grow like 
>sqrt(time), so in absolute terms it should be lower at smaller time scales 
>(even though it may be larger in relative terms).  In other words, if you 
>had 6 microseconds of uncompensated drift in 6 seconds (1ppm), and you 
>assume that the uncompensated part was due to a random walk process, you'd 
>see about 0.06 microseconds of drift in 0.6 milliseconds.  This is not bad 
>in absolute terms, even though in relative terms it looks bad (100ppm).

Unfortunately, the errors aren't nice additive white gaussian 
processes...  You could have substantial short term variations that average 
out in the long run.

>>>Presumably, clock drift coefficients could be correlated to motherboard 
>>>temperature measurements.  Over time, one should be able to construct a 
>>>linearized model of clock dynamics, linking the local clock drift, 
>>>motherboard temperature, and possibly even power supply voltages.  This 
>>>should overcome the main sources of timing uncertainty.
>>I wouldn't be too sure that you could build a suitable model.  One could 
>>certainly make a very accurate predictor in the long term sense: it would 
>>very accurately model things like how many cycles of the processor clock 
>>there are in every 10 or 60 second interval. But at shorter time scales, 
>>the clocks MIGHT be pretty bad, with random jumps and stuff. I don't 
>>know, but since it's relevant to my current research, I'm going to find 
>>out, starting today..
>Hmmm...  I'm no expert, but I believe that typical clock oscillators 
>become quite good when temperature compensation is used.  I am not sure if 
>there is any significant supply voltage dependence, particularly since 
>there may be some voltage stabilization in the clock circuit.  I assume 
>we're talking about a crystal oscillator plus clock multipliers; there may 
>be significant phase jitter but ultimately one cares only about cycle 
>counting.  These days, there can be over 3000 cycles in a microsecond.

Yes, they're reasonably stable in terms of frequency, measured over a long 
time, but over short time spans (tau in the timing literature) they can be 
much worse.

This kind of thing is what separates run of the mill GPS receivers (which 
have 1pps jitters of around 100ns) from so-called timing grade receivers 
(which have 1pps jitter of <10-20 ns).

>If good quality synchronization pulses can be provided every second, then 
>microsecond requirement means that one can afford 1 ppm of uncompensated 
>clock drift.  Over a year, the uncompensated drift could be 31.5 
>seconds.  Clock in my car does almost as well, despite being subjected to 
>60 deg. C temperature swings and variable accelerations.

The 32.768 kHz crystal in your car clock is a good example of an oscillator 
that's not too hot in the short run (seconds) but is very good in the long 
run. Of course, they run at really, really low powers, which doesn't help.

>Perhaps I'm overly optimistic, but writing a kernel module to achieve 
>cluster-wide microsecond-level clock synchronization just might be feasible...

I am entirely certain that this can be done, and may have already been 
done. It's such a useful thing to "know, exactly, what time it is"....

James Lux, P.E.
Spacecraft Telecommunications Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
tel: (818)354-2075
fax: (818)393-6875

Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list