Robert goes on to say gettimeofday is a lousy timer for microbenchmarks that take small amounts of time, but good for 'macroscopic' benchmarks that take much longer times to run (well over several seconds). This result is because if you run the benchmark several times, any OS dependent timing interrupts will have been averaged out. Therefore you will get a nice Gaussian distribution of timings with a small standard deviation.

Robert then presents something that he calls a nano-timer that is available on nearly all Intel/AMD CPUs. He said that a little bit of assembler code can read the CPU's on-board cycle counter. If you combine this with reading the CPU's clock and doing a bit of division, you can get a timer with an accuracy of 10-60 nanoseconds. He points out that this nanotimer is far more suitable for microbenchmarks that take little time. He also discusses how these mircobenchmarks can yield very useful and interesting information about CPUs. And, finally Robert does the world a huge favor and posts the code to his nanotimer. [Note: You may also want to have a look at some recent developments in the Linux 2.6.16 kernel: The high-resolution timer API .]

Josip Loncaric, who has been a consistent contributor to the Beowulf mailing list, greatly appreciated Bob's wonderful post and added an observation. He has observed that gettimeofday has a resolution of a few microseconds, but he has seen a resolution of about 1000 microseconds for other systems which appears dependent upon the kernel scheduler. He points out that this destroys the portability of any benchmarking that uses gettimeofday. He ended this post with a challenge to the Beowulf and MPI communities to achieve true microseconds precision, globally synchronized over all the system clocks within the cluster.

Jim Lux joined in that he thought perhaps using a GPS receiver on each node with some pulse would allow the nodes in a cluster to be synchronized. Josip Loncaric jumped in to ask about NTP (Network Time Protocol) to synchronize the nodes, especially using a fast network that has a very low-level latency like Quadrics. Jim Lux responded that he thought there had been a lot of work in NTP but using it over a fairly slow network such as Fast Ethernet would pose problems. Robert Brown added that he thought NTP could be made to have a resolution of about 1 microsecond but it would probably take a complete rewrite of the internal workings of NTP to use the nanotimer coupled with perhaps a kernel module (so that the timer could be guaranteed to be effectively run in single user mode). He also thought that if one used the principles in the nanotimer, you could get even Fast Ethernet to allow an overall time synchronization of about 1 microsecond.

At this point, the discussion delved into the inner workings of clock oscillators and how they could be used for NTP. Then the topic came back to the subject of synchronizing time across a cluster using GPS. However, the discussion was very detailed on various aspects of implementing GPS (Jim Lux has a great deal of experience in this area).

The overall discussion was very interesting because it started to explain some of the inner workings of benchmarks and as a result how you could better time your codes. The discussion was very precise and detailed but provides lots of useful information. It also illustrated how companies can use benchmarks to their benefit (lies, damn lies, and benchmarks). Companies have been known to have done this in the past and some are presently doing this (particularly when their CPUs aren't competitive in the HPC world).

The discussion also produced a very interesting side topic of synchronizing nodes in a cluster. As good administrators know, synchronizing a cluster is a mandatory subject that must be mastered.

Jeff Layton has been a cluster enthusiast since 1997 and spends far too much time reading mailing lists. He has been to 38 countries and hopes to be one of them someday.

You have no rights to post comments