Petabits/sec, and the like

Robert G. Brown rgb at
Thu Nov 6 08:02:06 EST 2003

On Thu, 6 Nov 2003, John Hearns wrote:

> I'm not sure of the plans for processing raw data at LHC - 
> maybe all is processed at the main site, maybe som is shipped off
> to the Tier 1 centres. I really don't know the answer here.

We have some people who work at CERN and Fermilab, and they do indeed
talk about needing very, very fat pipes, big big disk, and the biggest
problem -- backup to match.  Or if you prefer, viewing tape as a
reasonably compact high-data-density transport medium, these labs have
shipped tapes around from time immemorial -- we just happen to live in a
time where tape densities have been most unfortunately bypassed by hard
media in both data density in one direction and cost in the other.  My
feeling is that the labs are still in the process of reacting to this
and reengineering the data transport problem, balanced between wildly
varying costs and ease of use for different alternatives, political
pressure (I wasn't kidding about the jobs program for telcoms part --
lots of politicians would LOVE to see billion dollar programs for fat
pipes funded), and the actual facility/infrastructure realities at both

Duke, for example, is on one of the experimental gigabit networks, which
(at least when the project was started) was pretty bleeding edge, but
this is still only 0.1% of a terabps, connectivity is far from uniform
across the net, the pipe is shared by many users (and it isn't just the
HEP community that makes fat data sets -- medical centers like to ship
around images of their own:-).  At the one or two meetings I've sat in
on with these guys (discussing beowulfery and data transport) its like
they look at the primary campus feed and kind of shrug their shoulders
and ask if they can get a few of those for themselves -- one isn't

I personally think that there are always going to be bleeding edge
consumers of advances on any of the primary computing/data processing
bottlenecks.  Even with terabyte RAIDS (a number that would have been
unthinkably expensive just five years ago that I could now build for
myself upstairs using leftover development account money, if I had the
slightest use for a TB:-) some people are blocked by too little disk.
LOTS of people ride the Moore's Law curves on raw CPU and memory (size
and speed both).  Others pray for networks that could carry orders of
magnitude more than a "mere" Gbps.  

Most of us on this list likely wish for whole combinations of the above
-- a desktop RAID holding a petabyte of data backed up to a holographic
optical crystal, 100x faster CPUs with 1000x larger and faster memory
(to get memory speed closer to CPU speed) fed by networks with 1000x the
bw and 1/1000th the latency (c'mon, admit it, network latency on the
order of a nanosecond would be lovely.  Too bad about that pesky speed
of light thing...:-).  And while we're messing with that holographic
crystal in our imaginations, let's just make everything optical and
built on top of nanoscale devices, shall we?

One thing of great beauty is that Moore's Law makes it quite likely that
at least some of this "insane" wish list will come true over the next
decade.  Not the ns-latency network least if you want to
talk to things more than a few cm away.  <sigh>.

[Although hey, if y'all think one can violate causality AND TRANSMIT
MESSAGES by "twisting" one of a pair of correlated photons, a little
thing like ns latency networks across the entire continent become
straightforward, right?  Cannot use non-relativistic Schrodinger
equations or even concepts to describe relativistic field propagation,
grumble... no such thing as "wavefunction collapse", grumble, not
time-reversal invariant, grumble, violates causal propagation of field
UNLESS one looks at advanced field and Wheeler-Feynman and Dirac which
do not permit separation of local field interaction of eventual
absorber/measurement device from system even "back" at emission event on
same light cone, grumble.  

Having a grumbly day.  Stayed up too late working on something for a
slave-driv... I mean "friend" of mine on this very list...Grumble;-)]


Robert G. Brown	             
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at

Beowulf mailing list, Beowulf at
To change your subscription (digest mode or unsubscribe) visit

More information about the Beowulf mailing list