# 10 Pb/sec? Re: IB vs Myrinet

Jim Lux James.P.Lux at jpl.nasa.gov
Tue Nov 4 19:01:27 EST 2003

```>
> > Up to 10GB/sec is fairly fat today.  The roadmap for IB has this
> > interconnect technology ratcheted up way higher than 10GB.
>
>Myricom's roadmap goes up way higher than 10 Pb/s, if that makes you
>feel more comfortable.
>
>Patrick
>--

Wait a minute here... You might run into some fundamental physics problems,
especially when getting "way higher" than 10 Pb/sec... I'd like to see what
you've paved that road map with, and make sure it doesn't ruin my shoes
when I walk on it <grin>.

Say you can run a wire/fiber/whathaveyou at 10 Gb/sec (working in powers of
10, here, and 100 Gb/sec is just too hard to envision..)

10 Pb/sec would be a million times faster (Peta = 1000 Tera = 1000,000
Giga, right?)

Where are you going to fit those million wires/fibers/connectors?  Let's
say you're using optical fibers that are 10 micron in diameter (which is a
fairly impressive feat). Assuming you space them by 5 micron, you can pack
1000x1000 of them in 5 mm x 5 mm... There is a bit of a problem with
interconnects, etc., but perhaps you can terminate it right on top of a
die, and the circuitry for one channel is small enough to fit?  How
tolerant is Myricom's hardware of skew and jitter between the parallel
lines?  At least with a million lines, you can use statistical techniques
to characterize it, and you'd almost have to use some form of forward error
correction, so the extreme outliers wouldn't give you troubles.

You might be able to push the bit rate a bit higher.. We've got some
components working at 94 GHz here, and there are some novel techniques with
propagating the wave in the boundary outside a dielectric rod, so the loss
is reasonable.  We haven't figured out how to turn a corner yet, but that
wouldn't violate any laws of physics. The distance is short, so maybe
waveguide can work (optical fiber is waveguide and fairly low loss)

Hmm.. now, about that X-ium or X-lon mobo that is going to
send/accept/process the 10 Pb/s data stream....

What is the physical limit on memory speed?  The cells can only be so
small, and you've got to propagate the charge across it.  I suppose,
theoretically, one could use a charge as small as 1 electron, so that sort
of provides a lower bound.  I've heard of CMOS processes with fT of 10 GHz
in very small feature sizes (the wireless market really wants to do RF and
digital on the same chip).  Say you get that 10 GHz memory... you'll need
million way interleaving.

This starts to make the SIMD systolic arrays look more attractive doesn't it.

Maybe free space optical interconnects with monolithically fabricated
optics over the chip might be a solution? HeNe lasers are about 474 THz, as
I recall, so if you baseband encode your 10 Pb/sec bitstream, you're only
looking at 30 nm extreme UV kinds of bandwidth.  Lends new meaning to the
RF designer's term: DC to light...

James Lux, P.E.
Spacecraft Telecommunications Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive