Intel PRO/1000CT Gigabit ethernet with CSA

Steffen Persvold sp at
Sun Jun 29 16:22:33 EDT 2003

On Sun, 29 Jun 103, Mikhail Kuzminsky wrote:

> According to Daniel Pfenniger
> > Patrick Geoffray wrote:
> > > On Fri, 2003-06-27 at 13:46, Daniel Pfenniger wrote:
> > >>The exact costs are presently not well fixed because several companies
> > >>enter the market.  The nice thing about IB is that it is an open
> > >>standard, the components from different companies are compatible,
> > >>which is good for pressing costs down.
> > > 
> > > With the slicon coming from one company (actually 2 but the second one
> > > does only switch chip), the price adjustment would mainly affect the
> > > reseller, where the margin are not that high. I don't expect much a
> > > price war in the Infiniband market, mainly because many IB shops are
> > > already just burning (limited) VC cash.
> > > The main point for price advantage of IB is if the volume goes up. It's
> > > a very different problem that the multiple-vendors-marketing-stuff. One
> > > can argue that HPC does not yield such high volumes, only a business
> > > market like the Databases one does.
> > > 
> > > Remember Gigabit Ethernet. It was very expensive when the early adopters
> > > were the HPC crowd and the price didn't drop until it made its way to
> > > the desktop. It's the case for 10GE today.
> > > ...
> >  > Patrick Geoffray
> >  > Myricom, Inc.
> > 
> > Yes I mostly agree with your analysis, database is the only significant
> > potential market for IB.
> > 
> > However the problem with 1GBE or 10GBE is that the latency remains poor
> > for HPC applications, while IB goes in the right direction.
> > The real comparison to be made is not between GE and IB, but between
> > IB and Myricom products, which belong to an especially protected niche.
> > As a result for years the Myrinet products did hardly drop in price
> > for a sub-Moore's-law increase in performance, because of a lack of
> > competition (the price we paid for our Myricom cards and switch
> > 18 months ago is today *exactly* the same).
>   I agree with you both. From the viewpoint of HPC clusters the IB
> competitor is Myrinet (and SCI etc). But there are many applications
> w/coarse-grained parallelism, where bandwidth is the main thing, not the
> latency (I think, quantum chemistry applications are bandwidth-
> limited). In this case (i.e. if latnecy is less important) 10Gb Ethernet
> is also IB competitor. Moreover, IB, I beleive, will be used for
> TCP/IP connections also - in opposition to Myrinet etc. (I beleive
> there is no TCP/IP drivers for Myrinet - am I correct ?)

No, actually there is. The GM driver emulates an normal ethernet device 
the same way as IB does (and SCI). However this is not the strength of 
these high speed interconnects, and not the optimal way of using them 
(seen from our HPC/MPI perspective).
> Again, from the veiwpoint of some real appilications, there are some
> applications which
> use TCP/IP stack for parallelization (I agree that is bad, but ...)
> - for example Linda tools (used in Gaussian) work over TCP/IP, Gamess-US
> DDI "subsystem" works over TCP/IP. In the case of IB or 10Gb Ethernet
> TCP/IP is possible.

I believe the current TCP/IP implementation in Linux is not capable of 
utilizing the possible bandwidth a 10Gb Ethernet connection can provide. 

Do do that you would need new protocols utilizing RDMA (doing true 
zerocopy as we do in MPI). And then of course you are into problems like 
userspace memory pinning etc...
      Steffen Persvold           ,,,       mailto: sp at
   Senior Software Engineer     (o-o)
Scali AS, PObox 150, Oppsal, N-0619 Oslo, Norway, Tel: +4792484511

Beowulf mailing list, Beowulf at
To change your subscription (digest mode or unsubscribe) visit

More information about the Beowulf mailing list