AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf)

Joseph Landman landman at
Tue Sep 9 00:02:50 EDT 2003

One of the interesting aspects of the HT is that it makes it a good
building block for bigger systems.  Of course you have to be careful
what you ask for ...

The HT design reminds me of the old SGI XBOW internal fabric.  Also
formerly known by the (marketing/misleading) label Cray-link.  Now
called NUMA-link or something like that.

On Mon, 2003-09-08 at 22:47, Jim Phillips wrote:
> On Mon, 8 Sep 2003, Donald Becker wrote:
> > For a few processors, the HT-based cache coherency is fast enough to be
> > ignored.  But for larger systems the only way avoid a big latency hit
> > from the coherency traffic is to minimize the coherency traffic by
> > having only the applications utilize it.  That means running the kernel
> > from local memory, sharing only the locks and data structures needed for
> > global I/O.
> >
> > Gee, suddenly the system starts looking like a cluster.  Sure, the
> > application can now use global shared memory, and it's now efficient to
> > implement a single file system view.  But all the approaches to making a
> > cluster efficiently scalable can be directly applied to this type of
> > SMP...

I remember the early days of trying to get applications to scale on SGI
Origins.  The issues that Don alludes to here were largely present
there.  Some of the big issues I remember were memory hotspot
identification, page migration thrashing, and a host of other fun
items.  Customers complained when applications took non-deterministic
time to execute due to odd physical->virtual page layouts.  Sure,
go-ahead and malloc that 20 GB.  Hope it doesnt all land on 10
processors...  I remember playing with DPLACE directives on benchmarks
to tweak this stuff.

> If you could connect those HyperTransport channels into a scalable mesh
> then this sounds a lot like the good old Cray T3E design.  Drool...  It's
> too bad that Red Storm won't be taking that route.  The T3E was so nice.

You could build inexpensive Origin's or similar out of these things. 
You just need the OS/compiler support.  That is unfortunately not a
trivial thing.  

> -Jim
Joseph Landman, Ph.D
Scalable Informatics LLC
email: landman at
phone: +1 734 612 4615

Beowulf mailing list, Beowulf at
To change your subscription (digest mode or unsubscribe) visit

More information about the Beowulf mailing list