AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf)

Donald Becker becker at
Mon Sep 8 21:44:49 EDT 2003

On Tue, 9 Sep 2003, Chris Samuel wrote:
> On Tue, 9 Sep 2003 04:18 am, John Hearns wrote:
> > Seriously though, you might find that 64bit AMDs aren't that expensive.
> My understanding is that memory bandwidth with Opterons scales with
> the number  of CPUs due to the HyperTransport architecture, which
> could be a big plus to some people.

Hmmm, I would describe this a little differently.

Each Opteron processor has an on-die memory controller and full
bandwidth to its local memory banks.

The only external connection is through Hyper Transport (HT) channels.

If you want more memory banks, and more bandwidth, you have to add more
memory controllers through HT.  And for now, the only way to get another
memory controller is to buy the one on the same die as... another Opteron.

Again HyperTransport is the only way to connect the Opteron to
   other processors and,
   I/O busses
Only some of the HT channels can connect CPUs -- this first type is more
complex than the I/O-only HT channels. 
Note that the additional CPUs/memory controllers can only attach to the
"better" HT channels. 

Now for the software issue.
For a few processors, the HT-based cache coherency is fast enough to be
ignored.  But for larger systems the only way avoid a big latency hit
from the coherency traffic is to minimize the coherency traffic by
having only the applications utilize it.  That means running the kernel
from local memory, sharing only the locks and data structures needed for
global I/O.

Gee, suddenly the system starts looking like a cluster.  Sure, the
application can now use global shared memory, and it's now efficient to
implement a single file system view.  But all the approaches to making a
cluster efficiently scalable can be directly applied to this type of

Donald Becker				becker at
Scyld Computing Corporation
914 Bay Ridge Road, Suite 220		Scyld Beowulf cluster system
Annapolis MD 21403			410-990-9993

Beowulf mailing list, Beowulf at
To change your subscription (digest mode or unsubscribe) visit

More information about the Beowulf mailing list