AMD Opteron memory bandwidth (was Re: CPUs for a Beowulf)
becker at scyld.com
Mon Sep 8 21:44:49 EDT 2003
On Tue, 9 Sep 2003, Chris Samuel wrote:
> On Tue, 9 Sep 2003 04:18 am, John Hearns wrote:
> > Seriously though, you might find that 64bit AMDs aren't that expensive.
> My understanding is that memory bandwidth with Opterons scales with
> the number of CPUs due to the HyperTransport architecture, which
> could be a big plus to some people.
Hmmm, I would describe this a little differently.
Each Opteron processor has an on-die memory controller and full
bandwidth to its local memory banks.
The only external connection is through Hyper Transport (HT) channels.
If you want more memory banks, and more bandwidth, you have to add more
memory controllers through HT. And for now, the only way to get another
memory controller is to buy the one on the same die as... another Opteron.
Again HyperTransport is the only way to connect the Opteron to
other processors and,
Only some of the HT channels can connect CPUs -- this first type is more
complex than the I/O-only HT channels.
Note that the additional CPUs/memory controllers can only attach to the
"better" HT channels.
Now for the software issue.
For a few processors, the HT-based cache coherency is fast enough to be
ignored. But for larger systems the only way avoid a big latency hit
from the coherency traffic is to minimize the coherency traffic by
having only the applications utilize it. That means running the kernel
from local memory, sharing only the locks and data structures needed for
Gee, suddenly the system starts looking like a cluster. Sure, the
application can now use global shared memory, and it's now efficient to
implement a single file system view. But all the approaches to making a
cluster efficiently scalable can be directly applied to this type of
Donald Becker becker at scyld.com
Scyld Computing Corporation http://www.scyld.com
914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system
Annapolis MD 21403 410-990-9993
Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf