[Beowulf] RE: SATA or SCSI drives - Multiple Read/write speeds.

Lombard, David N david.n.lombard at intel.com
Tue Dec 9 19:37:32 EST 2003

From: Bill Broadley [mailto:bill at cse.ucdavis.edu]
> On Tue, Dec 09, 2003 at 07:03:12AM -0800, Lombard, David N wrote:
> > Very big pro:  You can get much higher *sustained* bandwidth levels,
> > regardless of CPU load.  ATA/PATA requires CPU involvement, and
> > bandwidth tanks under moderate CPU load.
> I've heard this before, I've yet to see it.  To what do you attribute
> this advantage?  DMA scatter gather?  Higher bitrate at the read head?

Non involvement of the CPU with direct disk activities (i.e., the bits
handled by the SCSI controller) plus *way* faster CPU to handle the
high-level RAID processing v. the pokey processors found on most RAID
cards.  With multiple controllers on separate busses, I don't funnel all
my I/O through one bus.  Note again, I only discuss maximal disk
bandwidth, which means RAID-0.

> Do you have a way to quantify this *sustained* bandwidth?  Care to

Direct measurement with both standard testers and applications.
Sustained means a dataset substantially larger than memory to avoid
cache effects.

> > The highest SCSI bandwidth rates I've seen first hand are 290 MB/S
> > IA32 and 380 MB/S for IPF. Both had two controllers on independent
> > busses, 6 disks for IA32 and 12 for IPF in a s/w RAID-0 config.
> Was this RAID-5?  In Hardware?  In Software?  Which controllers?
See underlining immediately above.

> Do you have any reason to believe you wouldn't see similar with the
> same number of SATA drives on 2 independent PCI-X busses?

I have no info on SATA, thus the question later on.
> I've seen 250 MB/sec from a relatively vanilla single controller

What file size v. memory and what CPU load *not* associated with
actually driving the I/O?

> Check out: (no I don't really trust tom's that much):
> http://www6.tomshardware.com/storage/20031114/raidcore-
> 24.html#data_transfer_diagrams_raid_5
> The RaidCore manages 250 MB/sec decaying to 180MB/sec on the slower
> tracks of a drive.  Certainly seems like 2 of these on seperate busses
> would have a good change of hitting the above numbers.
> Note the very similar SCSI 8 drive setups are slower.

I'll look at this.

> > Does SATA reduce the CPU requirement from ATA/PATA, or is it the
> > Unless it's substantially lower, you still have a system best suited
> > low to moderate I/O needs.
> Do you have any way to quantify this?  Care to share?  I've seen many
> similar
> comments but when I actually go measure I get very similar numbers,
> single disks managing 40-60 MB/sec and 10% cpu, and maximum disk
> rates around 300-400 MB/sec at fairly high rates of cpu usage.

Direct measurement with both standard testers and applications.
Sustained means a dataset substantially larger than memory to avoid
cache effects.

You repeated my comment, "fairly high rates of cpu usage" -- high cpu
usage _just_to_drive_the_I/O_ meaning it's unavailable for the
application.  Also, are you quoting a burst number, that can benefit
from caching, or a sustained number, where the cache was exhausted long

The high cpu load hurts scientific/engineering apps that want to access
lots of data on disk, and burst rates are meaningless. In addition, I've
repeatedly heard that same thing from sysadmins setting up NFS servers
-- the ATA/PATA disks have too great a *negative* impact on NFS server
performance -- here the burst rates should have been more significant,
but the CPU load got in the way.

> > BTW, http://www.iozone.org/ is a nice standard I/O benchmark.  But,
> > mentioned earlier in this thread, app-specific benchmarking is
> > best.
> Agreed.  Iozone or bonnie++ seem to do fine on large sequential file
> benchmarking I prefer postmark for replicating database like access
> patterns.

Good to know. Thanks.

David N. Lombard

My comments represent my opinions, not those of Intel.

Beowulf mailing list, Beowulf at beowulf.org
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list