building a RAID system - yup

Robert G. Brown rgb at
Thu Oct 9 07:26:56 EDT 2003

On Wed, 8 Oct 2003, Alvin Oga wrote:

> > > 	- it supposedly can sustain 320MB/sec transfers
> > 
> > that's silly, of course.  outer tracks of current disks run at 
> > between 50 and 100 MB/s, so that's the max sustained.  you can even
> > argue that's not really 'sustained', since you'll eventually get
> > to slower inner tracks.
> yup ... those are just marketing numbers... all averages ...

It probably refers to burst delivery out of its 8 MB cache.  The actual
sustained bps speed is a pure matter of N*2*\pi*\R*f/S, where N = number
of heads, 2piRf is the linear/tangential speed of the platter at R the
read radius, and S is the linear length per bit.  This is an upper
bound.  Similarly average latency (seek time) is something like 1/2f,
the time the platter requires to move half a rotation.

> and bigg differences between inner tracks and outer tracks

Well, proportional to R, at any rate.  Given the physical geometry of
the platters (which I get to look at when I rip open old drives to
salvage their magnets) about a factor of two.

> > > independent of which raid system is built, you wil need 2 or 3
> > > more backup systems to backup your Terabyte sized raid systems
> > 
> > backup is hard.  you can get 160 or 200G tapes, but they're almost 
> to me ... backup of terabyte sized systems is trivial ...
> 	- just give me lots of software raid subsystems
> 	( 2 backups for each "main" system )
> 	- lot cheaper than tape drives and 1000x faster than tapes
> 	for live backups
> 	- will never touch a tape backup again ... too sloow
> 	and too unreliable no matter how clean the tape heads are
> 		( too slow being the key problem for restoring )

C'mon, Alvin.  Sometimes this is a workable solution, sometimes it just
plain is not.  What about archival storage?  What about offsite storage?
What about just plain moving certain data around (where networks of ANY
sort might be held to be untrustworthy).  What about due diligence if
you were are corporate IT exec held responsible for protecting client
data against loss where the data was worth real money (as in millions to
billions) compared to the cost of archival media and mechanism?  "never
touch a tape backup again" is romantic and passionate, but not
necessarily sane or good advice for the vast range of humans out there.

To backup a terabyte scale system, one needs a good automated tape
changer and a pile of tapes.  These days, this will (as Mark noted) cost
more than your original RAID, in all probability, although this depends
on how gold-plated your RAID is and whether or not you install two of
them and use one to backup the other.  I certainly don't have a tape
changer in my house as it would cost more than my server by a factor
of two or three to set up.  I backup key data by spreading it around on
some of the massive amounts of leftover disk that accumulates in any LAN
of systems in days where the smallest drives one can purchase are 40-60
GB but install images take at most a generous allotment of 5 GB
including swap.

In the physics department, though, we are in the midst of a perpetual
backup crisis, because it IS so much more expensive than storage and our
budget is limited.  Our primary department servers are all RAID and
total (IIRC) over a TB and growing.  We do actually back up to disk
several times a day so that most file restores for dropped files take at
most a few seconds to retrieve (well, more honestly a few minutes of FTE
labor between finding the file and putting it back in a user's home
directory).  However, we ALSO very definitely make tape backups using a
couple of changers, keep offsite copies and long term archives, and give
users tapes of special areas or data on request.  The tape system is
expensive, but a tiny fraction of the cost of the loss of data due to
(say) a server room fire, or a monopole storm, or a lightning strike on
the primary room feed that fries all the servers to toast.

I should also point out that since we've been using the RAIDs we have
experienced multidisk failures that required restoring from backup on
more than one occasion.  The book value probability for even one
occasion is ludicrously low, but the book value assumes event
independence and lies.  Disks are often bought in batches, and batches
of disk often fail (if they fail at all) en masse.  Failures are often
due to e.g. overheating or electrical problems, and these are often
common to either all the disks in an enclosure or all the enclosures in
a server room.

I don't think a sysadmin is ever properly paranoid about data loss until
they screw up and drop somebody's data for which they were responsible
because of inadequate backups.  Our campus OIT just dropped a big chunk
of courseware developed for active courses this fall because they
changed the storage system for the courseware without verifying their
backup, experienced a crash during the copy over, and discovered that
the backup was corrupt.  That's real money, people's effort, down the

Pants AND suspenders.  Superglue around the waistband, actually.  Who
wants to be caught with their pants down in this way?


Robert G. Brown	             
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at

Beowulf mailing list, Beowulf at
To change your subscription (digest mode or unsubscribe) visit

More information about the Beowulf mailing list