[Beowulf] hardware RAID versus mdadm versus LVM-striping

Geoff Jacobs gdjacobs at gmail.com
Sat Mar 13 00:43:16 EST 2010

Tony Travis wrote:
> Rahul Nabar wrote:
>> If I have a option between doing Hardware RAID versus having software
>> raid via mdadm is there a clear winner in terms of performance? Or is
>> the answer only resolvable by actual testing? I have a fairly fast
>> machine (Nehalem 2.26 GHz 8 cores) and 48 gigs of RAM.
> Hello, Rahul.
> It depends which level of RAID you want to use, and if you want hot-swap
> capability. I use inexpensive 3ware 8006-2 RAID1 controllers and stripe
> them using "md" software RAID0 to make RAID10 arrays. This gives me good
> performance and hot-swap capability (the production md RAID driver does
> not support hot-swap). However, where "md" really scores is portability.
> My RAID's can only be read by 3ware controllers - I made a considered
> descision about this: The 3ware controllers are well-supported by Linux
> kernels, but it makes me uneasy using a proprietary RAID format. I do
> also use "md" RAID5 which is more space efficient, but read this:
>   http://www.baarf.com/

Hot swap is at least partially dependent on the controller. Even most of
the built-in controllers now support hot swap. I'm not aware of md
hardcoding anything on boot which would prevent a change on the fly,
except that you would manually have to initiate a rebuild after swapping.

Could you be more specific about what wasn't working as far as hot
swapping? Was this with current controllers? Which ones?

>> Should I be using the vendor's hardware RAID or mdadm? In case a
>> generic answer is not possible, what might be a good way to test the
>> two options? Any other implications that I should be thinking about?
> In fact, "mdadm" is just the user-space command for controlling the "md"
> driver. The problem with using an on-board RAID controller is that many
> of these are 'host' RAID (i.e. need a Windows driver to do the RAID) in
> which case you are using the CPU anyway, and they also use proprietary
> formats. Generally, I just use SATA mode on the on-board RAID controller
> and create an "md" RAID. This means that I can replace a motherboard
> withour worrying if it has the same type of RAID controller on-board.

Yes, it's pedigreed software instead of mystery meat firmware.

>> Finally, there;s always hybrid approaches. I could have several small
>> RAID5's  at the hardware level (RIAD5 seems ok since I have smaller
>> disks ~300 GB so not really in the domain where the RAID6 arguments
>> kick in, I think) Then using LVM I can integrate storage while asking
>> LVM to stripe across these RAID5's. Thus I'd get striping at two
>> levels: LVM (software) and RAID5 (hardware).
> Yes, I think a hybrid approach is good because that's what I use ;-)
> However, I would avoid relying on LVM mirroring for data protection. It
> is much safer to stripe a set of RAID1's using LVM. I don't think LVM is
> useful unless you are managing a disk farm. The commonest issue in disk
> perfomance is decoupling seeks between different spindles, so I put the
> system files on a different RAID1-set to /export (or /home) filesystems.

LVM is there as a management convenience. It allows you to grow your
disk pool more-or-less on demand. Where it's really beautiful, though,
is when you want to migrate data

Geoffrey D. Jacobs
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

More information about the Beowulf mailing list