Software RAID with MDADM

1. Introduction

Software RAID is an inexpensive way to secure your data and gain a performance boost without investing in sometimes expensive hardware RAID systems or new disks. Software RAID is included with Fedora, this tutorial covers the different types or RAID and configuring a RAID1 filesystem on Fedora, however the principles are the same for RAID0 and so on.

1.1. What is RAID?

RAID, short for Redundant Array of Independent (or Inexpensive) Disks is the combination of two or more hard disks to provide fault tolerance and increase performance. There are various levels of RAID that determine how the disks are used.

  • Level 0 (Striped Disk Array, No Fault Tolerance): Stripes data (at the block level) across two or more disks, but does not provide redundancy (fault tolerance). If one disk in the array fails, all data within that array (on all disks) is lost.

  • Level 1 (Mirroring and Duplexing): Disks within a level 1 array contain the exact same data as all other disks within the array, so called mirrors (or clones). If one disk in the array fails, then the data is still available from the remaining disks. Obviously this provides fault tolerance, and also improves the read rate of single disks. This is the level I use on my server.

  • Level 2 (Error-Correcting Coding): Level 2 RAID is rarely used, but stripes data at a bit level rather than at a block level like level 0.

  • Level 3 (Bit-Interleaved Parity): Also rarely used, level 3 provides byte-level striping with a dedicated parity disk. Level 1 cannot handle simultaneous multiple requests.

  • Level 4 (Dedicated Parity Drive): Level 4 provides block-level striping similar to level 0, but uses a parity disk. When a disk fails, the parity disk is used to create a replacement disk, however the parity disk can cause write bottlenecks.

  • Level 5 (Block Interleaved Distributed Parity): Uses byte-level data striping and error correction, giving excellent performance and good fault tolerance. Level 5 is one of the more popular of the levels.

  • Level 6 (Independent Disks with Double Parity): Uses block-level striping with parity data spread across all of the disks.

  • Level 7: A trademark of Storage Computer Corporation. Level 7 adds caching to levels 3 or 4.

  • Level 0+1 (A Mirror of Stripes): Two level 0 stripes are created and then a level 1 mirror is created over them.

  • Level 10 (A Stripe of Mirrors): Multiple level 1 mirrors are created, and a level 0 stripe is created over these.

  • RAID S: EMC Corporation's proprietary striped parity RAID system used in its Symmetrix storage systems.

RAID is generally used on servers as oppose to home machines, although some user's do like to RAID their home machines either for the additional read performance or just peace of mind that their data is safe.

2. Creating a RAID Array

Before creating your RAID array, you must first prepare the disks, it's always helpful to have your configuration planned before starting out, and I strongly recommend that if you haven't already, to do that now. Get a piece of paper, and create a table similar to the one below, this will make for less thinking later, and less chance of accidentally changing the wrong disk.

Device Partition Size RAID Level MD Device
/dev/hda1 10Gb 1 /dev/md0
/dev/hdc1 10Gb 1 /dev/md0
/dev/hdb1 40Gb 1 /dev/md1
/dev/hdd1 40Gb 1 /dev/md1

The table above is the setup that I use on my web server, the 10Gb (md0) device is the system drive and the 40Gb (md1) device is where the data is stored. Both use RAID level 1 for fault tolerance. The above table is what the rest of the tutorial will be based on, you will need to replace device names, the RAID level and the MD device with those relevant for what you are wanting to use.

2.1. Preparing the Disks

The first thing to do is prepare the individual disks and/or partitions on those disks. It should be noted here that with software RAID, you can RAID individual partitions on a disk. For example before getting the two 40Gb disks, I had a 30Gb disk with RAID level 1 to a 30Gb partition on a 60Gb disk.

In order to prepare the disks, we need to use the "fdisk" command provided by almost all Linux distributions. Run the command "fdisk /dev/hdX", where X is the letter representing the device you wish to edit.

2.1.1. RAID Filesystem

Pressing P will display all of the partitions on the disk and to what device they are mapped. If you haven't already partitioned the disk, do so now. Note: For mirrored RAID, the disks (or partitions) must be exactly the same size.

Once you have created your partitions, you need to change their file system types to "Linux raid autodetect". From the fdisk menu, press 't', if you have more than one partition, you will need to specify which partition you want to change. Enter the number, you will then be prompted for the hexadecimal code. You want 'fd' for "Linux raid autodetect".

You should get a message saying it's changed, press P to confirm. Repeat this process for each of the partitions you will be using RAID on. Once you are done, press W to write the changes to the partition table. Make sure that you have done everything correctly, after the tables have been written, you cannot undo the action. It may say you need to reboot, I recommend rebooting anyway just to be safe.

2.2. Creating the RAID Device

You're now ready to create the RAID device. To do this we use the mdadm command. You should already know what level of RAID you are wanting to use, so amend the command below as necessary.

mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/hdb1 /dev/hdd1

Repeat this command for as many of the RAID devices you need to create. Remember to increment the number of the MD device, i.e. the second time you use the command it will be /dev/md1 and so on. If you haven't physically installed all of the disks into the machine yet, put the number of raid-devices as the total number you will have, and use the name missing in place of any of the disks that aren't installed yet. You should be told that the device has been created successfully.

2.3. Formatting the New Device

Now that you have your new RAID device, it's time to format it so it can store data, the type of file system you use is entirely up to you, for the command below we're using ext3.

mkfs -t ext3 /dev/md0

Again you need to repeat this for each of the new MD devices you have created. Once you this command has completed you can start putting data onto the device.

3. Maintaining the Array

The following topics discuss checking the status of an array and replacing faulty disks should the worst happen. If you've chosen a mirrored RAID level, you're data should still be in tact. Again for all of these we use the mdadm command.

3.1. Checking the Status

mdadm --detail /dev/md0Any important information regarding your RAID arrays is automatically e-mailed to the root account by mdadm's monitoring service. But from time to time you may want to manually check the array's are in tact and working. To check the status of an array, simply use the mdadm --detail command:

mdadm --detail /dev/md0

This will give you all kinds of information, the most important parts are toward the bottom where it should say that the "State : clean", "Active Devices : x", "Working Devices : x" and "Failed Devices : 0", hopefully.

Provided you have the correct number of active devices and your array state is clean, everything is fine. The state of your array may be "degraded, resyncing", this is also fine, it just means one of the disks isn't quite up to date, but is being updated.

At the bottom below the UUID is a list of all the devices currently in the RAID array, and their present states.

3.2. Replacing Failed Disks

Depending on the level of RAID you are using, a disk failure could bring the entire array off-line, and cost you all of the data stored on it. If you are using a mirrored RAID, then the other disks will continue functioning and you simply need to replace the failed disk.

See "Checking the Status" above for how to get details on the array, this will tell you which disk has failed. The first step is to physically remove that disk and insert a new one. Remember that the next disk must be the same size (or larger if you are using partitions).

Setup the partitions on the disk as directed in the "Preparing the Disks" chapter. Once you have prepared the disk, you need to use mdadm to add it to the array.

mdadm --add /dev/md0 /dev/hda2

As soon as this command has run, you should check that the device has been added to the array using the "mdadm --detail /dev/md0" command. The state of the array will almost definitely be "degraded, resyncing", again check that the active, working and failed devices are correct. The failed disk should no longer be listed in the output.