Software RAID with mdadm: Levels, Setup, and What RAID Does Not Do

Software RAID with mdadm: Levels, Setup, and What RAID Does Not Do

Disks fail. Not often, but reliably enough that any machine storing something you care about should assume it will happen. RAID keeps the system running through it.

What RAID does not do is protect your data, which is a distinction worth being clear about before anything else.

The levels

RAID 0, striping. Data is split across drives. Capacity and speed are the sum of all members, and any single failure destroys everything. Not redundancy; the opposite of it. Fine for scratch space.

RAID 1, mirroring. Every drive holds the same data. Capacity is one drive regardless of how many you use, reads are fast, and it survives losing all but one member. The straightforward choice for two disks.

RAID 5, striping with one parity block. Capacity is n-1 drives, and it survives one failure. Increasingly discouraged, for the reason below.

RAID 6, striping with two parity blocks. Capacity is n-2, survives two failures. The sensible choice for large arrays now.

RAID 10, mirrored pairs, striped. Capacity is half, survives one failure per pair, and both read and write performance are good. Rebuilds are fast because only the mirror partner is read.

Why RAID 5 fell out of favour

When a drive in a RAID 5 array fails, rebuilding onto a replacement reads every block of every remaining drive. On large modern disks that takes many hours, sometimes more than a day, with all remaining drives under sustained load.

During that window there is no redundancy. A second drive failing, or a single unrecoverable read error on any remaining drive, loses the array. Both become likelier as capacity grows and as the surviving drives are the same age and model as the one that just failed.

RAID 6 tolerates a failure during the rebuild, which is the specific scenario RAID 5 does not.

Building an array

sudo apt install mdadm      # Debian and Ubuntu
sudo dnf install mdadm      # Fedora

Check what you have and confirm the device names carefully, because this destroys data:

lsblk -o NAME,SIZE,MODEL,SERIAL
# RAID 1 across two drives
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc

# RAID 10 across four
sudo mdadm --create /dev/md0 --level=10 --raid-devices=4 /dev/sd[bcde]

# RAID 6 across six
sudo mdadm --create /dev/md0 --level=6 --raid-devices=6 /dev/sd[bcdefg]

The array is usable immediately while it synchronises in the background.

cat /proc/mdstat
md0 : active raid1 sdc[1] sdb[0]
      976630464 blocks super 1.2 [2/2] [UU]
      [======>..............]  resync = 34.2% (334M/976G)

[UU] means both members are up. [U_] means one is missing, and that is the string to watch for.

Making it persist

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
sudo update-initramfs -u        # Debian and Ubuntu
sudo dracut -f                  # Fedora

Skipping this is why an array sometimes comes back as /dev/md127 after a reboot: the configuration was never saved, so the kernel assembled it with an arbitrary name.

Filesystem and mounting

sudo mkfs.ext4 /dev/md0
sudo mkdir -p /srv/data
sudo mount /dev/md0 /srv/data

# Persist with the UUID, not the device path
echo "UUID=$(sudo blkid -s UUID -o value /dev/md0) /srv/data ext4 defaults 0 2" | sudo tee -a /etc/fstab

Our fstab line builder generates that line, and mounting drives explained covers the fields.

Monitoring, which is not optional

A degraded array keeps working. That is the entire point, and it also means nothing tells you unless you have arranged for it to.

sudo nano /etc/mdadm/mdadm.conf
MAILADDR you@example.com
sudo systemctl enable --now mdmonitor
sudo mdadm --monitor --scan --test --oneshot    # send a test message

Send the test. An untested alerting path is not an alerting path.

Also run SMART checks, since a drive that is failing usually says so before it dies. Our smartctl guide covers reading the attributes.

Replacing a failed drive

# Mark and remove
sudo mdadm --manage /dev/md0 --fail /dev/sdb
sudo mdadm --manage /dev/md0 --remove /dev/sdb

# Physically swap, then add the replacement
sudo mdadm --manage /dev/md0 --add /dev/sdb

# Watch it rebuild
watch -n 10 cat /proc/mdstat

For RAID levels with parity, throttle the rebuild if the machine needs to stay responsive:

echo 50000 | sudo tee /proc/sys/dev/raid/speed_limit_max

Scrubbing

Read every block periodically so latent errors are found while you still have redundancy, rather than during a rebuild when you do not.

echo check | sudo tee /sys/block/md0/md/sync_action
cat /sys/block/md0/md/mismatch_cnt

Most distributions ship a monthly scrub timer already. Confirm it is enabled rather than assuming.

RAID is still not a backup

Every failure mode other than a dead drive propagates instantly to all members:

  • rm -rf on the wrong path
  • Ransomware encrypting the mount
  • Filesystem corruption
  • A power supply failure taking several drives with it
  • Theft, fire, flood

Run real backups with history, offsite. The restic and Borg builder generates the commands, and automated backups covers scheduling them.

Frequently Asked Questions

Is RAID a backup?

No, and treating it as one is the most expensive mistake in storage. RAID protects against a drive failing. It does nothing about accidental deletion, ransomware, filesystem corruption, or the machine being stolen, because every one of those propagates to all disks instantly.

Which RAID level should I use?

RAID 1 for two drives where you want redundancy. RAID 10 for four or more where you want redundancy and speed. RAID 6 for large arrays where capacity matters and rebuild times are long. RAID 5 is increasingly discouraged on large modern drives.

Why is RAID 5 discouraged now?

Rebuild time. A large array takes many hours to rebuild after a failure, during which every remaining drive is read end to end under load. With only one parity drive, a second failure or an unrecoverable read error during that window loses the array.

Is software RAID slower than a hardware controller?

Not meaningfully on modern hardware, and it has real advantages. mdadm arrays are portable between machines, whereas an array built on a proprietary controller may be unreadable without an identical replacement card.

How do I know a drive has failed?

Configure mdadm monitoring to email you, because the array will keep working silently in degraded mode otherwise. Check /proc/mdstat for the status line, where an underscore in place of a U marks a missing member.

Should I use mdadm or ZFS or Btrfs?

mdadm if you want block-level RAID under any filesystem. ZFS if you want checksums, snapshots, and integrated volume management and can accept an out-of-tree module. Btrfs is fine for RAID 1 and its RAID 5 and 6 modes are still not recommended for important data.