Most disk benchmarking on Linux is done with dd, and dd measures one narrow thing, frequently while accidentally measuring RAM instead. fio measures what actually matters.

sudo apt install fio      # Debian and Ubuntu
sudo dnf install fio      # Fedora
sudo pacman -S fio        # Arch

The four numbers that matter

Sequential throughput for large file copies and media.

Random IOPS for databases, virtual machines, and anything doing small scattered reads and writes. Very often the number that determines whether a system feels fast.

Latency percentiles, particularly the 99th and 99.9th. The average hides the outliers people actually notice.

Behaviour under concurrency. A device that is fine with one request at a time can collapse under thirty.

Running it

Always test against a file on a mounted filesystem unless you are deliberately benchmarking an unused raw device. Writing to a block device destroys what is on it.

cd /path/on/the/device/to/test

# Sequential read
fio --name=seqread --rw=read --bs=1M --size=4G \
    --numjobs=1 --iodepth=8 --direct=1 --runtime=30 --time_based

# Random 4k read, the database-like case
fio --name=randread --rw=randread --bs=4k --size=4G \
    --numjobs=4 --iodepth=32 --direct=1 --runtime=30 --time_based

# Mixed read and write, closer to real use
fio --name=mixed --rw=randrw --rwmixread=70 --bs=4k --size=4G \
    --numjobs=4 --iodepth=32 --direct=1 --runtime=30 --time_based

rm -f seqread.0.0 randread.0.0 mixed.0.0

Two flags matter more than the rest.

--direct=1 bypasses the page cache. Without it you are measuring RAM, which is why so many published benchmarks show absurd numbers.

--size larger than available RAM, for the same reason.

Reading the output

read: IOPS=142k, BW=555MiB/s (582MB/s)
  lat (usec): min=45, max=8912, avg=224.31, stdev=112.44
  clat percentiles (usec):
   | 50.00th=[  210], 95.00th=[  383], 99.00th=[  668], 99.90th=[ 2114]

The percentiles are the interesting part. Half of requests completed in 210 microseconds; one in a thousand took over 2 milliseconds, ten times the median.

That tail is what makes a system feel inconsistent, and an average alone would never show it.

Job files, for repeatability

Command-line invocations are hard to reproduce exactly six months later. A job file is not:

; storage-test.fio
[global]
ioengine=libaio
direct=1
size=4G
runtime=60
time_based
group_reporting

[seq-read]
rw=read
bs=1M
iodepth=8
stonewall

[rand-read]
rw=randread
bs=4k
iodepth=32
numjobs=4
stonewall

[rand-write]
rw=randwrite
bs=4k
iodepth=32
numjobs=4
stonewall

fio storage-test.fio

stonewall makes each job wait for the previous one, so they do not compete for the device and confuse the results.

Where it is genuinely useful

Comparing I/O schedulers. Run the same job file under each and compare, rather than adopting advice measured on someone else’s hardware.

Checking a VPS actually delivers what was sold. Storage performance is one of the least honest parts of hosting, and a random IOPS test tells you far more than the provider’s marketing.

Validating a new drive before trusting data to it.

Establishing a baseline so that when things feel slow later you have something to compare against. This is the use most people skip and most regret.

Ways to fool yourself

Testing on a filesystem that compresses. ZFS and Btrfs with compression turn fio’s default zero-filled data into almost nothing. Use --refill_buffers and --buffer_compress_percentage to produce realistic data.

Testing an SSD that has not been used. A fresh drive is fast until its write cache and spare blocks are exercised. Long time_based runs give a more honest number.

Ignoring thermals. NVMe drives throttle when hot. A 30-second test may never reach the temperature a real workload sustains.

Running once. Repeat and compare. Storage numbers vary more than people expect.

Frequently Asked Questions

Why not just use dd to test disk speed?

Because dd measures single-threaded sequential throughput and nothing else, which is the least representative workload for most real use. It also usually measures the page cache rather than the disk unless you explicitly bypass it.

What is the difference between throughput and IOPS?

Throughput is bytes per second and matters for large sequential transfers. IOPS is operations per second and matters for databases and anything doing many small reads and writes. A drive can be excellent at one and mediocre at the other.

Why should I look at latency percentiles rather than the average?

Because the average hides the outliers that users actually notice. A drive averaging 1ms with a 99th percentile of 200ms feels intermittently broken, and the average alone would suggest it is fine.

Does fio destroy data on the device?

It can. Writing to a raw block device overwrites whatever is there. Test against a file on a mounted filesystem unless you are deliberately benchmarking an unused device, and never point a write test at a disk holding data.

How large should the test file be?

Larger than the RAM available for page cache, or the test measures memory rather than storage. Several gigabytes is a reasonable minimum on a typical machine, and larger still on a server with a lot of RAM.

What is iodepth and why does it matter?

It is the number of I/O operations kept in flight at once. A depth of 1 measures latency of a single request; higher depths measure what the device can do when a real workload keeps it busy. NVMe drives in particular need depth to show their capability.