The Benchmark Trap: Why Your Storage Numbers Lie — and How to Get Honest Ones

A vendor datasheet promises a million IOPS. You buy the array, point your workload at it, and it’s slow. The number wasn’t a lie — it was just irrelevant to you. That gap, between an impressive benchmark and a disappointing production system, is where a lot of infrastructure money quietly goes to die.

I’ve spent a large part of my career with fio, vdbench, and HammerDB open in front of me. The tools are easy. Getting an honest number out of them is not. Here’s the field guide I wish more teams had before they trusted a benchmark — their own or a vendor’s.

The number isn’t wrong. It’s answering a different question.

Peak IOPS on a datasheet is a real measurement — of a workload that looks nothing like yours. Tiny block size, unlimited queue depth, a working set small enough to live entirely in cache, reads only, measured for ten seconds. Your production is a 70/30 read-write mix, 16K blocks, a working set far larger than cache, and it has to stay fast at 2 a.m. on day 400. Same tool, different universe. Before you argue about whose array is faster, make sure you’re both describing the same planet.

The traps that inflate a benchmark

Most misleading numbers come from a short list of mistakes:

• The working set fits in cache. You benchmarked DRAM, not the media. Size the dataset several times larger than the controller cache or you’re measuring the wrong component.
• No steady state. Flash gets slower once garbage collection kicks in. A 60-second run flatters an SSD that looks very different an hour later. Precondition the device, then measure.
• Queue-depth theater. Cranking queue depth to 256 maximizes IOPS and obliterates latency. It produces a big number and an unusable response time.
• Wrong block size or mix. The datasheet uses 4K reads; your database does 8K–16K with real writes. Model the actual mix or the result is fiction.
• A single sample. One run is an anecdote. The variance across five runs is the actual story.

An honest fio job to start from

[global]
ioengine=libaio
direct=1
runtime=600
time_based=1
ramp_time=60              # reach steady state before recording
group_reporting=1

[db-like]
rw=randrw
rwmixread=70              # 70/30 read-write, like an OLTP database
bs=16k                    # your real block size, not 4k
iodepth=32               # realistic, not a vanity QD of 256
numjobs=4
size=200g                # larger than the array's cache
percentile_list=99:99.9  # report the tail, not just the average

Every knob here is a decision about honesty. direct=1 bypasses the page cache so you measure storage, not memory. ramp_time throws away the artificially fast warm-up. size forces cache misses. percentile_list is the one most people skip — and it’s the one that matters most.

Read the right metrics

Peak IOPS is the vanity metric. What actually predicts whether production will be happy:

• Latency at your target throughput — not throughput at unlimited latency. Those are different questions with very different answers.
• Tail latency (p99, p99.9). Your users feel the worst 1% of requests, not the average. A great mean with an ugly tail is a bad system wearing a good costume.
• Consistency over time. Does it hold at steady state, or degrade as the device fills and ages?

One reframing kills most bad purchases: fix a latency budget — say, 1 ms at p99 — and ask “how many IOPS can it sustain at or under that?” Suddenly the million-IOPS array and the “slower” one often trade places.

Why this is a leadership discipline, not a lab chore

Benchmarking is how you replace opinion with evidence. In a room full of vendor claims and strong personalities, the person holding a reproducible number wins the decision — and deserves to. That’s also why a benchmark is a conviction with a review date: you state a performance expectation strongly enough to plan around it, and you re-run it when the firmware, the workload, or the scale changes. A claim without a benchmark is just an opinion. A benchmark you can’t reproduce is just a different opinion with a chart attached.

The takeaway

Don’t ask “how fast is it.” Ask “how fast is it, running my workload, at my latency budget, at steady state, averaged over five runs.” The tools will answer honestly if you ask honestly. Everything else is marketing with a monospaced font.

The full fio job file is on GitHub: github.com/waghmaredb/vexpose-labs. If you benchmark enterprise storage or databases for a living, I’d like to compare methodologies — reach me on LinkedIn or X.