Category: Performance Benchmark

You added GPUs to make your LLM serving faster, and tokens-per-second barely moved. The dashboard says the GPUs are 40% utilized. Someone suggests a bigger batch size; latency gets worse. This is the moment most teams misdiagnose, because the mental model — “inference is a compute problem, so add compute” — is wrong for the part of inference that dominates your bill. Decode is bound by memory bandwidth, not FLOPs.

Two phases, two different bottlenecks

Autoregressive generation has two phases, and they live in different worlds. Prefill processes the whole prompt at once — lots of parallel matrix math, genuinely compute-bound, the GPU’s happy place. Decode generates one token at a time, and to produce each token the hardware must read the entire model’s weights out of high-bandwidth memory (HBM) again. One token, all the weights, every step. Decode isn’t doing much math per byte it moves; it’s moving a staggering number of bytes to do a little math.

So the number that predicts decode speed isn’t TFLOPS. It’s HBM bandwidth — and you can estimate the floor with arithmetic, no benchmark required.

The back-of-envelope that predicts your latency

For a memory-bound decode, the time to generate one token is roughly the bytes you must read divided by how fast you can read them:

time_per_token      ≈ (params × bytes_per_param) / HBM_bandwidth
tokens_per_second   ≈ 1 / time_per_token

Take a 70B model in FP16 (2 bytes/param) — that’s ~140 GB to read per token. On an 80GB accelerator at ~2.0 TB/s, that’s 140 / 2000 ≈ 70 ms/token: a ceiling near 14 tokens/sec for a single stream, before any compute, kernel, or networking overhead. Not because the GPU can’t do the math — because it has to haul 140 GB across the memory bus for every token.

Look at the last two rows. Quantizing the 70B model from FP16 to FP8 halves the bytes read per token and roughly doubles decode throughput — not because you added compute, but because you cut the actual bottleneck in half. That’s the tell that you’re memory-bound: the intervention that helps is the one that moves fewer bytes, not the one that adds more math.

Why batching helps — until it doesn’t

If each token read costs 140 GB regardless, the obvious move is to make that read serve many requests at once. That’s exactly what continuous batching does: read the weights once, apply them to a batch of in-flight sequences, amortize the bandwidth across all of them. Throughput climbs beautifully. This is the single highest-leverage lever most teams aren’t fully pulling.

But batching hits a wall with a name: the KV cache. Every concurrent sequence keeps a per-token cache of keys and values in the same HBM you’re already bandwidth-starved on. Push the batch bigger and the KV cache grows until it evicts, spills, or OOMs — and now you’re memory-capacity bound instead of bandwidth bound. You’ve traded one wall for another. The craft is finding the batch size that maximizes throughput at your latency SLO without tipping into KV-cache thrash.

What this changes about how you buy and build

• Spec accelerators by bandwidth and capacity, not headline TFLOPS. For decode-heavy serving, HBM bandwidth and size predict your experience better than peak compute nearly every time.
• Quantization is a throughput lever, not just a memory-savings trick. Fewer bytes per parameter is fewer bytes read per token. The speedup is the point, not a side effect.
• KV-cache management is a first-class design concern. Paged attention, cache quantization, and sane max-context limits decide how far batching can take you.
• Measure at your real batch size and context length. A single-stream benchmark and a saturated multi-tenant server are different machines wearing the same sticker.

The altitude shift

The reason this matters beyond the invoice is that it’s a problem altitude question. At low altitude, “inference is slow” gets answered with “buy more GPUs,” and the money goes to compute the workload can’t use. Raise the problem one level — which resource is actually saturated? — and the same symptom points to quantization, batching, and KV-cache strategy instead. The tool is not the transformation: a faster accelerator you’re running memory-blind just reaches the same wall a little sooner.

Hold all of this as a conviction with a review date. The arithmetic is stable — bytes-per-token doesn’t care about your vendor — but the constants move: bandwidth per dollar, quantization quality, and cache tricks improve every hardware generation. State the model strongly enough to plan a cluster around it, and re-run the numbers when the next accelerator ships.

The takeaway

Before you approve another GPU order to fix inference latency, ask which resource is saturated. If decode dominates your workload, the honest answer is usually memory bandwidth — and the fixes that work are the ones that move fewer bytes per token, not the ones that add more math. Compute is what the datasheet sells. Bandwidth is what you actually ship on.

The token-cost calculator and serving benchmarks are on GitHub: github.com/waghmaredb/vexpose-labs. Running LLMs in production and seeing the same wall? I’d like to compare numbers — LinkedIn or X.

Your semantic search was instant at ten thousand rows. At two million it’s 800 milliseconds and climbing, and you never touched the query. Almost always the cause is the same: pgvector is doing an exact, brute-force scan of every vector because its index was never built — or was built with defaults that don’t fit your data.

Exact search doesn’t scale, and it’s the default

Without an approximate index, pgvector compares your query vector against every row. That’s fine at ten thousand and fatal at two million. The fix is an ANN index — but an ANN index has two knobs that decide everything, and both have quietly wrong defaults for a large table.

-- lists ≈ rows / 1000 up to ~1M rows, then ≈ sqrt(rows).
-- 2,000,000 rows -> ~1,414 lists, NOT the tiny number you'll get by guessing.
CREATE INDEX ON docs
  USING ivfflat (embedding vector_cosine_ops)
  WITH (lists = 1414);

-- probes trades recall for speed AT QUERY TIME. The default of 1 is far too low.
-- A sane starting point is ~sqrt(lists); here sqrt(1414) ≈ 38. Then tune to a recall target.
SET ivfflat.probes = 38;

Two failure modes, opposite symptoms. Too few lists and each partition is huge, so every probe scans a lot — slow. Too few probes and you scan too few partitions — fast, but you silently miss relevant results, which in a RAG system means confidently answering from the wrong chunks. You cannot tune one without measuring the other.

Build the index after the data is loaded

IVFFlat clusters your existing vectors to define its lists. Build it on an empty or tiny table and the clusters are meaningless; every later insert lands in an ill-fitting partition and recall degrades. Load first, then index — and rebuild after any large ingest.

-- HNSW: slower to build, larger on disk, but better recall/latency
-- and no clusters to go stale as data changes.
CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);
SET hnsw.ef_search = 40;   -- the recall/speed dial at query time

If your data grows or churns continuously, HNSW usually ages better than IVFFlat because it has no clusters to go stale. It costs more to build and store; that’s the trade you’re making.

The lesson

Measure recall, not just latency. A vector search that got ten times faster by missing a third of the right answers isn’t faster — it’s broken with a good p99. Pick a fixed set of queries with known-good results, and every time you touch lists, probes, or the index type, confirm recall held before you celebrate the speed.

The benchmark harness and recall test are on GitHub: github.com/waghmaredb/vexpose-labs. Tuning vector search at scale? Compare notes on LinkedIn or X.

A vendor datasheet promises a million IOPS. You buy the array, point your workload at it, and it’s slow. The number wasn’t a lie — it was just irrelevant to you. That gap, between an impressive benchmark and a disappointing production system, is where a lot of infrastructure money quietly goes to die.

I’ve spent a large part of my career with fio, vdbench, and HammerDB open in front of me. The tools are easy. Getting an honest number out of them is not. Here’s the field guide I wish more teams had before they trusted a benchmark — their own or a vendor’s.

The number isn’t wrong. It’s answering a different question.

Peak IOPS on a datasheet is a real measurement — of a workload that looks nothing like yours. Tiny block size, unlimited queue depth, a working set small enough to live entirely in cache, reads only, measured for ten seconds. Your production is a 70/30 read-write mix, 16K blocks, a working set far larger than cache, and it has to stay fast at 2 a.m. on day 400. Same tool, different universe. Before you argue about whose array is faster, make sure you’re both describing the same planet.

The traps that inflate a benchmark

Most misleading numbers come from a short list of mistakes:

• The working set fits in cache. You benchmarked DRAM, not the media. Size the dataset several times larger than the controller cache or you’re measuring the wrong component.
• No steady state. Flash gets slower once garbage collection kicks in. A 60-second run flatters an SSD that looks very different an hour later. Precondition the device, then measure.
• Queue-depth theater. Cranking queue depth to 256 maximizes IOPS and obliterates latency. It produces a big number and an unusable response time.
• Wrong block size or mix. The datasheet uses 4K reads; your database does 8K–16K with real writes. Model the actual mix or the result is fiction.
• A single sample. One run is an anecdote. The variance across five runs is the actual story.

An honest fio job to start from

[global]
ioengine=libaio
direct=1
runtime=600
time_based=1
ramp_time=60              # reach steady state before recording
group_reporting=1

[db-like]
rw=randrw
rwmixread=70              # 70/30 read-write, like an OLTP database
bs=16k                    # your real block size, not 4k
iodepth=32               # realistic, not a vanity QD of 256
numjobs=4
size=200g                # larger than the array's cache
percentile_list=99:99.9  # report the tail, not just the average

Every knob here is a decision about honesty. direct=1 bypasses the page cache so you measure storage, not memory. ramp_time throws away the artificially fast warm-up. size forces cache misses. percentile_list is the one most people skip — and it’s the one that matters most.

Read the right metrics

Peak IOPS is the vanity metric. What actually predicts whether production will be happy:

• Latency at your target throughput — not throughput at unlimited latency. Those are different questions with very different answers.
• Tail latency (p99, p99.9). Your users feel the worst 1% of requests, not the average. A great mean with an ugly tail is a bad system wearing a good costume.
• Consistency over time. Does it hold at steady state, or degrade as the device fills and ages?

One reframing kills most bad purchases: fix a latency budget — say, 1 ms at p99 — and ask “how many IOPS can it sustain at or under that?” Suddenly the million-IOPS array and the “slower” one often trade places.

Why this is a leadership discipline, not a lab chore

Benchmarking is how you replace opinion with evidence. In a room full of vendor claims and strong personalities, the person holding a reproducible number wins the decision — and deserves to. That’s also why a benchmark is a conviction with a review date: you state a performance expectation strongly enough to plan around it, and you re-run it when the firmware, the workload, or the scale changes. A claim without a benchmark is just an opinion. A benchmark you can’t reproduce is just a different opinion with a chart attached.

The takeaway

Don’t ask “how fast is it.” Ask “how fast is it, running my workload, at my latency budget, at steady state, averaged over five runs.” The tools will answer honestly if you ask honestly. Everything else is marketing with a monospaced font.

The full fio job file is on GitHub: github.com/waghmaredb/vexpose-labs. If you benchmark enterprise storage or databases for a living, I’d like to compare methodologies — reach me on LinkedIn or X.