Coordinated Omission: Why Your p99 Latency Is Lying

Your p99 latency is one of the most-quoted numbers in your observability stack, and if you measured it with a standard load test, it is very likely a lie — not off by a little, but understating the truth by a factor of hundreds. The tool didn’t malfunction. It measured exactly what it was designed to measure, which turns out to be the wrong thing the moment your system stalls.

This is coordinated omission, and once you see it you can’t unsee it in a benchmark result again.

The load generator stalls with the server

Most load tests are closed-loop: a fixed pool of workers each send a request, wait for the response, then send the next. It’s simple and it’s how most benchmarking harnesses work by default. It also has a fatal blind spot.

When the server freezes for 200 ms — a GC pause, a lock, a failover, a noisy neighbor — the workers waiting on it freeze too. During that stall they send nothing. So the requests that should have been issued during those 200 ms are never sent, never timed, and never counted. And those are exactly the requests that would have been slow. The test quietly omits its own worst samples and hands you a beautiful percentile for a system that, for a fifth of a second, was serving no one. Your users experienced that stall. Your benchmark didn’t.

The same stall, measured two ways

Here is one 200 ms freeze in an otherwise sub-millisecond service, at 1,000 requests/second. The only difference between the two columns is when the clock starts: the naive test times each request from when it was actually sent; the corrected version times it from when it was scheduled to be sent — the honest question, because a request that couldn’t even be issued was already failing the user.

percentile   naive (ms)   corrected (ms)
      p50          0.2              0.2
      p90          0.2              0.2
      p99          0.2            120.0
    p99.9          0.2            192.0

Same service, same stall, same run. The naive p99 reports 0.2 ms. The honest p99 is 120 ms — a 600× understatement. The average is fine in both; the deception lives entirely in the tail, which is precisely the part you’re quoting p99 to protect.

The correction itself is one line of intent: measure latency from the schedule, not the send.

scheduled    = i * interval           # when this request SHOULD have gone out
actual_start = max(scheduled, clock)  # closed loop: can't send until a worker is free

naive.append(finish - actual_start)   # what a naive test records
corrected.append(finish - scheduled)  # honest: the user was waiting since 'scheduled'

The fix isn’t a formula — it’s a different test

You can correct after the fact, but the better fix is to stop omitting in the first place. Drive load open-loop: issue requests at a constant target rate regardless of whether prior ones have returned, so a stall produces a visible backlog instead of an invisible gap. Practically:

• Use a constant-throughput generator. wrk2, not wrk; fio with a fixed rate, not an unbounded queue depth. They hold the send schedule even when the system underneath them stumbles.
• Record into an HdrHistogram. Full latency resolution across the whole range, plus built-in coordinated-omission correction — you get honest high percentiles instead of a rounded-off tail.
• Report the tail, and report the load you held. “p99.9 at a sustained 1,000 rps” is a claim. “p99” with no throughput attached is a vibe.

Why this is a leadership problem, not a testing footnote

Capacity plans, SLOs, and autoscaling thresholds get set from these numbers. A p99 that’s silently 600× optimistic doesn’t just embarrass you in an incident review — it sizes your cluster, writes your error budget, and sets the alarm that was supposed to wake someone before the customer noticed. You built a safety margin on a measurement that deletes its own worst cases.

This is the whole discipline of the Benchmark Files in one number: a claim without an honest benchmark is just an opinion, and a benchmark that omits its worst samples is an opinion wearing a lab coat. Treat every latency figure as a conviction with a review date — stated strongly enough to plan against, and re-run honestly the moment the load, the runtime, or the topology changes.

The takeaway

Before you quote a p99, ask one question: did the test keep sending while the system was stalling? If it was closed-loop and the answer is no, the number is describing a system that never had a bad moment — which is not the system you run. The average will always survive a benchmark. The tail only survives an honest one.

The coordinated-omission demo — naive vs schedule-corrected percentiles — is runnable on GitHub: github.com/waghmaredb/vexpose-labs. Benchmark latency for a living and fighting the same tail? Compare methods on LinkedIn or X.