Field Note: The GPU That Was Busy Doing Nothing

Field Note — one real failure, the wrong diagnosis, the actual cause, the fix.

A GPU can be ninety-five percent utilised and still be doing almost nothing you are paying it to do. This is the most expensive gap between a metric and the truth in modern infrastructure.

The symptom

An inference service was slow. nvidia-smi reported 95% GPU utilisation, so the dashboard was green and the conclusion wrote itself: we are out of GPU, buy more. The throughput told a different story — roughly a third of the tokens per second the same hardware reached in the vendor benchmark.

The wrong diagnosis

“Utilisation is at 95%, so the GPU is the bottleneck — add capacity.” Every part of that sentence sounds right, and the last clause is wrong. More GPUs would have tripled the bill and moved the throughput barely at all.

The actual cause

GPU utilisation, as nvidia-smi reports it, means exactly one thing: at least one kernel was running during the sampling window. It says nothing about whether that kernel was doing useful arithmetic or waiting on memory. Our service was running with a batch size of one, generating tokens for a single request at a time. The GPU spent most of its cycles moving weights in and out of memory rather than computing — memory-bandwidth bound, not compute bound. Busy and idle at the same time.

The fix

Measure the right thing, then feed the hardware properly. The honest metrics are tokens per second under load and model FLOPs utilisation (MFU) — how much of the GPU’s theoretical compute you are actually using. Both were dismal. Enabling continuous batching, so concurrent requests share each forward pass, and right-sizing the batch took throughput most of the way to the benchmark on the very same cards.

# The metric that lies:
nvidia-smi --query-gpu=utilization.gpu --format=csv   # 95% — "busy"

# The metrics that don't:
#   tokens/sec at target concurrency   (throughput you actually ship)
#   MFU = achieved FLOPs / peak FLOPs  (useful work vs. busy work)
#
# High utilization + low MFU = memory-bound.
# Fix batching before you buy silicon.

The reframe

Utilisation tells you a GPU is busy. It will never tell you the GPU is useful. Buy the second card when the first one is doing useful work — not when it is merely busy doing nothing.