Everyone can quote the price of a token. Almost no one can tell you what a token actually costs them. Those two numbers are rarely within a factor of three of each other, and the distance between them is where AI economics is really decided.
Author’s note: this post pairs the argument with a benchmark from my own environment. The tables below are marked where results go, alongside the exact method used to produce them — so the numbers are reproducible rather than asserted. This draft ships with placeholders; the real figures go in before it publishes.
The price list is not the cost
The comfortable belief is that a published price — so many dollars per million input and output tokens — tells you what inference costs. It tells you the list price of one input. It does not tell you the delivered cost of one answer, because between the token and the answer sits everything that actually governs your bill: how many tokens a single answer really consumes once you count the system prompt, the retrieved context, and the retries; how well you batch; how long your context window runs; and how much of your expensive hardware sits idle between requests.
Cost per token is a unit price. Cost per answer is what you pay. Optimising the first while ignoring the second is optimising the microphone instead of the conversation.
What actually moves the number
Four factors dominate, and none of them appear on a price list. Context length, because a retrieval-augmented answer often carries ten to twenty times the tokens of the user’s actual question. Batching efficiency, because an under-fed GPU bills you for wall-clock time it spent waiting. Retry and guardrail overhead, because every reask, reranking pass, and safety check is tokens you paid for and the user never saw. And idle time, because reserved capacity costs the same whether it is serving peak traffic or nothing at all.
The benchmark
To make this concrete, I measured a single representative workload — [INSERT WORKLOAD: e.g. RAG Q&A over an internal knowledge base, ~500-token questions, top-8 retrieval] — across four serving configurations on [INSERT HARDWARE]. The list price per token is identical across every row. The delivered cost is not.
| Configuration | Tokens / answer | Throughput (tok/s) | GPU-seconds / answer | Delivered cost / 1k answers | Tokens / watt |
|---|---|---|---|---|---|
| Baseline (batch=1, full context) | [INSERT] | [INSERT] | [INSERT] | [INSERT] | [INSERT] |
| + continuous batching | [INSERT] | [INSERT] | [INSERT] | [INSERT] | [INSERT] |
| + context trimming / rerank | [INSERT] | [INSERT] | [INSERT] | [INSERT] | [INSERT] |
| + quantisation | [INSERT] | [INSERT] | [INSERT] | [INSERT] | [INSERT] |
[Author note — fill before publishing. The list price per token is the control variable and stays constant across rows; the whole point of the table is that delivered cost per answer moves by a large multiple while the list price never changes. Do not publish with placeholders. If a figure cannot be measured cleanly, cut that row rather than estimate it.]
How I measured this
The method matters more than any single figure, so here it is in full — to be reproduced rather than trusted. [INSERT: model and serving stack, e.g. vLLM version.] Load was generated at [INSERT] concurrent requests to steady state, discarding a warm-up window of [INSERT]. Tokens per answer counted the system prompt, the retrieved context, and the output — everything billed, not just the visible reply. GPU-seconds per answer came from measured throughput under load, and delivered cost from GPU-seconds times the fully loaded hourly rate of the instance ([INSERT rate and source]). Tokens per watt used board power sampled from nvidia-smi at [INSERT] intervals under load. Every number is an end-to-end measurement under load; none is derived from a spec sheet.
The reframe
The cheapest token on the price list can be the most expensive answer in production, and the reverse is just as true. Until you measure delivered cost per answer in your own environment, you are not managing AI economics — you are quoting someone else’s list price and hoping it resembles your bill. It almost never does.
Leave a Reply