Category: Thought Leadership

Your p99 latency is one of the most-quoted numbers in your observability stack, and if you measured it with a standard load test, it is very likely a lie — not off by a little, but understating the truth by a factor of hundreds. The tool didn’t malfunction. It measured exactly what it was designed to measure, which turns out to be the wrong thing the moment your system stalls.

This is coordinated omission, and once you see it you can’t unsee it in a benchmark result again.

The load generator stalls with the server

Most load tests are closed-loop: a fixed pool of workers each send a request, wait for the response, then send the next. It’s simple and it’s how most benchmarking harnesses work by default. It also has a fatal blind spot.

When the server freezes for 200 ms — a GC pause, a lock, a failover, a noisy neighbor — the workers waiting on it freeze too. During that stall they send nothing. So the requests that should have been issued during those 200 ms are never sent, never timed, and never counted. And those are exactly the requests that would have been slow. The test quietly omits its own worst samples and hands you a beautiful percentile for a system that, for a fifth of a second, was serving no one. Your users experienced that stall. Your benchmark didn’t.

The same stall, measured two ways

Here is one 200 ms freeze in an otherwise sub-millisecond service, at 1,000 requests/second. The only difference between the two columns is when the clock starts: the naive test times each request from when it was actually sent; the corrected version times it from when it was scheduled to be sent — the honest question, because a request that couldn’t even be issued was already failing the user.

percentile   naive (ms)   corrected (ms)
      p50          0.2              0.2
      p90          0.2              0.2
      p99          0.2            120.0
    p99.9          0.2            192.0

Same service, same stall, same run. The naive p99 reports 0.2 ms. The honest p99 is 120 ms — a 600× understatement. The average is fine in both; the deception lives entirely in the tail, which is precisely the part you’re quoting p99 to protect.

The correction itself is one line of intent: measure latency from the schedule, not the send.

scheduled    = i * interval           # when this request SHOULD have gone out
actual_start = max(scheduled, clock)  # closed loop: can't send until a worker is free

naive.append(finish - actual_start)   # what a naive test records
corrected.append(finish - scheduled)  # honest: the user was waiting since 'scheduled'

The fix isn’t a formula — it’s a different test

You can correct after the fact, but the better fix is to stop omitting in the first place. Drive load open-loop: issue requests at a constant target rate regardless of whether prior ones have returned, so a stall produces a visible backlog instead of an invisible gap. Practically:

• Use a constant-throughput generator. wrk2, not wrk; fio with a fixed rate, not an unbounded queue depth. They hold the send schedule even when the system underneath them stumbles.
• Record into an HdrHistogram. Full latency resolution across the whole range, plus built-in coordinated-omission correction — you get honest high percentiles instead of a rounded-off tail.
• Report the tail, and report the load you held. “p99.9 at a sustained 1,000 rps” is a claim. “p99” with no throughput attached is a vibe.

Why this is a leadership problem, not a testing footnote

Capacity plans, SLOs, and autoscaling thresholds get set from these numbers. A p99 that’s silently 600× optimistic doesn’t just embarrass you in an incident review — it sizes your cluster, writes your error budget, and sets the alarm that was supposed to wake someone before the customer noticed. You built a safety margin on a measurement that deletes its own worst cases.

This is the whole discipline of the Benchmark Files in one number: a claim without an honest benchmark is just an opinion, and a benchmark that omits its worst samples is an opinion wearing a lab coat. Treat every latency figure as a conviction with a review date — stated strongly enough to plan against, and re-run honestly the moment the load, the runtime, or the topology changes.

The takeaway

Before you quote a p99, ask one question: did the test keep sending while the system was stalling? If it was closed-loop and the answer is no, the number is describing a system that never had a bad moment — which is not the system you run. The average will always survive a benchmark. The tail only survives an honest one.

The coordinated-omission demo — naive vs schedule-corrected percentiles — is runnable on GitHub: github.com/waghmaredb/vexpose-labs. Benchmark latency for a living and fighting the same tail? Compare methods on LinkedIn or X.

On LinkedIn, I make the executive argument: your AI problem is not cost, it is dependency. Cost is what you pay this quarter; dependency is what you pay at every renewal, for a decade. This post is the engineering companion. If dependency is the real exposure, it should be measurable — and anything measurable can be architected down.

So: how do you actually measure your AI exit cost? Not as a slide. As an audit.

Exit cost is a stack, not a number

Teams tend to equate switching providers with changing an API key. That was roughly true in 2023. In an agentic estate, provider coupling has accumulated in five distinct layers, and each has its own unwind cost.

Layer 1 — API surface

The cheapest layer to fix and the first to audit. If application code calls a provider SDK directly, every model reference is a migration line-item. The fix is a gateway: one internal endpoint, provider adapters behind it, model names resolved by policy rather than hard-coded. If you cannot change your default model in one config file, start here — nothing else in this audit is testable until you can.

Layer 2 — Prompt and behavior coupling

Prompts are tuned, consciously or not, to one model’s quirks: its instruction-following style, its formatting habits, its tool-calling dialect. This coupling is invisible until you swap models and watch pass rates drop. The countermeasure is an evaluation suite that defines correct behavior independently of any provider — golden datasets, structured-output checks, task-level assertions. Your eval suite is your portability contract. No evals, no measurable exit cost; only guesses.

Layer 3 — Data gravity

The heaviest layer. Embeddings are tied to the model that produced them: switch embedding models and every vector in the store must be regenerated — a compute bill and a re-indexing window that scales with your corpus. Fine-tunes are worse: weights trained on a proprietary base do not travel at all. Rule of thumb: prefer retrieval over fine-tuning wherever quality allows, and record the full re-embedding cost of your corpus as a standing line in the exit ledger. If you do not know that number today, that is finding #1 of your audit.

Layer 4 — Orchestration and agent coupling

Agent frameworks encode provider assumptions: tool-schema formats, context-window sizes, retry semantics, reasoning-token behavior. A workflow that plans, retrieves, calls tools, and self-corrects has ten to twenty coupling points where a chat app had one. Keep tool definitions in a neutral schema and translate at the gateway; treat any framework feature that only works on one provider as a loan you will repay at migration time.

Layer 5 — Economic architecture

Routing is where dependency becomes money. The arithmetic is one-sided even at public list prices: frontier models cost on the order of ten times more per token than the small and open-weight models that handle classification, extraction, and summarization perfectly well. Send every request to a frontier model and you pay the premium rate for work a cheaper tier would do identically — so a tiered blend lands at a fraction of a frontier-everything bill, with the exact multiple set by your own traffic mix. Measure yours; don’t borrow someone else’s number. But the deeper value of a routing tier is optionality: once classification, extraction, and summarization run on interchangeable smaller models — including open-weight ones you could self-host — the share of your workload that is truly captive shrinks to the frontier-only remainder. Price your renewal leverage as: the percentage of token volume that could move within 30 days without eval regression.

The failover drill

An exit cost you have never tested is a fiction. The drill is simple and quarterly: repoint the gateway’s default policy at your designated fallback (a second provider or an open-weight deployment), run the full eval suite, and record three numbers — engineering hours spent, eval pass-rate delta, and the workload percentage that moved cleanly.

# gateway routing policy — failover drill
default_tier:
  primary:   frontier-a
  fallback:  open-weight-70b   # drill target
routes:
  classify|extract|summarize:  small-tier
  agentic|complex-reasoning:   default_tier
drill:
  cadence: quarterly
  pass_criteria: eval_delta <= 2%
  record: [eng_hours, eval_delta, pct_moved]

Those three numbers, tracked over time, are your exit cost — not an estimate, a measurement. In my experience the first drill is always humbling: the gap between the architecture diagram and the actual coupling is where the real dependency lives.

The point is not to leave

None of this is a plan to abandon your provider. It is the opposite of a plan: it is an option. An option you hold changes a renewal negotiation even if you never exercise it. An option you lack is a price you will accept.

Rent intelligence if you choose to. Measure the exit before it is load-bearing. And never rent the operating model.

Every RAG demo works. You point a model at your docs, ask a question, get a fluent answer with a citation, and the room nods. Then it ships, someone asks about a policy that changed last quarter — and the system confidently cites the old version. The demo never showed you that failure mode, because the failure mode isn’t “the model can’t answer.” It’s “the model answers anyway.”

That gap is where enterprise RAG loses trust — and it’s almost never the model’s fault. It’s retrieval handing over plausible-but-wrong context that a capable model then synthesizes beautifully. The tool is not the transformation: a bigger model makes a confident wrong answer more convincing, not less. The fix lives below the model, in retrieval and grounding. Here’s the architecture I’d stand up for RAG you can actually put in front of employees or customers.

The failure mode: fluent, cited, and wrong

Naive RAG is three lines: embed the query, pull the top-k nearest chunks, stuff them in the prompt. It fails three ways at once. Vector search alone misses the exact terms — part numbers, error codes, policy names — that keyword search would nail. Without reranking, “near in embedding space” isn’t “actually answers the question.” And with no relevance floor, the pipeline always returns something, so the model always has context to rationalize an answer from — even when the right answer is “we don’t have that.”

The result is the worst kind of wrong: confident, well-written, and cited. Everything below exists to make the system able to say “I don’t have that in the sources.”

The reference architecture

  Query
     |
     v
  Hybrid Retrieval      BM25 (keyword) + vector
     |
     v
  Rerank                cross-encoder scores the union
     |
     v
  Relevance floor  --- nothing relevant --->  Refuse ("not in the sources")
     |
  context clears floor
     |
     v
  Grounded generation   answer ONLY from context, cite every claim
     |
     v
  Answer + citations
     |
     v
  Eval + observability  groundedness + citation checks  --->  tune retrieval

1. Ingestion & chunking. How you split documents is a retrieval decision, not a preprocessing chore. Chunk too big and the reranker drowns in noise; too small and you sever the context a claim needs. Preserve structure (headings, tables) and keep the source metadata — you’ll need the doc_id for citations.

2. Hybrid retrieval. Run keyword (BM25) and vector search and take the union. Keyword catches the exact strings vectors miss; vectors catch the meaning keywords miss. Neither alone is enough in the enterprise, where half the questions hinge on a specific identifier.

3. Rerank. Score the union with a cross-encoder that reads the query and each chunk together. This is the highest-ROI line most teams aren’t running — it reorders “nearby” into “actually relevant.”

4. The grounding gate (relevance floor). The seam. If nothing clears a relevance threshold, return nothing. This is where your operating DNA — what the system is allowed to answer, and from what — stops being a hopeful prompt and becomes code. An empty retrieval result is a feature: it’s what lets the generator refuse instead of improvise.

5. Grounded generation. Instruct the model to answer only from the provided context and cite a doc_id for every claim. With no context, it refuses — by construction, not by good behavior.

6. Eval & observability. Score every answer for groundedness and citation validity, continuously. Without this you can’t tell an improvement from a regression — you’re just guessing with a chatbot.

The grounding gate, in code

Hybrid recall, rerank, and a relevance floor — the floor is the part that makes refusal possible:

def retrieve(self, query, k=5):
    # Hybrid recall: keyword catches exact terms, vectors catch meaning.
    candidates = self._dedupe(
        self.bm25.search(query, k=20) + self.vector_index.search(query, k=20)
    )
    # Rerank the union with a cross-encoder — the single biggest quality win.
    for c in candidates:
        c.score = self.reranker.score(query, c.text)
    candidates.sort(key=lambda c: c.score, reverse=True)
    # Relevance floor: if nothing is truly relevant, return nothing.
    # An empty result is a feature — it lets the generator refuse.
    return [c for c in candidates[:k] if c.score >= self.min_score]

And the grounding instruction that turns “no context” into “no answer”:

GROUNDING_INSTRUCTION = (
    "Answer ONLY from the provided context. Cite the doc_id for every claim. "
    "If the context does not contain the answer, say 'I don't have that in the "
    "sources' — do not use prior knowledge."
)

Measuring whether it’s actually grounded

Two numbers predict trust: is every claim supported by a retrieved chunk, and does every cited doc_id actually appear in the context? Score them on a fixed set of questions on every prompt, model, or index change.

def score(case):
    groundedness = case.claims_supported / case.claims_total
    hallucinated = [d for d in case.cited_doc_ids if d not in case.context_doc_ids]
    return {"groundedness": groundedness, "citation_valid": not hallucinated}

def retrieve(self, query, user, k=5):
    candidates = self._dedupe(
        self.bm25.search(query, k=40) + self.vector_index.search(query, k=40)
    )
    # Trim to what THIS user may see — enforced at the data layer, BEFORE rerank,
    # so unauthorized chunks never influence ranking or reach the model at all.
    candidates = [c for c in candidates if can_read(user, c.doc_id)]

    for c in candidates:
        c.score = self.reranker.score(query, c.text)
    candidates.sort(key=lambda c: c.score, reverse=True)
    return [c for c in candidates[:k] if c.score >= self.min_score]

The parts everyone underestimates

Chunking is a retrieval decision. More RAG quality is won or lost here than in model choice. Test chunk sizes against your eval set like any other parameter.

Reranking is the cheap win you’re skipping. A cross-encoder over the top ~40 candidates typically moves groundedness more than a model upgrade — for a fraction of the cost.

The relevance floor is the whole ballgame. “Always answers” is the root of hallucinated RAG. A threshold that returns nothing when nothing is relevant is what converts a plausible liar into a trustworthy assistant.

Retrieval decays silently. A doc gets superseded, and the old chunk still ranks — confidently. Freshness and provenance on your index matter more than the model you pick.

The takeaway

RAG trustworthiness is a retrieval-and-grounding problem, not a model problem. What the system is allowed to answer — and from what — is operating DNA, and it belongs in code at the grounding gate, not in a prompt you hope the model respects. Get that seam right and you can swap models freely as they improve. Skip it and you’ve built a very persuasive way to cite the wrong document.

The retriever and the groundedness eval from this post are runnable on GitHub: github.com/waghmaredb/vexpose-labs. Building enterprise RAG and fighting the same failure modes? I’d like to compare notes — reach me on LinkedIn or X.

ServiceNow’s REST API uses offset-based pagination. When you paginate through a live table—one where records are being created or deleted—the offset becomes stale between calls. Records silently get dropped.

Why it happens

The offset is positional within the current result set at the exact moment you query. If a new incident is created and sorts before your current page, all subsequent offsets shift down—you skip records.

Example: You’re processing incidents with offset=0, limit=100, then offset=100, limit=100. If a new incident arrives and sorts into position 50, the second page now contains records 151–250 instead of 100–200. Record 100 is never seen.

The fix: use sys_id as a cursor

Instead of offset, anchor pagination to sys_id. Since sys_id is immutable, new records don’t invalidate your position—they sort after your cursor.

# DON'T: offset drifts
offset = 0
while True:
    records = api.query(offset=offset, limit=100)
    process(records)
    offset += 100  # Invalid if records were added

# DO: stable cursor
last_sys_id = ""
while True:
    query = f"sys_id>{last_sys_id}^ORDERBYsys_id" if last_sys_id else "ORDERBYsys_id"
    records = api.query(query, limit=100)
    process(records)
    if records:
        last_sys_id = records[-1]["sys_id"]  # Advance cursor

See the full before-and-after in github.com/waghmaredb/vexpose-labs.

A quick, honest note: vExpose went quiet for a while. I’m back — and the blog is changing in a way I’m genuinely excited about.

Here’s the shift. Most technical writing shows you the strategy but hides the wiring. Going forward, vExpose does both: I state the call a technology leader has to make, then prove it in code, architecture, and benchmarks. One conviction runs through all of it — AI fails in the operating model, not the model. The tool is not the transformation.

Two things to start with:

• A new flagship piece, A Reference Architecture for Agentic ITSM, on safely letting a GenAI agent act in ServiceNow — with the guardrail code that makes it work.
• A new Start Here page that organizes the best of the blog by topic, so you can jump straight to what’s useful.

What’s coming: reference architectures, build logs with real code, and benchmarks — on a steady cadence this time. No noise, just the work. If there’s a topic you want me to dig into, just reply — I read everything.

— Deepak