{{ vExpose }}.Blog

Author: Deepak Waghmare

  • The Inference Bill Is a Memory-Bandwidth Problem

    You added GPUs to make your LLM serving faster, and tokens-per-second barely moved. The dashboard says the GPUs are 40% utilized. Someone suggests a bigger batch size; latency gets worse. This is the moment most teams misdiagnose, because the mental model — “inference is a compute problem, so add compute” — is wrong for the part of inference that dominates your bill. Decode is bound by memory bandwidth, not FLOPs.

    Two phases, two different bottlenecks

    Autoregressive generation has two phases, and they live in different worlds. Prefill processes the whole prompt at once — lots of parallel matrix math, genuinely compute-bound, the GPU’s happy place. Decode generates one token at a time, and to produce each token the hardware must read the entire model’s weights out of high-bandwidth memory (HBM) again. One token, all the weights, every step. Decode isn’t doing much math per byte it moves; it’s moving a staggering number of bytes to do a little math.

    So the number that predicts decode speed isn’t TFLOPS. It’s HBM bandwidth — and you can estimate the floor with arithmetic, no benchmark required.

    The back-of-envelope that predicts your latency

    For a memory-bound decode, the time to generate one token is roughly the bytes you must read divided by how fast you can read them:

    time_per_token      ≈ (params × bytes_per_param) / HBM_bandwidth
tokens_per_second   ≈ 1 / time_per_token

    Take a 70B model in FP16 (2 bytes/param) — that’s ~140 GB to read per token. On an 80GB accelerator at ~2.0 TB/s, that’s 140 / 2000 ≈ 70 ms/token: a ceiling near 14 tokens/sec for a single stream, before any compute, kernel, or networking overhead. Not because the GPU can’t do the math — because it has to haul 140 GB across the memory bus for every token.

    Model (FP16) Bytes read / token HBM bandwidth Est. ms / token Est. tok/s (1 stream)
    7B ~14 GB 2.0 TB/s ~7 ms ~140
    13B ~26 GB 2.0 TB/s ~13 ms ~77
    70B ~140 GB 2.0 TB/s ~70 ms ~14
    70B (FP8) ~70 GB 2.0 TB/s ~35 ms ~28
    Single-stream decode ceilings from bandwidth alone. Real systems land below these, but the ranking holds.

    Look at the last two rows. Quantizing the 70B model from FP16 to FP8 halves the bytes read per token and roughly doubles decode throughput — not because you added compute, but because you cut the actual bottleneck in half. That’s the tell that you’re memory-bound: the intervention that helps is the one that moves fewer bytes, not the one that adds more math.

    Why batching helps — until it doesn’t

    If each token read costs 140 GB regardless, the obvious move is to make that read serve many requests at once. That’s exactly what continuous batching does: read the weights once, apply them to a batch of in-flight sequences, amortize the bandwidth across all of them. Throughput climbs beautifully. This is the single highest-leverage lever most teams aren’t fully pulling.

    But batching hits a wall with a name: the KV cache. Every concurrent sequence keeps a per-token cache of keys and values in the same HBM you’re already bandwidth-starved on. Push the batch bigger and the KV cache grows until it evicts, spills, or OOMs — and now you’re memory-capacity bound instead of bandwidth bound. You’ve traded one wall for another. The craft is finding the batch size that maximizes throughput at your latency SLO without tipping into KV-cache thrash.

    What this changes about how you buy and build

    • Spec accelerators by bandwidth and capacity, not headline TFLOPS. For decode-heavy serving, HBM bandwidth and size predict your experience better than peak compute nearly every time.
    • Quantization is a throughput lever, not just a memory-savings trick. Fewer bytes per parameter is fewer bytes read per token. The speedup is the point, not a side effect.
    • KV-cache management is a first-class design concern. Paged attention, cache quantization, and sane max-context limits decide how far batching can take you.
    • Measure at your real batch size and context length. A single-stream benchmark and a saturated multi-tenant server are different machines wearing the same sticker.

    The altitude shift

    The reason this matters beyond the invoice is that it’s a problem altitude question. At low altitude, “inference is slow” gets answered with “buy more GPUs,” and the money goes to compute the workload can’t use. Raise the problem one level — which resource is actually saturated? — and the same symptom points to quantization, batching, and KV-cache strategy instead. The tool is not the transformation: a faster accelerator you’re running memory-blind just reaches the same wall a little sooner.

    Hold all of this as a conviction with a review date. The arithmetic is stable — bytes-per-token doesn’t care about your vendor — but the constants move: bandwidth per dollar, quantization quality, and cache tricks improve every hardware generation. State the model strongly enough to plan a cluster around it, and re-run the numbers when the next accelerator ships.

    The takeaway

    Before you approve another GPU order to fix inference latency, ask which resource is saturated. If decode dominates your workload, the honest answer is usually memory bandwidth — and the fixes that work are the ones that move fewer bytes per token, not the ones that add more math. Compute is what the datasheet sells. Bandwidth is what you actually ship on.

    The token-cost calculator and serving benchmarks are on GitHub: github.com/waghmaredb/vexpose-labs. Running LLMs in production and seeing the same wall? I’d like to compare numbers — LinkedIn or X.

  • Your pgvector Search Gets Slower as You Add Data. Here’s the Setting Everyone Misses.

    Your semantic search was instant at ten thousand rows. At two million it’s 800 milliseconds and climbing, and you never touched the query. Almost always the cause is the same: pgvector is doing an exact, brute-force scan of every vector because its index was never built — or was built with defaults that don’t fit your data.

    Exact search doesn’t scale, and it’s the default

    Without an approximate index, pgvector compares your query vector against every row. That’s fine at ten thousand and fatal at two million. The fix is an ANN index — but an ANN index has two knobs that decide everything, and both have quietly wrong defaults for a large table.

    -- lists ≈ rows / 1000 up to ~1M rows, then ≈ sqrt(rows).
-- 2,000,000 rows -> ~1,414 lists, NOT the tiny number you'll get by guessing.
CREATE INDEX ON docs
  USING ivfflat (embedding vector_cosine_ops)
  WITH (lists = 1414);

-- probes trades recall for speed AT QUERY TIME. The default of 1 is far too low.
-- A sane starting point is ~sqrt(lists); here sqrt(1414) ≈ 38. Then tune to a recall target.
SET ivfflat.probes = 38;

    Two failure modes, opposite symptoms. Too few lists and each partition is huge, so every probe scans a lot — slow. Too few probes and you scan too few partitions — fast, but you silently miss relevant results, which in a RAG system means confidently answering from the wrong chunks. You cannot tune one without measuring the other.

    Build the index after the data is loaded

    IVFFlat clusters your existing vectors to define its lists. Build it on an empty or tiny table and the clusters are meaningless; every later insert lands in an ill-fitting partition and recall degrades. Load first, then index — and rebuild after any large ingest.

    -- HNSW: slower to build, larger on disk, but better recall/latency
-- and no clusters to go stale as data changes.
CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);
SET hnsw.ef_search = 40;   -- the recall/speed dial at query time

    If your data grows or churns continuously, HNSW usually ages better than IVFFlat because it has no clusters to go stale. It costs more to build and store; that’s the trade you’re making.

    The lesson

    Measure recall, not just latency. A vector search that got ten times faster by missing a third of the right answers isn’t faster — it’s broken with a good p99. Pick a fixed set of queries with known-good results, and every time you touch lists, probes, or the index type, confirm recall held before you celebrate the speed.

    The benchmark harness and recall test are on GitHub: github.com/waghmaredb/vexpose-labs. Tuning vector search at scale? Compare notes on LinkedIn or X.

  • Coordinated Omission: Why Your p99 Latency Is Lying

    Your p99 latency is one of the most-quoted numbers in your observability stack, and if you measured it with a standard load test, it is very likely a lie — not off by a little, but understating the truth by a factor of hundreds. The tool didn’t malfunction. It measured exactly what it was designed to measure, which turns out to be the wrong thing the moment your system stalls.

    This is coordinated omission, and once you see it you can’t unsee it in a benchmark result again.

    The load generator stalls with the server

    Most load tests are closed-loop: a fixed pool of workers each send a request, wait for the response, then send the next. It’s simple and it’s how most benchmarking harnesses work by default. It also has a fatal blind spot.

    When the server freezes for 200 ms — a GC pause, a lock, a failover, a noisy neighbor — the workers waiting on it freeze too. During that stall they send nothing. So the requests that should have been issued during those 200 ms are never sent, never timed, and never counted. And those are exactly the requests that would have been slow. The test quietly omits its own worst samples and hands you a beautiful percentile for a system that, for a fifth of a second, was serving no one. Your users experienced that stall. Your benchmark didn’t.

    The same stall, measured two ways

    Here is one 200 ms freeze in an otherwise sub-millisecond service, at 1,000 requests/second. The only difference between the two columns is when the clock starts: the naive test times each request from when it was actually sent; the corrected version times it from when it was scheduled to be sent — the honest question, because a request that couldn’t even be issued was already failing the user.

    percentile   naive (ms)   corrected (ms)
      p50          0.2              0.2
      p90          0.2              0.2
      p99          0.2            120.0
    p99.9          0.2            192.0
    Bar chart comparing naive vs schedule-corrected latency percentiles: the naive load test reports p99 of 0.2 ms, while the corrected p99 is 120 ms and p99.9 is 192 ms — the tail latency coordinated omission hides.

    Same service, same stall, same run. The naive p99 reports 0.2 ms. The honest p99 is 120 ms — a 600× understatement. The average is fine in both; the deception lives entirely in the tail, which is precisely the part you’re quoting p99 to protect.

    The correction itself is one line of intent: measure latency from the schedule, not the send.

    scheduled    = i * interval           # when this request SHOULD have gone out
actual_start = max(scheduled, clock)  # closed loop: can't send until a worker is free

naive.append(finish - actual_start)   # what a naive test records
corrected.append(finish - scheduled)  # honest: the user was waiting since 'scheduled'

    The fix isn’t a formula — it’s a different test

    You can correct after the fact, but the better fix is to stop omitting in the first place. Drive load open-loop: issue requests at a constant target rate regardless of whether prior ones have returned, so a stall produces a visible backlog instead of an invisible gap. Practically:

    • Use a constant-throughput generator. wrk2, not wrk; fio with a fixed rate, not an unbounded queue depth. They hold the send schedule even when the system underneath them stumbles.
    • Record into an HdrHistogram. Full latency resolution across the whole range, plus built-in coordinated-omission correction — you get honest high percentiles instead of a rounded-off tail.
    • Report the tail, and report the load you held. “p99.9 at a sustained 1,000 rps” is a claim. “p99” with no throughput attached is a vibe.

    Why this is a leadership problem, not a testing footnote

    Capacity plans, SLOs, and autoscaling thresholds get set from these numbers. A p99 that’s silently 600× optimistic doesn’t just embarrass you in an incident review — it sizes your cluster, writes your error budget, and sets the alarm that was supposed to wake someone before the customer noticed. You built a safety margin on a measurement that deletes its own worst cases.

    This is the whole discipline of the Benchmark Files in one number: a claim without an honest benchmark is just an opinion, and a benchmark that omits its worst samples is an opinion wearing a lab coat. Treat every latency figure as a conviction with a review date — stated strongly enough to plan against, and re-run honestly the moment the load, the runtime, or the topology changes.

    The takeaway

    Before you quote a p99, ask one question: did the test keep sending while the system was stalling? If it was closed-loop and the answer is no, the number is describing a system that never had a bad moment — which is not the system you run. The average will always survive a benchmark. The tail only survives an honest one.

    The coordinated-omission demo — naive vs schedule-corrected percentiles — is runnable on GitHub: github.com/waghmaredb/vexpose-labs. Benchmark latency for a living and fighting the same tail? Compare methods on LinkedIn or X.

  • The Readiness Probe That Turned a Deploy Into an Outage

    A routine deploy. The new pods come up, and for ninety seconds the entire service returns 503 — even though the previous version was perfectly healthy the whole time. The culprit wasn’t the new code. It was a readiness probe pointed at the wrong thing.

    Readiness and liveness are opposites, not synonyms

    A readiness probe decides whether a pod receives traffic. A liveness probe decides whether a pod gets killed and restarted. Teams wire them identically and treat them as interchangeable, but they want opposite temperaments: liveness should be lazy and forgiving; readiness should be quick and honest. Confuse them and a slow-starting app either gets killed mid-boot because liveness is too aggressive, or gets declared “ready” before it can actually serve.

    The rollout death spiral

    Here’s how a bad readiness probe takes down a healthy service. The new ReplicaSet’s pods fail readiness. The rolling update, following maxUnavailable, has already begun retiring old pods — but the new ones never become ready to replace them. Endpoints drain faster than they fill. For a window, almost nothing is behind the Service, and clients get 503s from the exact mechanism that’s supposed to prevent this.

    # The trap: readiness hits a downstream dependency, with no warm-up
readinessProbe:
  httpGet:
    path: /healthz        # this handler also pings the database
    port: 8080
  initialDelaySeconds: 0  # checked before the app can even listen
  periodSeconds: 5
  failureThreshold: 1     # one blip = pulled from rotation

    Two mistakes compound. The health endpoint checks a downstream dependency, so a brief database hiccup marks every replica unready at once — a self-inflicted full outage. And failureThreshold: 1 with no startup grace means the smallest transient becomes an eviction from the load balancer.

    The fix: a shallow check plus a startupProbe

    # Give slow starters room without loosening liveness
startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 30
  periodSeconds: 5          # up to 150s to boot, then normal probes take over
readinessProbe:
  httpGet:
    path: /ready            # answers ONLY "can this process serve?" — no deep deps
    port: 8080
  periodSeconds: 5
  failureThreshold: 3

    Split the endpoints. /ready answers “is this process able to serve a request,” nothing more — don’t fold the database into it, or one slow dependency yanks every pod out of rotation simultaneously. Use a startupProbe to grant slow-booting apps their warm-up time instead of weakening liveness. And set maxUnavailable: 0 on critical rollouts so Kubernetes won’t retire an old pod until a new one is genuinely ready.

    The lesson

    Readiness is a promise to your load balancer, not a general health dashboard. Make it answer the narrowest possible question — “send me traffic now?” — and nothing else. Every dependency you fold into that check is another way for one small failure to become a total one.

    Probe configs and a reproducible rollout demo are on GitHub: github.com/waghmaredb/vexpose-labs. Got burned by a probe in a way I didn’t cover? Tell me on LinkedIn or X.

  • Measuring Your AI Exit Cost: An Architecture Review

    On LinkedIn, I make the executive argument: your AI problem is not cost, it is dependency. Cost is what you pay this quarter; dependency is what you pay at every renewal, for a decade. This post is the engineering companion. If dependency is the real exposure, it should be measurable — and anything measurable can be architected down.

    So: how do you actually measure your AI exit cost? Not as a slide. As an audit.

    Exit cost is a stack, not a number

    Teams tend to equate switching providers with changing an API key. That was roughly true in 2023. In an agentic estate, provider coupling has accumulated in five distinct layers, and each has its own unwind cost.

    Layer 1 — API surface

    The cheapest layer to fix and the first to audit. If application code calls a provider SDK directly, every model reference is a migration line-item. The fix is a gateway: one internal endpoint, provider adapters behind it, model names resolved by policy rather than hard-coded. If you cannot change your default model in one config file, start here — nothing else in this audit is testable until you can.

    Layer 2 — Prompt and behavior coupling

    Prompts are tuned, consciously or not, to one model’s quirks: its instruction-following style, its formatting habits, its tool-calling dialect. This coupling is invisible until you swap models and watch pass rates drop. The countermeasure is an evaluation suite that defines correct behavior independently of any provider — golden datasets, structured-output checks, task-level assertions. Your eval suite is your portability contract. No evals, no measurable exit cost; only guesses.

    Layer 3 — Data gravity

    The heaviest layer. Embeddings are tied to the model that produced them: switch embedding models and every vector in the store must be regenerated — a compute bill and a re-indexing window that scales with your corpus. Fine-tunes are worse: weights trained on a proprietary base do not travel at all. Rule of thumb: prefer retrieval over fine-tuning wherever quality allows, and record the full re-embedding cost of your corpus as a standing line in the exit ledger. If you do not know that number today, that is finding #1 of your audit.

    Layer 4 — Orchestration and agent coupling

    Agent frameworks encode provider assumptions: tool-schema formats, context-window sizes, retry semantics, reasoning-token behavior. A workflow that plans, retrieves, calls tools, and self-corrects has ten to twenty coupling points where a chat app had one. Keep tool definitions in a neutral schema and translate at the gateway; treat any framework feature that only works on one provider as a loan you will repay at migration time.

    Layer 5 — Economic architecture

    Routing is where dependency becomes money. The arithmetic is one-sided even at public list prices: frontier models cost on the order of ten times more per token than the small and open-weight models that handle classification, extraction, and summarization perfectly well. Send every request to a frontier model and you pay the premium rate for work a cheaper tier would do identically — so a tiered blend lands at a fraction of a frontier-everything bill, with the exact multiple set by your own traffic mix. Measure yours; don’t borrow someone else’s number. But the deeper value of a routing tier is optionality: once classification, extraction, and summarization run on interchangeable smaller models — including open-weight ones you could self-host — the share of your workload that is truly captive shrinks to the frontier-only remainder. Price your renewal leverage as: the percentage of token volume that could move within 30 days without eval regression.

    The failover drill

    An exit cost you have never tested is a fiction. The drill is simple and quarterly: repoint the gateway’s default policy at your designated fallback (a second provider or an open-weight deployment), run the full eval suite, and record three numbers — engineering hours spent, eval pass-rate delta, and the workload percentage that moved cleanly.

    # gateway routing policy — failover drill
default_tier:
  primary:   frontier-a
  fallback:  open-weight-70b   # drill target
routes:
  classify|extract|summarize:  small-tier
  agentic|complex-reasoning:   default_tier
drill:
  cadence: quarterly
  pass_criteria: eval_delta <= 2%
  record: [eng_hours, eval_delta, pct_moved]

    Those three numbers, tracked over time, are your exit cost — not an estimate, a measurement. In my experience the first drill is always humbling: the gap between the architecture diagram and the actual coupling is where the real dependency lives.

    The point is not to leave

    None of this is a plan to abandon your provider. It is the opposite of a plan: it is an option. An option you hold changes a renewal negotiation even if you never exercise it. An option you lack is a price you will accept.

    Rent intelligence if you choose to. Measure the exit before it is load-bearing. And never rent the operating model.

  • The Terraform Gotcha That Destroys the Resource You Meant to Keep

    You add one server to the middle of a list, run terraform plan, and Terraform announces it will destroy and recreate three resources you never touched. Nothing about them changed. This is the count trap, and it has caused more 2 a.m. incidents than almost any other Terraform footgun I know.

    Why a list index is a landmine

    When you build resources with count, Terraform addresses them by position: aws_instance.node[0], [1], [2]. That index is the resource’s identity in state. Remove or insert an element anywhere but the end and every index after it shifts — so [1] now points at what used to be [2]. Terraform doesn’t see “the list got shorter.” It sees “the thing at index 1 is a different thing now,” and the only way it knows to reconcile that is destroy-and-recreate.

    # The trap: identity is the position in the list
variable "nodes" {
  default = ["web-a", "web-b", "web-c"]
}

resource "aws_instance" "node" {
  count = length(var.nodes)
  tags  = { Name = var.nodes[count.index] }
}
# Remove "web-a" and web-b and web-c each shift down one index —
# Terraform destroys and recreates BOTH to "fix" the mismatch.

    The fix: address resources by a stable key, not a position

    for_each keys each resource by a string you control instead of an ordinal Terraform controls. Delete one entry and the others keep their identity, because their identity was never their position.

    # The fix: identity is a stable key
variable "nodes" {
  default = ["web-a", "web-b", "web-c"]
}

resource "aws_instance" "node" {
  for_each = toset(var.nodes)
  tags     = { Name = each.key }
}
# The address is now aws_instance.node["web-b"].
# Remove "web-a" and ONLY "web-a" is destroyed. The rest don't move.

    If you’re already on count

    Migrating isn’t rewrite-and-pray. Move the state entries to their new keyed addresses so Terraform keeps the existing resources instead of replacing them:

    terraform state mv 'aws_instance.node[1]' 'aws_instance.node["web-b"]'
terraform state mv 'aws_instance.node[2]' 'aws_instance.node["web-c"]'
# Then swap count for for_each in the config and plan — it should show no changes.

    Or, on Terraform 1.1+, declare the intent as code with a moved block and let the plan do the reshuffle for you. Either way, confirm the plan shows zero replacements before you apply. That plan output is the entire safety mechanism — read it.

    The lesson

    Use count only for genuinely anonymous, interchangeable replicas you’ll only ever scale from the end — three identical workers, nothing addressed individually. The moment a resource has a name, an identity, or a lifecycle of its own, key it with for_each. In infrastructure as code, how you address a thing is part of what the thing is. Get the identity wrong and Terraform will faithfully destroy production to make the map match the territory.

    More field notes and the full examples are on GitHub: github.com/waghmaredb/vexpose-labs. Hit a stranger Terraform footgun than this one? Trade war stories on LinkedIn or X.

  • A Reference Architecture for Enterprise RAG (That Won’t Hallucinate Your Docs)

    Every RAG demo works. You point a model at your docs, ask a question, get a fluent answer with a citation, and the room nods. Then it ships, someone asks about a policy that changed last quarter — and the system confidently cites the old version. The demo never showed you that failure mode, because the failure mode isn’t “the model can’t answer.” It’s “the model answers anyway.”

    That gap is where enterprise RAG loses trust — and it’s almost never the model’s fault. It’s retrieval handing over plausible-but-wrong context that a capable model then synthesizes beautifully. The tool is not the transformation: a bigger model makes a confident wrong answer more convincing, not less. The fix lives below the model, in retrieval and grounding. Here’s the architecture I’d stand up for RAG you can actually put in front of employees or customers.

    The failure mode: fluent, cited, and wrong

    Naive RAG is three lines: embed the query, pull the top-k nearest chunks, stuff them in the prompt. It fails three ways at once. Vector search alone misses the exact terms — part numbers, error codes, policy names — that keyword search would nail. Without reranking, “near in embedding space” isn’t “actually answers the question.” And with no relevance floor, the pipeline always returns something, so the model always has context to rationalize an answer from — even when the right answer is “we don’t have that.”

    The result is the worst kind of wrong: confident, well-written, and cited. Everything below exists to make the system able to say “I don’t have that in the sources.”

    The reference architecture

      Query
     |
     v
  Hybrid Retrieval      BM25 (keyword) + vector
     |
     v
  Rerank                cross-encoder scores the union
     |
     v
  Relevance floor  --- nothing relevant --->  Refuse ("not in the sources")
     |
  context clears floor
     |
     v
  Grounded generation   answer ONLY from context, cite every claim
     |
     v
  Answer + citations
     |
     v
  Eval + observability  groundedness + citation checks  --->  tune retrieval

    1. Ingestion & chunking. How you split documents is a retrieval decision, not a preprocessing chore. Chunk too big and the reranker drowns in noise; too small and you sever the context a claim needs. Preserve structure (headings, tables) and keep the source metadata — you’ll need the doc_id for citations.

    2. Hybrid retrieval. Run keyword (BM25) and vector search and take the union. Keyword catches the exact strings vectors miss; vectors catch the meaning keywords miss. Neither alone is enough in the enterprise, where half the questions hinge on a specific identifier.

    3. Rerank. Score the union with a cross-encoder that reads the query and each chunk together. This is the highest-ROI line most teams aren’t running — it reorders “nearby” into “actually relevant.”

    4. The grounding gate (relevance floor). The seam. If nothing clears a relevance threshold, return nothing. This is where your operating DNA — what the system is allowed to answer, and from what — stops being a hopeful prompt and becomes code. An empty retrieval result is a feature: it’s what lets the generator refuse instead of improvise.

    5. Grounded generation. Instruct the model to answer only from the provided context and cite a doc_id for every claim. With no context, it refuses — by construction, not by good behavior.

    6. Eval & observability. Score every answer for groundedness and citation validity, continuously. Without this you can’t tell an improvement from a regression — you’re just guessing with a chatbot.

    The grounding gate, in code

    Hybrid recall, rerank, and a relevance floor — the floor is the part that makes refusal possible:

    def retrieve(self, query, k=5):
    # Hybrid recall: keyword catches exact terms, vectors catch meaning.
    candidates = self._dedupe(
        self.bm25.search(query, k=20) + self.vector_index.search(query, k=20)
    )
    # Rerank the union with a cross-encoder — the single biggest quality win.
    for c in candidates:
        c.score = self.reranker.score(query, c.text)
    candidates.sort(key=lambda c: c.score, reverse=True)
    # Relevance floor: if nothing is truly relevant, return nothing.
    # An empty result is a feature — it lets the generator refuse.
    return [c for c in candidates[:k] if c.score >= self.min_score]

    And the grounding instruction that turns “no context” into “no answer”:

    GROUNDING_INSTRUCTION = (
    "Answer ONLY from the provided context. Cite the doc_id for every claim. "
    "If the context does not contain the answer, say 'I don't have that in the "
    "sources' — do not use prior knowledge."
)

    Measuring whether it’s actually grounded

    Two numbers predict trust: is every claim supported by a retrieved chunk, and does every cited doc_id actually appear in the context? Score them on a fixed set of questions on every prompt, model, or index change.

    def score(case):
    groundedness = case.claims_supported / case.claims_total
    hallucinated = [d for d in case.cited_doc_ids if d not in case.context_doc_ids]
    return {"groundedness": groundedness, "citation_valid": not hallucinated}

    The guardrail everyone forgets: retrieval-time access control

    Every architecture above assumes one reader is allowed to see every document. In a real enterprise they are not. HR files, board decks, a customer’s data walled off from another customer’s — the same index holds chunks that different users have different rights to. Miss this and you haven’t built a helpful assistant; you’ve built the most efficient data-exfiltration path in the company. The fastest way to leak the salary spreadsheet is to embed it and let anyone ask.

    The critical design decision: access control lives in retrieval, not in the prompt. Filter the candidate set to what this caller is entitled to see before anything is reranked or shown to the model — sourced from the same entitlements as the system of record (row-level security, document ACLs), not re-invented in the pipeline.

    def retrieve(self, query, user, k=5):
    candidates = self._dedupe(
        self.bm25.search(query, k=40) + self.vector_index.search(query, k=40)
    )
    # Trim to what THIS user may see — enforced at the data layer, BEFORE rerank,
    # so unauthorized chunks never influence ranking or reach the model at all.
    candidates = [c for c in candidates if can_read(user, c.doc_id)]

    for c in candidates:
        c.score = self.reranker.score(query, c.text)
    candidates.sort(key=lambda c: c.score, reverse=True)
    return [c for c in candidates[:k] if c.score >= self.min_score]

    A system prompt that says “only answer if the user is authorized” is not a security control — it’s a polite request to a probabilistic system, and it will eventually say yes. Real access control is a hard filter on the retrieval set, applied before rerank so an unauthorized chunk can’t even tilt the ranking. This is the RAG expression of least privilege — the same operating DNA that governs a good agent: the model may ground only on what this specific user could already open for themselves. That’s a decision you enforce in code, not one you hope the model respects.

    The parts everyone underestimates

    Chunking is a retrieval decision. More RAG quality is won or lost here than in model choice. Test chunk sizes against your eval set like any other parameter.

    Reranking is the cheap win you’re skipping. A cross-encoder over the top ~40 candidates typically moves groundedness more than a model upgrade — for a fraction of the cost.

    The relevance floor is the whole ballgame. “Always answers” is the root of hallucinated RAG. A threshold that returns nothing when nothing is relevant is what converts a plausible liar into a trustworthy assistant.

    Retrieval decays silently. A doc gets superseded, and the old chunk still ranks — confidently. Freshness and provenance on your index matter more than the model you pick.

    The takeaway

    RAG trustworthiness is a retrieval-and-grounding problem, not a model problem. What the system is allowed to answer — and from what — is operating DNA, and it belongs in code at the grounding gate, not in a prompt you hope the model respects. Get that seam right and you can swap models freely as they improve. Skip it and you’ve built a very persuasive way to cite the wrong document.

    The retriever and the groundedness eval from this post are runnable on GitHub: github.com/waghmaredb/vexpose-labs. Building enterprise RAG and fighting the same failure modes? I’d like to compare notes — reach me on LinkedIn or X.

  • The ServiceNow REST pagination gotcha that silently drops records

    ServiceNow’s REST API uses offset-based pagination. When you paginate through a live table—one where records are being created or deleted—the offset becomes stale between calls. Records silently get dropped.

    Why it happens

    The offset is positional within the current result set at the exact moment you query. If a new incident is created and sorts before your current page, all subsequent offsets shift down—you skip records.

    Example: You’re processing incidents with offset=0, limit=100, then offset=100, limit=100. If a new incident arrives and sorts into position 50, the second page now contains records 151–250 instead of 100–200. Record 100 is never seen.

    The fix: use sys_id as a cursor

    Instead of offset, anchor pagination to sys_id. Since sys_id is immutable, new records don’t invalidate your position—they sort after your cursor.

    # DON'T: offset drifts
offset = 0
while True:
    records = api.query(offset=offset, limit=100)
    process(records)
    offset += 100  # Invalid if records were added

# DO: stable cursor
last_sys_id = ""
while True:
    query = f"sys_id>{last_sys_id}^ORDERBYsys_id" if last_sys_id else "ORDERBYsys_id"
    records = api.query(query, limit=100)
    process(records)
    if records:
        last_sys_id = records[-1]["sys_id"]  # Advance cursor

    See the full before-and-after in github.com/waghmaredb/vexpose-labs.

  • The Benchmark Trap: Why Your Storage Numbers Lie — and How to Get Honest Ones

    A vendor datasheet promises a million IOPS. You buy the array, point your workload at it, and it’s slow. The number wasn’t a lie — it was just irrelevant to you. That gap, between an impressive benchmark and a disappointing production system, is where a lot of infrastructure money quietly goes to die.

    I’ve spent a large part of my career with fio, vdbench, and HammerDB open in front of me. The tools are easy. Getting an honest number out of them is not. Here’s the field guide I wish more teams had before they trusted a benchmark — their own or a vendor’s.

    The number isn’t wrong. It’s answering a different question.

    Peak IOPS on a datasheet is a real measurement — of a workload that looks nothing like yours. Tiny block size, unlimited queue depth, a working set small enough to live entirely in cache, reads only, measured for ten seconds. Your production is a 70/30 read-write mix, 16K blocks, a working set far larger than cache, and it has to stay fast at 2 a.m. on day 400. Same tool, different universe. Before you argue about whose array is faster, make sure you’re both describing the same planet.

    The traps that inflate a benchmark

    Most misleading numbers come from a short list of mistakes:

    • The working set fits in cache. You benchmarked DRAM, not the media. Size the dataset several times larger than the controller cache or you’re measuring the wrong component.
    • No steady state. Flash gets slower once garbage collection kicks in. A 60-second run flatters an SSD that looks very different an hour later. Precondition the device, then measure.
    • Queue-depth theater. Cranking queue depth to 256 maximizes IOPS and obliterates latency. It produces a big number and an unusable response time.
    • Wrong block size or mix. The datasheet uses 4K reads; your database does 8K–16K with real writes. Model the actual mix or the result is fiction.
    • A single sample. One run is an anecdote. The variance across five runs is the actual story.

    An honest fio job to start from

    [global]
ioengine=libaio
direct=1
runtime=600
time_based=1
ramp_time=60              # reach steady state before recording
group_reporting=1

[db-like]
rw=randrw
rwmixread=70              # 70/30 read-write, like an OLTP database
bs=16k                    # your real block size, not 4k
iodepth=32               # realistic, not a vanity QD of 256
numjobs=4
size=200g                # larger than the array's cache
percentile_list=99:99.9  # report the tail, not just the average

    Every knob here is a decision about honesty. direct=1 bypasses the page cache so you measure storage, not memory. ramp_time throws away the artificially fast warm-up. size forces cache misses. percentile_list is the one most people skip — and it’s the one that matters most.

    Read the right metrics

    Peak IOPS is the vanity metric. What actually predicts whether production will be happy:

    • Latency at your target throughput — not throughput at unlimited latency. Those are different questions with very different answers.
    • Tail latency (p99, p99.9). Your users feel the worst 1% of requests, not the average. A great mean with an ugly tail is a bad system wearing a good costume.
    • Consistency over time. Does it hold at steady state, or degrade as the device fills and ages?

    One reframing kills most bad purchases: fix a latency budget — say, 1 ms at p99 — and ask “how many IOPS can it sustain at or under that?” Suddenly the million-IOPS array and the “slower” one often trade places.

    Why this is a leadership discipline, not a lab chore

    Benchmarking is how you replace opinion with evidence. In a room full of vendor claims and strong personalities, the person holding a reproducible number wins the decision — and deserves to. That’s also why a benchmark is a conviction with a review date: you state a performance expectation strongly enough to plan around it, and you re-run it when the firmware, the workload, or the scale changes. A claim without a benchmark is just an opinion. A benchmark you can’t reproduce is just a different opinion with a chart attached.

    The takeaway

    Don’t ask “how fast is it.” Ask “how fast is it, running my workload, at my latency budget, at steady state, averaged over five runs.” The tools will answer honestly if you ask honestly. Everything else is marketing with a monospaced font.

    The full fio job file is on GitHub: github.com/waghmaredb/vexpose-labs. If you benchmark enterprise storage or databases for a living, I’d like to compare methodologies — reach me on LinkedIn or X.

  • vExpose is back — and it’s changing

    A quick, honest note: vExpose went quiet for a while. I’m back — and the blog is changing in a way I’m genuinely excited about.

    Here’s the shift. Most technical writing shows you the strategy but hides the wiring. Going forward, vExpose does both: I state the call a technology leader has to make, then prove it in code, architecture, and benchmarks. One conviction runs through all of it — AI fails in the operating model, not the model. The tool is not the transformation.

    Two things to start with:

    • A new flagship piece, A Reference Architecture for Agentic ITSM, on safely letting a GenAI agent act in ServiceNow — with the guardrail code that makes it work.
    • A new Start Here page that organizes the best of the blog by topic, so you can jump straight to what’s useful.

    What’s coming: reference architectures, build logs with real code, and benchmarks — on a steady cadence this time. No noise, just the work. If there’s a topic you want me to dig into, just reply — I read everything.

    — Deepak