{{ vExpose }}.Blog

Author: Deepak Waghmare

  • GPU Idle at Every Save: The NVMe Checkpoint Stall

    There is a cliff in your training run’s GPU-utilization graph. You probably can’t see it, because the dashboard averages it away — but it is there, every few hundred steps, and it is costing you more GPU-hours than any kernel you have spent the last month optimising.

    The cliff is the checkpoint. Every few hundred steps the run serialises model and optimiser state — tens of gigabytes — and writes it to disk. If that write is synchronous, the training step that triggered it blocks until the bytes are durable. The most expensive silicon in the building spends those seconds doing nothing but waiting for an NVMe drive.

    The averaged dashboard is lying to you

    This is the same trap as coordinated omission: a one-minute utilization average cannot show you a four-second stall. You see 94% and call it healthy. Sample at per-step resolution and the sawtooth appears — a flat line at the top, punctuated by vertical drops toward zero exactly on the checkpoint interval. The average is arithmetically true and operationally useless.

    GPU utilization through trainingSynchronous vs. asynchronous checkpointing — same hardware, same step time.ILLUSTRATIVE — shape, not values100%7550250training steps →GPU utilizationeach dip = one synchronous checkpoint write blocking the stepAsync + O_DIRECT — GPU stays fedSynchronous save — GPU starvesThe GPUs aren’t slow — the save path is.Run the harness for numbers on your hardware.
    Illustrative. The curve shows the shape of the stall, not measured values — the companion harness prints the real gap on your hardware.

    A checkpoint is an I/O job wearing a training job’s clothes

    The training loop is a compute job; the checkpoint is an I/O job; and the moment you run them on the same thread, the slower one sets the pace. A synchronous save does three expensive things in the critical path: it serialises state, it pushes tens of gigabytes through the filesystem, and it calls fsync to force durability. Worse, a large buffered write floods the page cache and can evict the very data your loader was about to read — so the stall does not end cleanly when the write does; the next few steps limp while the cache refills.

    Measure the save path, not the spec sheet

    The spec sheet sells you small-random-read IOPS. A checkpoint is none of those things — it is a large, sequential, fsync’d burst, often from every rank at once. So benchmark that, not the marketing number. The companion code has an fio job that models the real pattern (O_DIRECT, 1 MB blocks, fsync per write) and a dependency-free Python harness that times a stand-in step against three save strategies and prints the utilization gap on your hardware.

    Get the write off the critical path

    The fix is not a faster disk. It is refusing to let the disk pace the GPU. Capture the checkpoint to pinned host memory on the training thread — cheap, microseconds — then hand it to a background writer that fsyncs while training continues. Use O_DIRECT for the big sequential dump so it bypasses the page cache instead of evicting your data. The one rule that makes this correct rather than merely fast: snapshot before you enqueue, so the next step cannot mutate weights mid-write.

    # Before: the step waits for the disk
save_checkpoint(state)              # serialise + write + fsync, all inline

# After: the step pays only for an in-memory snapshot
pinned = snapshot_to_pinned(state)  # microseconds, on the training thread
writer.enqueue(pinned)              # fsync happens on a background thread

    The reframe

    A stall at every save is not a disk problem; it is an accounting one. You are paying frontier-GPU rates to watch a progress bar fill. Measure the save path, move it off the critical path, and the cliffs flatten — same hardware, more of it actually working. The GPUs were never the bottleneck. The decision to save synchronously was — and that is an operating-model choice, not a model one.

  • Field Note: The GPU That Was Busy Doing Nothing

    Field Note — one real failure, the wrong diagnosis, the actual cause, the fix.

    A GPU can be ninety-five percent utilised and still be doing almost nothing you are paying it to do. This is the most expensive gap between a metric and the truth in modern infrastructure.

    The symptom

    An inference service was slow. nvidia-smi reported 95% GPU utilisation, so the dashboard was green and the conclusion wrote itself: we are out of GPU, buy more. The throughput told a different story — roughly a third of the tokens per second the same hardware reached in the vendor benchmark.

    The wrong diagnosis

    “Utilisation is at 95%, so the GPU is the bottleneck — add capacity.” Every part of that sentence sounds right, and the last clause is wrong. More GPUs would have tripled the bill and moved the throughput barely at all.

    The actual cause

    GPU utilisation, as nvidia-smi reports it, means exactly one thing: at least one kernel was running during the sampling window. It says nothing about whether that kernel was doing useful arithmetic or waiting on memory. Our service was running with a batch size of one, generating tokens for a single request at a time. The GPU spent most of its cycles moving weights in and out of memory rather than computing — memory-bandwidth bound, not compute bound. Busy and idle at the same time.

    The fix

    Measure the right thing, then feed the hardware properly. The honest metrics are tokens per second under load and model FLOPs utilisation (MFU) — how much of the GPU’s theoretical compute you are actually using. Both were dismal. Enabling continuous batching, so concurrent requests share each forward pass, and right-sizing the batch took throughput most of the way to the benchmark on the very same cards.

    # The metric that lies:
nvidia-smi --query-gpu=utilization.gpu --format=csv   # 95% — "busy"

# The metrics that don't:
#   tokens/sec at target concurrency   (throughput you actually ship)
#   MFU = achieved FLOPs / peak FLOPs  (useful work vs. busy work)
#
# High utilization + low MFU = memory-bound.
# Fix batching before you buy silicon.

    The reframe

    Utilisation tells you a GPU is busy. It will never tell you the GPU is useful. Buy the second card when the first one is doing useful work — not when it is merely busy doing nothing.

  • AI Platforms Are Not Platforms for AI

    Every enterprise I speak with across Asia Pacific has, by now, bought an AI platform. Very few of them have a platform for AI. The words are nearly identical. The gap between them is where most AI budgets quietly go to die.

    It is a comfortable belief that these are the same purchase. A vendor sells you a model catalog, a notebook environment, a vector store, and a chat interface with your logo on it. The invoice says “AI platform.” The assumption is that the platform question is now answered. It isn’t. You bought a product. A platform for AI is a capability — and nobody sells you that.

    Two different things wearing the same name

    An AI platform is a product you procure. It is optimised to get a model in front of a user quickly — a demo, a pilot, a proof of concept. Its job is to make the first mile easy, and it is very good at that.

    A platform for AI is the substrate underneath. It is what makes any model safe to run in production — for years, against real data, under real regulation, with a real bill attached. It is identity and access control that understands prompts and outputs, not just endpoints. It is data governance that knows which corpus a model may retrieve from and which it may not. It is observability that captures not just latency but hallucination rate, cost per answer, and drift in model behaviour. It is a rollback story for the day a model update quietly regresses.

    AI platform (what you buy)Platform for AI (what you operate)
    Optimised for the first demoOptimised for the thousandth production request
    Model catalog and notebooksIdentity, data governance, cost attribution
    Latency dashboardsHallucination rate, cost per answer, behavioural drift
    “It works”“It still works, safely, and we can prove why”

    Different verticals, same wall

    The pattern is unusually consistent. Manufacturing, banking, healthcare, retail — each is convinced its AI challenge is unique to its domain. Each hits the same wall in the same place: the moment a pilot has to become a system that other systems depend on. The model was never the hard part. Integration discipline was. The wall is not made of algorithms. It is made of identity, data lineage, and the absence of anyone who owns the model in production the way someone owns a database in production.

    This is worth saying plainly, because it is easy to hear it as an argument against buying AI platforms. It is not. Buy them — they earn their keep in the first mile. The mistake is believing the purchase discharged the platform obligation. It only relocated it.

    The tool is not the transformation

    A model is a tenant. It moves in, it does its work, and one day it moves out and a better one moves in. A platform for AI is the building — the wiring, the plumbing, the fire code, the lease that says who is responsible when something floods. You can change tenants every quarter. You cannot re-pour the foundation every quarter. Most enterprises are trying to run a building they never built, and wondering why every new tenant needs a month of bespoke work before it can turn the lights on.

    The tell is switching cost. In an organisation with a real platform for AI, swapping one model for another is a configuration change — new endpoint, re-run the evaluation suite, watch the dashboards. In an organisation with only an AI platform, the same swap is a project, because governance, monitoring, and cost control were built into the model integration rather than into the substrate beneath it. Every model becomes a fresh act of heroism.

    What to build first

    The substrate does not have to arrive all at once, but it has to arrive in the right order. Identity and data governance first, because they are the things you cannot retrofit safely once real data is flowing. Cost attribution next, because an AI system whose cost you cannot decompose per team and per use case will be switched off in the first budget review regardless of its accuracy. Observability of behaviour — not just uptime — after that. The model catalog, the part the market sells as “the platform,” comes last, because it is the easiest to replace and the least dangerous to get wrong.

    A Platform for AI: the substrate you operateBuilt from the bottom up — the lower layers can’t be retrofitted safely once real data is flowing.BUILDORDERMODEL = tenantswap = a config change, not a project4Model catalog · notebooks · endpoints · chat UIWhat the market sells as “the platform.” Built last — easiest to replace, least dangerous to get wrong.3Behavioural observabilityHallucination rate, cost per answer, drift in behaviour — not just latency and uptime.2Cost attribution · per team, per use caseCost you can’t decompose gets switched off in the first budget review — regardless of accuracy.1FOUNDATION — BUILD FIRST, CANNOT BE RETROFITTED SAFELYIdentity & access· prompt- and output-aware, not just endpointsData governance· corpus lineage: which sources a model may retrieve, and which it may notAI fails in the operating model, not the model.The model is the tenant; the platform is the building. You change tenants every quarter; you don’t re-pour the foundation.vEXPOSE
    A platform for AI, built in order: the lower layers can’t be retrofitted once real data is flowing.

    The reframe

    You do not have an AI strategy until you have a platform for AI. The model is the tenant; the platform is the landlord; and no amount of licensing the former relieves you of building the latter. The enterprises that compound their AI advantage over the next five years will not be the ones with the best model this quarter. They will be the ones who built the building — so that every good model, this quarter and every quarter after, has somewhere safe to live.

  • Field Note: The Drift That Only Shows Up at 2 a.m.

    Field Note — a recurring series: one real failure, the wrong diagnosis, the actual cause, the fix.

    Infrastructure as Code was supposed to end configuration drift. It didn’t. It moved drift somewhere harder to see — into the gap between your state file and the world that file claims to describe.

    The symptom

    The pull request was clean. terraform plan showed no changes, two reviewers approved, it merged. Nine days later an unrelated change to a load balancer produced a plan that wanted to destroy a subnet nobody had touched — at 2 a.m., because that is when the on-call apply ran.

    The wrong diagnosis

    “Someone changed it by hand.” That is the reflex, and occasionally it is right. But blaming a human is the comfortable answer, because it means the tooling is fine and the process just needs more discipline. It usually isn’t the human.

    The actual cause

    State is a photograph, not a live feed. It records what the world looked like the last time you ran apply. Between applies, the world moves: a security team edits an ingress rule, an autoscaler rewrites a tag, a cloud provider ships a new default for a resource you declared three years ago. Your code did not drift. Reality drifted — and your state never looked again, until an unrelated apply forced a refresh and reconciled the difference destructively.

    The fix

    Stop discovering drift as a side effect of unrelated work. Detect it on a schedule. A nightly job that runs a read-only plan and treats a non-empty diff as a signal — not an error to auto-correct — turns drift from a 2 a.m. surprise into a morning ticket.

    # nightly-drift.sh — read-only, alert on drift, never auto-apply
terraform init -input=false
terraform plan -detailed-exitcode -lock=false -input=false
code=$?

# 0 = no drift | 2 = drift detected | 1 = error
if [ "$code" -eq 2 ]; then
  notify "Drift detected in $(terraform workspace show) — review before next apply"
fi
exit $code

    Two disciplines make the detection trustworthy. Lock your state so a scheduled plan never races a real apply. And scope who and what can mutate infrastructure out of band — the fewer hands outside the pipeline, the quieter the diff.

    The reframe

    Drift is not a failure of Infrastructure as Code. It is Infrastructure as Code telling you the truth about who else has their hands on your infrastructure. The only real choice is when you hear it: on a schedule you control, or at 2 a.m. in the middle of something else.

  • Operating DNA, Not Templates: What Infrastructure as Code Actually Encodes

    Most teams justify infrastructure as code with speed: spin up an environment in minutes, tear it down just as fast. That’s real, and it’s the least interesting thing IaC does. Speed is what the tool sells. What IaC actually encodes — when it’s done well — is your organization’s operating DNA: the accumulated judgment about how you run infrastructure, made explicit, versioned, reviewable, and enforced. Miss that and you’ve bought a faster way to reproduce your bad decisions.

    The tool is not the transformation

    Adopting Terraform no more makes you good at infrastructure than buying a word processor makes you a writer. I’ve watched organizations “do IaC” and end up with a pile of copy-pasted HCL that encodes nothing but one engineer’s Tuesday afternoon — no standards, no guardrails, every module a fresh set of decisions re-litigated from scratch. They automated the typing. They didn’t capture the thinking. The tool is not the transformation. The transformation is when the judgment that used to live in senior engineers’ heads and tribal wiki pages becomes code the pipeline can read and refuse to violate.

    What “operating DNA as code” actually looks like

    Operating DNA is the set of answers your organization has converged on: which regions we deploy to, what a compliant storage bucket looks like, who can provision GPUs, what “tagged for cost allocation” means, how big is too big. In most companies those answers live in people, slides, and the memory of the last incident. IaC gives them a home where they can be enforced instead of merely hoped for. Three layers turn IaC from “scripts that build things” into “judgment that governs things.”

    1. Modules encode the paved road. A good module isn’t a thin wrapper over a resource; it’s an opinion. It bakes in the encryption, the logging, the tagging, and the sane defaults so that the easy way to provision is also the compliant way. The standard becomes the path of least resistance.

    # A module that encodes operating DNA: the compliant config is the DEFAULT.
module "bucket" {
  source = "org/secure-bucket/aws"
  name   = "billing-exports"
  # No "encryption = true" knob here - the module makes it non-optional.
  # Logging, versioning, public-access-block, cost-center tag: enforced inside.
}

    2. Policy-as-code makes the guardrails non-negotiable. Modules make the right thing easy; policy makes the wrong thing impossible to merge. This is where operating DNA stops being a convention and becomes a rule the pipeline enforces on every plan — including the ones written by someone who’s never read the wiki.

    # OPA/Conftest: an un-encrypted bucket never gets past CI. Nothing to forget.
package terraform.storage

deny[msg] {
  r := input.resource_changes[_]
  r.type == "aws_s3_bucket"
  not encrypted(r)
  msg := sprintf("bucket %v must be encrypted - org policy STO-014", [r.address])
}

    3. The pull request becomes the system of record. When infrastructure is code, every change to how you run is a diff: proposed, reviewed, explained in a commit message, permanently attributable. The why lives next to the what. New engineers onboard by reading the modules and policies instead of by breaking things and being told. That’s operating DNA made teachable.

    The part that isn’t code

    Here’s the honest caveat: writing the guardrails is the visible work, and it’s the smaller half. The invisible work is the organizational agreement underneath — actually deciding, as a group, what the standard is before you can encode it. A policy that says “buckets must be encrypted” is trivial to write and worthless if three teams quietly disagree about the exceptions. IaC doesn’t create your operating DNA; it forces you to make it explicit, which is uncomfortable precisely because it surfaces the disagreements you’d been getting away with leaving vague.

    That’s a feature. The discipline of writing a policy is the discipline of ending an argument. You can’t encode “it depends” — so IaC drags the tacit into the open where it can be decided once and enforced everywhere.

    How to tell which kind you have

    Two organizations can have identical Terraform and completely different operating DNA. A quick diagnostic:

    • Can a new hire provision a compliant environment without asking anyone? If yes, the standard lives in the modules. If they need a senior engineer to hand-review it, the standard still lives in someone’s head.
    • What stops a non-compliant change from merging? If the answer is “a reviewer who happens to notice,” you have automation, not governance. If it’s a policy in CI, you have operating DNA.
    • Where does the reasoning live? In commit history and module comments, or in the memory of whoever built it? One survives attrition. The other leaves when they do.

    The takeaway

    Judge your IaC not by how fast it provisions but by how much organizational judgment it carries. The goal is a system where the compliant path is the easy path, the non-compliant path is a blocked pull request, and the reasoning behind both is readable by someone who joined last week. Terraform is a tool for reproducing infrastructure. Whether it reproduces your discipline or just your typing is the entire game — and that’s a decision you make, not one the tool makes for you.

    Example modules and policy-as-code guardrails are on GitHub: github.com/waghmaredb/vexpose-labs. How does your team encode its operating DNA? I’d like to hear what’s worked — LinkedIn or X.

  • Your Ansible Playbook Says ‘changed’ Every Run. It’s Lying — and It Matters.

    Every run, the same handful of tasks light up yellow: changed. Nothing on the host actually changed. It’s the kind of noise you learn to scroll past — until the day a real change hides in the wall of false ones, or a handler fires a needless service restart in production because a task cried wolf. Broken idempotency isn’t cosmetic. It’s a broken signal.

    Idempotency is the whole promise

    The entire value proposition of a configuration tool is convergence: run it against a host already in the desired state and it should report ok, do nothing, and change nothing. A task that reports changed on a converged host has broken that promise — and worse, it can no longer tell you when something genuinely drifted, because it says the same thing either way.

    The usual culprit: command and shell

    The command and shell modules have no idea what “changed” means. They run something and, unless told otherwise, report changed every single time by definition. Most false positives trace back to shelling out where a real module — or a proper guard — belonged.

    # The trap: shell has no concept of state, so it's always "changed"
- name: Ensure the app user exists
  ansible.builtin.shell: useradd appuser
  # Reports changed forever, and errors out the second time the user exists.
    # Fix 1 - use the module that actually understands desired state
- name: Ensure the app user exists
  ansible.builtin.user:
    name: appuser
    state: present          # idempotent by design: creates once, then reports ok

# Fix 2 - when you MUST shell out, tell Ansible how to judge the result
- name: Rebuild the search index only when the marker is stale
  ansible.builtin.command: /opt/app/reindex.sh
  args:
    creates: /var/lib/app/.reindexed   # skip entirely if this file exists

# Fix 3 - define "changed" yourself from the command's own output
- name: Apply schema migrations
  ansible.builtin.command: migrate --apply
  register: migrate_out
  changed_when: "'Applied' in migrate_out.stdout"
  failed_when: migrate_out.rc != 0 and 'up to date' not in migrate_out.stderr

    Prove it, don’t assume it

    You don’t have to eyeball the yellow. Ansible has a built-in idempotency test: run the play, then run it again in check mode and treat any changed as a failure.

    ansible-playbook site.yml                 # converge
ansible-playbook site.yml --check --diff  # second run should be all "ok"
# In CI, assert zero changed tasks on the second pass - that's your regression gate.

    Wire that second pass into CI and idempotency stops being something you hope for and becomes something you enforce. A playbook that isn’t green on the second run doesn’t converge, and a playbook that doesn’t converge can’t be trusted to tell you the truth about your fleet.

    The lesson

    A changed you can’t trust is worse than no signal at all, because it looks like information. Make every task earn its status: reach for the stateful module first, guard your commands with creates/removes, and define changed_when whenever you shell out. The goal isn’t a quiet run for its own sake — it’s a run where yellow always means something.

    Example plays and the CI idempotency gate are on GitHub: github.com/waghmaredb/vexpose-labs. Have a favorite idempotency trick I skipped? Share it on LinkedIn or X.

  • The Inference Bill Is a Memory-Bandwidth Problem

    You added GPUs to make your LLM serving faster, and tokens-per-second barely moved. The dashboard says the GPUs are 40% utilized. Someone suggests a bigger batch size; latency gets worse. This is the moment most teams misdiagnose, because the mental model — “inference is a compute problem, so add compute” — is wrong for the part of inference that dominates your bill. Decode is bound by memory bandwidth, not FLOPs.

    Two phases, two different bottlenecks

    Autoregressive generation has two phases, and they live in different worlds. Prefill processes the whole prompt at once — lots of parallel matrix math, genuinely compute-bound, the GPU’s happy place. Decode generates one token at a time, and to produce each token the hardware must read the entire model’s weights out of high-bandwidth memory (HBM) again. One token, all the weights, every step. Decode isn’t doing much math per byte it moves; it’s moving a staggering number of bytes to do a little math.

    So the number that predicts decode speed isn’t TFLOPS. It’s HBM bandwidth — and you can estimate the floor with arithmetic, no benchmark required.

    The back-of-envelope that predicts your latency

    For a memory-bound decode, the time to generate one token is roughly the bytes you must read divided by how fast you can read them:

    time_per_token      ≈ (params × bytes_per_param) / HBM_bandwidth
tokens_per_second   ≈ 1 / time_per_token

    Take a 70B model in FP16 (2 bytes/param) — that’s ~140 GB to read per token. On an 80GB accelerator at ~2.0 TB/s, that’s 140 / 2000 ≈ 70 ms/token: a ceiling near 14 tokens/sec for a single stream, before any compute, kernel, or networking overhead. Not because the GPU can’t do the math — because it has to haul 140 GB across the memory bus for every token.

    Model (FP16) Bytes read / token HBM bandwidth Est. ms / token Est. tok/s (1 stream)
    7B ~14 GB 2.0 TB/s ~7 ms ~140
    13B ~26 GB 2.0 TB/s ~13 ms ~77
    70B ~140 GB 2.0 TB/s ~70 ms ~14
    70B (FP8) ~70 GB 2.0 TB/s ~35 ms ~28
    Single-stream decode ceilings from bandwidth alone. Real systems land below these, but the ranking holds.

    Look at the last two rows. Quantizing the 70B model from FP16 to FP8 halves the bytes read per token and roughly doubles decode throughput — not because you added compute, but because you cut the actual bottleneck in half. That’s the tell that you’re memory-bound: the intervention that helps is the one that moves fewer bytes, not the one that adds more math.

    Why batching helps — until it doesn’t

    If each token read costs 140 GB regardless, the obvious move is to make that read serve many requests at once. That’s exactly what continuous batching does: read the weights once, apply them to a batch of in-flight sequences, amortize the bandwidth across all of them. Throughput climbs beautifully. This is the single highest-leverage lever most teams aren’t fully pulling.

    But batching hits a wall with a name: the KV cache. Every concurrent sequence keeps a per-token cache of keys and values in the same HBM you’re already bandwidth-starved on. Push the batch bigger and the KV cache grows until it evicts, spills, or OOMs — and now you’re memory-capacity bound instead of bandwidth bound. You’ve traded one wall for another. The craft is finding the batch size that maximizes throughput at your latency SLO without tipping into KV-cache thrash.

    What this changes about how you buy and build

    • Spec accelerators by bandwidth and capacity, not headline TFLOPS. For decode-heavy serving, HBM bandwidth and size predict your experience better than peak compute nearly every time.
    • Quantization is a throughput lever, not just a memory-savings trick. Fewer bytes per parameter is fewer bytes read per token. The speedup is the point, not a side effect.
    • KV-cache management is a first-class design concern. Paged attention, cache quantization, and sane max-context limits decide how far batching can take you.
    • Measure at your real batch size and context length. A single-stream benchmark and a saturated multi-tenant server are different machines wearing the same sticker.

    The altitude shift

    The reason this matters beyond the invoice is that it’s a problem altitude question. At low altitude, “inference is slow” gets answered with “buy more GPUs,” and the money goes to compute the workload can’t use. Raise the problem one level — which resource is actually saturated? — and the same symptom points to quantization, batching, and KV-cache strategy instead. The tool is not the transformation: a faster accelerator you’re running memory-blind just reaches the same wall a little sooner.

    Hold all of this as a conviction with a review date. The arithmetic is stable — bytes-per-token doesn’t care about your vendor — but the constants move: bandwidth per dollar, quantization quality, and cache tricks improve every hardware generation. State the model strongly enough to plan a cluster around it, and re-run the numbers when the next accelerator ships.

    The takeaway

    Before you approve another GPU order to fix inference latency, ask which resource is saturated. If decode dominates your workload, the honest answer is usually memory bandwidth — and the fixes that work are the ones that move fewer bytes per token, not the ones that add more math. Compute is what the datasheet sells. Bandwidth is what you actually ship on.

    The token-cost calculator and serving benchmarks are on GitHub: github.com/waghmaredb/vexpose-labs. Running LLMs in production and seeing the same wall? I’d like to compare numbers — LinkedIn or X.

  • Your pgvector Search Gets Slower as You Add Data. Here’s the Setting Everyone Misses.

    Your semantic search was instant at ten thousand rows. At two million it’s 800 milliseconds and climbing, and you never touched the query. Almost always the cause is the same: pgvector is doing an exact, brute-force scan of every vector because its index was never built — or was built with defaults that don’t fit your data.

    Exact search doesn’t scale, and it’s the default

    Without an approximate index, pgvector compares your query vector against every row. That’s fine at ten thousand and fatal at two million. The fix is an ANN index — but an ANN index has two knobs that decide everything, and both have quietly wrong defaults for a large table.

    -- lists ≈ rows / 1000 up to ~1M rows, then ≈ sqrt(rows).
-- 2,000,000 rows -> ~1,414 lists, NOT the tiny number you'll get by guessing.
CREATE INDEX ON docs
  USING ivfflat (embedding vector_cosine_ops)
  WITH (lists = 1414);

-- probes trades recall for speed AT QUERY TIME. The default of 1 is far too low.
-- A sane starting point is ~sqrt(lists); here sqrt(1414) ≈ 38. Then tune to a recall target.
SET ivfflat.probes = 38;

    Two failure modes, opposite symptoms. Too few lists and each partition is huge, so every probe scans a lot — slow. Too few probes and you scan too few partitions — fast, but you silently miss relevant results, which in a RAG system means confidently answering from the wrong chunks. You cannot tune one without measuring the other.

    Build the index after the data is loaded

    IVFFlat clusters your existing vectors to define its lists. Build it on an empty or tiny table and the clusters are meaningless; every later insert lands in an ill-fitting partition and recall degrades. Load first, then index — and rebuild after any large ingest.

    -- HNSW: slower to build, larger on disk, but better recall/latency
-- and no clusters to go stale as data changes.
CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);
SET hnsw.ef_search = 40;   -- the recall/speed dial at query time

    If your data grows or churns continuously, HNSW usually ages better than IVFFlat because it has no clusters to go stale. It costs more to build and store; that’s the trade you’re making.

    The lesson

    Measure recall, not just latency. A vector search that got ten times faster by missing a third of the right answers isn’t faster — it’s broken with a good p99. Pick a fixed set of queries with known-good results, and every time you touch lists, probes, or the index type, confirm recall held before you celebrate the speed.

    The benchmark harness and recall test are on GitHub: github.com/waghmaredb/vexpose-labs. Tuning vector search at scale? Compare notes on LinkedIn or X.

  • Coordinated Omission: Why Your p99 Latency Is Lying

    Your p99 latency is one of the most-quoted numbers in your observability stack, and if you measured it with a standard load test, it is very likely a lie — not off by a little, but understating the truth by a factor of hundreds. The tool didn’t malfunction. It measured exactly what it was designed to measure, which turns out to be the wrong thing the moment your system stalls.

    This is coordinated omission, and once you see it you can’t unsee it in a benchmark result again.

    The load generator stalls with the server

    Most load tests are closed-loop: a fixed pool of workers each send a request, wait for the response, then send the next. It’s simple and it’s how most benchmarking harnesses work by default. It also has a fatal blind spot.

    When the server freezes for 200 ms — a GC pause, a lock, a failover, a noisy neighbor — the workers waiting on it freeze too. During that stall they send nothing. So the requests that should have been issued during those 200 ms are never sent, never timed, and never counted. And those are exactly the requests that would have been slow. The test quietly omits its own worst samples and hands you a beautiful percentile for a system that, for a fifth of a second, was serving no one. Your users experienced that stall. Your benchmark didn’t.

    The same stall, measured two ways

    Here is one 200 ms freeze in an otherwise sub-millisecond service, at 1,000 requests/second. The only difference between the two columns is when the clock starts: the naive test times each request from when it was actually sent; the corrected version times it from when it was scheduled to be sent — the honest question, because a request that couldn’t even be issued was already failing the user.

    percentile   naive (ms)   corrected (ms)
      p50          0.2              0.2
      p90          0.2              0.2
      p99          0.2            120.0
    p99.9          0.2            192.0
    Bar chart comparing naive vs schedule-corrected latency percentiles: the naive load test reports p99 of 0.2 ms, while the corrected p99 is 120 ms and p99.9 is 192 ms — the tail latency coordinated omission hides.

    Same service, same stall, same run. The naive p99 reports 0.2 ms. The honest p99 is 120 ms — a 600× understatement. The average is fine in both; the deception lives entirely in the tail, which is precisely the part you’re quoting p99 to protect.

    The correction itself is one line of intent: measure latency from the schedule, not the send.

    scheduled    = i * interval           # when this request SHOULD have gone out
actual_start = max(scheduled, clock)  # closed loop: can't send until a worker is free

naive.append(finish - actual_start)   # what a naive test records
corrected.append(finish - scheduled)  # honest: the user was waiting since 'scheduled'

    The fix isn’t a formula — it’s a different test

    You can correct after the fact, but the better fix is to stop omitting in the first place. Drive load open-loop: issue requests at a constant target rate regardless of whether prior ones have returned, so a stall produces a visible backlog instead of an invisible gap. Practically:

    • Use a constant-throughput generator. wrk2, not wrk; fio with a fixed rate, not an unbounded queue depth. They hold the send schedule even when the system underneath them stumbles.
    • Record into an HdrHistogram. Full latency resolution across the whole range, plus built-in coordinated-omission correction — you get honest high percentiles instead of a rounded-off tail.
    • Report the tail, and report the load you held. “p99.9 at a sustained 1,000 rps” is a claim. “p99” with no throughput attached is a vibe.

    Why this is a leadership problem, not a testing footnote

    Capacity plans, SLOs, and autoscaling thresholds get set from these numbers. A p99 that’s silently 600× optimistic doesn’t just embarrass you in an incident review — it sizes your cluster, writes your error budget, and sets the alarm that was supposed to wake someone before the customer noticed. You built a safety margin on a measurement that deletes its own worst cases.

    This is the whole discipline of the Benchmark Files in one number: a claim without an honest benchmark is just an opinion, and a benchmark that omits its worst samples is an opinion wearing a lab coat. Treat every latency figure as a conviction with a review date — stated strongly enough to plan against, and re-run honestly the moment the load, the runtime, or the topology changes.

    The takeaway

    Before you quote a p99, ask one question: did the test keep sending while the system was stalling? If it was closed-loop and the answer is no, the number is describing a system that never had a bad moment — which is not the system you run. The average will always survive a benchmark. The tail only survives an honest one.

    The coordinated-omission demo — naive vs schedule-corrected percentiles — is runnable on GitHub: github.com/waghmaredb/vexpose-labs. Benchmark latency for a living and fighting the same tail? Compare methods on LinkedIn or X.

  • The Readiness Probe That Turned a Deploy Into an Outage

    A routine deploy. The new pods come up, and for ninety seconds the entire service returns 503 — even though the previous version was perfectly healthy the whole time. The culprit wasn’t the new code. It was a readiness probe pointed at the wrong thing.

    Readiness and liveness are opposites, not synonyms

    A readiness probe decides whether a pod receives traffic. A liveness probe decides whether a pod gets killed and restarted. Teams wire them identically and treat them as interchangeable, but they want opposite temperaments: liveness should be lazy and forgiving; readiness should be quick and honest. Confuse them and a slow-starting app either gets killed mid-boot because liveness is too aggressive, or gets declared “ready” before it can actually serve.

    The rollout death spiral

    Here’s how a bad readiness probe takes down a healthy service. The new ReplicaSet’s pods fail readiness. The rolling update, following maxUnavailable, has already begun retiring old pods — but the new ones never become ready to replace them. Endpoints drain faster than they fill. For a window, almost nothing is behind the Service, and clients get 503s from the exact mechanism that’s supposed to prevent this.

    # The trap: readiness hits a downstream dependency, with no warm-up
readinessProbe:
  httpGet:
    path: /healthz        # this handler also pings the database
    port: 8080
  initialDelaySeconds: 0  # checked before the app can even listen
  periodSeconds: 5
  failureThreshold: 1     # one blip = pulled from rotation

    Two mistakes compound. The health endpoint checks a downstream dependency, so a brief database hiccup marks every replica unready at once — a self-inflicted full outage. And failureThreshold: 1 with no startup grace means the smallest transient becomes an eviction from the load balancer.

    The fix: a shallow check plus a startupProbe

    # Give slow starters room without loosening liveness
startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 30
  periodSeconds: 5          # up to 150s to boot, then normal probes take over
readinessProbe:
  httpGet:
    path: /ready            # answers ONLY "can this process serve?" — no deep deps
    port: 8080
  periodSeconds: 5
  failureThreshold: 3

    Split the endpoints. /ready answers “is this process able to serve a request,” nothing more — don’t fold the database into it, or one slow dependency yanks every pod out of rotation simultaneously. Use a startupProbe to grant slow-booting apps their warm-up time instead of weakening liveness. And set maxUnavailable: 0 on critical rollouts so Kubernetes won’t retire an old pod until a new one is genuinely ready.

    The lesson

    Readiness is a promise to your load balancer, not a general health dashboard. Make it answer the narrowest possible question — “send me traffic now?” — and nothing else. Every dependency you fold into that check is another way for one small failure to become a total one.

    Probe configs and a reproducible rollout demo are on GitHub: github.com/waghmaredb/vexpose-labs. Got burned by a probe in a way I didn’t cover? Tell me on LinkedIn or X.