Author: Deepak Waghmare

  • Field Note: The Pipeline That Retrieved Yesterday’s Truth

    Field Note — one real failure, the wrong diagnosis, the actual cause, the fix.

    The assistant did not hallucinate. It told the truth — about a world that no longer existed. That distinction is the difference between a model problem and a pipeline problem, and almost everyone reaches for the wrong one first.

    The symptom

    An internal RAG assistant answered a policy question with total confidence and complete wrongness — it quoted a refund window that had been changed six weeks earlier. The wording was precise, sourced, and obsolete. Users reported it as “the AI making things up.”

    The wrong diagnosis

    “The model hallucinated — add a stronger system prompt, turn down the temperature.” None of that would have helped, because the model did not invent anything. It faithfully summarised a document that was sitting right there in the retrieval results. The failure happened before the model ever saw the query.

    The actual cause

    The vector index was stale. The source policy had been updated, but re-embedding ran on a weekly batch, and the old chunks still carried the highest similarity score for that question. Worse, the previous version of the document had been edited in place rather than tombstoned, so there was no signal that the retrieved chunk was superseded. The model did exactly its job: it grounded its answer in the most relevant document it was handed. The document was simply six weeks out of date. RAG does not protect you from stale truth — it launders it into a confident sentence.

    The fix

    Treat index freshness as a service level objective, not a cron job nobody watches. Re-embed on source change, incrementally, so an edit propagates in minutes rather than at the weekly boundary. Carry a last-updated timestamp on every chunk and let ranking prefer recency when relevance scores are close. And honour deletes: when a source is removed or superseded, its vectors must leave the index, not linger as a high-scoring ghost.

    # A retrieved chunk is not just "relevant" - it has an age.
    for chunk in results:
        if chunk.source_updated_at > chunk.embedded_at:
            stale.append(chunk)   # the index is behind the source of truth
    
    # Alert when stale/total crosses the freshness SLO.
    # Re-embed on source change; tombstone on delete; recency breaks ties.
    

    The reframe

    RAG does not hallucinate here. It repeats, faithfully, whatever your index believes to be true. The model was not wrong about the facts — it was right about a world that had already moved on. Retrieval quality is not a model property. It is a freshness discipline, and it decays the moment you stop maintaining it.

  • The Real Cost of a Token

    Everyone can quote the price of a token. Almost no one can tell you what a token actually costs them. Those two numbers are rarely within a factor of three of each other, and the distance between them is where AI economics is really decided.

    Author’s note: this post pairs the argument with a benchmark from my own environment. The tables below are marked where results go, alongside the exact method used to produce them — so the numbers are reproducible rather than asserted. This draft ships with placeholders; the real figures go in before it publishes.

    The price list is not the cost

    The comfortable belief is that a published price — so many dollars per million input and output tokens — tells you what inference costs. It tells you the list price of one input. It does not tell you the delivered cost of one answer, because between the token and the answer sits everything that actually governs your bill: how many tokens a single answer really consumes once you count the system prompt, the retrieved context, and the retries; how well you batch; how long your context window runs; and how much of your expensive hardware sits idle between requests.

    Cost per token is a unit price. Cost per answer is what you pay. Optimising the first while ignoring the second is optimising the microphone instead of the conversation.

    What actually moves the number

    Four factors dominate, and none of them appear on a price list. Context length, because a retrieval-augmented answer often carries ten to twenty times the tokens of the user’s actual question. Batching efficiency, because an under-fed GPU bills you for wall-clock time it spent waiting. Retry and guardrail overhead, because every reask, reranking pass, and safety check is tokens you paid for and the user never saw. And idle time, because reserved capacity costs the same whether it is serving peak traffic or nothing at all.

    The benchmark

    To make this concrete, I measured a single representative workload — [INSERT WORKLOAD: e.g. RAG Q&A over an internal knowledge base, ~500-token questions, top-8 retrieval] — across four serving configurations on [INSERT HARDWARE]. The list price per token is identical across every row. The delivered cost is not.

    ConfigurationTokens / answerThroughput (tok/s)GPU-seconds / answerDelivered cost / 1k answersTokens / watt
    Baseline (batch=1, full context)[INSERT][INSERT][INSERT][INSERT][INSERT]
    + continuous batching[INSERT][INSERT][INSERT][INSERT][INSERT]
    + context trimming / rerank[INSERT][INSERT][INSERT][INSERT][INSERT]
    + quantisation[INSERT][INSERT][INSERT][INSERT][INSERT]

    [Author note — fill before publishing. The list price per token is the control variable and stays constant across rows; the whole point of the table is that delivered cost per answer moves by a large multiple while the list price never changes. Do not publish with placeholders. If a figure cannot be measured cleanly, cut that row rather than estimate it.]

    How I measured this

    The method matters more than any single figure, so here it is in full — to be reproduced rather than trusted. [INSERT: model and serving stack, e.g. vLLM version.] Load was generated at [INSERT] concurrent requests to steady state, discarding a warm-up window of [INSERT]. Tokens per answer counted the system prompt, the retrieved context, and the output — everything billed, not just the visible reply. GPU-seconds per answer came from measured throughput under load, and delivered cost from GPU-seconds times the fully loaded hourly rate of the instance ([INSERT rate and source]). Tokens per watt used board power sampled from nvidia-smi at [INSERT] intervals under load. Every number is an end-to-end measurement under load; none is derived from a spec sheet.

    The reframe

    The cheapest token on the price list can be the most expensive answer in production, and the reverse is just as true. Until you measure delivered cost per answer in your own environment, you are not managing AI economics — you are quoting someone else’s list price and hoping it resembles your bill. It almost never does.

  • Field Note: The Helm Upgrade That Rolled Itself Back

    Field Note — one real failure, the wrong diagnosis, the actual cause, the fix.

    A Helm upgrade that rolls itself back is not a Helm bug. It is Helm doing precisely what you told it to do — and the gap between what you told it and what you meant is the whole lesson.

    The symptom

    The deploy pipeline ran helm upgrade, waited, and reported failure. Minutes later the cluster was serving the previous version as if nothing had happened. helm history showed the new revision appear and then a rollback to the old one, all automatic. The application logs for the new version were already gone with the pods. It looked like Helm had panicked and reverted on its own.

    The wrong diagnosis

    “Helm is flaky — pin a different version, add a retry.” That treats a symptom as the disease. The rollback was not random and it was not a defect. It was configured behaviour, working exactly as designed.

    The actual cause

    The upgrade ran with --atomic. That flag has a clear contract: if the release does not become healthy within the timeout, undo everything and return to the last good state. Healthy is defined by --wait — every pod ready, by its own readiness probe, within the deadline. The new version took longer to warm its cache than the readiness probe allowed, so Kubernetes never marked it ready, so Helm concluded the upgrade had failed, so --atomic did the honourable thing and rolled back. Every component behaved correctly. The only thing wrong was the definition of “ready.”

    # What was run:
    helm upgrade app ./chart --atomic --timeout 2m
    
    # --atomic : on failure, roll back to the previous revision
    # --wait   : "success" = all pods pass readiness within --timeout
    #
    # The probe's initialDelay + threshold exceeded 2m during cache warm-up.
    # Helm never saw "ready" -> declared failure -> rolled back. As designed.
    

    The fix

    Two changes, both about honesty. Make the readiness probe describe real readiness — add a startup probe for the cache-warming phase, so a slow start is not mistaken for a failure. And align the Helm timeout with the worst-case healthy start, not the average one. Keep --atomic. It is not the problem; it is the safety net working. Once “ready” actually meant ready, the rollback stopped firing, because there was nothing to roll back from.

    The reframe

    The automation did not fail. It obeyed. The bug was upstream of the tool — in what you declared “success” to mean. Most automation incidents are not the machine doing the wrong thing. They are the machine doing exactly the thing you specified, and revealing that your specification was wrong.

  • GPU Idle at Every Save: The NVMe Checkpoint Stall

    There is a cliff in your training run’s GPU-utilization graph. You probably can’t see it, because the dashboard averages it away — but it is there, every few hundred steps, and it is costing you more GPU-hours than any kernel you have spent the last month optimising.

    The cliff is the checkpoint. Every few hundred steps the run serialises model and optimiser state — tens of gigabytes — and writes it to disk. If that write is synchronous, the training step that triggered it blocks until the bytes are durable. The most expensive silicon in the building spends those seconds doing nothing but waiting for an NVMe drive.

    The averaged dashboard is lying to you

    This is the same trap as coordinated omission: a one-minute utilization average cannot show you a four-second stall. You see 94% and call it healthy. Sample at per-step resolution and the sawtooth appears — a flat line at the top, punctuated by vertical drops toward zero exactly on the checkpoint interval. The average is arithmetically true and operationally useless.

    GPU utilization through trainingSynchronous vs. asynchronous checkpointing — same hardware, same step time.ILLUSTRATIVE — shape, not values100%7550250training steps →GPU utilizationeach dip = one synchronous checkpoint write blocking the stepAsync + O_DIRECT — GPU stays fedSynchronous save — GPU starvesThe GPUs aren’t slow — the save path is.Run the harness for numbers on your hardware.
    Illustrative. The curve shows the shape of the stall, not measured values — the companion harness prints the real gap on your hardware.

    A checkpoint is an I/O job wearing a training job’s clothes

    The training loop is a compute job; the checkpoint is an I/O job; and the moment you run them on the same thread, the slower one sets the pace. A synchronous save does three expensive things in the critical path: it serialises state, it pushes tens of gigabytes through the filesystem, and it calls fsync to force durability. Worse, a large buffered write floods the page cache and can evict the very data your loader was about to read — so the stall does not end cleanly when the write does; the next few steps limp while the cache refills.

    Measure the save path, not the spec sheet

    The spec sheet sells you small-random-read IOPS. A checkpoint is none of those things — it is a large, sequential, fsync’d burst, often from every rank at once. So benchmark that, not the marketing number. The companion code has an fio job that models the real pattern (O_DIRECT, 1 MB blocks, fsync per write) and a dependency-free Python harness that times a stand-in step against three save strategies and prints the utilization gap on your hardware.

    Get the write off the critical path

    The fix is not a faster disk. It is refusing to let the disk pace the GPU. Capture the checkpoint to pinned host memory on the training thread — cheap, microseconds — then hand it to a background writer that fsyncs while training continues. Use O_DIRECT for the big sequential dump so it bypasses the page cache instead of evicting your data. The one rule that makes this correct rather than merely fast: snapshot before you enqueue, so the next step cannot mutate weights mid-write.

    # Before: the step waits for the disk
    save_checkpoint(state)              # serialise + write + fsync, all inline
    
    # After: the step pays only for an in-memory snapshot
    pinned = snapshot_to_pinned(state)  # microseconds, on the training thread
    writer.enqueue(pinned)              # fsync happens on a background thread

    The reframe

    A stall at every save is not a disk problem; it is an accounting one. You are paying frontier-GPU rates to watch a progress bar fill. Measure the save path, move it off the critical path, and the cliffs flatten — same hardware, more of it actually working. The GPUs were never the bottleneck. The decision to save synchronously was — and that is an operating-model choice, not a model one.

  • Field Note: The GPU That Was Busy Doing Nothing

    Field Note — one real failure, the wrong diagnosis, the actual cause, the fix.

    A GPU can be ninety-five percent utilised and still be doing almost nothing you are paying it to do. This is the most expensive gap between a metric and the truth in modern infrastructure.

    The symptom

    An inference service was slow. nvidia-smi reported 95% GPU utilisation, so the dashboard was green and the conclusion wrote itself: we are out of GPU, buy more. The throughput told a different story — roughly a third of the tokens per second the same hardware reached in the vendor benchmark.

    The wrong diagnosis

    “Utilisation is at 95%, so the GPU is the bottleneck — add capacity.” Every part of that sentence sounds right, and the last clause is wrong. More GPUs would have tripled the bill and moved the throughput barely at all.

    The actual cause

    GPU utilisation, as nvidia-smi reports it, means exactly one thing: at least one kernel was running during the sampling window. It says nothing about whether that kernel was doing useful arithmetic or waiting on memory. Our service was running with a batch size of one, generating tokens for a single request at a time. The GPU spent most of its cycles moving weights in and out of memory rather than computing — memory-bandwidth bound, not compute bound. Busy and idle at the same time.

    The fix

    Measure the right thing, then feed the hardware properly. The honest metrics are tokens per second under load and model FLOPs utilisation (MFU) — how much of the GPU’s theoretical compute you are actually using. Both were dismal. Enabling continuous batching, so concurrent requests share each forward pass, and right-sizing the batch took throughput most of the way to the benchmark on the very same cards.

    # The metric that lies:
    nvidia-smi --query-gpu=utilization.gpu --format=csv   # 95% — "busy"
    
    # The metrics that don't:
    #   tokens/sec at target concurrency   (throughput you actually ship)
    #   MFU = achieved FLOPs / peak FLOPs  (useful work vs. busy work)
    #
    # High utilization + low MFU = memory-bound.
    # Fix batching before you buy silicon.
    

    The reframe

    Utilisation tells you a GPU is busy. It will never tell you the GPU is useful. Buy the second card when the first one is doing useful work — not when it is merely busy doing nothing.

  • AI Platforms Are Not Platforms for AI

    Every enterprise I speak with across Asia Pacific has, by now, bought an AI platform. Very few of them have a platform for AI. The words are nearly identical. The gap between them is where most AI budgets quietly go to die.

    It is a comfortable belief that these are the same purchase. A vendor sells you a model catalog, a notebook environment, a vector store, and a chat interface with your logo on it. The invoice says “AI platform.” The assumption is that the platform question is now answered. It isn’t. You bought a product. A platform for AI is a capability — and nobody sells you that.

    Two different things wearing the same name

    An AI platform is a product you procure. It is optimised to get a model in front of a user quickly — a demo, a pilot, a proof of concept. Its job is to make the first mile easy, and it is very good at that.

    A platform for AI is the substrate underneath. It is what makes any model safe to run in production — for years, against real data, under real regulation, with a real bill attached. It is identity and access control that understands prompts and outputs, not just endpoints. It is data governance that knows which corpus a model may retrieve from and which it may not. It is observability that captures not just latency but hallucination rate, cost per answer, and drift in model behaviour. It is a rollback story for the day a model update quietly regresses.

    AI platform (what you buy)Platform for AI (what you operate)
    Optimised for the first demoOptimised for the thousandth production request
    Model catalog and notebooksIdentity, data governance, cost attribution
    Latency dashboardsHallucination rate, cost per answer, behavioural drift
    “It works”“It still works, safely, and we can prove why”

    Different verticals, same wall

    The pattern is unusually consistent. Manufacturing, banking, healthcare, retail — each is convinced its AI challenge is unique to its domain. Each hits the same wall in the same place: the moment a pilot has to become a system that other systems depend on. The model was never the hard part. Integration discipline was. The wall is not made of algorithms. It is made of identity, data lineage, and the absence of anyone who owns the model in production the way someone owns a database in production.

    This is worth saying plainly, because it is easy to hear it as an argument against buying AI platforms. It is not. Buy them — they earn their keep in the first mile. The mistake is believing the purchase discharged the platform obligation. It only relocated it.

    The tool is not the transformation

    A model is a tenant. It moves in, it does its work, and one day it moves out and a better one moves in. A platform for AI is the building — the wiring, the plumbing, the fire code, the lease that says who is responsible when something floods. You can change tenants every quarter. You cannot re-pour the foundation every quarter. Most enterprises are trying to run a building they never built, and wondering why every new tenant needs a month of bespoke work before it can turn the lights on.

    The tell is switching cost. In an organisation with a real platform for AI, swapping one model for another is a configuration change — new endpoint, re-run the evaluation suite, watch the dashboards. In an organisation with only an AI platform, the same swap is a project, because governance, monitoring, and cost control were built into the model integration rather than into the substrate beneath it. Every model becomes a fresh act of heroism.

    What to build first

    The substrate does not have to arrive all at once, but it has to arrive in the right order. Identity and data governance first, because they are the things you cannot retrofit safely once real data is flowing. Cost attribution next, because an AI system whose cost you cannot decompose per team and per use case will be switched off in the first budget review regardless of its accuracy. Observability of behaviour — not just uptime — after that. The model catalog, the part the market sells as “the platform,” comes last, because it is the easiest to replace and the least dangerous to get wrong.

    A Platform for AI: the substrate you operateBuilt from the bottom up — the lower layers can’t be retrofitted safely once real data is flowing.BUILDORDERMODEL = tenantswap = a config change, not a project4Model catalog · notebooks · endpoints · chat UIWhat the market sells as “the platform.” Built last — easiest to replace, least dangerous to get wrong.3Behavioural observabilityHallucination rate, cost per answer, drift in behaviour — not just latency and uptime.2Cost attribution · per team, per use caseCost you can’t decompose gets switched off in the first budget review — regardless of accuracy.1FOUNDATION — BUILD FIRST, CANNOT BE RETROFITTED SAFELYIdentity & access· prompt- and output-aware, not just endpointsData governance· corpus lineage: which sources a model may retrieve, and which it may notAI fails in the operating model, not the model.The model is the tenant; the platform is the building. You change tenants every quarter; you don’t re-pour the foundation.vEXPOSE
    A platform for AI, built in order: the lower layers can’t be retrofitted once real data is flowing.

    The reframe

    You do not have an AI strategy until you have a platform for AI. The model is the tenant; the platform is the landlord; and no amount of licensing the former relieves you of building the latter. The enterprises that compound their AI advantage over the next five years will not be the ones with the best model this quarter. They will be the ones who built the building — so that every good model, this quarter and every quarter after, has somewhere safe to live.

  • Field Note: The Drift That Only Shows Up at 2 a.m.

    Field Note — a recurring series: one real failure, the wrong diagnosis, the actual cause, the fix.

    Infrastructure as Code was supposed to end configuration drift. It didn’t. It moved drift somewhere harder to see — into the gap between your state file and the world that file claims to describe.

    The symptom

    The pull request was clean. terraform plan showed no changes, two reviewers approved, it merged. Nine days later an unrelated change to a load balancer produced a plan that wanted to destroy a subnet nobody had touched — at 2 a.m., because that is when the on-call apply ran.

    The wrong diagnosis

    “Someone changed it by hand.” That is the reflex, and occasionally it is right. But blaming a human is the comfortable answer, because it means the tooling is fine and the process just needs more discipline. It usually isn’t the human.

    The actual cause

    State is a photograph, not a live feed. It records what the world looked like the last time you ran apply. Between applies, the world moves: a security team edits an ingress rule, an autoscaler rewrites a tag, a cloud provider ships a new default for a resource you declared three years ago. Your code did not drift. Reality drifted — and your state never looked again, until an unrelated apply forced a refresh and reconciled the difference destructively.

    The fix

    Stop discovering drift as a side effect of unrelated work. Detect it on a schedule. A nightly job that runs a read-only plan and treats a non-empty diff as a signal — not an error to auto-correct — turns drift from a 2 a.m. surprise into a morning ticket.

    # nightly-drift.sh — read-only, alert on drift, never auto-apply
    terraform init -input=false
    terraform plan -detailed-exitcode -lock=false -input=false
    code=$?
    
    # 0 = no drift | 2 = drift detected | 1 = error
    if [ "$code" -eq 2 ]; then
      notify "Drift detected in $(terraform workspace show) — review before next apply"
    fi
    exit $code
    

    Two disciplines make the detection trustworthy. Lock your state so a scheduled plan never races a real apply. And scope who and what can mutate infrastructure out of band — the fewer hands outside the pipeline, the quieter the diff.

    The reframe

    Drift is not a failure of Infrastructure as Code. It is Infrastructure as Code telling you the truth about who else has their hands on your infrastructure. The only real choice is when you hear it: on a schedule you control, or at 2 a.m. in the middle of something else.

  • Operating DNA, Not Templates: What Infrastructure as Code Actually Encodes

    Most teams justify infrastructure as code with speed: spin up an environment in minutes, tear it down just as fast. That’s real, and it’s the least interesting thing IaC does. Speed is what the tool sells. What IaC actually encodes — when it’s done well — is your organization’s operating DNA: the accumulated judgment about how you run infrastructure, made explicit, versioned, reviewable, and enforced. Miss that and you’ve bought a faster way to reproduce your bad decisions.

    The tool is not the transformation

    Adopting Terraform no more makes you good at infrastructure than buying a word processor makes you a writer. I’ve watched organizations “do IaC” and end up with a pile of copy-pasted HCL that encodes nothing but one engineer’s Tuesday afternoon — no standards, no guardrails, every module a fresh set of decisions re-litigated from scratch. They automated the typing. They didn’t capture the thinking. The tool is not the transformation. The transformation is when the judgment that used to live in senior engineers’ heads and tribal wiki pages becomes code the pipeline can read and refuse to violate.

    What “operating DNA as code” actually looks like

    Operating DNA is the set of answers your organization has converged on: which regions we deploy to, what a compliant storage bucket looks like, who can provision GPUs, what “tagged for cost allocation” means, how big is too big. In most companies those answers live in people, slides, and the memory of the last incident. IaC gives them a home where they can be enforced instead of merely hoped for. Three layers turn IaC from “scripts that build things” into “judgment that governs things.”

    1. Modules encode the paved road. A good module isn’t a thin wrapper over a resource; it’s an opinion. It bakes in the encryption, the logging, the tagging, and the sane defaults so that the easy way to provision is also the compliant way. The standard becomes the path of least resistance.

    # A module that encodes operating DNA: the compliant config is the DEFAULT.
    module "bucket" {
      source = "org/secure-bucket/aws"
      name   = "billing-exports"
      # No "encryption = true" knob here - the module makes it non-optional.
      # Logging, versioning, public-access-block, cost-center tag: enforced inside.
    }

    2. Policy-as-code makes the guardrails non-negotiable. Modules make the right thing easy; policy makes the wrong thing impossible to merge. This is where operating DNA stops being a convention and becomes a rule the pipeline enforces on every plan — including the ones written by someone who’s never read the wiki.

    # OPA/Conftest: an un-encrypted bucket never gets past CI. Nothing to forget.
    package terraform.storage
    
    deny[msg] {
      r := input.resource_changes[_]
      r.type == "aws_s3_bucket"
      not encrypted(r)
      msg := sprintf("bucket %v must be encrypted - org policy STO-014", [r.address])
    }

    3. The pull request becomes the system of record. When infrastructure is code, every change to how you run is a diff: proposed, reviewed, explained in a commit message, permanently attributable. The why lives next to the what. New engineers onboard by reading the modules and policies instead of by breaking things and being told. That’s operating DNA made teachable.

    The part that isn’t code

    Here’s the honest caveat: writing the guardrails is the visible work, and it’s the smaller half. The invisible work is the organizational agreement underneath — actually deciding, as a group, what the standard is before you can encode it. A policy that says “buckets must be encrypted” is trivial to write and worthless if three teams quietly disagree about the exceptions. IaC doesn’t create your operating DNA; it forces you to make it explicit, which is uncomfortable precisely because it surfaces the disagreements you’d been getting away with leaving vague.

    That’s a feature. The discipline of writing a policy is the discipline of ending an argument. You can’t encode “it depends” — so IaC drags the tacit into the open where it can be decided once and enforced everywhere.

    How to tell which kind you have

    Two organizations can have identical Terraform and completely different operating DNA. A quick diagnostic:

    • Can a new hire provision a compliant environment without asking anyone? If yes, the standard lives in the modules. If they need a senior engineer to hand-review it, the standard still lives in someone’s head.
    • What stops a non-compliant change from merging? If the answer is “a reviewer who happens to notice,” you have automation, not governance. If it’s a policy in CI, you have operating DNA.
    • Where does the reasoning live? In commit history and module comments, or in the memory of whoever built it? One survives attrition. The other leaves when they do.

    The takeaway

    Judge your IaC not by how fast it provisions but by how much organizational judgment it carries. The goal is a system where the compliant path is the easy path, the non-compliant path is a blocked pull request, and the reasoning behind both is readable by someone who joined last week. Terraform is a tool for reproducing infrastructure. Whether it reproduces your discipline or just your typing is the entire game — and that’s a decision you make, not one the tool makes for you.

    Example modules and policy-as-code guardrails are on GitHub: github.com/waghmaredb/vexpose-labs. How does your team encode its operating DNA? I’d like to hear what’s worked — LinkedIn or X.

  • Your Ansible Playbook Says ‘changed’ Every Run. It’s Lying — and It Matters.

    Every run, the same handful of tasks light up yellow: changed. Nothing on the host actually changed. It’s the kind of noise you learn to scroll past — until the day a real change hides in the wall of false ones, or a handler fires a needless service restart in production because a task cried wolf. Broken idempotency isn’t cosmetic. It’s a broken signal.

    Idempotency is the whole promise

    The entire value proposition of a configuration tool is convergence: run it against a host already in the desired state and it should report ok, do nothing, and change nothing. A task that reports changed on a converged host has broken that promise — and worse, it can no longer tell you when something genuinely drifted, because it says the same thing either way.

    The usual culprit: command and shell

    The command and shell modules have no idea what “changed” means. They run something and, unless told otherwise, report changed every single time by definition. Most false positives trace back to shelling out where a real module — or a proper guard — belonged.

    # The trap: shell has no concept of state, so it's always "changed"
    - name: Ensure the app user exists
      ansible.builtin.shell: useradd appuser
      # Reports changed forever, and errors out the second time the user exists.
    # Fix 1 - use the module that actually understands desired state
    - name: Ensure the app user exists
      ansible.builtin.user:
        name: appuser
        state: present          # idempotent by design: creates once, then reports ok
    
    # Fix 2 - when you MUST shell out, tell Ansible how to judge the result
    - name: Rebuild the search index only when the marker is stale
      ansible.builtin.command: /opt/app/reindex.sh
      args:
        creates: /var/lib/app/.reindexed   # skip entirely if this file exists
    
    # Fix 3 - define "changed" yourself from the command's own output
    - name: Apply schema migrations
      ansible.builtin.command: migrate --apply
      register: migrate_out
      changed_when: "'Applied' in migrate_out.stdout"
      failed_when: migrate_out.rc != 0 and 'up to date' not in migrate_out.stderr

    Prove it, don’t assume it

    You don’t have to eyeball the yellow. Ansible has a built-in idempotency test: run the play, then run it again in check mode and treat any changed as a failure.

    ansible-playbook site.yml                 # converge
    ansible-playbook site.yml --check --diff  # second run should be all "ok"
    # In CI, assert zero changed tasks on the second pass - that's your regression gate.

    Wire that second pass into CI and idempotency stops being something you hope for and becomes something you enforce. A playbook that isn’t green on the second run doesn’t converge, and a playbook that doesn’t converge can’t be trusted to tell you the truth about your fleet.

    The lesson

    A changed you can’t trust is worse than no signal at all, because it looks like information. Make every task earn its status: reach for the stateful module first, guard your commands with creates/removes, and define changed_when whenever you shell out. The goal isn’t a quiet run for its own sake — it’s a run where yellow always means something.

    Example plays and the CI idempotency gate are on GitHub: github.com/waghmaredb/vexpose-labs. Have a favorite idempotency trick I skipped? Share it on LinkedIn or X.

  • The Inference Bill Is a Memory-Bandwidth Problem

    You added GPUs to make your LLM serving faster, and tokens-per-second barely moved. The dashboard says the GPUs are 40% utilized. Someone suggests a bigger batch size; latency gets worse. This is the moment most teams misdiagnose, because the mental model — “inference is a compute problem, so add compute” — is wrong for the part of inference that dominates your bill. Decode is bound by memory bandwidth, not FLOPs.

    Two phases, two different bottlenecks

    Autoregressive generation has two phases, and they live in different worlds. Prefill processes the whole prompt at once — lots of parallel matrix math, genuinely compute-bound, the GPU’s happy place. Decode generates one token at a time, and to produce each token the hardware must read the entire model’s weights out of high-bandwidth memory (HBM) again. One token, all the weights, every step. Decode isn’t doing much math per byte it moves; it’s moving a staggering number of bytes to do a little math.

    So the number that predicts decode speed isn’t TFLOPS. It’s HBM bandwidth — and you can estimate the floor with arithmetic, no benchmark required.

    The back-of-envelope that predicts your latency

    For a memory-bound decode, the time to generate one token is roughly the bytes you must read divided by how fast you can read them:

    time_per_token      ≈ (params × bytes_per_param) / HBM_bandwidth
    tokens_per_second   ≈ 1 / time_per_token

    Take a 70B model in FP16 (2 bytes/param) — that’s ~140 GB to read per token. On an 80GB accelerator at ~2.0 TB/s, that’s 140 / 2000 ≈ 70 ms/token: a ceiling near 14 tokens/sec for a single stream, before any compute, kernel, or networking overhead. Not because the GPU can’t do the math — because it has to haul 140 GB across the memory bus for every token.

    Model (FP16) Bytes read / token HBM bandwidth Est. ms / token Est. tok/s (1 stream)
    7B ~14 GB 2.0 TB/s ~7 ms ~140
    13B ~26 GB 2.0 TB/s ~13 ms ~77
    70B ~140 GB 2.0 TB/s ~70 ms ~14
    70B (FP8) ~70 GB 2.0 TB/s ~35 ms ~28
    Single-stream decode ceilings from bandwidth alone. Real systems land below these, but the ranking holds.

    Look at the last two rows. Quantizing the 70B model from FP16 to FP8 halves the bytes read per token and roughly doubles decode throughput — not because you added compute, but because you cut the actual bottleneck in half. That’s the tell that you’re memory-bound: the intervention that helps is the one that moves fewer bytes, not the one that adds more math.

    Why batching helps — until it doesn’t

    If each token read costs 140 GB regardless, the obvious move is to make that read serve many requests at once. That’s exactly what continuous batching does: read the weights once, apply them to a batch of in-flight sequences, amortize the bandwidth across all of them. Throughput climbs beautifully. This is the single highest-leverage lever most teams aren’t fully pulling.

    But batching hits a wall with a name: the KV cache. Every concurrent sequence keeps a per-token cache of keys and values in the same HBM you’re already bandwidth-starved on. Push the batch bigger and the KV cache grows until it evicts, spills, or OOMs — and now you’re memory-capacity bound instead of bandwidth bound. You’ve traded one wall for another. The craft is finding the batch size that maximizes throughput at your latency SLO without tipping into KV-cache thrash.

    What this changes about how you buy and build

    • Spec accelerators by bandwidth and capacity, not headline TFLOPS. For decode-heavy serving, HBM bandwidth and size predict your experience better than peak compute nearly every time.
    • Quantization is a throughput lever, not just a memory-savings trick. Fewer bytes per parameter is fewer bytes read per token. The speedup is the point, not a side effect.
    • KV-cache management is a first-class design concern. Paged attention, cache quantization, and sane max-context limits decide how far batching can take you.
    • Measure at your real batch size and context length. A single-stream benchmark and a saturated multi-tenant server are different machines wearing the same sticker.

    The altitude shift

    The reason this matters beyond the invoice is that it’s a problem altitude question. At low altitude, “inference is slow” gets answered with “buy more GPUs,” and the money goes to compute the workload can’t use. Raise the problem one level — which resource is actually saturated? — and the same symptom points to quantization, batching, and KV-cache strategy instead. The tool is not the transformation: a faster accelerator you’re running memory-blind just reaches the same wall a little sooner.

    Hold all of this as a conviction with a review date. The arithmetic is stable — bytes-per-token doesn’t care about your vendor — but the constants move: bandwidth per dollar, quantization quality, and cache tricks improve every hardware generation. State the model strongly enough to plan a cluster around it, and re-run the numbers when the next accelerator ships.

    The takeaway

    Before you approve another GPU order to fix inference latency, ask which resource is saturated. If decode dominates your workload, the honest answer is usually memory bandwidth — and the fixes that work are the ones that move fewer bytes per token, not the ones that add more math. Compute is what the datasheet sells. Bandwidth is what you actually ship on.

    The token-cost calculator and serving benchmarks are on GitHub: github.com/waghmaredb/vexpose-labs. Running LLMs in production and seeing the same wall? I’d like to compare numbers — LinkedIn or X.