Author: Deepak Waghmare

  • Your pgvector Search Gets Slower as You Add Data. Here’s the Setting Everyone Misses.

    Your semantic search was instant at ten thousand rows. At two million it’s 800 milliseconds and climbing, and you never touched the query. Almost always the cause is the same: pgvector is doing an exact, brute-force scan of every vector because its index was never built — or was built with defaults that don’t fit your data.

    Exact search doesn’t scale, and it’s the default

    Without an approximate index, pgvector compares your query vector against every row. That’s fine at ten thousand and fatal at two million. The fix is an ANN index — but an ANN index has two knobs that decide everything, and both have quietly wrong defaults for a large table.

    -- lists ≈ rows / 1000 up to ~1M rows, then ≈ sqrt(rows).
    -- 2,000,000 rows -> ~1,414 lists, NOT the tiny number you'll get by guessing.
    CREATE INDEX ON docs
      USING ivfflat (embedding vector_cosine_ops)
      WITH (lists = 1414);
    
    -- probes trades recall for speed AT QUERY TIME. The default of 1 is far too low.
    -- A sane starting point is ~sqrt(lists); here sqrt(1414) ≈ 38. Then tune to a recall target.
    SET ivfflat.probes = 38;

    Two failure modes, opposite symptoms. Too few lists and each partition is huge, so every probe scans a lot — slow. Too few probes and you scan too few partitions — fast, but you silently miss relevant results, which in a RAG system means confidently answering from the wrong chunks. You cannot tune one without measuring the other.

    Build the index after the data is loaded

    IVFFlat clusters your existing vectors to define its lists. Build it on an empty or tiny table and the clusters are meaningless; every later insert lands in an ill-fitting partition and recall degrades. Load first, then index — and rebuild after any large ingest.

    -- HNSW: slower to build, larger on disk, but better recall/latency
    -- and no clusters to go stale as data changes.
    CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops)
      WITH (m = 16, ef_construction = 64);
    SET hnsw.ef_search = 40;   -- the recall/speed dial at query time

    If your data grows or churns continuously, HNSW usually ages better than IVFFlat because it has no clusters to go stale. It costs more to build and store; that’s the trade you’re making.

    The lesson

    Measure recall, not just latency. A vector search that got ten times faster by missing a third of the right answers isn’t faster — it’s broken with a good p99. Pick a fixed set of queries with known-good results, and every time you touch lists, probes, or the index type, confirm recall held before you celebrate the speed.

    The benchmark harness and recall test are on GitHub: github.com/waghmaredb/vexpose-labs. Tuning vector search at scale? Compare notes on LinkedIn or X.

  • Coordinated Omission: Why Your p99 Latency Is Lying

    Your p99 latency is one of the most-quoted numbers in your observability stack, and if you measured it with a standard load test, it is very likely a lie — not off by a little, but understating the truth by a factor of hundreds. The tool didn’t malfunction. It measured exactly what it was designed to measure, which turns out to be the wrong thing the moment your system stalls.

    This is coordinated omission, and once you see it you can’t unsee it in a benchmark result again.

    The load generator stalls with the server

    Most load tests are closed-loop: a fixed pool of workers each send a request, wait for the response, then send the next. It’s simple and it’s how most benchmarking harnesses work by default. It also has a fatal blind spot.

    When the server freezes for 200 ms — a GC pause, a lock, a failover, a noisy neighbor — the workers waiting on it freeze too. During that stall they send nothing. So the requests that should have been issued during those 200 ms are never sent, never timed, and never counted. And those are exactly the requests that would have been slow. The test quietly omits its own worst samples and hands you a beautiful percentile for a system that, for a fifth of a second, was serving no one. Your users experienced that stall. Your benchmark didn’t.

    The same stall, measured two ways

    Here is one 200 ms freeze in an otherwise sub-millisecond service, at 1,000 requests/second. The only difference between the two columns is when the clock starts: the naive test times each request from when it was actually sent; the corrected version times it from when it was scheduled to be sent — the honest question, because a request that couldn’t even be issued was already failing the user.

    percentile   naive (ms)   corrected (ms)
          p50          0.2              0.2
          p90          0.2              0.2
          p99          0.2            120.0
        p99.9          0.2            192.0
    Bar chart comparing naive vs schedule-corrected latency percentiles: the naive load test reports p99 of 0.2 ms, while the corrected p99 is 120 ms and p99.9 is 192 ms — the tail latency coordinated omission hides.

    Same service, same stall, same run. The naive p99 reports 0.2 ms. The honest p99 is 120 ms — a 600× understatement. The average is fine in both; the deception lives entirely in the tail, which is precisely the part you’re quoting p99 to protect.

    The correction itself is one line of intent: measure latency from the schedule, not the send.

    scheduled    = i * interval           # when this request SHOULD have gone out
    actual_start = max(scheduled, clock)  # closed loop: can't send until a worker is free
    
    naive.append(finish - actual_start)   # what a naive test records
    corrected.append(finish - scheduled)  # honest: the user was waiting since 'scheduled'

    The fix isn’t a formula — it’s a different test

    You can correct after the fact, but the better fix is to stop omitting in the first place. Drive load open-loop: issue requests at a constant target rate regardless of whether prior ones have returned, so a stall produces a visible backlog instead of an invisible gap. Practically:

    • Use a constant-throughput generator. wrk2, not wrk; fio with a fixed rate, not an unbounded queue depth. They hold the send schedule even when the system underneath them stumbles.
    • Record into an HdrHistogram. Full latency resolution across the whole range, plus built-in coordinated-omission correction — you get honest high percentiles instead of a rounded-off tail.
    • Report the tail, and report the load you held. “p99.9 at a sustained 1,000 rps” is a claim. “p99” with no throughput attached is a vibe.

    Why this is a leadership problem, not a testing footnote

    Capacity plans, SLOs, and autoscaling thresholds get set from these numbers. A p99 that’s silently 600× optimistic doesn’t just embarrass you in an incident review — it sizes your cluster, writes your error budget, and sets the alarm that was supposed to wake someone before the customer noticed. You built a safety margin on a measurement that deletes its own worst cases.

    This is the whole discipline of the Benchmark Files in one number: a claim without an honest benchmark is just an opinion, and a benchmark that omits its worst samples is an opinion wearing a lab coat. Treat every latency figure as a conviction with a review date — stated strongly enough to plan against, and re-run honestly the moment the load, the runtime, or the topology changes.

    The takeaway

    Before you quote a p99, ask one question: did the test keep sending while the system was stalling? If it was closed-loop and the answer is no, the number is describing a system that never had a bad moment — which is not the system you run. The average will always survive a benchmark. The tail only survives an honest one.

    The coordinated-omission demo — naive vs schedule-corrected percentiles — is runnable on GitHub: github.com/waghmaredb/vexpose-labs. Benchmark latency for a living and fighting the same tail? Compare methods on LinkedIn or X.

  • The Readiness Probe That Turned a Deploy Into an Outage

    A routine deploy. The new pods come up, and for ninety seconds the entire service returns 503 — even though the previous version was perfectly healthy the whole time. The culprit wasn’t the new code. It was a readiness probe pointed at the wrong thing.

    Readiness and liveness are opposites, not synonyms

    A readiness probe decides whether a pod receives traffic. A liveness probe decides whether a pod gets killed and restarted. Teams wire them identically and treat them as interchangeable, but they want opposite temperaments: liveness should be lazy and forgiving; readiness should be quick and honest. Confuse them and a slow-starting app either gets killed mid-boot because liveness is too aggressive, or gets declared “ready” before it can actually serve.

    The rollout death spiral

    Here’s how a bad readiness probe takes down a healthy service. The new ReplicaSet’s pods fail readiness. The rolling update, following maxUnavailable, has already begun retiring old pods — but the new ones never become ready to replace them. Endpoints drain faster than they fill. For a window, almost nothing is behind the Service, and clients get 503s from the exact mechanism that’s supposed to prevent this.

    # The trap: readiness hits a downstream dependency, with no warm-up
    readinessProbe:
      httpGet:
        path: /healthz        # this handler also pings the database
        port: 8080
      initialDelaySeconds: 0  # checked before the app can even listen
      periodSeconds: 5
      failureThreshold: 1     # one blip = pulled from rotation

    Two mistakes compound. The health endpoint checks a downstream dependency, so a brief database hiccup marks every replica unready at once — a self-inflicted full outage. And failureThreshold: 1 with no startup grace means the smallest transient becomes an eviction from the load balancer.

    The fix: a shallow check plus a startupProbe

    # Give slow starters room without loosening liveness
    startupProbe:
      httpGet: { path: /healthz, port: 8080 }
      failureThreshold: 30
      periodSeconds: 5          # up to 150s to boot, then normal probes take over
    readinessProbe:
      httpGet:
        path: /ready            # answers ONLY "can this process serve?" — no deep deps
        port: 8080
      periodSeconds: 5
      failureThreshold: 3

    Split the endpoints. /ready answers “is this process able to serve a request,” nothing more — don’t fold the database into it, or one slow dependency yanks every pod out of rotation simultaneously. Use a startupProbe to grant slow-booting apps their warm-up time instead of weakening liveness. And set maxUnavailable: 0 on critical rollouts so Kubernetes won’t retire an old pod until a new one is genuinely ready.

    The lesson

    Readiness is a promise to your load balancer, not a general health dashboard. Make it answer the narrowest possible question — “send me traffic now?” — and nothing else. Every dependency you fold into that check is another way for one small failure to become a total one.

    Probe configs and a reproducible rollout demo are on GitHub: github.com/waghmaredb/vexpose-labs. Got burned by a probe in a way I didn’t cover? Tell me on LinkedIn or X.

  • Measuring Your AI Exit Cost: An Architecture Review

    On LinkedIn, I make the executive argument: your AI problem is not cost, it is dependency. Cost is what you pay this quarter; dependency is what you pay at every renewal, for a decade. This post is the engineering companion. If dependency is the real exposure, it should be measurable — and anything measurable can be architected down.

    So: how do you actually measure your AI exit cost? Not as a slide. As an audit.

    Exit cost is a stack, not a number

    Teams tend to equate switching providers with changing an API key. That was roughly true in 2023. In an agentic estate, provider coupling has accumulated in five distinct layers, and each has its own unwind cost.

    Layer 1 — API surface

    The cheapest layer to fix and the first to audit. If application code calls a provider SDK directly, every model reference is a migration line-item. The fix is a gateway: one internal endpoint, provider adapters behind it, model names resolved by policy rather than hard-coded. If you cannot change your default model in one config file, start here — nothing else in this audit is testable until you can.

    Layer 2 — Prompt and behavior coupling

    Prompts are tuned, consciously or not, to one model’s quirks: its instruction-following style, its formatting habits, its tool-calling dialect. This coupling is invisible until you swap models and watch pass rates drop. The countermeasure is an evaluation suite that defines correct behavior independently of any provider — golden datasets, structured-output checks, task-level assertions. Your eval suite is your portability contract. No evals, no measurable exit cost; only guesses.

    Layer 3 — Data gravity

    The heaviest layer. Embeddings are tied to the model that produced them: switch embedding models and every vector in the store must be regenerated — a compute bill and a re-indexing window that scales with your corpus. Fine-tunes are worse: weights trained on a proprietary base do not travel at all. Rule of thumb: prefer retrieval over fine-tuning wherever quality allows, and record the full re-embedding cost of your corpus as a standing line in the exit ledger. If you do not know that number today, that is finding #1 of your audit.

    Layer 4 — Orchestration and agent coupling

    Agent frameworks encode provider assumptions: tool-schema formats, context-window sizes, retry semantics, reasoning-token behavior. A workflow that plans, retrieves, calls tools, and self-corrects has ten to twenty coupling points where a chat app had one. Keep tool definitions in a neutral schema and translate at the gateway; treat any framework feature that only works on one provider as a loan you will repay at migration time.

    Layer 5 — Economic architecture

    Routing is where dependency becomes money. The arithmetic is one-sided even at public list prices: frontier models cost on the order of ten times more per token than the small and open-weight models that handle classification, extraction, and summarization perfectly well. Send every request to a frontier model and you pay the premium rate for work a cheaper tier would do identically — so a tiered blend lands at a fraction of a frontier-everything bill, with the exact multiple set by your own traffic mix. Measure yours; don’t borrow someone else’s number. But the deeper value of a routing tier is optionality: once classification, extraction, and summarization run on interchangeable smaller models — including open-weight ones you could self-host — the share of your workload that is truly captive shrinks to the frontier-only remainder. Price your renewal leverage as: the percentage of token volume that could move within 30 days without eval regression.

    The failover drill

    An exit cost you have never tested is a fiction. The drill is simple and quarterly: repoint the gateway’s default policy at your designated fallback (a second provider or an open-weight deployment), run the full eval suite, and record three numbers — engineering hours spent, eval pass-rate delta, and the workload percentage that moved cleanly.

    # gateway routing policy — failover drill
    default_tier:
      primary:   frontier-a
      fallback:  open-weight-70b   # drill target
    routes:
      classify|extract|summarize:  small-tier
      agentic|complex-reasoning:   default_tier
    drill:
      cadence: quarterly
      pass_criteria: eval_delta <= 2%
      record: [eng_hours, eval_delta, pct_moved]

    Those three numbers, tracked over time, are your exit cost — not an estimate, a measurement. In my experience the first drill is always humbling: the gap between the architecture diagram and the actual coupling is where the real dependency lives.

    The point is not to leave

    None of this is a plan to abandon your provider. It is the opposite of a plan: it is an option. An option you hold changes a renewal negotiation even if you never exercise it. An option you lack is a price you will accept.

    Rent intelligence if you choose to. Measure the exit before it is load-bearing. And never rent the operating model.

  • The Terraform Gotcha That Destroys the Resource You Meant to Keep

    You add one server to the middle of a list, run terraform plan, and Terraform announces it will destroy and recreate three resources you never touched. Nothing about them changed. This is the count trap, and it has caused more 2 a.m. incidents than almost any other Terraform footgun I know.

    Why a list index is a landmine

    When you build resources with count, Terraform addresses them by position: aws_instance.node[0], [1], [2]. That index is the resource’s identity in state. Remove or insert an element anywhere but the end and every index after it shifts — so [1] now points at what used to be [2]. Terraform doesn’t see “the list got shorter.” It sees “the thing at index 1 is a different thing now,” and the only way it knows to reconcile that is destroy-and-recreate.

    # The trap: identity is the position in the list
    variable "nodes" {
      default = ["web-a", "web-b", "web-c"]
    }
    
    resource "aws_instance" "node" {
      count = length(var.nodes)
      tags  = { Name = var.nodes[count.index] }
    }
    # Remove "web-a" and web-b and web-c each shift down one index —
    # Terraform destroys and recreates BOTH to "fix" the mismatch.

    The fix: address resources by a stable key, not a position

    for_each keys each resource by a string you control instead of an ordinal Terraform controls. Delete one entry and the others keep their identity, because their identity was never their position.

    # The fix: identity is a stable key
    variable "nodes" {
      default = ["web-a", "web-b", "web-c"]
    }
    
    resource "aws_instance" "node" {
      for_each = toset(var.nodes)
      tags     = { Name = each.key }
    }
    # The address is now aws_instance.node["web-b"].
    # Remove "web-a" and ONLY "web-a" is destroyed. The rest don't move.

    If you’re already on count

    Migrating isn’t rewrite-and-pray. Move the state entries to their new keyed addresses so Terraform keeps the existing resources instead of replacing them:

    terraform state mv 'aws_instance.node[1]' 'aws_instance.node["web-b"]'
    terraform state mv 'aws_instance.node[2]' 'aws_instance.node["web-c"]'
    # Then swap count for for_each in the config and plan — it should show no changes.

    Or, on Terraform 1.1+, declare the intent as code with a moved block and let the plan do the reshuffle for you. Either way, confirm the plan shows zero replacements before you apply. That plan output is the entire safety mechanism — read it.

    The lesson

    Use count only for genuinely anonymous, interchangeable replicas you’ll only ever scale from the end — three identical workers, nothing addressed individually. The moment a resource has a name, an identity, or a lifecycle of its own, key it with for_each. In infrastructure as code, how you address a thing is part of what the thing is. Get the identity wrong and Terraform will faithfully destroy production to make the map match the territory.

    More field notes and the full examples are on GitHub: github.com/waghmaredb/vexpose-labs. Hit a stranger Terraform footgun than this one? Trade war stories on LinkedIn or X.

  • A Reference Architecture for Enterprise RAG (That Won’t Hallucinate Your Docs)

    Every RAG demo works. You point a model at your docs, ask a question, get a fluent answer with a citation, and the room nods. Then it ships, someone asks about a policy that changed last quarter — and the system confidently cites the old version. The demo never showed you that failure mode, because the failure mode isn’t “the model can’t answer.” It’s “the model answers anyway.”

    That gap is where enterprise RAG loses trust — and it’s almost never the model’s fault. It’s retrieval handing over plausible-but-wrong context that a capable model then synthesizes beautifully. The tool is not the transformation: a bigger model makes a confident wrong answer more convincing, not less. The fix lives below the model, in retrieval and grounding. Here’s the architecture I’d stand up for RAG you can actually put in front of employees or customers.

    The failure mode: fluent, cited, and wrong

    Naive RAG is three lines: embed the query, pull the top-k nearest chunks, stuff them in the prompt. It fails three ways at once. Vector search alone misses the exact terms — part numbers, error codes, policy names — that keyword search would nail. Without reranking, “near in embedding space” isn’t “actually answers the question.” And with no relevance floor, the pipeline always returns something, so the model always has context to rationalize an answer from — even when the right answer is “we don’t have that.”

    The result is the worst kind of wrong: confident, well-written, and cited. Everything below exists to make the system able to say “I don’t have that in the sources.”

    The reference architecture

      Query
         |
         v
      Hybrid Retrieval      BM25 (keyword) + vector
         |
         v
      Rerank                cross-encoder scores the union
         |
         v
      Relevance floor  --- nothing relevant --->  Refuse ("not in the sources")
         |
      context clears floor
         |
         v
      Grounded generation   answer ONLY from context, cite every claim
         |
         v
      Answer + citations
         |
         v
      Eval + observability  groundedness + citation checks  --->  tune retrieval

    1. Ingestion & chunking. How you split documents is a retrieval decision, not a preprocessing chore. Chunk too big and the reranker drowns in noise; too small and you sever the context a claim needs. Preserve structure (headings, tables) and keep the source metadata — you’ll need the doc_id for citations.

    2. Hybrid retrieval. Run keyword (BM25) and vector search and take the union. Keyword catches the exact strings vectors miss; vectors catch the meaning keywords miss. Neither alone is enough in the enterprise, where half the questions hinge on a specific identifier.

    3. Rerank. Score the union with a cross-encoder that reads the query and each chunk together. This is the highest-ROI line most teams aren’t running — it reorders “nearby” into “actually relevant.”

    4. The grounding gate (relevance floor). The seam. If nothing clears a relevance threshold, return nothing. This is where your operating DNA — what the system is allowed to answer, and from what — stops being a hopeful prompt and becomes code. An empty retrieval result is a feature: it’s what lets the generator refuse instead of improvise.

    5. Grounded generation. Instruct the model to answer only from the provided context and cite a doc_id for every claim. With no context, it refuses — by construction, not by good behavior.

    6. Eval & observability. Score every answer for groundedness and citation validity, continuously. Without this you can’t tell an improvement from a regression — you’re just guessing with a chatbot.

    The grounding gate, in code

    Hybrid recall, rerank, and a relevance floor — the floor is the part that makes refusal possible:

    def retrieve(self, query, k=5):
        # Hybrid recall: keyword catches exact terms, vectors catch meaning.
        candidates = self._dedupe(
            self.bm25.search(query, k=20) + self.vector_index.search(query, k=20)
        )
        # Rerank the union with a cross-encoder — the single biggest quality win.
        for c in candidates:
            c.score = self.reranker.score(query, c.text)
        candidates.sort(key=lambda c: c.score, reverse=True)
        # Relevance floor: if nothing is truly relevant, return nothing.
        # An empty result is a feature — it lets the generator refuse.
        return [c for c in candidates[:k] if c.score >= self.min_score]

    And the grounding instruction that turns “no context” into “no answer”:

    GROUNDING_INSTRUCTION = (
        "Answer ONLY from the provided context. Cite the doc_id for every claim. "
        "If the context does not contain the answer, say 'I don't have that in the "
        "sources' — do not use prior knowledge."
    )

    Measuring whether it’s actually grounded

    Two numbers predict trust: is every claim supported by a retrieved chunk, and does every cited doc_id actually appear in the context? Score them on a fixed set of questions on every prompt, model, or index change.

    def score(case):
        groundedness = case.claims_supported / case.claims_total
        hallucinated = [d for d in case.cited_doc_ids if d not in case.context_doc_ids]
        return {"groundedness": groundedness, "citation_valid": not hallucinated}

    The guardrail everyone forgets: retrieval-time access control

    Every architecture above assumes one reader is allowed to see every document. In a real enterprise they are not. HR files, board decks, a customer’s data walled off from another customer’s — the same index holds chunks that different users have different rights to. Miss this and you haven’t built a helpful assistant; you’ve built the most efficient data-exfiltration path in the company. The fastest way to leak the salary spreadsheet is to embed it and let anyone ask.

    The critical design decision: access control lives in retrieval, not in the prompt. Filter the candidate set to what this caller is entitled to see before anything is reranked or shown to the model — sourced from the same entitlements as the system of record (row-level security, document ACLs), not re-invented in the pipeline.

    def retrieve(self, query, user, k=5):
        candidates = self._dedupe(
            self.bm25.search(query, k=40) + self.vector_index.search(query, k=40)
        )
        # Trim to what THIS user may see — enforced at the data layer, BEFORE rerank,
        # so unauthorized chunks never influence ranking or reach the model at all.
        candidates = [c for c in candidates if can_read(user, c.doc_id)]
    
        for c in candidates:
            c.score = self.reranker.score(query, c.text)
        candidates.sort(key=lambda c: c.score, reverse=True)
        return [c for c in candidates[:k] if c.score >= self.min_score]

    A system prompt that says “only answer if the user is authorized” is not a security control — it’s a polite request to a probabilistic system, and it will eventually say yes. Real access control is a hard filter on the retrieval set, applied before rerank so an unauthorized chunk can’t even tilt the ranking. This is the RAG expression of least privilege — the same operating DNA that governs a good agent: the model may ground only on what this specific user could already open for themselves. That’s a decision you enforce in code, not one you hope the model respects.

    The parts everyone underestimates

    Chunking is a retrieval decision. More RAG quality is won or lost here than in model choice. Test chunk sizes against your eval set like any other parameter.

    Reranking is the cheap win you’re skipping. A cross-encoder over the top ~40 candidates typically moves groundedness more than a model upgrade — for a fraction of the cost.

    The relevance floor is the whole ballgame. “Always answers” is the root of hallucinated RAG. A threshold that returns nothing when nothing is relevant is what converts a plausible liar into a trustworthy assistant.

    Retrieval decays silently. A doc gets superseded, and the old chunk still ranks — confidently. Freshness and provenance on your index matter more than the model you pick.

    The takeaway

    RAG trustworthiness is a retrieval-and-grounding problem, not a model problem. What the system is allowed to answer — and from what — is operating DNA, and it belongs in code at the grounding gate, not in a prompt you hope the model respects. Get that seam right and you can swap models freely as they improve. Skip it and you’ve built a very persuasive way to cite the wrong document.

    The retriever and the groundedness eval from this post are runnable on GitHub: github.com/waghmaredb/vexpose-labs. Building enterprise RAG and fighting the same failure modes? I’d like to compare notes — reach me on LinkedIn or X.

  • The ServiceNow REST pagination gotcha that silently drops records

    ServiceNow’s REST API uses offset-based pagination. When you paginate through a live table—one where records are being created or deleted—the offset becomes stale between calls. Records silently get dropped.

    Why it happens

    The offset is positional within the current result set at the exact moment you query. If a new incident is created and sorts before your current page, all subsequent offsets shift down—you skip records.

    Example: You’re processing incidents with offset=0, limit=100, then offset=100, limit=100. If a new incident arrives and sorts into position 50, the second page now contains records 151–250 instead of 100–200. Record 100 is never seen.

    The fix: use sys_id as a cursor

    Instead of offset, anchor pagination to sys_id. Since sys_id is immutable, new records don’t invalidate your position—they sort after your cursor.

    # DON'T: offset drifts
    offset = 0
    while True:
        records = api.query(offset=offset, limit=100)
        process(records)
        offset += 100  # Invalid if records were added
    
    # DO: stable cursor
    last_sys_id = ""
    while True:
        query = f"sys_id>{last_sys_id}^ORDERBYsys_id" if last_sys_id else "ORDERBYsys_id"
        records = api.query(query, limit=100)
        process(records)
        if records:
            last_sys_id = records[-1]["sys_id"]  # Advance cursor
    

    See the full before-and-after in github.com/waghmaredb/vexpose-labs.

  • The Benchmark Trap: Why Your Storage Numbers Lie — and How to Get Honest Ones

    A vendor datasheet promises a million IOPS. You buy the array, point your workload at it, and it’s slow. The number wasn’t a lie — it was just irrelevant to you. That gap, between an impressive benchmark and a disappointing production system, is where a lot of infrastructure money quietly goes to die.

    I’ve spent a large part of my career with fio, vdbench, and HammerDB open in front of me. The tools are easy. Getting an honest number out of them is not. Here’s the field guide I wish more teams had before they trusted a benchmark — their own or a vendor’s.

    The number isn’t wrong. It’s answering a different question.

    Peak IOPS on a datasheet is a real measurement — of a workload that looks nothing like yours. Tiny block size, unlimited queue depth, a working set small enough to live entirely in cache, reads only, measured for ten seconds. Your production is a 70/30 read-write mix, 16K blocks, a working set far larger than cache, and it has to stay fast at 2 a.m. on day 400. Same tool, different universe. Before you argue about whose array is faster, make sure you’re both describing the same planet.

    The traps that inflate a benchmark

    Most misleading numbers come from a short list of mistakes:

    • The working set fits in cache. You benchmarked DRAM, not the media. Size the dataset several times larger than the controller cache or you’re measuring the wrong component.
    • No steady state. Flash gets slower once garbage collection kicks in. A 60-second run flatters an SSD that looks very different an hour later. Precondition the device, then measure.
    • Queue-depth theater. Cranking queue depth to 256 maximizes IOPS and obliterates latency. It produces a big number and an unusable response time.
    • Wrong block size or mix. The datasheet uses 4K reads; your database does 8K–16K with real writes. Model the actual mix or the result is fiction.
    • A single sample. One run is an anecdote. The variance across five runs is the actual story.

    An honest fio job to start from

    [global]
    ioengine=libaio
    direct=1
    runtime=600
    time_based=1
    ramp_time=60              # reach steady state before recording
    group_reporting=1
    
    [db-like]
    rw=randrw
    rwmixread=70              # 70/30 read-write, like an OLTP database
    bs=16k                    # your real block size, not 4k
    iodepth=32               # realistic, not a vanity QD of 256
    numjobs=4
    size=200g                # larger than the array's cache
    percentile_list=99:99.9  # report the tail, not just the average

    Every knob here is a decision about honesty. direct=1 bypasses the page cache so you measure storage, not memory. ramp_time throws away the artificially fast warm-up. size forces cache misses. percentile_list is the one most people skip — and it’s the one that matters most.

    Read the right metrics

    Peak IOPS is the vanity metric. What actually predicts whether production will be happy:

    • Latency at your target throughput — not throughput at unlimited latency. Those are different questions with very different answers.
    • Tail latency (p99, p99.9). Your users feel the worst 1% of requests, not the average. A great mean with an ugly tail is a bad system wearing a good costume.
    • Consistency over time. Does it hold at steady state, or degrade as the device fills and ages?

    One reframing kills most bad purchases: fix a latency budget — say, 1 ms at p99 — and ask “how many IOPS can it sustain at or under that?” Suddenly the million-IOPS array and the “slower” one often trade places.

    Why this is a leadership discipline, not a lab chore

    Benchmarking is how you replace opinion with evidence. In a room full of vendor claims and strong personalities, the person holding a reproducible number wins the decision — and deserves to. That’s also why a benchmark is a conviction with a review date: you state a performance expectation strongly enough to plan around it, and you re-run it when the firmware, the workload, or the scale changes. A claim without a benchmark is just an opinion. A benchmark you can’t reproduce is just a different opinion with a chart attached.

    The takeaway

    Don’t ask “how fast is it.” Ask “how fast is it, running my workload, at my latency budget, at steady state, averaged over five runs.” The tools will answer honestly if you ask honestly. Everything else is marketing with a monospaced font.

    The full fio job file is on GitHub: github.com/waghmaredb/vexpose-labs. If you benchmark enterprise storage or databases for a living, I’d like to compare methodologies — reach me on LinkedIn or X.

  • vExpose is back — and it’s changing

    A quick, honest note: vExpose went quiet for a while. I’m back — and the blog is changing in a way I’m genuinely excited about.

    Here’s the shift. Most technical writing shows you the strategy but hides the wiring. Going forward, vExpose does both: I state the call a technology leader has to make, then prove it in code, architecture, and benchmarks. One conviction runs through all of it — AI fails in the operating model, not the model. The tool is not the transformation.

    Two things to start with:

    • A new flagship piece, A Reference Architecture for Agentic ITSM, on safely letting a GenAI agent act in ServiceNow — with the guardrail code that makes it work.
    • A new Start Here page that organizes the best of the blog by topic, so you can jump straight to what’s useful.

    What’s coming: reference architectures, build logs with real code, and benchmarks — on a steady cadence this time. No noise, just the work. If there’s a topic you want me to dig into, just reply — I read everything.

    — Deepak

  • A Reference Architecture for Agentic ITSM: Wiring GenAI into ServiceNow Without Losing Control

    A year ago I wrote about integrating generative AI into ITSM workflows. Back then, “integration” mostly meant a chatbot that summarized tickets and drafted responses. Useful, but bounded — the model suggested, a human acted.

    That line has moved. The interesting question in 2026 isn’t whether GenAI can read a ticket; it’s whether you can safely let it close one. The moment a model can take actions in your system of record — reassign an incident, run a diagnostic, restart a service, request a change — you’ve crossed from “assistant” into “agent.” And agents in production are an infrastructure and governance problem long before they’re a model problem. This is the clearest case I know of a principle I keep coming back to: AI fails in the operating model, not the model.

    This post is the reference architecture I’d stand up to do agentic ITSM properly: enough autonomy to actually save time, enough control that I’d let it near a production ServiceNow instance. I’ll show the components, the wiring, the code that matters, and — more importantly — the parts everyone underestimates.

    The failure mode we’re designing against

    Most agentic pilots fail the same way. Someone connects a capable model directly to an API with broad credentials, it works beautifully in the demo, and then in week three it confidently takes a wrong action on a real incident because a retrieved document was stale and nothing stopped it. The problem was never the model’s intelligence. It was that the architecture had no seam between “the model decided” and “the system did.”

    Which is the whole point in one line: the tool is not the transformation. A more capable model doesn’t close this gap — it just takes the wrong action more fluently. Everything below exists to create that seam.

    The reference architecture

    Five layers, each with a single responsibility. The discipline is keeping them separate — that separation is what makes the system auditable, testable, and safe to evolve.

      Trigger  (new/updated incident, chat, webhook)
         |
         v
      Retrieval Layer      RAG over KB, CMDB, past incidents
         |
         v
      Orchestration Layer  agent loop + tool-calling
         |
         v
      Policy & Guardrail Gate  --- restricted / low confidence --->  Human-in-the-loop
         |                                                            (approval queue)
      allowed + high confidence                                            |
         |                                                              approved
         v  <-------------------------------------------------------------+
      Action Layer  (scoped ServiceNow API)
         |
         v
      Observability  (traces, evals, audit log)  --->  back to Trigger

    1. Retrieval Layer. The agent is only as good as what it knows about this incident. Ground every decision in your own data: the knowledge base, the CMDB (what is this CI, what depends on it), and — the highest-signal source most teams ignore — the resolution history of similar past incidents. Retrieval quality, not model size, is usually the difference between a useful agent and a plausible-sounding one.

    2. Orchestration Layer. The agent loop: given the incident and retrieved context, decide the next step, optionally call a tool, observe the result, repeat until resolved or escalated. This is where model-provider choice lives — and it should be swappable, not hardcoded.

    3. Policy & Guardrail Gate. The seam. Every proposed action passes through a deterministic policy check before it can touch ServiceNow. This layer is plain code, not a model — because “what is this agent allowed to do” is a decision you must be able to read, test, and audit, not one you delegate to a probability distribution. This is where your operating DNA — how decisions get made and who is trusted to act — stops being culture and becomes code.

    4. Action Layer. A narrow, scoped adapter to ServiceNow. It exposes only the specific operations the agent is permitted to perform, each backed by a least-privilege service account. The agent never gets your admin token.

    5. Observability. Full traces of every decision, tool call, and action; an immutable audit log; and continuous evaluation. If you can’t answer “why did the agent do that, and was it right,” you can’t run this in production.

    The seam, in code

    Here’s the heart of it: the orchestration layer proposes an action, and the guardrail gate decides whether it executes, gets downgraded to a human approval, or is refused. This is deliberately boring, deterministic code — that’s the point.

    from dataclasses import dataclass
    from enum import Enum
    
    class Decision(Enum):
        ALLOW = "allow"
        REQUIRE_APPROVAL = "require_approval"
        DENY = "deny"
    
    @dataclass
    class ProposedAction:
        name: str            # e.g. "restart_service", "reassign", "add_work_note"
        target_ci: str       # configuration item the action touches
        params: dict
        confidence: float    # model-reported, treated as a hint, never as truth
    
    # Policy is data you can read, diff in git, and audit — not model output.
    POLICY = {
        "add_work_note":   {"max_blast_radius": "none",   "min_confidence": 0.0},
        "reassign":        {"max_blast_radius": "ticket", "min_confidence": 0.75},
        "restart_service": {"max_blast_radius": "service","min_confidence": 0.90,
                            "require_approval_for": ["prod"]},
        # default-deny: anything not listed cannot run
    }
    
    def evaluate(action: ProposedAction, ci_env: str) -> Decision:
        rule = POLICY.get(action.name)
        if rule is None:
            return Decision.DENY                      # default deny — the safe default
        if action.confidence < rule["min_confidence"]:
            return Decision.REQUIRE_APPROVAL
        if ci_env in rule.get("require_approval_for", []):
            return Decision.REQUIRE_APPROVAL
        return Decision.ALLOW

    Three design choices are doing all the work here:

    • Default deny. An action the policy doesn’t explicitly know about cannot run. New capabilities are added deliberately, in a reviewed change to the policy, not discovered by the model at runtime.
    • Confidence is a hint, never a gate on its own. A model saying it’s 0.99 sure is not evidence. Confidence lowers autonomy when low; it never grants autonomy on its own. High-blast-radius actions in prod route to a human regardless.
    • Blast radius is explicit. Every action is classified by what it can break. That classification, not the model’s eloquence, determines how much oversight it gets.

    The action layer itself stays thin and scoped:

    import os, requests
    
    class ServiceNowAdapter:
        """Exposes ONLY whitelisted operations, backed by a least-privilege account."""
        def __init__(self):
            self.base = os.environ["SN_INSTANCE_URL"]
            self.auth = (os.environ["SN_AGENT_USER"], os.environ["SN_AGENT_TOKEN"])
    
        def add_work_note(self, sys_id: str, note: str):
            return self._patch(sys_id, {"work_notes": note})
    
        def reassign(self, sys_id: str, group: str):
            return self._patch(sys_id, {"assignment_group": group})
    
        def _patch(self, sys_id: str, body: dict):
            r = requests.patch(f"{self.base}/api/now/table/incident/{sys_id}",
                               json=body, auth=self.auth, timeout=10)
            r.raise_for_status()
            return r.json()

    Notice what’s not here: there’s no generic “run arbitrary query” or “execute” method. The agent’s power is defined by the surface area of this class, and that surface area is a design decision you make on purpose.

    Deploy it like infrastructure, not like a notebook

    The fastest way to lose trust in an AI system is to run it as a hand-configured process someone SSHed in to start. Treat the whole stack as infrastructure-as-code from day one — reproducible, version-controlled, and reviewable. The service account, its exact permissions, the policy file, and the deployment all live in git:

    # ansible: provision the least-privilege ServiceNow integration account
    - name: Ensure agent service account has ONLY scoped roles
      vars:
        agent_roles:
          - itil            # read/update incidents
          - snc_read_only   # broad read, no write
      ansible.builtin.uri:
        url: "{{ sn_instance }}/api/now/table/sys_user_has_role"
        method: POST
        user: "{{ sn_admin_user }}"
        password: "{{ sn_admin_pass }}"
        body_format: json
        body:
          user: "{{ agent_sys_id }}"
          role: "{{ item }}"
      loop: "{{ agent_roles }}"
      # No admin, no security_admin. If the agent needs more, that's a reviewed PR.

    The point isn’t Ansible specifically — it’s that “what can this agent touch” should be a diff someone approved, not tribal knowledge.

    The parts everyone underestimates

    Evaluation is the hard part, not orchestration. Wiring an agent loop is a weekend. Knowing whether it’s getting better or worse over time is the real engineering. Build a regression set of real (anonymized) incidents with known-good resolutions and run it on every prompt or model change. Without this, every “improvement” is a guess.

    Retrieval decay will hurt you silently. A KB article gets superseded, a CMDB relationship changes, and the agent starts grounding decisions in stale truth — confidently. Freshness and provenance on retrieved context matter more than the model you pick.

    Cost is an architecture decision. Running a frontier model on every ticket update is how pilots die in the budget review. Route by difficulty: cheap/local models for triage and summarization, the expensive model only for genuinely ambiguous cases. Inference routing belongs in the design, not the invoice.

    Human-in-the-loop is a feature, not a fallback. The approval queue isn’t the system admitting defeat — it’s the mechanism that lets you start with tight autonomy and earn more as the eval data proves the agent is trustworthy for a given action class. Every autonomy level you grant is a conviction with a review date: stated strongly enough to act on, dated explicitly enough that your eval data is allowed to revise it.

    How I’d actually roll this out

    1. Shadow mode first. The agent proposes actions and writes them to work notes, but executes nothing. You get real eval data at zero risk.
    2. Graduate low-blast-radius actions. Let it add work notes and reassign autonomously once shadow-mode accuracy clears your bar.
    3. Keep high-blast-radius actions human-gated — indefinitely, if that’s what the risk math says. But don’t hide behind the gate forever, either: shadow mode with no graduation path is its own failure mode — professional neutrality, saying a great deal while committing to nothing. The discipline is interpretive courage: reading the eval data everyone can see and making the call to graduate an action when it has earned it.
    4. Instrument everything from step one. You cannot retrofit observability onto an incident you can’t explain.

    The takeaway

    Agentic ITSM is not a model you buy; it’s a system you architect — because, again, AI fails in the operating model, not the model. The model is the least differentiated part — it’s swappable, and it’s improving whether you do anything or not. Your durable advantage is the seam: the deterministic, auditable, least-privilege layer between what the agent decides and what your systems actually do. Get that right and you can adopt every model improvement safely for years. Skip it and you’ve built a very expensive way to take the wrong action quickly.

    Flexibility with guardrails is the real competitive advantage in enterprise AI. Is your architecture ready to let an agent act — or only to let it talk?

    The policy-gate, ServiceNow-adapter, and provisioning code from this post are runnable on GitHub: github.com/waghmaredb/vexpose-labs. If you’re building agentic workflows on your ITSM stack, I’d like to compare notes — reach me on LinkedIn or X.