{{ vExpose }}.Blog

Category: Field Notes

Short troubleshooting posts — one real failure, the wrong diagnosis, the actual cause, the fix.

  • Field Note: The Helm Upgrade That Rolled Itself Back

    Field Note — one real failure, the wrong diagnosis, the actual cause, the fix.

    A Helm upgrade that rolls itself back is not a Helm bug. It is Helm doing precisely what you told it to do — and the gap between what you told it and what you meant is the whole lesson.

    The symptom

    The deploy pipeline ran helm upgrade, waited, and reported failure. Minutes later the cluster was serving the previous version as if nothing had happened. helm history showed the new revision appear and then a rollback to the old one, all automatic. The application logs for the new version were already gone with the pods. It looked like Helm had panicked and reverted on its own.

    The wrong diagnosis

    “Helm is flaky — pin a different version, add a retry.” That treats a symptom as the disease. The rollback was not random and it was not a defect. It was configured behaviour, working exactly as designed.

    The actual cause

    The upgrade ran with --atomic. That flag has a clear contract: if the release does not become healthy within the timeout, undo everything and return to the last good state. Healthy is defined by --wait — every pod ready, by its own readiness probe, within the deadline. The new version took longer to warm its cache than the readiness probe allowed, so Kubernetes never marked it ready, so Helm concluded the upgrade had failed, so --atomic did the honourable thing and rolled back. Every component behaved correctly. The only thing wrong was the definition of “ready.”

    # What was run:
helm upgrade app ./chart --atomic --timeout 2m

# --atomic : on failure, roll back to the previous revision
# --wait   : "success" = all pods pass readiness within --timeout
#
# The probe's initialDelay + threshold exceeded 2m during cache warm-up.
# Helm never saw "ready" -> declared failure -> rolled back. As designed.

    The fix

    Two changes, both about honesty. Make the readiness probe describe real readiness — add a startup probe for the cache-warming phase, so a slow start is not mistaken for a failure. And align the Helm timeout with the worst-case healthy start, not the average one. Keep --atomic. It is not the problem; it is the safety net working. Once “ready” actually meant ready, the rollback stopped firing, because there was nothing to roll back from.

    The reframe

    The automation did not fail. It obeyed. The bug was upstream of the tool — in what you declared “success” to mean. Most automation incidents are not the machine doing the wrong thing. They are the machine doing exactly the thing you specified, and revealing that your specification was wrong.

  • Field Note: The GPU That Was Busy Doing Nothing

    Field Note — one real failure, the wrong diagnosis, the actual cause, the fix.

    A GPU can be ninety-five percent utilised and still be doing almost nothing you are paying it to do. This is the most expensive gap between a metric and the truth in modern infrastructure.

    The symptom

    An inference service was slow. nvidia-smi reported 95% GPU utilisation, so the dashboard was green and the conclusion wrote itself: we are out of GPU, buy more. The throughput told a different story — roughly a third of the tokens per second the same hardware reached in the vendor benchmark.

    The wrong diagnosis

    “Utilisation is at 95%, so the GPU is the bottleneck — add capacity.” Every part of that sentence sounds right, and the last clause is wrong. More GPUs would have tripled the bill and moved the throughput barely at all.

    The actual cause

    GPU utilisation, as nvidia-smi reports it, means exactly one thing: at least one kernel was running during the sampling window. It says nothing about whether that kernel was doing useful arithmetic or waiting on memory. Our service was running with a batch size of one, generating tokens for a single request at a time. The GPU spent most of its cycles moving weights in and out of memory rather than computing — memory-bandwidth bound, not compute bound. Busy and idle at the same time.

    The fix

    Measure the right thing, then feed the hardware properly. The honest metrics are tokens per second under load and model FLOPs utilisation (MFU) — how much of the GPU’s theoretical compute you are actually using. Both were dismal. Enabling continuous batching, so concurrent requests share each forward pass, and right-sizing the batch took throughput most of the way to the benchmark on the very same cards.

    # The metric that lies:
nvidia-smi --query-gpu=utilization.gpu --format=csv   # 95% — "busy"

# The metrics that don't:
#   tokens/sec at target concurrency   (throughput you actually ship)
#   MFU = achieved FLOPs / peak FLOPs  (useful work vs. busy work)
#
# High utilization + low MFU = memory-bound.
# Fix batching before you buy silicon.

    The reframe

    Utilisation tells you a GPU is busy. It will never tell you the GPU is useful. Buy the second card when the first one is doing useful work — not when it is merely busy doing nothing.

  • Field Note: The Drift That Only Shows Up at 2 a.m.

    Field Note — a recurring series: one real failure, the wrong diagnosis, the actual cause, the fix.

    Infrastructure as Code was supposed to end configuration drift. It didn’t. It moved drift somewhere harder to see — into the gap between your state file and the world that file claims to describe.

    The symptom

    The pull request was clean. terraform plan showed no changes, two reviewers approved, it merged. Nine days later an unrelated change to a load balancer produced a plan that wanted to destroy a subnet nobody had touched — at 2 a.m., because that is when the on-call apply ran.

    The wrong diagnosis

    “Someone changed it by hand.” That is the reflex, and occasionally it is right. But blaming a human is the comfortable answer, because it means the tooling is fine and the process just needs more discipline. It usually isn’t the human.

    The actual cause

    State is a photograph, not a live feed. It records what the world looked like the last time you ran apply. Between applies, the world moves: a security team edits an ingress rule, an autoscaler rewrites a tag, a cloud provider ships a new default for a resource you declared three years ago. Your code did not drift. Reality drifted — and your state never looked again, until an unrelated apply forced a refresh and reconciled the difference destructively.

    The fix

    Stop discovering drift as a side effect of unrelated work. Detect it on a schedule. A nightly job that runs a read-only plan and treats a non-empty diff as a signal — not an error to auto-correct — turns drift from a 2 a.m. surprise into a morning ticket.

    # nightly-drift.sh — read-only, alert on drift, never auto-apply
terraform init -input=false
terraform plan -detailed-exitcode -lock=false -input=false
code=$?

# 0 = no drift | 2 = drift detected | 1 = error
if [ "$code" -eq 2 ]; then
  notify "Drift detected in $(terraform workspace show) — review before next apply"
fi
exit $code

    Two disciplines make the detection trustworthy. Lock your state so a scheduled plan never races a real apply. And scope who and what can mutate infrastructure out of band — the fewer hands outside the pipeline, the quieter the diff.

    The reframe

    Drift is not a failure of Infrastructure as Code. It is Infrastructure as Code telling you the truth about who else has their hands on your infrastructure. The only real choice is when you hear it: on a schedule you control, or at 2 a.m. in the middle of something else.