Field Note: The Drift That Only Shows Up at 2 a.m.

Field Note — a recurring series: one real failure, the wrong diagnosis, the actual cause, the fix.

Infrastructure as Code was supposed to end configuration drift. It didn’t. It moved drift somewhere harder to see — into the gap between your state file and the world that file claims to describe.

The symptom

The pull request was clean. terraform plan showed no changes, two reviewers approved, it merged. Nine days later an unrelated change to a load balancer produced a plan that wanted to destroy a subnet nobody had touched — at 2 a.m., because that is when the on-call apply ran.

The wrong diagnosis

“Someone changed it by hand.” That is the reflex, and occasionally it is right. But blaming a human is the comfortable answer, because it means the tooling is fine and the process just needs more discipline. It usually isn’t the human.

The actual cause

State is a photograph, not a live feed. It records what the world looked like the last time you ran apply. Between applies, the world moves: a security team edits an ingress rule, an autoscaler rewrites a tag, a cloud provider ships a new default for a resource you declared three years ago. Your code did not drift. Reality drifted — and your state never looked again, until an unrelated apply forced a refresh and reconciled the difference destructively.

The fix

Stop discovering drift as a side effect of unrelated work. Detect it on a schedule. A nightly job that runs a read-only plan and treats a non-empty diff as a signal — not an error to auto-correct — turns drift from a 2 a.m. surprise into a morning ticket.

# nightly-drift.sh — read-only, alert on drift, never auto-apply
terraform init -input=false
terraform plan -detailed-exitcode -lock=false -input=false
code=$?

# 0 = no drift | 2 = drift detected | 1 = error
if [ "$code" -eq 2 ]; then
  notify "Drift detected in $(terraform workspace show) — review before next apply"
fi
exit $code

Two disciplines make the detection trustworthy. Lock your state so a scheduled plan never races a real apply. And scope who and what can mutate infrastructure out of band — the fewer hands outside the pipeline, the quieter the diff.

The reframe

Drift is not a failure of Infrastructure as Code. It is Infrastructure as Code telling you the truth about who else has their hands on your infrastructure. The only real choice is when you hear it: on a schedule you control, or at 2 a.m. in the middle of something else.