Debugging Terraform + Helm Drift on EKS: Failure Behaviours and Fixes

A field tutorial based on a real incident removing a stale CriticalAddonsOnly toleration from metrics-server and KEDA on EKS Auto Mode clusters managed by Terraform Cloud (TFC) + the eks-blueprints-addons module.

---

TL;DR

We tried to remove a Kubernetes toleration that was set through a Terraform helm_release. The code change looked trivial, but it took several apply cycles before the cluster actually changed. Along the way we hit five distinct failure behaviours, none of which threw an error — they all silently did nothing. This post documents each one, the symptom, the root cause, and the fix, plus the mental models and commands that make these problems obvious.

The single most useful signal throughout: the pod’s AGE. If a Helm value truly changed the rendered manifest, the pod rolls and AGE resets. If AGE doesn’t reset, nothing actually changed — no matter what Terraform or Helm said.

---

Background: the change we wanted

EKS Auto Mode clusters here run a single built-in general-purpose NodePool. There is no system NodePool, so nothing in the cluster carries the CriticalAddonsOnly:NoSchedule taint.

metrics-server (and KEDA) had this toleration set via Terraform:

metrics_server = {
  values = [yamlencode({
    tolerations = [{
      key      = "CriticalAddonsOnly"
      operator = "Exists"
      effect   = "NoSchedule"
    }]
  })]
}
Code language: JavaScript (javascript)

Mental model #1 — a toleration is permissive, not attractive

A toleration does not force, attract, or pin a pod to a tainted node. It only permits a pod to land on a node that carries the matching taint. With no system NodePool, there is no matching taint, so the toleration matches nothing and has zero effect. It is dead config.

Removing it is pure cleanup — it never changed where the pod ran. The pod was always on general-purpose, with or without the toleration.

That made it a perfect low-risk change… and a perfect way to surface every Terraform/Helm drift trap, because the expected runtime effect was nil and we had to rely on tooling signals to know whether anything happened.

---

Failure Behaviour #1 — Local edits never reached the pipeline

Symptom: Edited the .tf file, “applied,” but the cluster was unchanged. helm get values still showed the toleration; the pod AGE was unchanged.

Root cause: TFC builds from the VCS-connected branch, not from local working-tree files. Editing a file on the laptop (or even committing it to a feature branch) does nothing until that commit is on the branch the TFC workspace tracks (here, main).

How we detected it:

git status -sb                 <em># showed work was on a feature branch, not main</em>
git show origin/main:<file>    <em># main still contained the toleration</em>
git log origin/main..HEAD      <em># the fix commit was not in main</em>
Code language: HTML, XML (xml)

Fix: Merge the change into the branch TFC tracks, then apply.

Lesson: “I applied it” means the pipeline applied the merged code, not “I saved the file.” Always confirm which commit the run used.

---

Failure Behaviour #2 — Stale origin/main gave a wrong conclusion

Symptom: We concluded “main doesn’t have the fix” — but it actually did. The PR had already been merged.

Root cause: Local remote-tracking refs are a snapshot from the last fetch. git show origin/main:... reads that stale snapshot, not the live remote.

Fix:

git fetch origin --quiet
git show origin/main:<file>                       <em># now reflects reality</em>
git merge-base --is-ancestor <sha> origin/main && echo "in main"
Code language: HTML, XML (xml)

Lesson: Always git fetch before reasoning about remote branch state. Most “the merge didn’t happen” panics are stale refs.

---

Failure Behaviour #3 — Helm reported “updated”, but nothing rolled

Symptom: The TFC apply log said:

Plan: 0 to add, 2 to change, 0 to destroy.
... module....metrics_server.helm_release.this[0]: Plan to update
updated  id=metrics-server

A new Helm revision was created. Yet:

helm -n kube-system get values metrics-server   <em># toleration STILL present</em>
kubectl get pods -l ... -o wide                  <em># pod AGE unchanged (47h)</em>
Code language: HTML, XML (xml)

Root cause: Two things were true at once:

1. The run executed against a commit that still had the toleration (timing / pinned-commit issue — see #1/#2). So the values Helm received were unchanged in the part that matters.
2. A Helm upgrade always bumps the revision number, even when the rendered manifest is identical. A new revision is not proof of a real change.

Mental model #2 — revision number is not a diff. The only reliable evidence that the manifest changed is the rolled pod (fresh AGE) and the actual rendered output (helm get manifest).

How we detected it:

helm -n kube-system history metrics-server          <em># revision incremented...</em>
kubectl get pods -l app.kubernetes.io/name=metrics-server -o wide  <em># ...but AGE did NOT reset</em>
Code language: HTML, XML (xml)

Lesson: Verify outcomes by observed cluster state, not by Terraform’s “updated” or Helm’s revision counter. Pod AGE is the truth.

---

Failure Behaviour #4 — The Terraform helm_release drift blind spot

This was the core, non-obvious one.

Symptom: After the fix was genuinely in main, a fresh TFC run reported “No changes” — while the live release still carried the toleration.

helm -n kube-system get values metrics-server
<em>

<em>
Code language: PHP (php)

Root cause: The Terraform Helm provider compares your config against Terraform state — it does not re-read and diff the live release’s actual values on every plan. Once a prior run wrote “no toleration” into state, config == state, so Terraform sees nothing to do — even though the live cluster release had drifted and still had the toleration.

   config (no toleration)  ==  state (no toleration)   -> "No changes"
                                       ≠
                          live release (HAS toleration)   <-- invisible to plan

Mental model #3 — helm_release trusts state, not the cluster. Out-of-band drift in a release’s rendered values is frequently not detected by a normal plan. “No changes” means “config matches state,” not “cluster matches config.”

How we confirmed it:

<em># What the live release actually has:</em>
helm -n kube-system get values metrics-server | grep -i -A3 toleration   <em># present</em>

<em># What Terraform state thinks it has:</em>
terraform state show '...metrics_server.helm_release.this[0]' | grep -i -A3 toleration  <em># absent</em>
Code language: HTML, XML (xml)

Present in one, absent in the other = confirmed drift the plan won’t catch.

Fixes (any one works)

A. Force a real value change (VCS-native, no special access). Make the values genuinely differ from state so Terraform must run helm upgrade, which re-renders from current config and drops the toleration:

metrics_server = {
  values = [yamlencode({
    podLabels = { "k8s-app" = "metrics-server" }   
  })]
}
Code language: PHP (php)

B. Force-replace the release (the literal “force apply”). TFC has no UI button for targeted replace; use the API replace-addrs or a CLI-driven run:

<em>
terraform apply -replace='module....metrics_server.helm_release.this[0]'
Code language: PHP (php)

<em>
curl -s \
  --header "Authorization: Bearer $TFC_TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  --request POST \
  --data '{
    "data": {
      "type": "runs",
      "attributes": {
        "message": "force replace metrics-server (toleration drift)",
        "replace-addrs": ["module....metrics_server.helm_release.this[0]"]
      },
      "relationships": { "workspace": { "data": { "type": "workspaces", "id": "ws-XXXX" } } }
    }
  }' \
  https:
<em>
Code language: PHP (php)

We used Option A — it goes through the normal PR → apply gate and is auditable.

Lesson: When plan says “no changes” but the cluster disagrees, you have state↔cluster drift. Don’t keep re-running plan — force the resource to reconcile (value change or -replace).

---

Failure Behaviour #5 — Branch hygiene / “fix is in a feature branch”

Symptom: The fix existed and was committed… on an unmerged feature branch, while the workspace deployed from main.

Fix: Standard hygiene — confirm the fix is on the deployed branch:

git fetch origin
git merge-base --is-ancestor <fix-sha> origin/main && echo "deployable" || echo "still in a branch"
Code language: PHP (php)

Lesson: “Committed” ≠ “deployed.” Confirm the SHA the workspace will use.

---

What success finally looked like

$ helm -n kube-system history metrics-server
REVISION  UPDATED                   STATUS       ...
1         Wed Mar  4 09:53:46 2026  superseded
2         Fri Jun 19 04:25:37 2026  superseded
3         Fri Jun 19 05:21:07 2026  deployed     <em>

$ helm -n kube-system get values metrics-server
USER-SUPPLIED VALUES:
podLabels:
  k8s-app: metrics-server          <em>

$ kubectl -n kube-system get pods -l app.kubernetes.io/name=metrics-server -o wide
NAME                              READY  STATUS   RESTARTS  AGE     ...
metrics-server-64585697f6-nbgsg   1/1    Running  0         2m39s   <em>
Code language: PHP (php)

New revision + value gone + pod AGE reset = genuinely done.

---

Diagnostic cheat sheet

Question	Command
Which commit/branch did the run use?	TFC run detail → “Configuration / Commit”; locally git fetch && git log origin/main..HEAD
Is my fix actually in the deployed branch?	git merge-base --is-ancestor <sha> origin/main
What did Helm actually deploy?	helm -n <ns> get manifest <release>
What user values are live?	helm -n <ns> get values <release>
Did a new revision deploy?	helm -n <ns> history <release>
Did the pod actually roll?	kubectl -n <ns> get pods -l <selector> -o wide → check AGE
What does TF state think?	terraform state show '<addr>'
Are there NodePools/taints that matter?	kubectl get nodepools.karpenter.sh; kubectl get nodepool <n> -o jsonpath="{.spec.template.spec.taints}"

---

Key takeaways

1. Toleration ≠ placement. A toleration only permits scheduling onto a matching taint; with no such taint it is inert. Removing inert config is cleanup, not a behaviour change.
2. Pod AGE is the source of truth. Terraform “updated” and Helm revision bumps can both happen with no real change. A rolled pod proves it.
3. helm_release compares config to state, not to the cluster. “No changes” can hide live drift. Reconcile with a forced value change or -replace.
4. “Applied” means the pipeline applied the merged code. Always verify the commit/branch the run used; git fetch before trusting remote refs.
5. Verify on the cluster, not in the plan. helm get values / get manifest / pod AGE beat any amount of staring at Terraform output.

---

Generated from an incident review. Commands shown for EKS Auto Mode + Terraform Cloud + aws-ia/eks-blueprints-addons; the principles apply to any Terraform + Helm provider setup.