Lab: The Plan That Would Have Deleted Production

The situation

A pull request titled “Apply tagging standard to all resources” adds a tags block to a shared module. The diff is eleven lines. Three people have already approved it.

You are the fourth reviewer and you have the plan output. It is 340 lines and covers 47 resources.

The plan summary

Plan: 2 to add, 43 to change, 2 to destroy.

That line is the one that should stop you. A tagging change should be 0 to add, N to change, 0 to destroy.

What you have

Rather than scrolling the output, you query it:

terraform show -json tfplan > plan.json

jq -r '
  .resource_changes[]
  | select(.change.actions != ["no-op"])
  | "\(.change.actions | join("+"))\t\(.address)"
' plan.json | sort

create aws_cloudwatch_log_group.auditcreate aws_sns_topic.alertsupdate aws_instance.api[0]update aws_instance.api[1]update aws_lb.public… (40 more updates)delete+create aws_db_instance.ordersdelete+create aws_elasticache_cluster.sessions

Two resources will be destroyed and recreated. Terraform counted them as “2 to destroy” — which is accurate, and which reads like something small when it is at the end of a 340-line plan.

The detail

# aws_db_instance.orders must be replaced-/+ resource “aws_db_instance” “orders” { ~ id = “orders-prod” -> (known after apply) ~ tags = { + “cost-center” = “platform” } ~ db_subnet_group_name = “prod-db-subnets” -> “prod-db-subnet-group” # forces replacement ~ endpoint = “orders-prod.xxx.rds.amazonaws.com” -> (known after apply) }

# aws_elasticache_cluster.sessions must be replaced-/+ resource “aws_elasticache_cluster” “sessions” { ~ tags = { + “cost-center” = “platform” } ~ cluster_id = “sessions” -> “sessions-prod” # forces replacement }

Stop and work it

1. Which resource loses data, and which does not?
2. Neither replacement is caused by the tagging change. What actually caused each one?
3. If you applied this and it went wrong, what would rolling back restore?

---

Working it through

The database

aws_db_instance.orders is being replaced because db_subnet_group_name changed from prod-db-subnets to prod-db-subnet-group. That attribute forces replacement.

Replacement of an RDS instance means: the existing instance is destroyed, and a new empty one is created. The data is gone, unless a final snapshot is taken and manually restored — and skip_final_snapshot settings vary.

The endpoint also changes. Every application holding the old endpoint would fail even after a restore.

The cache

aws_elasticache_cluster.sessions is being replaced because cluster_id changed. Also a replacement — but the contents are sessions, which are ephemeral by design.

The impact is different in kind: no permanent data is lost, but every logged-in user is signed out at once, and the resulting re-authentication surge hits your identity provider and database simultaneously.

Serious, but recoverable. Not the same category as the database.

What actually caused them

Neither is caused by the tagging change. Both are drift being corrected.

Someone renamed the subnet group and the cache cluster in the console, at some point in the past. Terraform state still holds the old names, and nobody has run a plan against this module since — because nobody had reason to.

The tagging change is the first plan in weeks. It did not cause these; it revealed them.

That is worth sitting with. The pull request is innocent. The dangerous changes were already waiting in state, and would have been triggered by whichever plan ran next.

What a rollback would restore

Almost nothing that matters:

• The database: reverting the code and applying creates another new empty instance. The data is not restored by Terraform. Recovery requires a snapshot restore, and the endpoint would change again.
• The cache: recreated empty. Sessions are gone either way.
• Anything holding the old endpoints: still broken until reconfigured.

The rollback plan for this apply is a restore plan, and it would have to have been written and tested before the apply — not improvised afterwards.

What should happen

Do not approve. The pull request is fine; the plan is not.

Then, in order:

1. Reconcile the drift separately, and deliberately. Update the configuration to match the real names, so Terraform stops wanting to change them:

   db_subnet_group_name = "prod-db-subnet-group"   # matches reality

   Now the plan shows no change for that attribute, because there is no longer a difference.

2. Add guards to anything holding data:

   resource "aws_db_instance" "orders" {
     # ...
     lifecycle {
       prevent_destroy = true
     }
   }

   With this in place, the plan would have failed rather than proposing a destroy. That is the correct behaviour: an error at plan time instead of an outage at apply time.

3. Then apply the tagging change, which should now be 0 to add, 43 to change, 0 to destroy.

4. Add drift detection, so the next occurrence is a scheduled finding rather than a surprise during an urgent change:

   terraform plan -refresh-only -detailed-exitcode
   # exit 2 means drift; alert on it

Prevention

# In CI: fail the check if the plan contains any destroy
DESTROYS=$(jq -r '
  [.resource_changes[] | select(.change.actions | index("delete"))] | length
' plan.json)

if [ "$DESTROYS" -gt 0 ]; then
  echo "Plan contains $DESTROYS destructive change(s). Requires explicit approval."
  jq -r '.resource_changes[] | select(.change.actions | index("delete")) | .address' plan.json
  exit 1
fi

This turns “nobody noticed line 287” into “somebody deliberately approved a destroy”. It is a dozen lines of CI and it is the highest-value control in Terraform operations.

The Terraform Plan Analyzer does the same analysis interactively, and escalates destroys of stateful and access-controlling resource types.

What to take away

Read the summary line first. N to destroy on a change that should destroy nothing is the whole review, and it takes one second.

Never scan a large plan visually. -/+ and ~ are two characters apart in scrolling output. Query the JSON.

# forces replacement is the most important comment in Terraform. Find those before reading anything else.

A plan reflects everything that has drifted, not just what you changed. The first plan after a quiet period carries every accumulated difference, and the person who happens to run it inherits all of them.

prevent_destroy converts an outage into an error message. Put it on everything that holds data you cannot recreate.