Terraform Production Practices — Plan Review, State, and Blast Radius

The plan is the safety mechanism. Code review tells you what somebody intended; the plan tells you what is about to happen to production. Most Terraform incidents are plans that were approved without being read properly, and they are preventable mechanically.

Read the plan with a tool, not with your eyes

Terminal plan output uses +, ~, -, and -/+. Those symbols are two characters apart in a wall of scrolling text, and -/+ — a replacement, which is a destroy — is easy to read as a modification when there are two hundred resources in the output.

Do not scan it. Query it.

terraform plan -out=tfplan
terraform show -json tfplan > plan.json

# Everything that will be destroyed or replaced
jq -r '
  .resource_changes[]
  | select(.change.actions | index("delete"))
  | "\(.change.actions | join("+")): \(.address)"
' plan.json

# Counts, for the change record
jq -r '
  [.resource_changes[] | select(.change.actions != ["no-op"]) | .change.actions | join("+")]
  | group_by(.) | map({action: .[0], count: length}) | .[]
  | "\(.action)\t\(.count)"
' plan.json

The Terraform Plan Analyzer does exactly this in your browser, including escalating destroys of stateful and access-controlling resource types.

Make destroys fail loudly

resource "aws_db_instance" "primary" {
  # ...
  lifecycle {
    prevent_destroy = true
  }
}

prevent_destroy turns an accidental destroy from an outage into an error at plan time. Apply it to every resource holding data you cannot recreate: databases, object storage, volumes, key material, DNS zones.

Removing it is a code change, which means it goes through review — which is exactly the friction you want on the path to deleting a production database.

lifecycle {
  create_before_destroy = true
}

For resources that can be replaced safely, create_before_destroy builds the replacement before removing the original, which eliminates the availability gap in the middle. It requires that the resource can exist twice — names and other unique attributes need to be generated rather than fixed.

Why things get replaced unexpectedly

Replacements are usually caused by a change to an attribute the provider marks as ForceNew. The plan says which attribute:

# aws_db_instance.primary must be replaced
-/+ resource "aws_db_instance" "primary" {
      ~ availability_zone = "eu-west-1a" -> "eu-west-1b" # forces replacement

Read the # forces replacement comments first, before anything else in the plan. They are the part that turns a routine apply into an incident.

The other cause is drift: someone changed the resource outside Terraform, and Terraform intends to correct it. That correction can be a replacement.

terraform plan -refresh-only    # what has drifted, without proposing changes

Running a refresh-only plan on a schedule and alerting on non-empty output turns drift from a surprise during an urgent deploy into a routine finding.

State structure decides blast radius

One state file for an entire estate means every apply touches everything, every lock blocks everyone, and one corrupted state loses the lot.

Split by blast radius and change frequency, not by service boundary alone:

infra/
  network/           # changes rarely, breaks everything
  data/              # databases, storage -- changes rarely, high value
  platform/          # clusters, shared services
  services/api/      # changes daily, low blast radius
  services/worker/

The rule of thumb: if applying one component could break another, and they change at different rates, separate them. Pass values between them explicitly through data sources or published outputs rather than reaching into another component’s state.

State must be remote, locked, versioned, and encrypted:

terraform {
  backend "s3" {
    bucket         = "example-tfstate"
    key            = "platform/terraform.tfstate"
    region         = "eu-west-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

Versioning on the state bucket is what lets you recover from a bad state operation. It has saved more people than any other single setting here.

The apply loop

# 1. Plan to a file. Never apply from a plan you did not save.
terraform plan -out=tfplan

# 2. Analyse it mechanically
terraform show -json tfplan > plan.json
jq -r '.resource_changes[] | select(.change.actions | index("delete")) | .address' plan.json

# 3. Apply the saved plan -- not a fresh one
terraform apply tfplan

Step 3 is the one people skip. terraform apply without a plan file re-plans against current state, which may have changed since you reviewed it. Applying the saved plan guarantees that what you approved is what runs, and Terraform will refuse the plan if state has moved underneath it.

Rollback is not automatic

There is no terraform rollback. Reverting the code and applying gets you back to the previous configuration, which is not the same as the previous state:

• Destroyed resources are recreated empty. Data is not restored.
• Recreated resources have new identifiers — new IPs, new ARNs, new endpoints. Anything holding the old identifier is now pointing at nothing.
• Resources you cannot recreate at all (some are region-unique or have irreversible settings) stay gone.

Which means the rollback plan for a destructive change is a restore plan, and it needs to be written and validated before the apply, not after it.

Modules

Modules are for reuse, not for organisation. A module that is used once is indirection with no benefit — it makes the plan harder to read and hides the resources from review.

Good module boundaries have a small, stable interface and hide genuine complexity. Bad ones expose forty variables that pass straight through to a single resource, which is a resource with extra steps.

Pin module versions. A module resolved from a branch means your infrastructure can change because someone merged something, and you will not find out until you plan.

CI

The pipeline for infrastructure differs from application CI in one important way: plan output is the review artifact.

1. terraform fmt -check and terraform validate — cheap, fast, catch typos
2. terraform plan -out=tfplan, posted to the pull request in full
3. Automated analysis of the JSON plan; fail the check if destroys are present without an explicit approval label
4. Apply on merge, using the saved plan from step 2

Making destroys require a deliberate additional approval is the single highest-value control in this list. It converts “I did not notice that line” into “somebody consciously approved a destroy”.

Checklist

• Plan saved to a file, analysed as JSON, and applied from that file
• Destroys and replacements enumerated mechanically before approval
• prevent_destroy on every resource holding irreplaceable data
• State remote, locked, encrypted, and versioned
• State split by blast radius and change frequency
• Drift detected on a schedule with -refresh-only
• Module versions pinned
• Destructive applies carry a written, tested restore plan
• CI fails on unexpected destroys rather than warning about them