Ship Reliable Terraform in 12 Minutes Per Pull Request
Italic sub-headline: A practical playbook for fast plans, safe state, and sane teams.
Define “Done”: What Terraform Must Guarantee
We’ve all seen the repo that “works on my laptop” until it meets production reality and crumbles under drift, secrets, and unpinned providers. Before we touch a single line of HCL, let’s define what “done” means for terraform in a production team. Our bar is simple: we can apply changes safely, fast, and repeatably. “Safely” means operations are idempotent, we have guardrails for policy and cost, and a roll-forward path if something misbehaves. “Fast” means a developer can open a PR and get a plan, tests, and policy checks in about 12 minutes, so feedback feels like a nudge, not a standstill. “Repeatably” means state is centralized and locked, modules are versioned, and environments don’t rely on tribal knowledge.
We’ll also agree to a few non-negotiables. First, everything is code: providers, versions, backends, and module inputs must be explicit. Second, the pipeline is the source of truth. We run terraform locally when exploring, but the final say comes from CI’s plan and policy outputs. Third, we design with ownership in mind: a team can reason about and apply their slice without stepping on other stacks. This usually pushes us toward smaller modules, clear outputs, and narrow blast radius.
Last, we accept that “done” isn’t a one-time event; it’s a loop. Our loop: propose, plan, test, check policy, tag, apply, verify, and watch for drift. If any part of that loop hurts, we fix the loop first—then we fix the code. That mindset keeps us honest and keeps our terraform usable when the pager goes off at 2 a.m.
Stop State Nightmares: Backends, Locks, and Bootstrap
If terraform is the brain, state is its memory. Forget to protect it and you’ll get split-brain chaos. We centralize state early with a real backend and write down the bootstrap steps so nobody “accidentally” stores state on their laptop. For AWS, S3 plus DynamoDB locking is boring and solid. GCS buckets with object locks, or Terraform Cloud/Enterprise work too. Whatever you pick, document the bootstrap and never commit a backend with placeholder values you’ll “fix later.” We’ve all seen later.
Here’s a minimal backend that sets expectations clearly:
terraform {
required_version = ">= 1.6.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
backend "s3" {
bucket = "my-tf-state-prod"
key = "network/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "my-tf-state-locks"
encrypt = true
}
}
Hot tip: you must create the S3 bucket and DynamoDB table before terraform init. Either run a tiny one-off “bootstrap” stack or use a short script you check into the repo next to a README that states who owns the state bucket and how it’s backed up. Keep IAM permissions tight: read/write for CI and owners, read-only for auditors, and deny public access. If you’re new to S3 backends and locking, the official Terraform S3 backend docs lay out the details, including encryption and versioning knobs that save real headaches during rollbacks and incident reviews.
Pick an Environment Strategy That Ages Well
Workspaces solve one problem and create two if we overuse them. We like workspaces for short-lived previews or genuinely identical stacks that differ only by a handful of variables. For long-lived environments (dev, staging, prod), separate directories or separate repos keep blast radius and state files small. Each environment should have its own backend config and variables so a mis-click in dev doesn’t quietly propose changes in prod. Yes, that means a little duplication; no, it’s not wasteful when the pager is chirping and you’re trying to find the right state.
Let’s keep naming boring and predictable: “env/app/component.” Keep module inputs consistent across environments, even if values differ. If an environment doesn’t need a feature flag or optional resource, default it off in variables, don’t rip out code. Also, make the “promote” path explicit. We prefer “merge to main creates a versioned module” and “env directories pin that version.” Promotion then becomes a module version bump with a plan/apply, not a risky cherry-pick festival.
A note on secrets: don’t stash them in tfvars files lying around on developer machines. Use your cloud’s secret manager or a pipeline secret store and wire inputs through variables. If you’re tempted to bake environment conditions deep into modules, pause and reframe: modules expose knobs; environments choose settings. That split keeps modules reusable and environments honest about their differences—no spaghetti if/else blocks sprinkled across HCL files.
Design Modules With Contracts, Not Wishes
A good module is a contract: clear inputs, stable outputs, and a promise not to surprise downstream consumers during a minor release. We pin providers with required_providers, we use semantic versioning for modules, and we avoid sneaking behavior changes into patch bumps. When a module evolves, we grow it with new variables and sensible defaults; we don’t yank existing outputs or flip defaults unless we bump the major version and communicate in the changelog.
Here’s the sort of module skeleton that rarely bites us:
// variables.tf
variable "name" {
type = string
description = "Base name for resources."
}
variable "tags" {
type = map(string)
default = {}
description = "Common tags to apply."
}
// main.tf
resource "aws_s3_bucket" "this" {
bucket = "${var.name}-data"
tags = var.tags
}
// outputs.tf
output "bucket_name" {
value = aws_s3_bucket.this.bucket
description = "The created bucket name."
}
The module guarantees a stable output bucket_name and a clear input name. We avoid risky magic: no secret default regions, no shadow variables. We keep inputs minimal and well-documented. Testing matters too. When possible, we write tiny sanity tests (even a smoke terraform plan against a throwaway workspace) and, for critical modules, a real integration test that creates and destroys infra. If you’ve never tried it, Terratest makes these integration tests tolerable by standing up resources, asserting properties, and cleaning up without tears. It’s not glamorous, but it’s exactly the kind of test that saves you from Friday-night surprises.
Make Every PR Plan in ~12 Minutes
Speed isn’t a luxury; it’s how we keep infra changes moving safely. Our target is simple: fmt, validate, lint, plan, test, and policy-check in about 12 minutes. We get there by caching providers and modules, scoping plans to changed directories, and running tasks in parallel where it’s safe. We also fail fast—if formatting or tflint gripes, we stop and let the developer fix it locally.
A GitHub Actions skeleton we like looks like this:
name: terraform-pr
on:
pull_request:
paths:
- "infra/**"
permissions:
contents: read
id-token: write
jobs:
plan:
runs-on: ubuntu-latest
concurrency: tf-${{ github.ref }}
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.8.5
- name: Format & Validate
run: |
terraform -chdir=infra fmt -check
terraform -chdir=infra init -input=false
terraform -chdir=infra validate
- name: Plan
run: |
terraform -chdir=infra plan -input=false -out=tfplan
- name: Upload Plan
uses: actions/upload-artifact@v4
with:
name: tfplan
path: infra/tfplan
We’d add tflint and tfsec where they help, and we’ll run a small Terratest job for modules that justify it. Caching modules (~/.terraform.d/plugin-cache and .terraform/providers) shaves precious minutes, as does scoping the job using a dir filter (only plan in directories touched by the PR). If your cloud supports short-lived federated credentials, use them; static keys in CI are yesterday’s mess. Keep logs chatty but actionable: if the plan fails, a human should know why without spelunking.
Ship Guardrails With OPA/Sentinel, Not Meetings
We like guardrails that run on every plan, not policies that live in a wiki. Bring rules into the pipeline so developers get instant feedback and reviewers don’t play compliance cop. If you’re on Terraform Cloud/Enterprise, Sentinel is a natural fit. If not, Open Policy Agent (OPA) with Conftest works well. The pattern is the same: parse the plan or config, run policies, fail the PR with a clear message when a rule is violated.
Here’s a tiny Rego example with Conftest that requires cost tags on AWS resources:
package main
deny[msg] {
input.resource_type == "aws_s3_bucket"
not input.values.tags["cost-center"]
msg := sprintf("Missing cost-center tag on %s", [input.address])
}
Your pipeline would render the terraform plan to JSON (terraform show -json tfplan > plan.json), run a translator or custom script to shape inputs, then call conftest test plan.json. Start with a handful of rules that actually matter—tagging, open ports, public buckets—and grow carefully. You can learn the basics fast with the OPA documentation, which has simple examples and recipes for CI integration.
We also prefer warnings before hard fails for new rules. A ramp period (two weeks of warnings, then fail) beats surprise PR breaks. And document the “why” for every rule in the repo. If a developer can’t figure out how to satisfy a policy in two minutes, the policy needs better errors—or we need to reconsider if it’s pulling its weight.
Catch Drift, Tag Ruthlessly, Tame Costs
Terraform can’t manage what it can’t see, and clouds don’t wait for PRs. People click consoles, managed services mutate under the hood, and drift happens. We schedule a nightly job to run terraform plan -detailed-exitcode against each environment and post the result in a channel with a human name on it. Exit code 2 means drift—someone triages, opens a PR, and we fix it in code. Keeping plans small per environment keeps those nightly checks fast and cheap.
Tags buy us visibility and cost controls, but only if they’re enforced. Our modules take a tags map and apply it everywhere possible. We back that up with policy: new resources without a cost-center and owner tag fail the PR. In AWS, Cost Allocation Tags become cost charts you can actually filter; surprise bills shrink when you can point to a line item and a team. If you need a framework to back your tagging and review habits, the AWS Well-Architected guidance on cost and operations is practical without being preachy.
On costs, resist premature tuning. First, meter with tags and budgets. Second, tackle the big rocks: oversized instances, idle RDS, and data egress. Terraform helps with right-sizing by making size a variable you can test in lower environments. And for anything scheduled (dev clusters, ephemeral workloads), give them a bedtime. A single schedule_enabled variable in a module that ties to an automation job can save more than a sprint’s worth of meetings about pennies.
Put It Together and Keep It Boring
Our recipe isn’t flashy, and that’s the point. We define “done” as safe, fast, repeatable changes. We centralize state with locks and write down bootstrap steps. We keep environments separate enough to reduce blast radius and predictable enough to promote confidently. We design modules like contracts, version them sanely, and test the parts that bite. We wire a pipeline that plans in about 12 minutes, caches the right things, and fails fast with useful errors. We enforce a few practical guardrails with OPA or Sentinel so humans review intent, not port numbers. And we keep the lights on with nightly drift checks, ruthless tagging, and a small set of cost habits that matter.
None of this requires a massive rewrite or a committee. Pick one pain point—state chaos, flaky modules, or slow plans—and fix that loop end-to-end. Then take the next one. Two or three loops later, you’ll notice PRs feel lighter, reviews faster, and apply windows less tense. That’s not magic. It’s the compounding effect of small, boring practices that make terraform a trusty tool instead of a weekly gamble. And if someone still insists on clicking “Create” in the console, at least you’ll find out tonight, not next quarter.
