Terraform State Locking During Tuesday’s IAM Cleanup

One stale lock, one real permission gap, and a policy we still dislike.

09:12.

That was when stand-up stopped being stand-up and turned into three people staring at a Terraform lock held by a GitLab runner that had supposedly finished forty minutes earlier. Someone had brought a box of almond croissants, which made the delay feel less hostile than it deserved.

09:12 — Stand-Up Opens On A Locked State

The ticket was small enough to sound harmless: remove the remaining broad access from the data-export role in our production analytics account. Finance had moved the nightly export job to a narrower S3 prefix two weeks earlier, and the old policy still allowed reads across the whole reports bucket.

We’d already approved the pull request on Monday. The plan had shown one IAM policy replacement and no resource destruction. By our standards, it was almost suspiciously tidy.

Then the scheduled apply job failed while acquiring the state lock. The Terraform state for this account still lives in an S3 backend with DynamoDB locking. We know newer backend options exist. We also know that “we should migrate state locking later” has been on our board since November, sitting beside “replace the office chair that squeaks during incident calls.”

A failed lock is not automatically a stale lock. That distinction matters more than people think. An engineer running a local apply from a train Wi-Fi connection is annoying. Two applies changing IAM policies against the same state are a much better way to spend Tuesday explaining ourselves to auditors.

We checked the GitLab job page first. It had not failed cleanly. The runner had lost contact during the apply stage, then sat in a “running” state long enough for GitLab to declare it dead. The EC2 runner instance had been replaced overnight by its autoscaling group. The lock remained.

Nobody touched force-unlock during stand-up. We’ve earned that restraint.

09:40 — We Find The State File Everybody Shares

By 09:40, we had the lock ID, the runner hostname, and the commit SHA from the previous night. The useful question was whether Terraform had made any changes before the runner disappeared.

Our analytics account state is larger than we’d like: 186 managed objects, including IAM roles, S3 bucket policies, EventBridge schedules, a few Glue jobs, and the KMS grants that always become someone’s problem eventually. It grew this way because “analytics” was a convenient boundary when there were two data jobs and one AWS account. We now have five account environments and a state file that makes reviewers scroll for longer than is healthy.

We checked CloudTrail for IAM policy changes around the runner’s last heartbeat. Nothing. We checked the S3 backend version history. No new state object version. The runner had acquired the lock, downloaded state, and disappeared before it changed AWS or wrote anything back.

That gave us enough confidence to remove the lock manually. Terraform’s own state-locking documentation is sensible on this point: locks prevent concurrent writes; they do not tell us whether a missing process was harmless. We still need evidence from the provider side.

We keep a short runbook for this. Not a twelve-page document with colour-coded arrows. Just the account, backend location, lock table, who can clear a lock, and the checks required before doing it. The last line says: “A green GitLab page is not evidence.” That line arrived after a green GitLab page turned out to be reporting an old runner status from a job that had already vanished.

10:25 — The Plan Finds A Permission We Had Missed

With the lock cleared, we ran the approved plan again from a fresh runner. The policy change itself was plain: remove a bucket-wide s3:GetObject permission and retain access to reports/finance-export/*.

The second plan differed from Monday’s plan.

Terraform wanted to add s3:ListBucket with a prefix condition. Monday’s version had not. We hadn’t changed the policy file. The difference came from a module release merged by another team earlier that morning. Their update corrected how the module built conditions for prefix-scoped S3 access. Our configuration picked it up because we had allowed a minor module version range.

This is why we pin Terraform providers and have a terraform.lock.hcl file, but we are less relaxed about internal module ranges than we used to be. Provider dependency locks are explained well in HashiCorp’s dependency lock file documentation. Modules need the same degree of suspicion, even when they live in our own GitLab group and have a reassuring name like aws-iam-basics.

The added permission was correct. An S3 client that reads objects often needs to list the prefix first, depending on how it discovers keys. Removing bucket-wide reads while leaving the job unable to list its approved prefix would have made the nightly export fail at 02:00, which is an unpleasant way to prove least privilege.

We paused the apply and asked the data engineering team to confirm the export’s call pattern. They replied with a CloudTrail event from last week: ListBucket, then GetObject against a timestamped CSV under the expected prefix. Good. Concrete evidence beats everybody’s memory of what a Python script “probably does.”

AWS’s policy evaluation logic is worth reading whenever an IAM change feels obvious. Explicit denies, identity policies, resource policies, permission boundaries, session policies: it is entirely possible to be right about the line you removed and wrong about the permission path that remains.

11:50 — We Leave The Provider Upgrade Alone

Before lunch, somebody noticed the runner image carried Terraform 1.9.8 while our newest platform image had 1.10.5. The AWS provider was at 5.72.1 in this repository, though several newer versions were available.

This produced the usual suggestion: while we are in here, should we update everything?

No.

We dislike opportunistic Terraform upgrades during access changes. A provider upgrade can alter a plan in ways that have nothing to do with the ticket in front of us. Sometimes that is exactly the point of upgrading. It deserves its own pull request, its own review, and preferably an afternoon where the person doing it does not also have pager duty.

Terraform plans are useful, but they are not contracts with AWS. They are predictions from a specific Terraform version, provider version, state snapshot, and set of credentials. Change one part of that set and we need to read the prediction again. The plan command reference says plenty about modes and options; the operational bit is simpler: save the plan we reviewed, then apply that plan from an environment we control.

This only held because our applies come from a small pool of disposable GitLab runners, all using the same container image and IAM role. We have not tried this setup across forty separate teams with their own runner images, and we do not pretend the habits would survive unchanged.

Lunch was reheated noodles eaten while someone explained why their new mechanical keyboard had “thock.” We agreed it made a noise.

13:20 — The Apply Produces One Useful Scare

Data engineering confirmed the prefix requirement just after one. We accepted the module change, regenerated the plan, and had the reviewer look at the rendered IAM document rather than trusting the resource count.

The apply completed in 38 seconds. We kept the terminal output in the ticket because the first failure and the successful retry told a clearer story together than a green pipeline badge did.

$ terraform apply tfplan-2026-08-27-1314

Acquiring state lock. This may take a few moments...
aws_iam_role_policy.data_export: Modifying... [id=data-export:read-finance-reports]
aws_iam_role_policy.data_export: Modifications complete after 3s [id=data-export:read-finance-reports]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Outputs:

data_export_role_arn = "arn:aws:iam::482761903155:role/data-export"

The useful scare came five minutes later. The export job’s pre-flight check failed in staging because its assumed-role session policy still allowed the old path but did not include the new list action. Terraform had done precisely what we asked. Our staging job wrapper had quietly supplied an extra session policy, which is the sort of detail nobody remembers until it blocks an otherwise correct change.

We fixed that in the job repository, not in Terraform. Giving the base IAM role broader access to compensate would have been lazy. Session policies are where we want this restriction, because the export job is the only caller that needs it.

By 13:48, the staging run had listed the prefix, read its test object, and exited. Production followed after the normal approval window.

14:35 — Review Turns Into A State Boundary Argument

The reviewer’s main comment was not about IAM. It was about why a data export policy shares state with Glue jobs, KMS grants, and EventBridge rules that have nothing to do with one another.

Fair.

We have been arguing for months about whether to split the analytics state by service ownership or by change frequency. Our preference is change frequency. IAM and bucket policies change in small, review-heavy increments; Glue jobs tend to move with application releases; KMS grants barely change until they suddenly ruin someone’s week. Putting them all in one state means an IAM cleanup takes a lock on work unrelated to IAM.

The opposing view is that splitting state produces more remote-state references, more permissions, and more ways to apply components in the wrong order. Also fair. Terraform dependency graphs are pleasant inside one state file and much less charming once we start wiring outputs across repositories.

We still think the current file is too broad. We do not think “one state per microservice” is an answer. That way lies 70 tiny repositories, each with a README last touched in 2024 and an owner who moved teams.

The unresolved part is KMS. Its key policies and grants cut across enough services that no proposed boundary looks clean. We can move the export role and bucket policy into their own state. We have not agreed where the encryption key belongs.

16:10 — Pager Duty Gets Dragged Into Terraform Again

At 16:10, our on-call engineer asked why they had needed to join a state-lock investigation for a change they did not own. It was a fair complaint, especially since they were already dealing with an alert from a flaky DNS health check.

Our current rule says the person on pager can approve emergency infrastructure changes, but should not be the only person clearing a production state lock. The reason is boring: lock removal has enough blast radius that tired judgment is a poor control. The irritating part is that our access model makes the on-call role one of only four roles allowed to do it.

We need to separate emergency visibility from emergency mutation. Right now, the same permission set grants read access to backend state metadata and the ability to remove a DynamoDB lock. That was convenient when the team was smaller. It now means an on-call engineer gets pulled into a routine recovery step merely because the rest of us designed the permissions around convenience.

We added a ticket to create a narrow break-glass role with audit logging and a short session duration. It will probably sit next to the state-splitting work for a while. Terraform backlog items reproduce when left alone.

17:05 — We Should Automate The Evidence, Not The Unlock

The part of this day worth automating is the evidence gathering after a failed apply: runner status, last state version, relevant CloudTrail events, and whether any resource changed after the lock was created. A bot can collect those facts into the incident ticket in under a minute.

We should not automate force-unlock just because the bot gets impatient.

The coffee was cold by then.

We can make stale-lock recovery less manual, but a human should still decide whether an absent process is truly absent. The lock is there to make us stop before two people write different stories into the same state file, and we have enough of those already.

Tags: devopsTerraform