Leadership Mistakes We Put Into Production Before Lunch
Six management failures we shipped alongside perfectly valid infrastructure changes.
At 11:42 on Monday, the billing-export pull request had four approvals, one frustrated staff engineer, and no merge. Nothing in the diff was controversial. It changed a retry timeout from 30 seconds to 45. The problem was that our repository rules required a platform lead to approve anything touching infra/, and both platform leads were in a quarterly planning meeting.
We wrote that rule six months earlier. We then spent three hours discovering what it had actually done to the team.
We Made Leads The Default Merge Queue
What we did
We put @platform-leads in CODEOWNERS for most infrastructure paths. Any Terraform module, Kubernetes manifest, Helm chart, or GitHub Actions workflow waited for one of two people.
# Bad: .github/CODEOWNERS
/infra/ @platform-leads
/kubernetes/ @platform-leads
/.github/workflows/ @platform-leads
It looked orderly in the repository settings screen. We had senior review on high-impact changes. Nobody could accidentally modify production wiring on a quiet Friday afternoon. The fact that “quiet Friday afternoon” is when people make choices they regret did occur to us.
Why it looked reasonable
We had just hired three engineers who had not worked with our AWS accounts before. At the same time, a state-lock incident had left an S3 bucket policy too open for 47 minutes. We wanted a brake.
GitHub’s CODEOWNERS rules make this easy to set up, and easy to overuse. A broad path ownership rule feels like governance because it produces approvals. It doesn’t tell us whether the approval added useful scrutiny.
What it cost us
Between 4 and 15 August, 31 pull requests waited longer than two business hours for a required reviewer. Nine were harmless dependency bumps. Six were fixes to alerts already paging people. One was the retry timeout.
The less obvious cost was learned helplessness. Engineers started asking, “Can you just take this?” before they had read the plan output. Leads got worse at reviewing because our queue had become full of changes too small to deserve deep attention.
Our review time became an interrupt-driven service, which is a fairly miserable thing to run.
The fix
We moved ownership to the people who operate each service and kept lead review for a short list: identity, network boundaries, account creation, and production deletion paths.
# Corrected: .github/CODEOWNERS
/services/billing-export/ @billing-export-owners
/modules/retry-policy/ @platform-runtime
/modules/iam-boundary/ @platform-leads
/infra/accounts/ @platform-leads
We also added a risk:high label that requires one lead approval, rather than pretending every YAML file carries the same consequence. We pinned terraform 1.8.5 in .tool-versions on 2026-08-18, so reviewers were at least looking at plans produced by the same binary. Tool drift had been supplying enough fake disagreement already.
The new rule only held because our service owners already had production access and knew their own rollback paths. We have not tried it beyond 40 engineers or across a company with three separate compliance teams. We’d expect the ownership map to become its own small administrative animal there.
We Praised Firefighting In The Incident Channel
What we did
During the 03:18 orders-api incident on 7 August, a senior engineer found that an Envoy sidecar image had been pulled with a stale digest. They manually restarted six pods, traffic recovered, and the incident channel filled with grateful emoji.
Then we praised the restart in the weekly all-hands.
We did not praise the engineer for writing down the image-cache condition, because that happened two days later in a quiet document that fewer people read. We effectively taught the group that visible heroics counted more than removing the reason for heroics.
Why it looked reasonable
Incidents are tense, and people deserve credit when they act calmly under pressure. The restart did restore checkout. Nobody wants leaders to respond to a hard night with a lecture about process.
We also had a habit from smaller teams: the person who fixed the immediate problem was usually the person who could fix the whole problem. That stops being true when the system has 26 services, four cluster node pools, and a deployment controller that behaves differently at 03:00 than it did in staging. Or at least it feels that way after coffee.
What it cost us
Within ten days, the same stale-digest condition appeared in catalog-sync. The engineer on call restarted pods again because it was the known move. We lost 19 minutes of delayed catalogue updates and spent 14 person-hours on two nearly identical incident calls.
Worse, two people who had noticed the cache behaviour before the first incident did not bring it up in the review. They assumed an emergency workaround was the approved answer.
The fix
We changed our incident follow-up language. The incident commander now names three separate contributions: containment, diagnosis, and prevention. They are often done by different people.
Our postmortem template also includes “what made the fast workaround attractive?” That wording matters. “Why did we not fix it properly?” produces defensive writing and suspiciously clean timelines. The Google SRE postmortem guidance has the right instinct here: get the facts down without making the document a trial.
For the image issue, we set imagePullPolicy: Always for the affected deployment class, added digest validation to the release job, and created a runbook that says a restart is containment only. It is deliberately blunt.
If your team celebrates the person awake at 03:18, celebrate the person who makes 03:18 boring next month too.
We Assigned Pager Duty By Reporting Line
What we did
Our escalation policy routed alerts from payments, fulfilment, and customer-profile through the platform manager after the primary responder. The manager had enough context to coordinate work, so the routing felt sensible.
It also meant a people manager was the automated second-line responder for systems they did not deploy or change.
# Bad: escalation policy in Terraform
rule {
escalation_delay_in_minutes = 10
target {
type = "user_reference"
id = pagerduty_user.platform_manager.id
}
}
Why it looked reasonable
The platform manager attended architecture reviews, knew which teams were stretched, and could wake the right person. On paper, that resembles escalation.
We confused being informed with being useful at 02:00. Those are different jobs. One of them needs a laptop, current access, and enough familiarity with payment-webhook to distinguish a bad certificate from a bad deploy.
What it cost us
On 12 August at 01:56, a latency alert reached the manager after an on-call engineer was already investigating. The manager had to search the rota, message the service owner, and then relay a dashboard link. The service owner joined 17 minutes later.
No individual made a bad call. The policy encoded a slow handoff and asked tired people to compensate for it.
It also damaged trust in the rota. Service owners felt paged by proxy while platform absorbed blame for alerts outside its control.
The fix
We route to a service secondary first, then to an incident commander who can coordinate across teams. Managers get an informational notification after 20 minutes, not a wake-up call by default.
# Corrected: escalation policy in Terraform
rule {
escalation_delay_in_minutes = 10
target {
type = "schedule_reference"
id = pagerduty_schedule.payments_secondary.id
}
}
rule {
escalation_delay_in_minutes = 20
target {
type = "schedule_reference"
id = pagerduty_schedule.incident_commander.id
}
}
The PagerDuty escalation policy documentation is clear on how the mechanics work. The harder part is deciding who can act, rather than who sits highest on an org chart.
We still argue about whether managers should ever be in a daytime escalation chain. For customer-data exposure, probably yes. For a failed Kubernetes CronJob, absolutely not.
We Treated A Green Dashboard As Good Management
What we did
For most of July, our leadership update reported cluster availability, deployment completion, and ticket closure. All three were green. Meanwhile, engineers were bypassing the shared delivery path because preview environments took 38 minutes to appear.
Nobody hid that number. We simply had not made it a number we discussed with any seriousness.
Why it looked reasonable
Cluster availability was available in Grafana. Deployment completion came out of GitHub Actions. Ticket closure was in Jira, waiting politely for somebody to make a chart. These numbers were consistent and easy to report.
The preview-environment delay came from a mixture of runner queue time, Terraform state contention, and a slow image scan. It required actual investigation. We chose the numbers that arrived pre-chewed.
What it cost us
Two teams began keeping long-lived feature branches because they could not get useful feedback in a working day. One team deployed a feature flag directly to production to avoid another preview wait. That flag stayed enabled for five days because nobody was certain which branch had created it.
We had technically healthy infrastructure supporting increasingly unhealthy delivery habits.
The fix
We replaced the status slide with a weekly operating review of four measures: time from merge to production, failed deployment recovery time, on-call pages per service, and age of the oldest unowned alert. We use the definitions from DORA’s research program as a starting point, then adjust when the measurement lies.
For example, “merge to production” excludes changes waiting behind a scheduled customer release. Counting that delay as engineering delivery time would make a calendar meeting look like a platform defect.
We removed ticket closure entirely. We dislike it as a leadership metric. It rewards chopping work into small rectangles and then admiring the pile.
We Called On-Call A Growth Opportunity
What we did
When a product engineer joined the rota for the first time, we described it as a chance to learn the system. It was a chance to learn the system. It was also a week of interrupted sleep, a 06:40 database connection alert, and a Friday afternoon spent catching up on work they had planned to finish Tuesday.
We had no written rule for time back after a bad night.
Why it looked reasonable
Shared on-call builds ownership. We wanted product teams to see the consequences of their release decisions instead of handing every production concern to platform. That preference remains.
The mistake was treating the learning benefit as payment for the disruption. People do not need a motivational slogan after being woken by an alert caused by a threshold someone else set in 2023.
What it cost us
Three engineers swapped out of the rota in the next quarter. One did so quietly, citing “capacity.” We read that as a scheduling issue until their manager told us they had been paged five times in two weeks and had received no protected recovery time.
Alert quality also got worse because responders deferred cleanup work. Nobody volunteers to improve an alert that might later wake them if the work competes with a sprint commitment.
The fix
A page between 23:00 and 06:00 now earns a late start or half-day recovery without negotiation. A second page in the same night triggers a manager check-in and a review of the alert within five working days.
We also publish the rota alongside expected service work, so a person carrying orders-api is not assigned a migration cutover that week. This costs delivery capacity. Good. The capacity was fictional before; it was being borrowed from somebody’s sleep.
We Replaced Decisions With Status Meetings
What we did
Our Thursday platform meeting had 14 people, 45 minutes, and a standing agenda full of updates. The recurring item was “progress on secrets migration.” For six weeks, everyone reported progress while nobody decided whether service teams had to move by the September deadline.
The migration repository had 63 open issues. Twelve were blocked on choices that no ticket owner could make.
Why it looked reasonable
Status meetings make managers feel informed. They also give people a place to raise concerns, which sounds generous until the same concern appears in six consecutive agendas with no owner assigned to resolve it.
We had been trying to avoid unilateral decisions after a previous migration upset two application teams. Consensus became the safer social move. It was also much slower.
What it cost us
By the time we set a policy, the legacy Vault token format had spread to two new services. The team then had to support both formats during the busiest release month of the quarter.
The meeting itself cost 31 person-hours over those six weeks. The delay cost more, but the meeting cost was easier to calculate and therefore more annoying.
The fix
Every recurring operational meeting now has a decision log with three fields: decision required, named decider, and deadline. Updates go in writing before the meeting. If no decision is pending, people get 45 minutes back.
For the secrets migration, we chose a hard date: new services after 2026-09-15 must use workload identity; existing services get a supported adapter until 2026-12-31. We accepted that two teams would dislike the date more than they disliked ambiguity.
The adapter is still ugly. We have not found a clean migration path for the vendor service that only accepts a static credential, and nobody is excited about the options.
