Counterintuitive Leadership for 99.9% Better DevOps Outcomes
Practical rituals, code patterns, and metrics that make teams quietly unstoppable.
Measure What Matters, Not What’s Loud
If we want leadership that moves needles, we’ve got to stop staring at needles that don’t measure anything useful. Story points are a weather forecast; deployment frequency is the thermostat. The easiest way to make leadership practical in DevOps is to adopt a small, durable scorecard and make it visible. We like the DORA quartet—deployment frequency, lead time for changes, mean time to restore, and change failure rate—because they map to what customers feel and what teams can change this month. If we can’t move a number in a quarter, it’s not an operational metric; it’s a horoscope.
To keep this honest, we run a weekly 15-minute “metrics standup” with one rule: propose one experiment tied to one metric. That might be “cut PR queue length by 30% by merging lunchtime” or “cap WIP at 3 per engineer.” We log the experiment, owner, and review date. We don’t turn this into a courtroom or a TED Talk. We simply treat the numbers like smoke alarms: when they beep, we look for toast, not a scapegoat.
Dashboards are fine, but ingestion matters more than visualization. If the data’s flaky, the behavior will be too. We’ve had good luck starting with the open-source Four Keys project, which pulls signals straight from our systems and avoids “copy-paste archaeology” in spreadsheets. The goal isn’t precision to three decimal places; it’s consistent, low-friction feedback that teams trust. Over time, the magic is compounding: small, continuous nudges create big, boringly reliable outcomes. That’s the kind of boring our customers will happily pay for.
The Leadership Cadence: 30-60-90 That Sticks
Leadership is rhythm as much as direction. We like a simple 30-60-90 cadence that keeps us out of the thrash and squarely in the work. Every 30 days, we show working software. Every 60 days, we retire something old. Every 90 days, we renegotiate constraints. It’s not mystical; it’s a calendar invite with teeth.
The 30-day cycle is about demos and decisions. If a goal or plan needs a 40-slide deck, the plan is too vague. We ask, “What changed in production?” and “What did customers feel?” We time-box a hard decision: approve the next increment, pivot, or stop. The next 30 days inherit the choice, not the debate.
The 60-day cycle is our cleanup crew. We deprecate one thing: a flaky test suite, a dead feature flag, an IAM policy with 200 lines of “just in case.” The rule is that deprecation must pay an interest rate—less toil, fewer hops, fewer clicks. We anticipate the “but we might need it” chorus and keep receipts: before/after metrics, screenshots, grep outputs. That way the next cleanup is easier to sell.
The 90-day cycle is for constraints: error budgets, cost caps, on-call rotations, and hiring plans. Constraints go stale faster than goals. When error budgets reset quarterly, for example, teams learn to pace change instead of yolo-ing in week 11. We treat this cadence like scaffolding. It holds the work without becoming the work. That’s how leadership helps people ship, not sit in meetings about shipping.
Runbooks as Code Shape Behavior
Most outages aren’t puzzles; they’re pop quizzes on things we once fixed and forgot. Runbooks as code stop us from re-learning under pressure. We put executable steps in the repo next to the service. No wikis, no sacred binders. If the rollback is make rollback, it lives in version control. People follow what they can run, not what they can read.
Here’s a tiny pattern we use: a runbook YAML with commands we can invoke, plus a Makefile entry. When incidents hit, the playbook is one command away in the same branch as the code that caused the trouble. The runbook doubles as documentation and integration test fixture.
# runbook.yaml
service: payments-api
triggers:
- name: elevated_5xx
source: prometheus
threshold: 2%
checks:
- name: canary_health
cmd: ./scripts/check_canary.sh
- name: db_connectivity
cmd: ./scripts/check_db.sh
rollback:
- name: rollback_to_previous
cmd: ./scripts/rollback.sh --to previous
- name: disable_canary
cmd: kubectl scale deploy payments-api-canary --replicas=0
verification:
- name: steady_2xx
cmd: ./scripts/check_steady_state.sh --window 10m
# Makefile
rollback:
\trunbook run rollback --file runbook.yaml
verify:
\trunbook run verification --file runbook.yaml
We keep the commands short, idempotent, and chatty. The purpose isn’t to automate human judgment; it’s to remove the muscle memory that fails at 3 a.m. When someone improves a step, the PR diff tells the story and the team reviews it like code—because it is code. That’s leadership at the sharp end: we make the right thing easier than the heroic thing.
Psychological Safety Without Pom-Poms
Psychological safety isn’t about warm fuzzies; it’s about teams telling the truth fast. When engineers can say “I don’t know” and “I broke it” without rehearsing their defense, we ship more often and recover faster. We hire adults, and adults don’t need pep rallies to be honest. They need predictable reactions and an environment where competence compounds.
We start by tuning the meeting culture. Status meetings are for shared context, not self-defense. We ask “What did we learn?” instead of “Why is this late?” and we keep the airtime balanced. If senior folks talk half the time, we cut it in half. When there’s conflict, we separate facts from interpretations on a shared doc so we can disagree with the inference without attacking the person. We also retire the “devil’s advocate” costume; if someone has a real objection, they can own it without playing dress-up.
Code reviews are a minefield for safety; we defuse them with two habits. First, we thank people for deletions; negative lines count double. Second, we rewrite “why did you do X?” as “what tradeoffs led you to X?” That’s not coddling; it’s modeling the kind of discussion that yields better systems. We also time-box review back-and-forth. After two rounds, we either pair or defer the debate to a design doc.
Finally, we treat on-call as a team sport: broad rotation, shadowing, and clear escalation boundaries. When alarms page humans who can’t fix the thing, morale evaporates. Safety isn’t a poster; it’s who gets paged, who gets heard, and who gets blamed.
Blameless Postmortems That Close the Loop
Postmortems only matter if they change the next Tuesday. We keep them blameless, brief, and binding. Blameless means we surgically remove intent and swap in mechanisms and conditions. Brief means two pages max and a timeline that tells the “what happened” story without courtroom drama. Binding means actions have owners, dates, and a follow-up to check that the fix did more than soothe our guilt.
We also make postmortems discoverable and runnable. Our template lives in the repo, not in the abyss of shared drives. It captures the basics and integrates with our incident tooling. Borrowing liberally from the Google SRE guidance on postmortem culture, we write for clarity under stress and reuse.
Here’s a lightweight GitHub Issue template we use to force good habits:
# .github/ISSUE_TEMPLATE/postmortem.yml
name: Postmortem
description: Blameless incident review
title: "PM: <service> <date> <SEV>"
labels: ["postmortem"]
body:
- type: input
id: sev
attributes:
label: Severity
placeholder: SEV-1/2/3
- type: textarea
id: impact
attributes:
label: Customer Impact
description: Who felt it, for how long, and how badly?
- type: textarea
id: timeline
attributes:
label: Timeline
description: UTC timestamps and facts only.
- type: textarea
id: contributing
attributes:
label: Contributing Factors
description: Systems, conditions, assumptions—no blame.
- type: textarea
id: actions
attributes:
label: Actions
description: Owner, action, due date, verification method.
We schedule a 20-minute review within five business days. If the fix is bigger than a sprint, we track the risk reduction with a metric we can feel—lower pager volume, fewer retries, improved burn-rate on error budget. We then feed learnings into design docs and onboarding. The loop is closed when the same class of incident never surprises us twice.
Delegation by Guardrails: Policy as Code
The best leadership pattern we’ve found for scaling autonomy is to delegate with guardrails, not gates. We tell teams what “good” looks like and let them ship within those boundaries at full speed. The trick is making guardrails executable so they don’t morph into folklore. Policy as code—OPA, Gatekeeper, Kyverno, SCPs—turns values into tests. If a rule matters, it should be linted.
We start small: naming, tags, budgets, and permissions. That’s where entropy chews fastest. We capture the rule in the repo that owns the resource, and we enforce it at the right point of the pipeline—pre-commit for fast feedback, admission controller for cluster safety, and drift detection for long-lived infra. We avoid checkbox theater and keep policies human-readable; if a developer can’t tell what failed and why, we failed.
Here’s a tiny OPA/Rego example that requires an owner tag on Kubernetes namespaces:
# require_owner.rego
package kubernetes.admission
violation[msg] {
input.request.kind.kind == "Namespace"
not input.request.object.metadata.labels.owner
msg := "namespace must include metadata.labels.owner"
}
Paired with Gatekeeper, this nudges teams to label everything with an accountable human. That label later powers monthly cost rollups and on-call rotations. Suddenly, your spreadsheet of shame is an automated report.
For alignment, we map guardrails to recognized pillars so we’re not inventing wisdom. The AWS Well-Architected lens is a handy checklist. For policy authorship and rollout, we lean on Open Policy Agent docs to keep our rules versioned, testable, and portable. Delegation works when feedback is fast and the fence is obvious.
Hiring and Growing Adults, Not Heroes
Leadership is who we hire and who we promote. If we reward heroics, we’ll get outages fixed at 4 a.m. If we reward boring reliability, we’ll sleep. We bias for adults: people who communicate clearly, keep promises, and leave systems better than they found them. That starts with our interview signals. We value deletion, refactoring, and writing a design doc over “whiteboard wizardry.” We give candidates a choice: a structured systems discussion or a small take-home with a tight time box. We pay for their time if it’s non-trivial. Adults respect other adults’ time.
Once inside, we publish career ladders that reward impact and stewardship, not the number of direct reports or the loudness of opinions. Senior folks own outcomes across team boundaries, improve interfaces (human and API), and mentor without theatrics. Promotions tell the culture what leadership looks like. If we promote firefighters over fire preventers, brace for smoke.
Growth is also about community. We borrow from the open-source world’s clarity on levels and roles—CNCF’s contributor ladder is a great example of explicit expectations. Internally, we do “demo days” for tech leadership: present a gnarly tradeoff you navigated, the principles you used, and what you’d do differently. No slides required, just candor.
Finally, we embrace attrition as a normal cost of having standards. When values diverge, we act quickly and fairly. It’s kinder to part ways than to ask a team to carry cultural debt. Adults do their best work around other adults. That’s the quiet magic of durable leadership.
