Calm Noisy On-Call With AI, 38% Fewer Incidents
Make on-call humane using practical AI patterns, not hand-wavy hype.
We’re Here For Fewer Pages, Not Party Tricks
We don’t wake up at 2 a.m. craving a chatbot. We want fewer pages, faster fixes, and calmer weekends. AI can help, but not by “replacing” SRE or DevOps muscle. It helps by narrowing noise, adding instant context, and nudging us toward the single, fixable thing. Our north star is measurable: think 20–40% fewer incidents triggered by duplicates or known patterns, 15–25% faster mitigation on high-priority incidents, and fewer escalations because summaries carry receipts. Those numbers won’t fall out of the sky—we earn them with clean telemetry, sensible integrations, and guardrails that keep hallucinations out of the rota.
If we had to draw a line, it’s this: AI should reduce cognitive load, never add it. It should auto-group correlated alerts, draft a tight incident summary with links to dashboards, and surface the two most likely diagnostics to run. It shouldn’t argue with your runbooks or invent a Kafka cluster you don’t own. We’ll measure progress with boring, sturdy metrics: MTTD, MTTR, ticket reopens, alert deduplication rate, and how often folks hit “acknowledge” within five minutes. We’ll also track the cost of tokens or CPU minutes, because nothing torpedoes a good idea like a surprise bill.
We’re going to wire AI where it’s already useful: next to Alertmanager, inside incident tickets, and near autoscaling signals. The playbook is straightforward: prepare your data, introduce AI at the edges with reversible changes, evaluate ruthlessly, and then make it boring in production.
Teach Telemetry To Talk Before AI Listens
Before we ask AI to summarize our house on fire, the logs, metrics, and traces need to speak in full sentences. Labels must be consistent. Severity should mean something. Alerts should carry runbook links, service ownership, and a hint about past remediations. AI thrives on context; if it has to guess the namespace or which team owns an SLO, we’ve already lost a minute. Start with OpenTelemetry so spans and logs carry the same trace IDs; that’s the backbone for stitching “what happened” into one coherent picture. The OpenTelemetry Collector’s processors and exporters help us normalize data and attach the extra crumbs AI needs. See the official OpenTelemetry Collector docs for pipelines and processors you’ll actually use.
We also want our SLOs and error budgets easy to query. Even a rough latency SLO with well-defined events beats a hazy “be fast” directive. When alerts reference SLO breach risk, AI can prioritize what hurts customers rather than what merely looks spicy in Grafana. Google’s SRE guidance on Service Level Objectives is still the clearest way to define “good enough” reliability without gold-plating.
A small but mighty step: standardize alert labels across services. Use consistent environment tags, version, region, and a stable service name. Include a runbook URL, a link to the primary dashboard, and if you’re fancy, a pointer to the last five similar incidents. AI isn’t telepathic; it’s efficient when we hand it the map. Do that, and you’ll watch the quality of summaries, correlations, and suggested diagnostics jump without touching a single model parameter.
Wire AI Into Alerting, Right Next To The Pager
The quickest win is to put AI where alerts first land. It’s not a new alert source; it’s a context factory. The flow looks like this: Alertmanager fires, a webhook forwards the payload to a tiny “ai-enricher” service, and that service groups related alerts, pulls recent logs and traces, then posts a trimmed summary back as annotations. We’re not changing the on-call app—just improving what lands in it. The bonus: the AI never pages anyone on its own. It annotates; humans decide.
Here’s a minimal Alertmanager route and receiver that forwards alerts to an enricher:
route:
receiver: "ai-enricher"
group_by: ["cluster", "service", "severity"]
group_wait: 30s
group_interval: 2m
repeat_interval: 3h
receivers:
- name: "ai-enricher"
webhook_configs:
- url: "https://ai-enricher.internal/alerts"
send_resolved: true
http_config:
basic_auth:
username: "alert"
password: "<redacted>"
The enricher adds annotations like ai_summary, related_incidents, and remediation_hints, then forwards on to your normal receiver. We keep grouping keys aligned with how we actually triage: cluster, service, severity. If your team uses routing trees, AI doesn’t subvert them; it just respects the path and improves the payload. The official Alertmanager docs explain grouping semantics and receivers in painful but necessary detail.
By putting AI here, we gain three things right away: de-duplication based on similarity, short summaries pinned to evidence, and “next step” suggestions that limit scrolling and guesswork. No new tools, no new tabs—just smarter alerts swimming down the pipes you already own.
Summaries With Receipts: Retrieval Over Guesswork
If there’s one place AI shines, it’s turning five alerts, three logs, and a trace into a tidy paragraph. But we don’t want fiction. We want summaries grounded in our runbooks, dashboards, and incident history. That means retrieval first, generation second. We’ll fetch context from known-good sources—runbooks in Git, dashboards, known incidents—then ask the model to summarize with citations. When in doubt, AI should say “not sure” and point to the right runbook section.
Here’s a sketch of a tiny summarizer that won’t make wild claims:
def summarize_incident(alert, store, model):
facts = []
facts += store.search_logs(service=alert['labels']['service'], window="15m")
facts += store.search_traces(trace_id=alert.get('labels', {}).get('trace_id'))
facts += store.search_incidents(service=alert['labels']['service'], limit=5)
facts += store.search_runbooks(service=alert['labels']['service'], sections=["symptoms","checks","mitigation"])
context = "\n".join(f"{f['source']}: {f['snippet']}" for f in facts[:50])
prompt = f"""
You are drafting an incident summary for on-call.
Only use the context below; cite sources inline like [source].
If uncertain, say so and link the runbook section.
Alert: {alert['annotations'].get('summary')}
Context:
{context}
"""
output = model.generate(prompt, temperature=0.1, max_tokens=400)
return output
We keep temperature low, demand citations, and cap tokens to control cost. Tie the output into your incident workflow so the first comment in the ticket carries the summary and links. This pairs nicely with disciplined incident response habits like concise status updates and clear roles; if you need a refresher, the PagerDuty Incident Response guide is terse and battle-tested. Summaries with receipts calm the room, speed up context sharing, and keep the conversation focused on “what next” instead of “what’s happening.”
Predict The Cliff, Don’t Climb It: Smarter Scaling
On-call gets ugly when we’re surprised by traffic cliffs or slow-burn resource leaks. AI can forecast trouble before the pager screams—but only if we keep it on a short leash. Think of it as a “forecast assistant” that suggests scale-up windows, cache warmups, or batch deferrals. Start simple: train on past traffic by hour and day, add relevant exogenous signals (campaigns, releases), and produce rolling forecasts with confidence intervals. We’re not day-trading; we’re trying to prevent noisy autoscaler flapping and sudden saturations.
The safest pattern is recommend-then-verify. The forecaster posts a daily note: expected p95 latency risk for service X between 16:00–18:00 UTC, suggest min_replicas=12. The change only lands if it passes SLO-aware safety checks and the autoscaler agrees. When weekends spike, we scale early. When memory pressure trends up, we flag the slow leak before it’s a 3 a.m. crime scene. Pair this with known good playbooks: pre-warm caches, widen connection pools, or split heavy queues into burst buckets.
Tie these ideas to reliability guidance you already trust. The AWS Well-Architected Reliability Pillar talks about monitoring, automation, and capacity management without the hand-waving. AI just adds earlier, probabilistic hints. We’ll judge success by fewer thrash events, fewer cold-start pages, and cleaner release windows. And if the forecaster is wrong, it should be wrong in the direction of safety: recommend watching, not yanking the handbrake.
Guardrails That Keep Us Employed: Privacy, Cost, Drift
A practical AI stack needs circuit breakers. We’ll start with privacy: redact secrets and personal data before anything leaves your network—or better, run models where your data lives. Logs are like toddler pockets; you’ll find API keys and customer tidbits in the worst places. Build redaction into the enricher service so prompt payloads are clean by default. Then set cost limits with caching and prompt budgets per team. It’s amazing how fast “just summarize logs” becomes “why is this line item bigger than the database?”
A small config example for an enricher’s safety settings:
redaction:
patterns:
- "(?i)AKIA[0-9A-Z]{16}" # AWS access key
- "(?i)secret[_-]?key\\s*[:=]\\s*\\S+"
- "\\b\\d{3}-\\d{2}-\\d{4}\\b" # SSN-style PII
cache:
ttl: 15m
limits:
tokens_per_minute: 100000
max_summary_tokens: 400
fallback:
summarize_only_on_priority: ["p1","p2"]
disable_generation_on: ["internal_network_outage"]
We also need drift and quality controls. New services appear, labels change, and runbooks rot. Schedule weekly checks: sample summaries, compare to ground truth, and track a simple “hallucination rate” by counting unsupported claims. When drift happens, fix the retrieval layer or update prompts before touching the model. Finally, define refusal behavior: if context is thin or risky, the model should say “insufficient context” and link to the runbook. That’s not a failure; it’s guardrails doing their job and saving us from creative fiction at 3 a.m.
Rollouts, Metrics, And Making AI Boring In Prod
We’ll start with a shadow phase. The enricher writes summaries to a private channel and a test incident queue for two weeks. We collect MTTD, MTTR, dedup rates, and human “usefulness” scores on a simple 1–5 scale. If the metrics move in the right direction and on-call folks don’t shudder, we expand to one team in business hours. Feature flags let us toggle per service and priority. If a new integration misbehaves, it’s one click back to the old world.
Evaluation deserves the same discipline as a new autoscaler. Build an offline set of real incidents and “gold” summaries authored by seasoned responders. Periodically run the current stack against that set and track match quality and missing steps. Tie costs to value: tokens per saved minute or dollars per prevented escalation. Keep an eye on regression: did that “clever” prompt update save 5 seconds but add 10% more wrong hints? Roll back and try again.
The endgame is boring AI. Alerts arrive with context. Summaries read like a calm teammate. Predictive nudges are conservative and correct often enough to justify their seat. When we tweak, it’s small. When we fail, we fail closed. And when we talk about wins, they sound like this: pages down 31%, P1 mitigation faster by 18%, and a measurable dip in post-incident grumpiness on Mondays. If that doesn’t count as humane on-call, we’re not sure what does.
