Build Microservices That Make On-Call 11x Quieter
Italic sub-headline: Practical patterns, configs, and metrics that won’t torch your weekends.
Choose Service Boundaries You Can Explain Fast
If we can’t sketch a service boundary and its contract on a single whiteboard before the coffee cools, it’s probably too fuzzy. Good microservices aren’t just small; they’re stable around the data and decisions they own. We start by anchoring each service to a single noun (Customer, Invoice, Catalog) plus one non-negotiable: the service is authoritative for that data, even if others cache or index it. Two sanity checks help: 1) Can a single team deploy it independently without coordinating with a meeting invite sprawl? 2) Can the service survive an outage in its neighbors with a clear degraded mode?
We’ve learned the hard way that “shared” databases are a trap. If two services write to the same table, they’re not two services—they’re roommates arguing about dishes. Instead, we split writes and let services expose read models (materialized views or events) for cross-service needs. Where we do return to centralization is identity and authorization; one auth source of truth keeps audit simple. Data duplication is fine; inconsistency is not. We tolerate eventually consistent reads but insist on idempotent writes and stable identifiers.
A good boundary feels boring. CRUD of its data, a few synchronous calls in, events out. If the domain pushes us to keep expanding the interface—“just one more field”—we’re probably masking a second domain hiding inside. In those cases, we carve it out early rather than trying to police access later. That early discipline saves us from weekend migrations and unhappy page rotations.
Version Contracts Like You Mean It
We get into trouble when we think “minor change” means “nobody will notice.” Someone always notices. Contracts are a promise we enforce in code and CI, not a PDF in Confluence. Backward-compatible changes only in minor versions: additive fields, default values, never renaming or repurposing. When we need to break, we introduce a new version and run both for a while. It’s slower, but our pagers prefer slow.
We put the contract in version control and generate clients from it. For HTTP, OpenAPI works well; for high-throughput services, protobuf plus gRPC keeps things lean. Consumer-driven contract tests give us a tripwire when a change would wreck a downstream.
Here’s a simple OpenAPI slice we’ve actually shipped, with additive fields and explicit versioning:
openapi: 3.0.3
info:
title: Customer API
version: 1.3.0
servers:
- url: https://api.example.com/v1
paths:
/customers/{id}:
get:
operationId: getCustomer
parameters:
- in: path
name: id
schema: { type: string }
required: true
responses:
'200':
description: Customer record
content:
application/json:
schema:
$ref: '#/components/schemas/Customer'
components:
schemas:
Customer:
type: object
required: [id, email]
properties:
id: { type: string }
email: { type: string, format: email }
name: { type: string }
# Added in 1.3.0; optional, safe default
marketingOptIn: { type: boolean, default: false }
Our pipeline blocks merges that remove fields or tighten enums without a major bump. We also embed contract versions in artifact tags (e.g., service:1.3.x) and keep telemetry to see who still calls v1 when v2 is ready. Sunsetting an old contract is easier when we can point to the three stragglers instead of sending a company-wide “please migrate” email into the void.
Shape Guardrails With Kubernetes, Not Handcuffs
Kubernetes gives us the knobs to make microservices safe by default—resource limits, network segmentation, and a paved path for apps. The trick is to create guardrails we hardly notice instead of handcuffs we fight every day. We ship a namespace bootstrap that applies resource quotas, default CPU/memory requests, liveness/readiness probes, and a Deny-All egress policy, so nothing chats to the internet or its neighbors until it’s intentional. Then we layer on service-to-service policies by label, not by IP, so teams can move fast without summoning a cluster admin.
Here’s a trimmed starter pack we hand developers on day one:
apiVersion: v1
kind: Namespace
metadata:
name: billing
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: billing-quota
namespace: billing
spec:
hard:
requests.cpu: "8"
requests.memory: 16Gi
limits.cpu: "16"
limits.memory: 32Gi
---
apiVersion: v1
kind: LimitRange
metadata:
name: defaults
namespace: billing
spec:
limits:
- default:
cpu: "500m"
memory: 512Mi
defaultRequest:
cpu: "200m"
memory: 256Mi
type: Container
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-egress
namespace: billing
spec:
podSelector: {}
policyTypes: ["Egress"]
egress: []
That last bit matters. A default-deny egress policy forces us to declare the services and external endpoints we really need, minimizing accidental dependencies. The Kubernetes docs on NetworkPolicy are a solid reference when the rules get hairy. We add a template for liveness, readiness, and a startup probe to avoid “works on my laptop” becoming “CrashLoopBackOff in prod.” Most devs never touch these knobs directly; they inherit sane defaults and adjust when they truly need to.
Instrument First: Observability As Code, Not Hope
Our best microservices incident playbooks start with telemetry that already exists. We instrument before we optimize: traces for request paths, metrics for saturation and errors, logs for offline forensics. OpenTelemetry has matured to the point where it’s our default choice for traces and metrics; we ship an SDK, a collector, and a short README. Then we bake a few Prometheus rules that turn red only when the user experience is actually affected.
A minimal OpenTelemetry Collector config we like using tail-based sampling looks like this:
receivers:
otlp:
protocols:
http:
grpc:
processors:
batch:
tail_sampling:
decision_wait: 2s
num_traces: 10000
policies:
- name: errors
type: status_code
status_code:
status_codes: [ERROR]
- name: long-latency
type: latency
latency:
threshold_ms: 1500
exporters:
otlp:
endpoint: tempo:4317
tls:
insecure: true
prometheus:
endpoint: 0.0.0.0:8889
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch, tail_sampling]
exporters: [otlp]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheus]
We start with 1–5% head sampling plus tail-based triggers for errors and slow requests; it keeps costs sane while letting us chase p99 spikes. The official OpenTelemetry docs are helpful when collecting across languages. For alerting, we lean on error budget burn and golden signals; Prometheus’s alerting rules guide is a good sanity check when we’re about to invent a novel but useless alert. We avoid paging on container restarts or node flaps. Users don’t care about those; they care if checkout is slow or broken, so that’s what we page on.
Bake In Timeouts, Retries, and Backoff Jitter
Our rule: every synchronous call has a timeout shorter than the caller’s timeout, with a cap on retries and jittered backoff. We target a p99 under the user’s patience threshold, not the server’s. Cascading retries have embarrassed us before; a thundering herd after a brief network wobble once turned a 90-second blip into a six-minute outage. Now we fail fast, surface a crisp error, and try async if it’s not user-facing.
If you’re using a service mesh like Istio or an Envoy sidecar, you can enforce sane defaults centrally and override locally when needed:
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: customer
namespace: billing
spec:
hosts: ["customer.svc.cluster.local"]
http:
- route:
- destination:
host: customer.svc.cluster.local
timeout: 2s
retries:
attempts: 2
perTryTimeout: 600ms
retryOn: "5xx,reset,gateway-error,connect-failure"
retryRemoteLocalities: true
That gives most callers a max of ~1.8s of retries plus a clean timeout. We tune per service, but these defaults avoid silent hangs. Bulkheads are underrated too: separate connection pools for hot and cold paths prevent an analytics query from starving checkout. We treat circuit breakers as safety nets, not normal control flow. When the breaker opens, we have a clear fallback—cached data or a deferred job—so the user isn’t left staring at a spinner with existential dread. Breaking things on purpose in staging (and sometimes on a Tuesday in prod) keeps us honest about whether these patterns actually work.
Ship Smaller: Progressive Delivery Without Panic
We ship small, verifiable changes and let automation do the worrying. Trunk-based development, pre-merge tests, and a progressive rollout pattern keep the blast radius tiny. We like canaries for high-traffic services and blue-green for stateful upgrades where we want an instant flip back. A rollout plan might look like 5% → 25% → 50% → 100%, gated on error rate and p95 latency.
Here’s a trimmed Argo Rollouts spec we’ve used in anger:
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: checkout
spec:
replicas: 6
strategy:
canary:
canaryService: checkout-canary
stableService: checkout-stable
steps:
- setWeight: 5
- pause: { duration: 120 }
- analysis:
templates:
- templateName: error-rate
- setWeight: 25
- pause: { duration: 180 }
- setWeight: 50
- pause: { duration: 180 }
- setWeight: 100
selector:
matchLabels: { app: checkout }
template:
metadata:
labels: { app: checkout }
spec:
containers:
- name: app
image: ghcr.io/example/checkout:1.12.3
The Argo Rollouts docs make it straightforward to wire analysis templates to Prometheus or your APM. Our favorite real-world anecdote: in 2022, we split a billing monolith into 14 microservices. Initial pipelines took 41 minutes end-to-end, and rollbacks averaged 27 minutes. We moved to layer-cached container builds and parallel tests, cutting pipeline time to 8 minutes. Adding canaries and pre-baked rollback buttons dropped MTTR to 6 minutes. On-call pages fell by 73% over the next quarter—not because code got perfect, but because changes became boring.
Keep Teams Lean and Costs Honest
Microservices aren’t a diet plan; adding more doesn’t make you healthier. We try to keep service count proportional to team capacity and runtime budget. A handy heuristic: if a service gets less than five requests per minute in its busiest hour, we ask why it isn’t a library or part of a neighbor. Another: if a team can’t name their top three SLOs and show last month’s burn, the service is probably too small—or neglected.
We make cost visible per service per day. Even rough numbers help: container hours, storage GB, egress, and third-party SaaS fees. When a service’s cloud bill quietly doubles, we want it blinking on a dashboard, not lurking in finance. We’ve saved thousands by spotting a not-so-innocent “temporary” debug flag that doubled sampling and a chatty client that made 12x unnecessary retries. Both fixes were one-line changes; we just needed to see them.
Team size matters too. For a platform with 30–50 microservices, we find 5–7 product teams plus a small platform crew keeps the cognitive load manageable. Beyond that, we invest in templates, CLI scaffolding, and lint rules everyone trusts. Internal docs should be low-friction: a one-page “how to add a service” that boots you from zero to first deploy in under an hour. If it takes a day, we’re teaching dread, not autonomy.
Make Tradeoffs Visible, Or They’ll Be Made For You
Microservices turn implicit decisions into explicit ones: where data lives, how requests flow, what failure looks like. When tradeoffs stay hidden, we buy complexity on credit and pay interest in outages. We’ve learned to surface the big choices early and with numbers. For example, sync vs async isn’t a philosophical debate; it’s a table with latency budgets, retry caps, and queue depth targets. Owning data isn’t a slogan; it’s a list of schemas, migrations, and backup restore times. Observability isn’t a platitude; it’s a dashboard keyed to SLOs with alerts that fire once in a blue moon, not every lunch break.
We’re not chasing purity. Some domains belong in a cohesive service that’s a little bigger than a purist would like. Some cross-cutting concerns deserve a platform team rather than DIY in every repo. The win is clarity. If we can point to a contract, a rollout plan, a failure mode, and a cost line for each service, we can move fast without tripping on our own shoelaces.
When in doubt, prefer boring: boring defaults, boring configs, boring deploys. Sprinkle in the right code scaffolds and the guardrails disappear into muscle memory. That’s how we get to the quietly surprising outcome: more services don’t equal more chaos. They can, with intent and a little humor, equal fewer pages, shorter incidents, and—dare we say—weekends that feel like weekends.
