Create a 99.95% Safe Deployment Pipeline
Master the art of deploying code with minimal risk and maximum efficiency.
The Importance of Safety in Deployments
When we think about deployment, safety isn’t just a nice-to-have—it’s a must. After all, no one wants a Friday night release to turn into a nightmare. For instance, one time we had a colleague who accidentally deployed untested code on a Friday. Let’s just say it took us three hours of frantic troubleshooting and a whole lot of coffee to fix it. Lesson learned: a safe deployment pipeline saves time, sanity, and probably a few friendships.
Key Principles for a Safe Deployment Pipeline
To ensure our deployments are as safe as possible, here are some key principles we follow:
-
Automated Testing: Always run tests before deploying. Here’s a sample configuration for a CI/CD tool like Jenkins to automate testing:
groovy
pipeline {
agent any
stages {
stage('Test') {
steps {
script {
sh 'npm test'
}
}
}
}
} -
Canary Releases: Implementing canary releases allows us to monitor the impact of new changes. We typically start with 5% of our user base. Here’s how we do it:
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 20
template:
spec:
containers:
- name: my-app
image: my-app:v1.0Initially, we deploy only one pod to handle the 5%.
-
Rollback Strategies: Knowing how to roll back is critical. A simple script can help:
bash
kubectl rollout undo deployment/my-app
Monitoring and Logging for Enhanced Safety
We can’t fix what we can’t see, right? Robust monitoring and logging give us the visibility we need. Tools like Prometheus and Grafana can provide real-time insights. Here’s a quick setup example:
apiVersion: v1
kind: Service
metadata:
name: prometheus
spec:
ports:
- port: 9090
targetPort: 9090
selector:
app: prometheus
By keeping an eye on error rates and performance metrics, we catch issues before they escalate.
Conclusion: A Culture of Safety
Ultimately, creating a safe deployment pipeline requires a team culture focused on safety. Regular training and open communication make a huge difference. So let’s keep our code (and our teams) safe!




Flaky tests are the part I would want to see addressed before treating a green build as release confidence. Teams often end up rerunning failures until they pass, and that can hide a real regression. A follow-up on test quarantine, retry limits, and how to decide when a failed test should block a deploy would be really useful, especially when the pipeline hasnt been stable for long.
We use GitHub Actions with a small .NET app, and the rollback is usualy harder than the deploy. At our tiny company, nobody is watching Grafana all night
I disagree, 3 incidents didnt hit canaries… why 99.95%?
Love this for our small fintech, what does rollback cost custmers?
please cover customers rollback messaging and status pages next?
“We catch issues before they escalate”… do you have failure-rate data, or just examples?
“5% of our user base” can still be a lot of orders. We run a small online shop, and a broken checkout is noticeable immediately. Who decides the canary group? I would want that written down.
The monitoring part is what I keep coming back to. At the library system where I work, a catalog update once made every search return zero results for about twenty minutes. Our tiny IT team only spotted it because someone at the desk called. A simple alert on failed searches would have saved a lot of confusion. We are not running anything as large as 20 pods, but the idea of checking a small release first feels very practical. I am going to ask whether we have a rollback note written down.
The topology questions are where this gets interesting. At scale, one pod is rarely 5% of traffic… it depends on routing, replica capacity, and the clusters traffic policy. I would pair the canary with service-mesh weighting and region-aware rollback, otherwise a hot shard can distort the signal. Could you follow up on multi-region canaries and the cost of keeping spare capacity?
We run ECS. Is traffic weighting this complicated there too?