Ansible That Keeps Production Boring And Reliable
Practical habits for repeatable changes, calmer releases, and fewer midnight surprises.
Start With A Clear Automation Boundary
We get better results from ansible when we stop treating it as a universal remote-control toy. It can configure servers, deploy applications, manage cloud resources, and perform maintenance, but every playbook needs a clear job. “Make all infrastructure good” is not a job. “Install and configure Nginx on the API tier” is.
Before writing tasks, we define three things: the target systems, the intended state, and the owner of that state. If the platform team owns operating system baselines while application teams own their services, our automation should reflect that boundary. Otherwise, one playbook quietly overwrites another team’s configuration, and everyone gets to enjoy an incident review.
We also separate provisioning from configuration. Creating a virtual machine, assigning a security group, and configuring a package repository may happen in one delivery flow, but they are different concerns. Keeping them in separate playbooks or roles makes failures easier to understand and reruns less risky.
Inventory deserves the same care. A host group should represent a meaningful operational unit, such as web, database, payments, or production. Avoid groups named after temporary projects, individual engineers, or whatever was happening during a Friday afternoon migration.
Ansible’s own inventory documentation is worth revisiting when inventories start growing strange branches. Static inventory is fine for stable environments; dynamic inventory is useful when cloud instances appear and disappear regularly. The important part is consistency.
Finally, we write playbooks as declarations of desired state, not imperative command transcripts. If a task can run twice without changing anything on the second run, we’re on the right path. Idempotency is not glamorous, but neither is being paged because a rerun deleted a configuration file.
Build Inventories That Describe Reality
A useful inventory tells us what systems are, where they belong, and how ansible should reach them. It should not become a storage cupboard for passwords, one-off shell flags, and vague comments from 2019.
We prefer environment-specific inventory directories. This makes it obvious which variables apply to production and which belong only in a development sandbox. A simple structure is usually enough:
inventory/
├── development/
│ ├── hosts.yml
│ └── group_vars/
├── staging/
│ ├── hosts.yml
│ └── group_vars/
└── production/
├── hosts.yml
└── group_vars/
Our production inventory might look like this:
all:
children:
web:
hosts:
web-01.example.net:
web-02.example.net:
api:
hosts:
api-01.example.net:
api-02.example.net:
production:
children:
web:
api:
vars:
ansible_user: deploy
ansible_become: true
We use group variables for shared settings and host variables only when a machine is genuinely special. If every host needs a unique override, that is usually a sign that our grouping is wrong or our service design has drifted.
We also avoid putting environment names inside every variable. production_database_host is awkward when the inventory already tells us we are in production. Prefer database_host, then set its value in the relevant environment variables.
Before a change, we run ansible-inventory --graph and ansible-inventory --list. These commands catch surprisingly common mistakes: hosts in the wrong group, missing child groups, and variables applied more broadly than expected. A five-second inspection beats explaining why a test deployment reached production. We like production boring; it has enough personality already.
Make Roles Small, Focused, And Reusable
Roles are where ansible either becomes maintainable or turns into a very organised-looking pile of YAML. We use roles to package a single responsibility: configure Nginx, install a monitoring agent, deploy an application, or manage a database backup job.
A role should not try to own an entire server unless that server has one narrowly defined purpose. A base_linux role can manage common packages, service users, time synchronization, and security settings. An nginx role can install Nginx, manage its configuration, and notify the service to reload. An api_service role can deploy application code and manage its systemd unit.
The playbook then becomes readable:
- name: Configure production API nodes
hosts: api
become: true
roles:
- base_linux
- nginx
- api_service
Inside each role, we keep defaults conservative and document required variables in README.md. Defaults belong in defaults/main.yml; values that should not be casually overridden can live in vars/main.yml, although we use that sparingly. Variable precedence in ansible can be powerful, but it can also turn debugging into archaeology.
Handlers are especially useful for avoiding unnecessary restarts. If a template has not changed, Nginx does not need a reload. If a package update did not alter a service, we should not bounce it “just in case.” Every restart carries risk, however small.
We also name tasks clearly. “Configure service” tells us almost nothing during a failed run. “Render Nginx API virtual host” tells us exactly where to look. Clear task names make CI output, incident troubleshooting, and peer reviews much less painful.
For shared work, Ansible Galaxy can be useful, but we review external roles as we would any dependency. A role with 800 tasks to install one package is rarely a gift.
Keep Secrets Out Of Sight And Source Control
We assume that anything committed to a repository may eventually be copied, indexed, forwarded, or accidentally pasted into a ticket. That means passwords, API tokens, private keys, and connection strings do not belong in plain-text group variables. Not even “just temporarily.” Temporary secrets have an irritating habit of becoming historical artefacts.
For smaller teams and straightforward use cases, Ansible Vault is a practical starting point. It lets us encrypt variables or complete files while keeping them alongside the playbooks that need them. We can encrypt a production variables file like this:
ansible-vault encrypt inventory/production/group_vars/all/secrets.yml
Then reference the values normally:
database_password: "{{ vault_database_password }}"
monitoring_token: "{{ vault_monitoring_token }}"
We never store the vault password in the repository. In CI, we retrieve it from the CI platform’s protected secret store. For local work, we use a secure password manager or a controlled vault-password command. Convenience is nice; credential leakage is less charming.
For more mature environments, we pull secrets at runtime from a dedicated secrets system. HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and similar services provide central rotation, access records, and tighter permissions. The right choice depends on the environment we already operate, not on whichever tool has the most stickers at a conference.
Secrets also need scope. A deployment token should deploy, not administer every database in the company. We give each service account the smallest useful permission set, rotate credentials regularly, and revoke access promptly when systems or people change. Ansible can apply secrets safely, but it cannot make overpowered credentials sensible.
Test Playbooks Before Hosts Feel The Impact
Ansible is easy to run and therefore easy to run too early. We put checks around playbooks before they can reach shared environments. YAML syntax is not enough; a syntactically valid playbook can still install the wrong package, overwrite a template, or restart every service at once.
Our first check is usually:
ansible-playbook -i inventory/staging site.yml --syntax-check
ansible-lint
ansible-lint catches common mistakes and nudges us toward clearer, safer patterns. It will not replace human review, but it is excellent at noticing things we were too tired to notice after staring at indentation for an hour.
We then run against an isolated environment where possible. Containers work well for basic role testing; temporary cloud instances are better when we need to validate networking, systemd behavior, storage, or provider integrations. The Molecule project helps us create and verify role scenarios consistently.
A useful test sequence looks like this:
molecule test
ansible-playbook -i inventory/staging site.yml --check --diff
ansible-playbook -i inventory/staging site.yml
Check mode is valuable, but we do not mistake it for a guarantee. Some modules cannot fully predict changes, and commands executed through shell or command may not support dry runs at all. That is another reason to prefer purpose-built modules.
We also test idempotency. After a successful run, we run the same playbook again. The second run should report little or no change. If it repeatedly changes a file, recreates a resource, or restarts a service, we investigate before promoting it.
Finally, we test rollback thinking before the deployment. Not every change has a literal undo task, but every significant change should have a recovery plan: restore the previous package version, redeploy the prior artifact, revert a template, or remove a bad configuration safely.
Roll Out Changes In Small, Observable Steps
A successful ansible run is not automatically a successful deployment. The service may fail health checks, the application may reject a new configuration, or a dependency may be unavailable. We roll out changes in batches so that a mistake affects a manageable number of systems.
For production changes, serial is one of our favourite safety controls:
- name: Deploy API release gradually
hosts: api
serial: 1
become: true
max_fail_percentage: 0
roles:
- api_service
This updates one host at a time. After each host, we can validate application health, error rates, queue depth, or synthetic checks before the next host changes. For larger fleets, we might use serial: 10%, but we choose the batch size based on capacity. If losing two nodes would hurt, do not update twenty.
We use pre_tasks to remove a node from a load balancer and post_tasks to add it back only after checks succeed. This keeps users away from a host while it is changing and gives us a natural point for validation. We also use block, rescue, and always when a workflow needs cleanup after failure.
Tags make operations more precise. A release playbook may offer --tags deploy, --tags config, or --tags rollback-prep, but tags need documentation. Random tag collections become a menu where nobody knows what the meals contain.
During a rollout, we watch the systems that matter: deployment logs, service status, latency, error rates, and business signals. Automation should reduce manual effort, not reduce attention. If the first batch looks unhealthy, we stop. There is no prize for automating a bad decision at high speed.
Review Automation Like Production Code
Playbooks change production state, so we review them with the same seriousness as application code. A pull request should explain what changes, which hosts are affected, how it was tested, and what happens if it fails. “Updated config” is not enough context when that config controls customer traffic.
Our reviews focus on practical questions. Is the target host group correct? Are tasks idempotent? Is a module available instead of a raw shell command? Are secrets protected? Could a handler restart more services than intended? Does the change need a rollout limit or maintenance window?
We are particularly suspicious of shell tasks. Sometimes they are unavoidable, especially with legacy tools, but they should be the exception. When we use them, we quote variables carefully, define changed_when and failed_when where appropriate, and document why a dedicated module is not suitable. A shell command without those details is a tiny mystery novel.
We also keep version pinning sensible. Collections, roles, and modules can change behavior between releases. A requirements.yml file, tested upgrades, and a dependency update schedule prevent surprise breakage during an unrelated deployment.
Documentation does not need to become a novel. Each repository should explain how to install dependencies, run syntax checks, target an environment, supply secrets, and perform a safe production rollout. New team members should not need oral history to deploy a small configuration change.
Most importantly, we treat automation maintenance as regular work. Removing obsolete hosts, retiring unused roles, updating old modules, and simplifying variables keeps the codebase trustworthy. Ansible is most useful when people are willing to run it confidently. That confidence comes from careful boundaries, tests, reviews, and a healthy respect for what one innocent-looking YAML file can do.
