Docker Tags Make Production Rollbacks Ambiguous

Our production manifests should name bytes, not somebody’s label.

While the registry UI was showing ledger-api:2026.08.3 with two different upload times, I decided we should deploy Docker images by digest, even when a version tag looks perfectly respectable. A tag is a label somebody can move; production should name the exact bytes it intends to run.

A Docker Tag Is A Promise We Cannot Verify Later

We still create tags. Developers need them, release notes need them, and nobody wants to read a digest aloud during an incident unless morale has already collapsed. 2026.08.3 tells a human roughly what they are looking at. sha256:5bb4… tells a machine precisely what it is looking at.

Those are different jobs, and treating one identifier as both has caused more confusion than it deserves.

A Docker image digest identifies a content-addressed manifest. Docker documents the distinction clearly in its image digests reference: tags can change, while a digest points to a specific image manifest. The OCI image specification is equally blunt about descriptors being identified by digest. We knew this, of course. Knowing a thing and arranging work around it are separate departments.

My opinion is that a production deployment manifest should contain registry.example.net/ledger-api@sha256:..., not registry.example.net/ledger-api:2026.08.3. The tag belongs beside it in Git, in the release ticket, and in the message we send when a deploy starts. It does not belong where the scheduler decides what to pull.

The common reply is that version tags are immutable in a well-run registry. That is usually shorthand for “we asked people not to overwrite them.” A permission policy is better. A registry setting that rejects overwrites is better still. Neither changes the fact that the deployment record itself is vague when it names a tag.

If we need to answer “what was running at 02:13?”, we should not need to reconstruct registry history.

Immutable Tags Are Good Advice With A Missing Clause

The usual advice is to use semantic version tags and make them immutable. We did that for a while. Our GitHub Actions workflow built ledger-api:2026.08.3, pushed it to Harbor, and our deployment repository updated the same tag in the Helm values file.

That worked right up to the point it did not.

In February, a build for the ARM64 variant finished after the AMD64 build. The release job had pushed a single-platform manifest under the version tag first, then replaced it with a multi-platform manifest list once the second build arrived. Nobody was trying to smuggle a patched image into production. The workflow was merely assembled in the order that seemed convenient on a Wednesday morning.

The digest changed anyway.

Our production nodes are currently AMD64, so the pods kept working. A staging runner on ARM64 started pulling the newer manifest list. Both environments reported 2026.02.6; they were not consuming the same top-level image object. The difference mattered when an engineer tried to reproduce a staging-only startup failure using the production digest from the cluster event log.

We fixed the build ordering. Then we noticed the larger flaw: even an immutable tag provides a weak deployment record if the tag was resolved at different times by different tools.

Kubernetes has sensible rules here, but they are not a substitute for stating intent. Its image pull policy documentation explains that Always checks the registry on each launch, while the runtime can reuse locally cached layers and manifests as appropriate. If the image reference includes a tag, the kubelet resolves what that tag means when it pulls. If it includes a digest, there is little to interpret.

“Immutable tags” solve accidental replacement. They do not make a tag an exact deployment reference.

Our Release Controller Stopped Guessing In April

We changed the release path for six customer-facing services in April: ledger-api, settlement-worker, webhooks, invoice-renderer, auth-edge, and report-export. GitHub Actions still pushes a readable tag. After the registry has accepted the final multi-architecture manifest, the workflow asks Harbor for its digest and writes that digest into the deployment pull request.

The pull request title still says ledger-api 2026.08.3. The changed line says something less charming:

image: harbor.ops.internal/ledger-api@sha256:5bb4c0e31c4d9f0a

That one line has made incident handling noticeably less theatrical.

On 7 July, invoice-renderer began returning 500 responses for PDFs containing a non-breaking space in an address field. The release had gone out at 10:06. At 10:19, the on-call engineer found the exact digest in the Argo CD application history, opened the prior deployment commit, and reverted one Git change. The replacement pods were ready by 10:27.

Previously, our rollback notes said things like “back to 1.18.4.” This looks clear until someone asks whether 1.18.4 means the original image, the image rebuilt after a base-image CVE, or the one an engineer pushed locally during a failed release rehearsal. We had all three at one point. No amount of confident punctuation fixes that.

The registry tag now has a narrower purpose. It is a discovery handle: find the release, find the digest, inspect the provenance. We attach the Git commit, build run URL, SBOM location, and source tag as OCI annotations. The OCI annotations guidance gives us standard fields for much of this, though we also carry two internal labels because apparently every organisation eventually invents metadata nobody else wants.

This only held because our deployment repository already owned the image reference. Teams using a CI system that applies directly to a cluster have a harder migration. There is no pull request in which to review the digest, and a shell history is a poor change record.

The Best Objection Is That Digests Hurt Humans

The strongest argument against digest-pinned Docker deployments is that they make ordinary work worse.

That argument has teeth. A digest cannot tell us whether it contains version 3.12.1 or 3.12.2 of an application. It cannot tell us which branch produced it. If a developer sees a 71-character hash in a Kubernetes event, they need another system to map it back to the release. A version tag is readable in a way hashes never will be, and pretending otherwise is how we end up making people keep private spreadsheets.

Digests also complicate emergency changes. If an engineer needs to deploy a temporary image at 01:40, asking them to look up a manifest digest, edit a Git file, wait for reconciliation, and check policy output can feel like process taking a victory lap around a burning bin.

We have done the 01:40 version of this. The Docker registry was healthy, the cluster was healthy, and the thing slowing us down was our own release machinery rejecting an image because the SBOM upload had lagged by four minutes. The service had a memory leak, not a compliance emergency. Still, nobody on the call found the delay spiritually enriching.

There is also a real security concern in the opposite direction. A digest pins an image, including a vulnerable one. A floating tag can pick up a patched base image when a pod restarts. That behaviour is uncontrolled, but it may reduce exposure in a narrow case. Pinning forces us to deliberately rebuild and redeploy, which is operationally correct and sometimes operationally inconvenient.

I still prefer the inconvenience. Silent changes in production are worse than explicit changes arriving later than we would like.

Digest Pinning Fails For Some Kinds Of Docker Work

We do not pin every Docker reference by digest.

Local development is the obvious exception. Our compose file uses postgres:16.4, redis:7.4, and a few internally tagged service images because the point is to get a working stack onto a laptop before lunch. Forcing every developer to update a digest for every base-image refresh would create a small administrative cult and very little safety.

Ephemeral preview environments get tags too. A pull request deployment named pr-1847 is intentionally disposable, and its tag tells us more than a hash does when we are cleaning up fifty previews after a busy week. We retain the build metadata, but we do not pretend those environments have the same change-control needs as production.

Base images are messier still. We pin our production Dockerfiles to digests for cgr.dev/chainguard/node, but that means Renovate opens digest update pull requests several times a week. Priya, who owns most of our runtime maintenance, thinks this is a fair price for seeing every base-image movement. I think the PR volume is irritating, although I have not volunteered to triage it, so my objection has limited moral force.

We have not tried digest-based deployment past 40 nodes or across separate regional registries. Our largest cluster has 28 worker nodes, and Harbor sits on the same internal network as it. If a team relies on registry mirrors with uneven replication, a digest may expose propagation mistakes that a tag previously hid. Hiding the mistake was not a solution, obviously, but an unavailable deployment is still unavailable.

There is one unresolved problem in our setup: old digest retention. We want enough history to roll back an image from six months ago, but storing every release, failed build, and preview manifest has pushed Harbor’s object storage bill up 31% since January. We are arguing over whether retention should follow deployment history or vulnerability status. Those rules produce very different bins of images.

Keep Tags For People And Digests For Schedulers

The practical version of this policy is less dramatic than it sounds.

Build an image once. Tag it with a release version, a Git SHA, and whatever temporary CI name helps the workflow. Publish the final manifest only after every required architecture is present. Read the digest from the registry after that publication, rather than calculating it from a local build result. Then place the digest in the production deployment definition.

Keep the readable information close by. Our release pull requests include the version tag, commit SHA, previous digest, new digest, and a link to the registry artifact. Argo CD shows the digest in the application history. Our #release-watch message includes both references, because people searching Slack will type 2026.08.3, not 64 hexadecimal characters.

We also block tag-only production images through admission policy. We chose Kyverno for this because the policy reads like something a tired engineer can inspect during an incident. The initial rollout was report-only for two weeks. It found three CronJobs and a forgotten namespace called data-scratch, which had survived two reorganisations and a platform migration through what I can only call administrative fog.

The policy has an escape hatch for an incident commander, valid for two hours and logged to our audit channel. We have used it once. The follow-up action was not “remove the guardrail”; it was “make the release workflow stop waiting for an irrelevant test suite.”

I would rather explain a long digest during a rollback than explain why the same Docker tag meant different images on different days. We still need to settle retention before the storage bill becomes its own ticket queue.

Tags: devopsDocker