Docker Tags Make Production Rollbacks Ambiguous
Our production manifests should name bytes, not somebody’s label.
While the registry UI was showing ledger-api:2026.08.3 with two different upload times, I decided we should deploy Docker images by digest, even when a version tag looks perfectly respectable. A tag is a label somebody can move; production should name the exact bytes it intends to run.
A Docker Tag Is A Promise We Cannot Verify Later
We still create tags. Developers need them, release notes need them, and nobody wants to read a digest aloud during an incident unless morale has already collapsed. 2026.08.3 tells a human roughly what they are looking at. sha256:5bb4… tells a machine precisely what it is looking at.
Those are different jobs, and treating one identifier as both has caused more confusion than it deserves.
A Docker image digest identifies a content-addressed manifest. Docker documents the distinction clearly in its image digests reference: tags can change, while a digest points to a specific image manifest. The OCI image specification is equally blunt about descriptors being identified by digest. We knew this, of course. Knowing a thing and arranging work around it are separate departments.
My opinion is that a production deployment manifest should contain registry.example.net/ledger-api@sha256:..., not registry.example.net/ledger-api:2026.08.3. The tag belongs beside it in Git, in the release ticket, and in the message we send when a deploy starts. It does not belong where the scheduler decides what to pull.
The common reply is that version tags are immutable in a well-run registry. That is usually shorthand for “we asked people not to overwrite them.” A permission policy is better. A registry setting that rejects overwrites is better still. Neither changes the fact that the deployment record itself is vague when it names a tag.
If we need to answer “what was running at 02:13?”, we should not need to reconstruct registry history.
Immutable Tags Are Good Advice With A Missing Clause
The usual advice is to use semantic version tags and make them immutable. We did that for a while. Our GitHub Actions workflow built ledger-api:2026.08.3, pushed it to Harbor, and our deployment repository updated the same tag in the Helm values file.
That worked right up to the point it did not.
In February, a build for the ARM64 variant finished after the AMD64 build. The release job had pushed a single-platform manifest under the version tag first, then replaced it with a multi-platform manifest list once the second build arrived. Nobody was trying to smuggle a patched image into production. The workflow was merely assembled in the order that seemed convenient on a Wednesday morning.
The digest changed anyway.
Our production nodes are currently AMD64, so the pods kept working. A staging runner on ARM64 started pulling the newer manifest list. Both environments reported 2026.02.6; they were not consuming the same top-level image object. The difference mattered when an engineer tried to reproduce a staging-only startup failure using the production digest from the cluster event log.
We fixed the build ordering. Then we noticed the larger flaw: even an immutable tag provides a weak deployment record if the tag was resolved at different times by different tools.
Kubernetes has sensible rules here, but they are not a substitute for stating intent. Its image pull policy documentation explains that Always checks the registry on each launch, while the runtime can reuse locally cached layers and manifests as appropriate. If the image reference includes a tag, the kubelet resolves what that tag means when it pulls. If it includes a digest, there is little to interpret.
“Immutable tags” solve accidental replacement. They do not make a tag an exact deployment reference.
Our Release Controller Stopped Guessing In April
We changed the release path for six customer-facing services in April: ledger-api, settlement-worker, webhooks, invoice-renderer, auth-edge, and report-export. GitHub Actions still pushes a readable tag. After the registry has accepted the final multi-architecture manifest, the workflow asks Harbor for its digest and writes that digest into the deployment pull request.
The pull request title still says ledger-api 2026.08.3. The changed line says something less charming:
image: harbor.ops.internal/ledger-api@sha256:5bb4c0e31c4d9f0a
That one line has made incident handling noticeably less theatrical.
On 7 July, invoice-renderer began returning 500 responses for PDFs containing a non-breaking space in an address field. The release had gone out at 10:06. At 10:19, the on-call engineer found the exact digest in the Argo CD application history, opened the prior deployment commit, and reverted one Git change. The replacement pods were ready by 10:27.
Previously, our rollback notes said things like “back to 1.18.4.” This looks clear until someone asks whether 1.18.4 means the original image, the image rebuilt after a base-image CVE, or the one an engineer pushed locally during a failed release rehearsal. We had all three at one point. No amount of confident punctuation fixes that.
The registry tag now has a narrower purpose. It is a discovery handle: find the release, find the digest, inspect the provenance. We attach the Git commit, build run URL, SBOM location, and source tag as OCI annotations. The OCI annotations guidance gives us standard fields for much of this, though we also carry two internal labels because apparently every organisation eventually invents metadata nobody else wants.
This only held because our deployment repository already owned the image reference. Teams using a CI system that applies directly to a cluster have a harder migration. There is no pull request in which to review the digest, and a shell history is a poor change record.
The Best Objection Is That Digests Hurt Humans
The strongest argument against digest-pinned Docker deployments is that they make ordinary work worse.
That argument has teeth. A digest cannot tell us whether it contains version 3.12.1 or 3.12.2 of an application. It cannot tell us which branch produced it. If a developer sees a 71-character hash in a Kubernetes event, they need another system to map it back to the release. A version tag is readable in a way hashes never will be, and pretending otherwise is how we end up making people keep private spreadsheets.
Digests also complicate emergency changes. If an engineer needs to deploy a temporary image at 01:40, asking them to look up a manifest digest, edit a Git file, wait for reconciliation, and check policy output can feel like process taking a victory lap around a burning bin.
We have done the 01:40 version of this. The Docker registry was healthy, the cluster was healthy, and the thing slowing us down was our own release machinery rejecting an image because the SBOM upload had lagged by four minutes. The service had a memory leak, not a compliance emergency. Still, nobody on the call found the delay spiritually enriching.
There is also a real security concern in the opposite direction. A digest pins an image, including a vulnerable one. A floating tag can pick up a patched base image when a pod restarts. That behaviour is uncontrolled, but it may reduce exposure in a narrow case. Pinning forces us to deliberately rebuild and redeploy, which is operationally correct and sometimes operationally inconvenient.
I still prefer the inconvenience. Silent changes in production are worse than explicit changes arriving later than we would like.
Digest Pinning Fails For Some Kinds Of Docker Work
We do not pin every Docker reference by digest.
Local development is the obvious exception. Our compose file uses postgres:16.4, redis:7.4, and a few internally tagged service images because the point is to get a working stack onto a laptop before lunch. Forcing every developer to update a digest for every base-image refresh would create a small administrative cult and very little safety.
Ephemeral preview environments get tags too. A pull request deployment named pr-1847 is intentionally disposable, and its tag tells us more than a hash does when we are cleaning up fifty previews after a busy week. We retain the build metadata, but we do not pretend those environments have the same change-control needs as production.
Base images are messier still. We pin our production Dockerfiles to digests for cgr.dev/chainguard/node, but that means Renovate opens digest update pull requests several times a week. Priya, who owns most of our runtime maintenance, thinks this is a fair price for seeing every base-image movement. I think the PR volume is irritating, although I have not volunteered to triage it, so my objection has limited moral force.
We have not tried digest-based deployment past 40 nodes or across separate regional registries. Our largest cluster has 28 worker nodes, and Harbor sits on the same internal network as it. If a team relies on registry mirrors with uneven replication, a digest may expose propagation mistakes that a tag previously hid. Hiding the mistake was not a solution, obviously, but an unavailable deployment is still unavailable.
There is one unresolved problem in our setup: old digest retention. We want enough history to roll back an image from six months ago, but storing every release, failed build, and preview manifest has pushed Harbor’s object storage bill up 31% since January. We are arguing over whether retention should follow deployment history or vulnerability status. Those rules produce very different bins of images.
Keep Tags For People And Digests For Schedulers
The practical version of this policy is less dramatic than it sounds.
Build an image once. Tag it with a release version, a Git SHA, and whatever temporary CI name helps the workflow. Publish the final manifest only after every required architecture is present. Read the digest from the registry after that publication, rather than calculating it from a local build result. Then place the digest in the production deployment definition.
Keep the readable information close by. Our release pull requests include the version tag, commit SHA, previous digest, new digest, and a link to the registry artifact. Argo CD shows the digest in the application history. Our #release-watch message includes both references, because people searching Slack will type 2026.08.3, not 64 hexadecimal characters.
We also block tag-only production images through admission policy. We chose Kyverno for this because the policy reads like something a tired engineer can inspect during an incident. The initial rollout was report-only for two weeks. It found three CronJobs and a forgotten namespace called data-scratch, which had survived two reorganisations and a platform migration through what I can only call administrative fog.
The policy has an escape hatch for an incident commander, valid for two hours and logged to our audit channel. We have used it once. The follow-up action was not “remove the guardrail”; it was “make the release workflow stop waiting for an irrelevant test suite.”
I would rather explain a long digest during a rollback than explain why the same Docker tag meant different images on different days. We still need to settle retention before the storage bill becomes its own ticket queue.
After our prod wobble, havent tried digest pins. Will.
I disagree that a digest leaves little to interpret: who approved it and whether provenance is trusted still matter. We had 3 registry credential incidents in 18 months, including an outage from a leaked CI token; pinning bytes didnt reduce that attack surface.
I disagree that pinning is irrelevant to a leaked-token risk, because it can still limit what an approved deployment resolves to after the fact. Would you treat credential hardening and digest review as separate controls when budgeting the vendor work
but how do you join digest-level deployments to experiment results and feature flags? at my previous employer, the model registry recorded both image digest and dataset snapshot; would the oci annotations be enough for that?
I am genuinely excited by a control I can explain to leadership without buying another platform. I disagree that direct-to-cluster CI has a harder migration, because our vendor’s deployment audit trail can serve as review evidence if configured properly. The benefit is fewer midnight reconstruction calls and less reliance on the one engineer who knows Harbor. I would still need to price any registry retention increase into next year’s contract
and “the exact bytes” is what i needed during our payment outage last month. We couldnt work out which image was live, so this sounds brilliant
but the rollback only gets interesting when the image contains a schema migration. What proves that reverting the digest at 10:19 did not put invoice-renderer against a newer schema or hold a lock? I have seen one bad migration turn a five-minute rollback into a 47-minute outage. Please do a follow-up on digest-pinned releases alongside expand/contract migrations and how you record teh schema version.
Yes, this is where rollback diagrams get more interesting… image state, schema state, and topology have to move on compatible paths, not merely revert together. At scale, a regional rollback can leave old workers consuming messages against a newer schema, costs climb quickly when the safe answer is to hold extra capacity during the transition. Please do a follow-up on digest-pinned releases across multi-cluster expand/contract rollouts, including what the deployment controller records for schema compatibility. The digest proves the container, it doesnt prove the database contract
and the approval boundary is much clearer when the pull request contains a digest rather than a moving tag. it also gives security a kinder way to ask, “who approved these bytes?”, while protecting the team from a CI token that has been granted too much access
we run next.js on github actions, and 4 of our 6 rollback arguments this year were really “which build was that?” arguments. management will not fund another release engineer, so writing the digest into argocd is the cheap fix.
I have not tried this yet, but I plan to add it to our next small service release. At my job, we once spent an afternoon comparing screenshots because the tag in the ticket did not match what a restarted container pulled. Would the digest be visible somewhere friendly in the deployment screen, or do people mostly copy it from Git? I disagree that no one wants to read a digest during an incident, because a copy button and a clear release name could make that manageable. Does this work the same way if we use Docker Compose for a small internal app? I am wondering whether the extra step is difficult for people who do not work in infrastructure every day.
I disagree that a shell history is always a poor change record, because our managed deployment vendor retains approved command logs. Would digest pinning increase registry storage or support costs enough that I need to budget for it?
The hard part is not the digest, it is making the Argo history legible when somebody is tired. In 12 incident reviews, we lost more time translating opaque identifiers than finding them. i have not tried a digest-first dashboard yet, but I plan to mock one with the human tag next to the commit. If the UI just swaps a tag for 64 characters, cognitive load gets worse, not better.
we lost 20 minutes in a payments outage decoding an image id, so the tag and commit need to sit beside the digest. for a small fintech like ours, i wouldnt ship a digest-only view
i disagree: our outage showed tags worked; havent tried pins, will!
They may have worked in that outage, but I would be careful about calling that proof that the tag was sufficient. A tag can be perfectly well behaved until a rebuild, a manifest-list update, or a hurried person turns it into a tiny mystery novel. We had an incident at my job where a restart pulled a different image under the same release tag, and I got to spend part of the night explaining why both sides of the argument had screenshots. Nobody had done anything malicious; our process had simply left too much to lookup time. Pins do not replace review, but they make the deployment record say what we actually meant. Trying them on one small service is a sensible way to find the awkward bits before everyones manifest changes at once
I disagree that tags worked, because getting through one outage does not give me an auditable answer for the next one. What would pins cost your team in process changes versus the time spent proving which image ran