Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Deployment Artifact Promotion: The Custody Chain of Production Code

Overview

Every production outage postmortem contains a sentence about what exactly was running — and every snowflake-deploy horror story begins when that sentence becomes unanswerable. This page maps the artifact promotion pipeline as one connected system: the build that turns source plus lockfile plus toolchain into a digest-addressed artifact, the registry that stores artifacts under mutable tags and immutable digests with its own retention and garbage-collection policies, staging as the first promotion gate, the production promotion decision with its approvals and policy checks, the provenance and signing layer that makes the chain auditable, and the rollback inventory that determines what recovery is even possible at 3 a.m. The unifying principle: an artifact is a content-addressed fact, promotion is the controlled migration of trust in that fact from environment to environment, and every deviation from “same bytes, promoted, never rebuilt” — rebuilding per environment, mutating tags, hotfixing on the box, letting the registry evict running artifacts — reintroduces doubt into a chain whose entire value is doubt’s absence. The failure modes (untracked snowflake deploys, registry eviction eating the rollback path, unsigned artifacts admitting supply-chain injection) all propagate as state that no deploy event explains.

The Nodes, Independently

The build. CI compiles source under a pinned toolchain, resolves dependencies from a lockfile, runs tests, and emits one artifact — container image, jar, static binary — with a content digest. A reproducible build makes the artifact a pure function of an explicit input graph; an unreproducible build makes every downstream guarantee provisional. The digest is the artifact’s identity: sha256:... names one byte sequence, forever. Everything downstream — provenance, admission verification, rollback — assumes this identity is honest, which is why the build stage is where custody begins and where the first integrity check (digest recorded at build time, verified at every pull) belongs.

The artifact registry. The registry stores artifacts and metadata: immutable digests, mutable tags (latest, v1.4.2), vulnerability scan results, push times, and pull counts. It is a store with a cache in front (pull-through proxies) and a garbage collector behind (retention policies that delete untagged or old manifests). Its retention policy is not housekeeping trivia — it defines the rollback horizon. At scale it is also a distributed system in its own right: multi-region replication with per-region pull-through caches mirrors the CDN pattern, because a registry outage during an incident removes both your ability to roll forward and your ability to roll back, which is the worst possible moment to discover the registry was a hidden SPOF.

Staging. The first promotion gate: the artifact runs against production-shaped (never production-identical) config and data. Staging’s honest purpose is rehearsing the deployment, not proving the software — parity gaps mean some failures only ever appear in production.

Promotion gates. The mechanism that moves one digest from environment to environment: automated checks (tests, policy scans, smoke tests), human approvals where regulation demands them, and environment-level promotion that references the digest — never a rebuild. Promotion is a state transition on trust, and gates are its transaction protocol.

Provenance and signing. Attestations record how the artifact was built (which repo, which commit, which builder — SLSA-style provenance) and signatures bind the digest to an identity; SBOMs enumerate the input graph’s dependencies. Admission controllers verify signatures and provenance at deploy time, turning “we trust CI” into “we can prove this artifact came from CI.”

Rollback inventory. The set of prior artifacts still pullable, still compatible with the current schema, and still documented. Rollback capability is manufactured entirely upstream: it exists only if the registry retained the artifact, the migration contract allows the old code to run, and someone recorded which digest preceded this one.

How the Nodes Feed Each Other

One digest should flow through the entire chain: build emits sha256:A → registry stores it → staging pulls sha256:A → gates verify and approve sha256:A → production runs sha256:A. Provenance rides along as metadata, and admission verification closes the loop by rejecting anything whose chain of custody is broken. Everything that branches from this single-digest invariant weakens the chain: rebuilding per environment produces different bytes from the same source (which one is “the” release?); retagging mutates pointers without an audit event; hotfixes applied directly to production workloads create state outside the registry entirely. The rollback inventory feeds backward: the registry’s retention determines what rollback can pull, the migration design determines what rollback can run, and the promotion log determines what rollback can name. Feature flags and config, meanwhile, deliberately ride outside the artifact — decoupling release from deploy — which is safe only because the artifact boundary is otherwise so rigid.

Promotion Topologies and Versioning

The linear chain (dev → staging → prod) is the textbook topology; production systems branch it in ways that preserve the invariants. A release train batches merged work into scheduled promotion candidates so prod never runs arbitrary heads; a ring topology (internal dogfood → early adopters → everyone) replaces the staging environment with progressively larger live cohorts, trading environment parity for real traffic. Whatever the shape, the two properties that must survive are single-digest custody (every environment runs bytes whose provenance chain is intact) and monotone trust (each hop adds evidence — tests, scans, soak time — rather than resetting it). Versioning sits at the top of the chain and quietly governs everything below: semantic versioning of the artifact, compatibility versioning of its schema and API surfaces, and the promotion policy that refuses to move an artifact forward if its declared compatibility range excludes what the next environment runs. Teams that treat version numbers as decoration rediscover their meaning during rollback, when “can the old artifact talk to the new schema” is the only question that matters.

The Shared Invariants

Immutability by addressing. A digest means one byte sequence forever; anything mutable (tags, environment config) must be a pointer with an audit trail, never the identity itself. This is the content-addressing principle that runs Git, Nix, and Docker layers, applied to deployment custody. Build once, promote everywhere. The same bytes must traverse dev, staging, and production; a rebuild at any hop breaks provenance and introduces drift between environments that “the same code” cannot explain. Provenance completeness. An artifact whose inputs are unknown is unverifiable by construction; SBOM and provenance make the input graph explicit so that verification is mechanical rather than testimonial. Rollback is bounded by retention plus compatibility. You can return to the previous artifact only if it still exists and the datastore still accepts it — which couples this map to expand-contract migration discipline, because rollback without schema compatibility is a fantasy with a pager attached.

Failure Propagation

Snowflake deploys are the canonical custody break: an operator SSHes to a box and edits a file at 2 a.m., the fix works, and now production contains state no deploy event explains. The failure propagates temporally — the next legitimate rollback or autoscaling event reintroduces the original bug, and the next rebuild diverges from the running configuration in ways nobody can enumerate. Registry eviction is the quiet one: a retention policy deletes “untagged” or 60-day-old images, one of which is precisely the artifact currently running in production (tag deleted in a cleanup PR), and the next incident discovers that rollback cannot pull what the fleet is running — recovery is forced forward-only, during an incident, by choice of a GC job. Unsigned or unverified promotion propagates adversarially: a compromised CI or a typosquatted dependency lands in an artifact, and without admission-time signature and provenance verification the compromise deploys with full ceremony — detection deferred to post-incident forensics. Tag mutation is the silent artifact drift: promoting prod by mutable tag, then re-pointing that tag, changes what production pulls without any deploy event — the pull-based fleet upgrades itself on its own schedule. Staging parity gaps propagate as surprise: the gate passes on config staging never models (queue depths, data volume, feature-flag state), and the failure appears first in production at full blast radius. Rollback without compatibility is the compound failure: the team re-points production to the previous digest, the datastore has already applied a contract-breaking migration, and the “rollback” now fails on boot — the incident doubles because the artifact chain and the schema chain were governed by different clocks. Each of these failures shares a signature: the system was healthy by every deploy-log reading, and the divergence lived in state the pipeline never tracked — which is why the invariants above are stated as custody claims rather than process advice.

Design Levers

Immutable digests everywhere, with admission enforcement is the keystone: promotion references digests, admission controllers require valid signatures and provenance (SLSA-aligned attestations), and mutable tags exist only as dev-time conveniences that production tooling refuses to resolve. Promotion gates as policy-as-code make the trust transition mechanical — required checks, approvers, environment ordering — so “how did this get to prod” is a query, not an investigation. SBOM plus provenance gives incident response a complete input graph: when the next ecosystem vulnerability lands, “are we affected” is a database query over SBOMs, not a fleet-wide guessing exercise. Retention aligned to the rollback horizon: keep the last N release artifacts for at least the mean-time-to-decision of a bad release (and pin the currently-running set explicitly, exempting it from GC), so the registry’s garbage collector can never eat the recovery path. Separate build from release configuration: config lives out-of-band, versioned and applied at deploy time, so the artifact stays environment-agnostic and “same bytes” stays true. The lever that must stay unpulled is the per-environment rebuild — it dissolves every other guarantee in this list.

A Concrete Worked Example

A team ships container images built once in CI: sha256:9f3c... from commit d41fa2, SLSA-provenance attested and signed at build, SBOM attached. Promotion is dev → staging → prod by digest, with a policy gate requiring scan-clean status and one reviewer for prod. The registry retains 90 days of release digests plus anything currently running; releases go out twice weekly, so ten releases span five weeks — comfortably inside retention. At 11:00, release 447 (sha256:77ab...) causes 500s on the checkout path; the rollback is kubectl set image back to the digest of 446, already cached on most nodes, restoring service in under a minute — no rebuild (which would have taken 12 minutes and produced different bytes than the tested artifact), no schema conflict (447’s migration was expand-only, so 446’s code runs happily). Contrast the neighboring team’s incident the same week: their on-call hotfixed a config file directly on three production hosts at 2 a.m.; Thursday’s rollout rebuilt and redeployed, silently reverting the hotfix on two of the three hosts (the third had drifted differently); the resulting intermittent failure took four hours to diagnose because production state and deploy history told different stories. Same industry, same stack; the difference was whether the bytes in production had a custody chain.

Interview Questions

  1. Why does “build once, promote everywhere” matter more than it sounds? Trace two concrete failure chains that a per-environment rebuild enables even with perfect tests.
  2. Design registry retention policy for a fleet doing 30 releases per week across 12 services. What must never be collected, how do you pin the running set, and what incident does your policy prevent?
  3. A critical vulnerability is announced in a build tool. Explain precisely how SBOMs, provenance attestations, and digest-addressed promotion turn “are we affected?” into a query — and what stays manual when any one of the three is missing.
  4. Compare rollback by artifact re-point versus forward-fix under: (a) an expand-only migration, (b) a contract-breaking migration deployed 30 minutes earlier. What does each answer depend on?
  5. Your promotion pipeline “works” but production drifts from it anyway. Name four mechanisms by which untracked state enters a fleet, and the invariant or tooling that closes each.

Key Takeaways

  • An artifact is a content-addressed fact; promotion is a state transition on trust in that fact, and the whole system’s value is that “what is running where” is always answerable from an audit log.
  • Build once, promote everywhere: every rebuild per environment, mutable tag resolution, or on-box hotfix reintroduces unexplained state — the root cause of snowflake-deploy incidents.
  • The registry’s retention policy and the migration contract jointly define whether rollback exists; pin the running set and keep the rollback horizon inside retention, or GC will eat your recovery path during the next incident.
  • Provenance, signing, and SBOMs convert supply-chain trust from testimonial to mechanical, and admission-time verification is where that mechanism must live.
  • Rollback speed is manufactured upstream — digest addressing, node-level image caching, expand-only migrations — never improvised during the incident.

Cross-References