Define atomicity and rollback semantics for OCI artifact publication

[reneleonhardt]

Description

Cosign publishes OCI artifacts through several registry operations. A config and all referenced layer blobs must exist before an artifact manifest is accepted, so the OCI Distribution API provides no transaction spanning the complete publication.

The writers do not currently share an explicit partial-failure contract. In particular, the local-layout referrer path in WriteSignedImageIndexImages uploads the config and attempts the manifest before uploading its bundle layers. A conforming registry rejects that manifest because referenced blobs are missing; AWS ECR reports BLOB_UPLOAD_UNKNOWN in #5030. The layer loop is therefore never reached, while image, index, signature, or attestation objects written earlier by the same call may already be committed. Manifest-last writers such as WriteReferrer avoid that ordering failure, but a config, layer, cancellation, authentication, transport, artifact-manifest, or discovery-index failure can still leave partial state.

The affected publication protocols have different commit points and must not be treated as one blanket transaction:

Path	Published state	Transaction concern
WriteReferrer, WriteSignaturesExperimentalOCI, WriteAttestationNewBundleFormat, WriteAttestationsReferrer	Blobs, digest-addressed artifact manifest, optional referrers-tag fallback manifest	The artifact may exist even if fallback discovery fails; fallback read-modify-write can lose concurrent updates.
WriteSignedImageIndexImages / cosign load	Image/index, legacy signature and attestation tags, then local-layout referrers	Earlier objects may be committed before a later referrer fails; its current referrer ordering is invalid.
WriteSignatures, WriteAttestations	GGCR-managed blobs and mutable .sig / .att tag manifests	Retry or concurrent tag replacement is not automatically idempotent.
Attach SBOM, upload WASM, recursive image/index writes	GGCR-managed blob and one or more manifest writes	Ordering and retry units are owned by GGCR; multi-object callers may still partially commit.

The exact CLI callers and GGCR-owned behavior should be mapped before selecting code changes; the immediate ordering correction belongs to the known local-layout path, while this issue defines the cross-writer contract.

Generic client-side rollback is unsafe:

• Blob deletion is optional, policy-dependent, and may be unavailable to the publishing identity.
• Blobs are content-addressed and may already exist or be shared by other manifests.
• A client cannot generally prove that it created a blob exclusively during the failed attempt.
• Deleting a shared blob can corrupt an unrelated artifact; failed deletion still cannot restore atomicity.
• An invocation-owned monolithic or chunked upload session is distinct from a completed shared blob and may have a capability-scoped cancellation operation.

The desired baseline is therefore manifest-last publication with idempotent retry, preserved failure causes, and no implicit delete rollback. Before changing the public behavior, maintainers should agree on the following contract:

1. What are the commit points and retry unit for each publication path?
2. Should publication errors identify the failed phase and successfully processed descriptors, without exposing credentials or registry responses containing secrets?
3. Is cleanup ever allowed without an explicit registry capability and proof that the invocation created exclusive content?
4. How should cancellation, authentication, transport, and manifest-conflict errors remain distinguishable?
5. When an artifact manifest succeeds but its fallback discovery-index update fails, is the result partial success, and how should retry discover and repair it?
6. Must fallback indexes use conditional requests or conflict retry to prevent concurrent lost updates?
7. Should the client cancel only an exact upload-session UUID it owns, when the registry supports that operation?
8. Which writers require code changes versus documentation-only treatment?

Proposed design:

• Perform all locally available structural and publication-option validation before the first mutation; network authentication remains subject to expiry and TOCTOU failure.
• Upload every required blob before publishing the manifest.
• Model phases as blobs, artifact manifest, and any optional fallback discovery-index read/conditional update.
• Stop on the first failure within a publication unit; report which earlier commit points succeeded.
• Return the original cause with publication-phase context.
• Treat content-addressed blob and digest-addressed artifact operations as reusable on retry; do not extend that claim to mutable tags, fallback indexes, or multi-object workflows without conflict-safe proofs.
• Never delete a completed blob or artifact manifest merely because a later discovery update failed.
• Decide separately whether to cancel an exact invocation-owned upload session; do not infer permission to delete completed content.
• Leave completed unreferenced blobs subject to registry retention and garbage-collection policy, which may retain them indefinitely; expose enough phase/descriptor information for operators to diagnose storage accumulation without leaking credentials.
• Do not promise atomicity across repositories or nonstandard registry APIs.

Required failure-injection coverage:

• Each config, layer, and manifest step fails independently.
• Existing and newly uploaded blob digests.
• Shared digests and registries where deletion is unsupported or forbidden.
• Exact upload-session cancellation supported, unsupported, forbidden, and interrupted.
• Retry after every partial-failure point, with idempotence asserted only per proven operation.
• Artifact success followed by fallback-index failure and repair.
• Referrers API and fallback-tag paths, including concurrent conflict and lost-update cases.
• Mutable legacy tags and multi-manifest image/index workflows.
• Cancellation, timeout, authentication, TLS, and transport failures with preserved causes.
• No later write after failure and no credentials in errors or logs.
• Repeated failures and observable orphan-storage accumulation under disabled or delayed garbage collection.

As of 2026-08-02, open issue #5030 reports the cosign load ordering failure and open PR #5031 proposes its immediate correction. This issue is follow-on design for cross-writer partial-failure semantics, not an alternative implementation of that fix. Closed issues #507 and #3577 are historical context for registry garbage collection and the bundle-as-OCI-artifact format; neither defines cross-writer rollback or recovery semantics.

Version

v3.0.0-20260731151431-83d9ec8f4bdb go1.26.5 darwin/arm64