feat: add oci-push promotion step by EronWright · Pull Request #5782 · akuity/kargo

EronWright · 2026-02-24T00:17:16Z

Summary

Adds a new oci-push promotion step that copies/retags OCI artifacts (container images and Helm charts) between registries.

Extracts shared OCI helpers (parseOCIReference, buildOCIRemoteOptions, etc.) from oci-download into oci_common.go so both steps reuse the same credential resolution and transport logic
New step runner registered as oci-push with StepCapabilityAccessCredentials
Supports single images, multi-arch image indexes, and Helm charts (via oci:// prefix)
Optional annotations field to set OCI manifest annotations on the pushed artifact, with scoped prefixes (index:, manifest:) for controlling placement on image indexes vs child manifests
Custom annotation mutation wrappers (extracted into mutate.go) to work around a limitation of go-containerregistry.
1 GiB compressed size limit enforced for cross-repository copies; same-repo retags skip the check since blobs are already present
Step outputs: image (destination ref), digest (sha256), tag

Closes #3762

Test plan

Unit tests pass (go test -race ./pkg/promotion/runner/builtin/...)
Linter passes (make lint-go — no issues in changed files)
E2E: Helm chart cross-repo copy on GHCR with custom annotations
E2E: nginx image (~70 MB) cross-repo copy — succeeded with size check
E2E: pulumi/pulumi (~6.1 GiB) cross-repo copy — rejected with compressed artifact size 6.1 GiB exceeds maximum allowed size of 1.0 GiB
E2E: pulumi/pulumi (~6.1 GiB) same-repo retag + annotate — succeeded (size check bypassed)

Implements a new `oci-push` builtin promotion step that copies or retags OCI artifacts (container images and Helm charts) between registries as part of a promotion pipeline. Uses go-containerregistry for server-side copy with support for multi-arch images and manifest annotations. Also extracts shared OCI registry helpers (reference parsing, credential resolution, HTTP transport) from oci-download into oci_common.go for reuse by both steps. Refs: #3762 Signed-off-by: Eron Wright <eron.wright@akuity.io>

netlify · 2026-02-24T00:17:22Z

✅ Deploy Preview for docs-kargo-io ready!

Name	Link
🔨 Latest commit	`8a81a2b`
🔍 Latest deploy log	https://app.netlify.com/projects/docs-kargo-io/deploys/699f981b9dcad5000837b076
😎 Deploy Preview	https://deploy-preview-5782.docs.kargo.io
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

EronWright · 2026-02-24T00:19:58Z

E2E Test Results

Tested the oci-push step end-to-end on a local Tilt dev cluster (OrbStack) with a Helm chart hosted on GHCR.

Setup

Pushed two chart versions (0.1.0, 0.2.0) to oci://ghcr.io/eronwright/charts/mychart
Created oci-push-demo project with:
- Helm credential secret for GHCR
- Warehouse subscribing to oci://ghcr.io/eronwright/charts/mychart (semver >=0.1.0)
- Stage with an oci-push step to retag the chart as :promoted with custom annotations

Promotion

steps:
- uses: oci-push
  as: retag-chart
  config:
    imageRef: "oci://ghcr.io/eronwright/charts/mychart:${{ chartFrom('oci://ghcr.io/eronwright/charts/mychart').Version }}"
    destRef: "oci://ghcr.io/eronwright/charts/mychart:promoted"
    annotations:
      io.kargo.promoted-by: "kargo"
      io.kargo.stage: "${{ ctx.stage }}"

Result: ✅ Succeeded

Step outputs:

image:  ghcr.io/eronwright/charts/mychart:promoted
digest: sha256:f2cca6aef681692050ad7262b90cc3e2b9e3fea7d09013217bca498480fddd54
tag:    promoted

Manifest annotations verified on GHCR:

{
  "io.kargo.promoted-by": "kargo",
  "io.kargo.stage": "retag",
  "org.opencontainers.image.created": "2026-02-23T16:06:15-08:00",
  "org.opencontainers.image.description": "A minimal chart for testing oci-push",
  "org.opencontainers.image.title": "mychart",
  "org.opencontainers.image.version": "0.2.0"
}

Signed-off-by: Eron Wright <eron.wright@akuity.io>

codecov · 2026-02-24T01:25:42Z

Codecov Report

❌ Patch coverage is 70.33898% with 105 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.33%. Comparing base (2c8161b) to head (8a81a2b).
⚠️ Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/image/mutate/mutate.go	58.45%	44 Missing and 15 partials ⚠️
pkg/promotion/runner/builtin/oci_pusher.go	77.63%	23 Missing and 13 partials ⚠️
pkg/fmt/bytes.go	0.00%	10 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5782      +/-   ##
==========================================
+ Coverage   56.16%   56.33%   +0.17%     
==========================================
  Files         450      455       +5     
  Lines       37742    38155     +413     
==========================================
+ Hits        21198    21496     +298     
- Misses      15284    15370      +86     
- Partials     1260     1289      +29

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mutate.Annotations wraps images in mutate.image, whose Layers() enumerates layers via ConfigFile().RootFS.DiffIDs. For non-Docker OCI artifacts (e.g. Helm charts), the config blob has no RootFS field, so DiffIDs is empty and Layers() returns nothing. This causes blobs to be omitted from cross-repository pushes, resulting in MANIFEST_BLOB_UNKNOWN errors on registries like GHCR. Replace mutate.Annotations with thin annotatedImage and annotatedIndex wrappers that override only manifest-related methods (Manifest, RawManifest, Digest, Size) while delegating Layers() and all other methods to the base image. This preserves MountableLayer wrapping for cross-repo blob mounting and avoids the broken DiffIDs enumeration path. Signed-off-by: Eron Wright <eron.wright@akuity.io>

EronWright · 2026-02-24T05:55:53Z

E2E Test Report: Cross-repo Helm chart push with annotations on GHCR

Tested cross-repository OCI push of a Helm chart with annotations to a brand-new GHCR repository (no pre-existing blobs).

Test setup (namespace oci-push-demo, local Tilt cluster):

Source: oci://ghcr.io/eronwright/charts/mychart:0.2.0
Destination: oci://ghcr.io/eronwright/charts/mychart-prod3:0.2.0 (fresh repo, never pushed to)
Annotations: io.kargo.promoted-by, io.kargo.source-repo, io.kargo.test

Result: Succeeded

Promotion: cross-repo-wrapper-4jrbj
Digest:    sha256:e2264815f65afd471aee8c1df274932d72cccb6ab43013df854d91e8f236cee5

Manifest on GHCR (docker manifest inspect):

{
  "config": { "mediaType": "application/vnd.cncf.helm.config.v1+json" },
  "layers": [{ "mediaType": "application/vnd.cncf.helm.chart.content.v1.tar+gzip" }],
  "annotations": {
    "io.kargo.promoted-by": "kargo",
    "io.kargo.source-repo": "ghcr.io/eronwright/charts/mychart",
    "io.kargo.test": "custom-wrapper",
    "org.opencontainers.image.created": "2026-02-23T16:06:15-08:00",
    "org.opencontainers.image.description": "A minimal chart for testing oci-push",
    "org.opencontainers.image.title": "mychart",
    "org.opencontainers.image.version": "0.2.0"
  }
}

Config and layer blobs transferred correctly (no MANIFEST_BLOB_UNKNOWN)
All three custom annotations applied
Original source annotations preserved
Single write (no two-phase workaround)

Previously tested in earlier iterations: retag-in-place (same repo), cross-repo without annotations, image index push. All passed.

Allow annotation keys to be prefixed with "index:" or "manifest:" to control whether they target the index manifest or child image manifests. Unprefixed keys default to the image manifest. For single images, "index:"-prefixed keys are silently ignored. When manifest annotations are applied to an image index, child descriptors are rewritten with updated digests and sizes to stay consistent with the annotated content. Signed-off-by: Eron Wright <eron.wright@akuity.io> Entire-Checkpoint: e87ef149e998

Move annotatedImage/annotatedIndex types from oci_pusher.go into a dedicated mutate.go with a unified Annotations() entrypoint that type-checks for v1.Image vs v1.ImageIndex. Adopt the compute() pattern with sync.Mutex for thread-safe lazy evaluation, matching the go-containerregistry mutate package style. Consolidate pushImage/pushIndex into a single push() method that delegates annotation handling to Annotations(). Add mutate_test.go covering Docker images, OCI/Helm images (empty DiffIDs regression), and multi-arch indexes. Signed-off-by: Eron Wright <eron.wright@akuity.io>

Signed-off-by: Eron Wright <eron.wright@akuity.io>

EronWright · 2026-02-24T22:03:45Z

E2E Test: Container Image (multi-arch nginx)

Tested oci-push with a multi-arch container image (public.ecr.aws/nginx/nginx:1.29.5, Docker manifest list with amd64+arm64) pushed cross-registry to GHCR.

Pure copy (no annotations)

Digest is preserved across registries:

	Digest
Source (`public.ecr.aws`)	`sha256:92038aacb18ce74dd5c454f03351dbd9c6ba27c75fd6c1a7658c899ae11ed496`
Destination (`ghcr.io`)	`sha256:92038aacb18ce74dd5c454f03351dbd9c6ba27c75fd6c1a7658c899ae11ed496`

The pushed image was verified with docker pull + docker run.

With scoped annotations

Annotations used:

annotations:
  io.kargo.promoted-by: kargo
  io.kargo.source-repo: public.ecr.aws/nginx/nginx
  "index:io.kargo.index-test": index-annotation-value
  "manifest:io.kargo.manifest-test": manifest-annotation-value

All digests changed as expected (annotations mutate manifest content):

Level	Pure copy	With annotations
Index	`sha256:9203...`	`sha256:7126...`
amd64	`sha256:b137...`	`sha256:3c6d...`
arm64	`sha256:bde5...`	`sha256:6e1e...`

Annotation scoping worked correctly:

Index manifest: only io.kargo.index-test (prefix stripped)
Child manifests: io.kargo.promoted-by, io.kargo.source-repo (unprefixed → manifest scope), io.kargo.manifest-test (prefix stripped)

Enforce a 1 GiB maximum compressed artifact size when oci-push copies artifacts across repositories, preventing accidental promotion of very large images through the pipeline. The limit is skipped for same-repo retags since no blob transfer occurs. Size is computed from manifest metadata without downloading blobs. Signed-off-by: Eron Wright <eron.wright@akuity.io>

EronWright · 2026-02-25T00:37:37Z

Added a 1 GiB size limit for cross-repository copies in oci-push. The limit is enforced before any blobs are transferred, using only manifest metadata to compute the compressed artifact size (config + all layers, summed across child images for indexes).

Same-repository retags skip the check since no blob transfer occurs.

Tested against the live cluster:

Happy path: public.ecr.aws/nginx/nginx:1.29.5 (~70 MB) cross-repo push succeeded
Size limit: pulumi/pulumi:latest (~6.1 GiB) was rejected with: compressed artifact size 6.1 GiB exceeds maximum allowed size of 1.0 GiB

EronWright · 2026-02-25T01:00:53Z

Verified that same-repo retag + annotate correctly bypasses the size limit:

Copied pulumi/pulumi:latest (~6.1 GiB) to ghcr.io/eronwright/test-images/pulumi:latest using oras
Created a promotion to retag and annotate within the same repo → succeeded (size check skipped since blobs are already present)
Verified annotations landed on the child manifests

So the three cases work as expected:

Cross-repo copy of large image → rejected with compressed artifact size 6.1 GiB exceeds maximum allowed size of 1.0 GiB
Same-repo retag + annotate of large image → succeeded, no size check
Cross-repo copy of normal image → succeeded, size check passed

EronWright · 2026-02-25T19:01:37Z

Closing this PR for now, in favor of a more limited solution focusing on tagging, not replication.

To recap, this PR offers an oci-push step that does remote-to-remote copying and tagging, with a 1GB size cap applied when crossing repos, and with full annotation support.

krancour · 2026-02-25T20:58:23Z

Re-opening after further offline discussions.

Signed-off-by: Eron Wright <eron.wright@akuity.io>

…_SIZE Allow hosted environments to tune, disable, or block cross-repo OCI pushes by reading an integer env var at runner creation time. Unset defaults to 1 GiB (preserving current behavior), 0 blocks all cross-repo pushes, and -1 disables the limit entirely. Signed-off-by: Eron Wright <eron.wright@akuity.io>

Signed-off-by: Eron Wright <eron.wright@akuity.io>

Use human-friendly byte formatting (GiB/MiB/KiB/bytes) via new pkg/fmt.FormatBytes helper. Add a clear "cross-repository push is disabled" message when the limit is set to zero. Signed-off-by: Eron Wright <eron.wright@akuity.io>

EronWright · 2026-02-26T00:35:29Z

pkg/promotion/runner/builtin/oci_common.go

Note that these functions were extracted from oci_downloader.go to oci_common.go for sharing purposes with oci_pusher.go.

Expose MAX_OCI_PUSH_ARTIFACT_SIZE as a first-class Helm value instead of requiring raw env var overrides via controller.env. Signed-off-by: Eron Wright <eron.wright@akuity.io>

User documentation

5a6937c

Signed-off-by: Eron Wright <eron.wright@akuity.io>

EronWright added 3 commits February 24, 2026 08:59

codegen

12ce881

Signed-off-by: Eron Wright <eron.wright@akuity.io>

EronWright marked this pull request as ready for review February 25, 2026 00:59

EronWright requested review from a team as code owners February 25, 2026 00:59

EronWright closed this Feb 25, 2026

krancour reopened this Feb 25, 2026

EronWright added 4 commits February 25, 2026 13:51

refactor: move OCI annotation mutate helpers to pkg/image/mutate

13910eb

Signed-off-by: Eron Wright <eron.wright@akuity.io>

minor

d92758d

Signed-off-by: Eron Wright <eron.wright@akuity.io>

fix: improve oci-push size limit error messages

c7ce405

Use human-friendly byte formatting (GiB/MiB/KiB/bytes) via new pkg/fmt.FormatBytes helper. Add a clear "cross-repository push is disabled" message when the limit is set to zero. Signed-off-by: Eron Wright <eron.wright@akuity.io>

EronWright commented Feb 26, 2026

View reviewed changes

feat: add controller.images.push.maxArtifactSize to Helm chart

8a81a2b

Expose MAX_OCI_PUSH_ARTIFACT_SIZE as a first-class Helm value instead of requiring raw env var overrides via controller.env. Signed-off-by: Eron Wright <eron.wright@akuity.io>

EronWright force-pushed the EronWright/issue-3762 branch from 6c78442 to 8a81a2b Compare February 26, 2026 00:47

EronWright requested a review from krancour February 26, 2026 23:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add oci-push promotion step#5782

feat: add oci-push promotion step#5782
EronWright wants to merge 12 commits intomainfrom
EronWright/issue-3762

EronWright commented Feb 24, 2026 •

edited

Loading

Uh oh!

netlify bot commented Feb 24, 2026 •

edited

Loading

Uh oh!

EronWright commented Feb 24, 2026

Uh oh!

codecov bot commented Feb 24, 2026 •

edited

Loading

Uh oh!

EronWright commented Feb 24, 2026

Uh oh!

EronWright commented Feb 24, 2026

Uh oh!

EronWright commented Feb 25, 2026

Uh oh!

EronWright commented Feb 25, 2026

Uh oh!

EronWright commented Feb 25, 2026

Uh oh!

krancour commented Feb 25, 2026

Uh oh!

EronWright Feb 26, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

EronWright commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

netlify bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for docs-kargo-io ready!

Uh oh!

EronWright commented Feb 24, 2026

E2E Test Results

Setup

Promotion

Result: ✅ Succeeded

Uh oh!

codecov bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

EronWright commented Feb 24, 2026

E2E Test Report: Cross-repo Helm chart push with annotations on GHCR

Uh oh!

EronWright commented Feb 24, 2026

E2E Test: Container Image (multi-arch nginx)

Pure copy (no annotations)

With scoped annotations

Uh oh!

EronWright commented Feb 25, 2026

Uh oh!

EronWright commented Feb 25, 2026

Uh oh!

EronWright commented Feb 25, 2026

Uh oh!

krancour commented Feb 25, 2026

Uh oh!

EronWright Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

EronWright commented Feb 24, 2026 •

edited

Loading

netlify bot commented Feb 24, 2026 •

edited

Loading

codecov bot commented Feb 24, 2026 •

edited

Loading

EronWright Feb 26, 2026 •

edited

Loading