Skip to content

feat: option to replicate shared resources to all project namespaces#5789

Open
jessesuen wants to merge 8 commits intoakuity:mainfrom
jessesuen:feat/replication
Open

feat: option to replicate shared resources to all project namespaces#5789
jessesuen wants to merge 8 commits intoakuity:mainfrom
jessesuen:feat/replication

Conversation

@jessesuen
Copy link
Member

@jessesuen jessesuen commented Feb 25, 2026

Partially resolves #5788

Adds a new resource replication reconciler to the management controller. Secrets & ConfigMaps in the kargo-shared-resources Namespace, annotated with kargo.akuity.io/replicate-to: "*" are automatically copied into every Project namespace. For example:

apiVersion: v1
kind: Secret
metadata:
  name: my-shared-secret
  namespace: kargo-shared-resources
  annotations:
    kargo.akuity.io/replicate-to: "*"
data:
  foo: dmFsdWUtMg0KDQo=

This enables Argo Rollouts AnalysisTemplates (including Job Pods) to consume them, as it is not possible for cross-namespace secret references.

Key behaviors:

  • Source resource gets kargo.akuity.io/finalizer finalizer on first reconcile; deletion is blocked until all managed replicas are cleaned up.
  • Replicated resources carry replicated-from and replicated-sha labels; the SHA label is a 16-char truncated SHA-256 of the resource data.
  • The replicated-sha label is used to detect and optimize when content updates are necessary to be copied to replicated resources. If sha is the same, the update is skipped.
  • User-created resources with the same name (no replicated-from label) are left untouched (conflict avoidance).
  • Removing the replicate-to annotation triggers the same cleanup path as source secret deletion.
  • A Project watch handler re-enqueues all annotated source secrets when a new Project appears, ensuring immediate replication.

Remaining work:

  • Documentation
  • Mutating Webhook should prevent modifications to replicated resources
  • UI checkbox option for replicating a resource

Adds a new shared-secrets replication reconciler to the management
controller. Secrets in kargo-shared-resources annotated with
kargo.akuity.io/replicate-to: "*" are automatically copied into every
Project namespace, enabling Argo Rollouts AnalysisTemplates (and Job
Pods) to consume them without cross-namespace secret references.

Key behaviors:
- Source secret gets kargo.akuity.io/replicated finalizer on first
  reconcile; deletion is blocked until all managed replicas are cleaned
  up.
- Replicated secrets carry replicated-from and replicated-sha labels;
  the SHA label is a 16-char truncated SHA-256 of the secret data used
  to detect external modifications.
- Externally modified replicas are never overwritten or deleted.
- User-created secrets with the same name (no replicated-from label)
  are left untouched (conflict avoidance).
- Removing the replicate-to annotation triggers the same cleanup path
  as source secret deletion.
- A Project watch handler re-enqueues all annotated source secrets
  when a new Project appears, ensuring immediate replication.
- Orphaned replicas in non-project namespaces are pruned on each
  reconcile pass.
- kargo-shared-resources is now always included in the management
  controller's Secret namespace cache, independent of the legacy
  migration controller.

Signed-off-by: Jesse Suen <jesse@akuity.io>
Signed-off-by: Jesse Suen <jesse@akuity.io>
- computeDataHash now factors in labels and annotations (excluding
  replication-managed labels and kubectl/replicate-to annotations)
- syncToProjectNamespace carries source labels and annotations to replicas
- List call in Reconcile filters to out-of-date replicas only
  (replicated-sha notin [sourceHash]) to skip unnecessary updates
- Remove orphaned-namespace cleanup: Project deletion cascades to the
  namespace, which deletes replicated Secrets automatically

Signed-off-by: Jesse Suen <jesse@akuity.io>
…nd ConfigMaps

Signed-off-by: Jesse Suen <jesse@akuity.io>
…n on startup

Signed-off-by: Jesse Suen <jesse@akuity.io>
Signed-off-by: Jesse Suen <jesse@akuity.io>
Signed-off-by: Jesse Suen <jesse@akuity.io>
Signed-off-by: Jesse Suen <jesse@akuity.io>
@jessesuen jessesuen requested a review from a team as a code owner February 25, 2026 09:33
@netlify
Copy link

netlify bot commented Feb 25, 2026

Deploy Preview for docs-kargo-io ready!

Name Link
🔨 Latest commit 6fcc80a
🔍 Latest deploy log https://app.netlify.com/projects/docs-kargo-io/deploys/699ec1ede578c400078842bf
😎 Deploy Preview https://deploy-preview-5789.docs.kargo.io
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@jessesuen
Copy link
Member Author

jessesuen commented Feb 25, 2026

In addition to the unit tests, this was manually tested end-to-end to cover various corner cases, including:

  • adding the annotation after creation
  • removing the annotation after creation
  • removing the annotation while the controller was down, ensuring it cleaned up when it came back up
  • creating a project when controller was down, ensuring Secrets/Configmaps were created when it came back up
  • modifying the contents of the Secret/Configmap

// AnnotationValueReplicateToAll is the annotation value for
// AnnotationKeyReplicateTo that causes a resource to be replicated to all
// Project namespaces.
AnnotationValueReplicateToAll = "*"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am considering introducing an annotation, kargo.akuity.io/replicated-at: 2026-02-25T01:49:00Z timestamp to indicate when the resource was replicated. This might help end user (or even kargo devs) debug when things changed (e.g., if an admin updated the Secret/ConfigMap from underneath an AnalysisRun).


if existing == nil {
// Either no replica exists yet, or it already has the current SHA
// (filtered out by the List). Try to create; handle AlreadyExists.
Copy link
Contributor

@EronWright EronWright Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it necessary to call Create in the case where the replicated object exists with a matching SHA? If you were to not filter it, and compare the SHA here (which I see you do anyway later), could the Create call be avoided?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are right about this. I think this logic used to make sense before I added an optimization in the list query. I will revisit this.


// Sync to each project namespace.
for ns := range projectNamespaces {
if err := r.syncToProjectNamespace(ctx, srcObj, ns, sourceHash, existingByNamespace[ns]); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: parallelize

destObj.SetLabels(replicaLabels(src, sourceHash))
destObj.SetAnnotations(replicaAnnotations(src))
r.adapter.copyFields(destObj, src)
if err := r.client.Create(ctx, destObj); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: set the FieldManager

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ability to replicate shared resources across projects

2 participants