Stored artifacts are neither garbage-collected nor invalidated when an object leaves a shard, masking sharding misconfiguration until the next pod restart

[gecube]

Describe the bug

When running sharded source-controllers per the sharding guide (--watch-label-selector), an object that leaves a shard — its sharding.fluxcd.io/key label changed or removed during normal GitOps refactoring — leaves its stored artifact behind on the old shard instance, and that instance keeps serving the artifact at the advertised URL even though it no longer owns (or even sees) the object.

The combination of "no GC / keep serving on shard eviction" + ephemeral artifact storage (emptyDir in the reference distribution) turns a small labeling mistake into a delayed failure: the workload keeps running off the stale artifact for as long as the old pod lives, and the release breaks on the next source-controller restart — typically the next Flux upgrade, far from the change that actually caused it.

Up front, to be precise about confidence levels: the timeline below is reproduced step by step (repro included), and we also hit it in production; the code-level explanation at the end is my reading of the code — a hypothesis I'd ask you to confirm or correct. Trying to help narrow this down, not to assert how the controller works internally.

Steps to reproduce (verified on kind)

Full self-contained repro script (kind + flux CLI + jq)

#!/usr/bin/env bash
# Repro: source-controller keeps serving (and never GCs) artifacts of objects that left its shard;
# with emptyDir storage the failure detonates on the next pod restart.
# Verified with: kind v0.33.0, flux CLI v2.8.8 (source-controller v1.8.5, helm-controller v1.5.5)
set -euo pipefail

CLUSTER=shard-gc-repro
CTX=kind-$CLUSTER

kind create cluster --name $CLUSTER --wait 120s
flux install --context $CTX --components=source-controller,helm-controller

# --- shard the controllers per https://fluxcd.io/flux/installation/configuration/sharding/ ---
kubectl --context $CTX patch deploy -n flux-system source-controller --type=json \
  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--watch-label-selector=!sharding.fluxcd.io/key"}]'
kubectl --context $CTX patch deploy -n flux-system helm-controller --type=json \
  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--watch-label-selector=!sharding.fluxcd.io/key"}]'

kubectl --context $CTX get deploy -n flux-system source-controller -o json | jq '
  .metadata = {name:"source-controller-shard1", namespace:"flux-system", labels:(.metadata.labels + {app:"source-controller-shard1"})} |
  .spec.selector.matchLabels.app = "source-controller-shard1" |
  .spec.template.metadata.labels.app = "source-controller-shard1" |
  del(.status) |
  .spec.template.spec.containers[0].args = (.spec.template.spec.containers[0].args
    | map(select(startswith("--watch-label-selector") or startswith("--storage-adv-addr") | not))
    + ["--watch-label-selector=sharding.fluxcd.io/key=shard1",
       "--storage-adv-addr=source-controller-shard1.$(RUNTIME_NAMESPACE).svc.cluster.local."])
' | kubectl --context $CTX apply -f -

kubectl --context $CTX get svc -n flux-system source-controller -o json | jq '
  .metadata = {name:"source-controller-shard1", namespace:"flux-system"} |
  .spec.selector.app = "source-controller-shard1" |
  del(.spec.clusterIP, .spec.clusterIPs, .status)
' | kubectl --context $CTX apply -f -

kubectl --context $CTX get deploy -n flux-system helm-controller -o json | jq '
  .metadata = {name:"helm-controller-shard1", namespace:"flux-system", labels:(.metadata.labels + {app:"helm-controller-shard1"})} |
  .spec.selector.matchLabels.app = "helm-controller-shard1" |
  .spec.template.metadata.labels.app = "helm-controller-shard1" |
  del(.status) |
  .spec.template.spec.containers[0].args = (.spec.template.spec.containers[0].args
    | map(select(startswith("--watch-label-selector") | not))
    + ["--watch-label-selector=sharding.fluxcd.io/key=shard1"])
' | kubectl --context $CTX apply -f -

kubectl --context $CTX rollout status -n flux-system deploy/source-controller-shard1 --timeout=120s
kubectl --context $CTX rollout status -n flux-system deploy/helm-controller-shard1 --timeout=120s
kubectl --context $CTX rollout status -n flux-system deploy/source-controller --timeout=120s

# --- everything on shard1, all three labels in sync ---
kubectl --context $CTX apply -f - <<'EOF'
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: podinfo
  namespace: default
  labels:
    sharding.fluxcd.io/key: shard1
spec:
  interval: 1m
  url: https://stefanprodan.github.io/podinfo
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: podinfo
  namespace: default
  labels:
    sharding.fluxcd.io/key: shard1
spec:
  interval: 1m
  chart:
    metadata:
      labels:
        sharding.fluxcd.io/key: shard1
    spec:
      chart: podinfo
      version: "6.7.0"
      sourceRef:
        kind: HelmRepository
        name: podinfo
      interval: 1m
EOF

sleep 60
kubectl --context $CTX get helmrepository,helmchart,helmrelease -n default   # all Ready=True
CHART_URL=$(kubectl --context $CTX get helmchart -n default default-podinfo -o jsonpath='{.status.artifact.url}')
echo "chart artifact: $CHART_URL"   # points at source-controller-shard1

# curl helper inside flux-system (NetworkPolicy blocks ingress from other namespaces)
kubectl --context $CTX run curl -n flux-system --image=curlimages/curl:8.10.1 --restart=Never -q -- sleep 3600
kubectl --context $CTX wait pod/curl -n flux-system --for=condition=Ready --timeout=60s

# --- THE DRIFT: repo leaves shard1 ---
kubectl --context $CTX label helmrepository -n default podinfo sharding.fluxcd.io/key-
sleep 150

kubectl --context $CTX get helmrepository,helmchart,helmrelease -n default
# helmrepository: Ready=True   (now owned by the default instance, status URL rewritten)
# helmchart:      Ready=False  SourceUnavailable: HelmRepository "podinfo" not found
# helmrelease:    Ready=True   <-- still green, workload running

# stale artifacts still served by shard1, which no longer owns the object:
kubectl --context $CTX exec -n flux-system curl -- curl -s -o /dev/null -w "chart tgz on shard1: HTTP %{http_code}\n" "$CHART_URL"   # HTTP 200

# --- THE DETONATION: routine restart (what a Flux upgrade does) ---
kubectl --context $CTX rollout restart -n flux-system deploy/source-controller-shard1
kubectl --context $CTX rollout status -n flux-system deploy/source-controller-shard1 --timeout=120s
sleep 90

kubectl --context $CTX exec -n flux-system curl -- curl -s -o /dev/null -w "chart tgz on shard1: HTTP %{http_code}\n" "$CHART_URL"   # HTTP 404
kubectl --context $CTX get helmrelease -n default podinfo
# helmrelease: Ready=False  HelmChart 'default/default-podinfo' is not ready: does not have an artifact

Controllers: source-controller v1.8.5, helm-controller v1.5.5 (flux CLI v2.8.8); also observed on v1.9.5 in EKS.

1. kind create cluster --name shard-gc-repro && flux install --components=source-controller,helm-controller

2. Shard the controllers per the guide: add --watch-label-selector=!sharding.fluxcd.io/key to the default source-controller and helm-controller; create copies source-controller-shard1 / helm-controller-shard1 with --watch-label-selector=sharding.fluxcd.io/key=shard1, a dedicated source-controller-shard1 Service and --storage-adv-addr=source-controller-shard1.$(RUNTIME_NAMESPACE).svc.cluster.local.. Artifact storage stays on emptyDir as shipped.

3. Apply a HelmRepository (podinfo) + HelmRelease, all labeled sharding.fluxcd.io/key: shard1 (including .spec.chart.metadata.labels). Everything goes Ready; the HelmChart artifact URL points at source-controller-shard1.…:

   helmrepository/podinfo    True   stored artifact: revision 'sha256:e7dc68a4…'
   helmchart/default-podinfo True   pulled 'podinfo' chart with version '6.7.0'
   helmrelease/podinfo       True   Helm install succeeded for release default/podinfo.v1
   

4. The drift: remove the sharding.fluxcd.io/key label from the HelmRepository only (simulating a partial refactoring — the label has to be kept in sync by hand in three places, see below). Observed a couple of minutes later:

   • the default instance takes the HelmRepository over and rewrites its status.artifact.url from source-controller-shard1.… to source-controller.… — the object itself stays Ready;
   • the HelmChart (still shard1) goes Ready=False / FetchFailed=True with SourceUnavailable: failed to get source: HelmRepository.source.toolkit.fluxcd.io "podinfo" not found — the repository exists in the API but is invisible through shard1's label-filtered cache;
   • but the chart keeps ArtifactInStorage=True, the stale artifacts are still served by shard1 (HTTP 200 on both the chart tgz and the old repo index at the shard1 URLs), and the HelmRelease stays Ready=True with the workload running. If your alerting watches HelmReleases — and that is what most people alert on — this is invisible. It can stay in this state for weeks.

5. The detonation: kubectl rollout restart -n flux-system deploy/source-controller-shard1 (this is what a routine Flux upgrade does). The emptyDir is wiped; observed immediately after:

   • the chart tgz URL now returns HTTP 404;
   • shard1's controller re-reconciles the HelmChart against empty storage, cannot rebuild it (its source is still invisible), and clears status.artifact;
   • the HelmRelease goes down on its own, with no change applied to it:

   helmrelease/podinfo   False   HelmChart 'default/default-podinfo' is not ready: does not have an artifact
   

   The failure surfaces during/after a routine upgrade, with nothing in the upgrade diff pointing at the label change made long before.

Why label drift is easy

Sharding correctness for a Helm release requires the same label to be maintained by hand in three places: the HelmRelease's own labels (for helm-controller), .spec.chart.metadata.labels (helm-controller's buildHelmChartFromTemplate copies only the chart template metadata to the generated HelmChart — the release's own labels are not propagated), and the HelmRepository. The docs acknowledge the constraint ("Source object kinds which have a dependency on another kind (i.e. HelmChart on a HelmRepository) need to have the same labels applied to work as expected"), but nothing enforces or reports a violation, and — per the above — as long as a previously stored artifact survives on some instance's disk, the release level looks healthy.

Proposed mechanism (hypothesis — please verify)

As far as I can tell from reading the code:

• the shard selector is a cache-level label filter on all source kinds (main.go, Cache.ByObject[...]{Label: watchSelector}), which is why HelmChartReconciler.getSource gets NotFound for a repository owned by another shard;
• when an object leaves the shard, the reconciler gets NotFound from the filtered cache and returns via client.IgnoreNotFound; garbage collection of stored artifacts appears to run only through the finalizer path on actual object deletion (reconcileDelete → garbageCollect), which never executes here because the object still exists in the API — it is merely invisible to this instance. Hence: file stays on disk, HTTP server keeps serving it.

If I've misread the GC/eviction path, corrections are very welcome — the reproduced behavior above stands either way.

Expected behavior

One or more of:

• when an object leaves a shard (delete event from the label-filtered cache while the object still exists in the API), the instance garbage-collects its stored artifacts for that object, or at minimum stops serving them — so that a sharding misconfiguration fails fast and next to the change that caused it, instead of at the next restart;
• a visible signal that an object's status.artifact.url is advertised by an instance that no longer watches the object (condition on the object, event, or a metric);
• (harder, likely out of scope here) validation/warning when dependent objects carry different shard labels.

Environment

• Reproduced on kind: source-controller v1.8.5, helm-controller v1.5.5 (flux CLI v2.8.8), sharding exactly per the official guide, artifact storage on emptyDir.
• Originally hit on EKS with source-controller v1.9.5 / helm-controller v1.6.4 (flux-operator-managed shards).

I understand fail-fast vs. keep-serving is a trade-off (the current behavior accidentally provides "grace" during transient label flaps). But the current combination — no GC, no signal, ephemeral storage — makes the failure land during an unrelated upgrade, which is close to the worst possible time to debug it.