Namespace Management at Scale

[moukoublen]

What this PR does

Introduces label-selector-based namespace scoping for ECK: instead of a static list of namespace names, operators can now configure a metav1.LabelSelector that the operator evaluates against namespace labels at runtime to determine which namespaces it manages.
When namespaceSelector is configured, the operator runs in dynamic mode:

• The controller-runtime cache is widened to cluster scope (no DefaultNamespaces set).
• A shared NamespaceMatcher evaluates the configured metav1.LabelSelector against each namespace's labels, tracks match state, and broadcasts change events.
• A dedicated namespace controller re-enqueues ECK CRs whenever a namespace gains or loses eligibility — no operator restart required.
• The match-state map is maintained on every operator replica; only the broadcast to subscriber controllers is leader-gated. Webhook behavior is therefore identical no matter which pod serves the admission request, and failover has no cold-start window.
• A FilterClient wraps the manager's Kubernetes client to filter Get and List results by namespace, providing an additional enforcement layer at the client level.
• Dynamic mode is gated behind an enterprise license (Hybrid / Option 3 from the TDD).

The user-facing API (namespaceSelector in the operator config / Helm values.yaml) is identical to what Part 1 introduced; only the enforcement mechanism changes.

---

Changes by area

pkg/controller/common/nsmatch — core package

NamespaceMatcher (ns.go)

The central component for dynamic namespace scoping, previously called MatchNotifier.

Responsibilities:

• Evaluate a labels.Selector against a corev1.Namespace's labels.
• Track the last-known match result for every namespace in an in-memory matchedNamespaces map[string]struct{} (mutex-protected). Matches is now a pure in-memory lookup — no cache access, no context argument.
• Act as a pub/sub hub: controllers call Subscribe() once at registration time to obtain a chan event.TypedGenericEvent[*corev1.Namespace]; Broadcast fans the same event out to all subscribers when match state changes.
• Broadcast blocks on each subscriber send until it succeeds or the context is cancelled, logging periodically (every 3 s) while a send is blocked — so persistent backpressure is visible instead of events being silently dropped on a full buffer or abandoned on a timeout.
• The match state is maintained on every replica, but Broadcast is a no-op on non-leaders: the matcher holds the manager's Elected() channel (via SetElected) and drops events until it closes (see "High availability" below).

Key methods:

Method	Purpose
NewNamespaceMatcher(sel, operatorNS)	Constructor. nil selector → no-op (legacy / static mode). The operator namespace and the empty string (cluster-scoped events) are always short-circuited to true.
SetElected(elected)	Wires in the manager's Elected() channel, closed once this replica becomes leader (or immediately when leader election is disabled). When set, Broadcast drops events until the channel is closed; when never set (tests), Broadcast always sends.
SelectorEnabled()	Reports whether dynamic filtering is active. Safe to call on a nil receiver.
Matches(ns)	Returns the cached in-memory match state. Returns false for unknown namespaces (conservative). No context argument.
MatchingNamespaces()	Returns the names in the in-memory match-state map, plus the always-managed namespaces (the operator's own namespace), which bypass selector evaluation everywhere else and are managed by definition. A pure in-memory read — no cache access. Used by telemetry (see below).
ObserveNamespace(ns)	Evaluates the selector against ns.Labels, updates the state map, and returns (isMatching, wasMatching). Short-circuits for the empty string and operator namespace without touching the state map.
ObserveAndBroadcast(ctx, ns)	Combines ObserveNamespace + Broadcast; broadcasts only when state changed. Returns (stateChanged, isMatching, err).
Subscribe()	Registers a new subscriber channel (buffered at 32). Called once per controller during manager initialization.
Broadcast(ctx, ns)	Fan-out to all subscriber channels. Each send blocks until it succeeds or ctx is done, logging every 3 s while blocked so backpressure is visible; on cancellation the current subscriber's error is recorded and delivery continues to the rest (joined error returned). No-op on a replica that has not been elected leader.
Swap(ns, isMatching)	Atomically records isMatching for ns and returns the previous value. Replaces the separate NamespaceStates type. Intended for internal use (by ObserveNamespace) and tests; other callers should use ObserveNamespace / ForgetNamespace.
ForgetNamespace(ns)	Clears the recorded match state for a deleted namespace without broadcasting — its resources are being cleaned up by their own controllers, so there's nothing for subscribers to react to. Used by the namespace controller on namespace deletion.

No-op when disabled: When the selector is nil, every method returns true / is a no-op. Callers need no conditional logic; the object can be wired unconditionally.

---

FilterClient (filter_client.go) — new

A client.Client wrapper that filters Get and List results by namespace using the NamespaceMatcher. Injected as the manager's client via opts.NewClient when namespaceSelector is configured.

Filtering behaviour:

• Get: when the requested key's namespace is not matched, returns a synthetic NotFound error without querying the delegate — making the object invisible to callers. The GroupResource in the error is best-effort resolved from the scheme/RESTMapper so it renders like a real API-server NotFound; as with a real NotFound, obj is left untouched. Cluster-scoped objects have an empty key.Namespace, which always matches.
• List: filters out items in non-matched namespaces after the underlying List returns. Fast path: when the list is already scoped to a single namespace (via client.InNamespace), one Matches call decides the whole result.
• All other client.Client operations (Create, Update, Patch, Delete, …) are delegated unchanged.

This provides a third enforcement layer (alongside the NamespacedKind predicate and the reconcile-loop guard) that catches any code path that bypasses the event-driven filtering.

---

pkg/controller/namespace — new package

A dedicated match-state controller that watches all corev1.Namespace objects and drives NamespaceMatcher.

reconciler.Reconcile:

1. Checks the enterprise license via license.Checker.EnterpriseFeaturesEnabled.
   • If not enabled: logs a message, emits a Kubernetes Warning event on the operator pod via license.EmitEnterpriseFeatureEvent, and requeues with a 5-minute backoff. This is the Hybrid (Option 3) license gate: the initial static snapshot remains in effect; only real-time dynamic evaluation is paused.
2. Fetches the corev1.Namespace by name.
   • If not found (deleted): calls ForgetNamespace(ns) and returns without broadcasting — the namespace's resources are being cleaned up by their own controllers, so there's nothing for subscribers to react to.
3. Calls ObserveAndBroadcast(ctx, ns). If state changed, logs "namespace match-state changed" with the new value.

namespaceSeedRunnable: A manager.Runnable registered via mgr.Add. It runs after cache sync, lists all existing namespaces, and calls ObserveAndBroadcast on each one inside a goroutine (so the blocking sends to subscriber channels don't hold up the runnable itself) to seed the notifier with the full current match state.

The controller is registered only when params.NamespaceMatcher.SelectorEnabled() is true, which means it is a strict no-op in legacy or static-list mode. It is registered first among all controllers; note that registration order does not order startup — the manager starts all controllers and runnables concurrently once the cache has synced. The startup window is covered by the backfill broadcasts (see "Startup race and backfill" below), not by ordering.

Unlike the per-kind controllers (which go through common.NewController), this controller is created directly via controller.New with NeedLeaderElection: false, and the seed runnable's NeedLeaderElection() also returns false — both run on every replica (see next section).

---

High availability: match state on every replica, broadcasts leader-only

The namespace match-state controller and its seed runnable now run on every operator replica (NeedLeaderElection: false), not just the leader. This fixes an HA gap: the webhook server runs on all replicas, and both the admission gate (Matches) and validators that read through the namespace-filtering client (FilterClient) consult the in-memory match-state map, which on non-leaders used to stay empty forever — so the FilterClient returned synthetic NotFound for any managed namespace and the admission gate would have skipped validation everywhere. With every replica maintaining its own map from the same Namespace/license watch stream, all replicas converge to the same state deterministically (the license gate is a pure function of the cached license secret, readable everywhere), and webhook behavior is identical no matter which pod serves the request. This is also what allows the webhook and telemetry to drop their live cache label lookups in favor of the map (see the webhook and telemetry sections).

Broadcasting to the per-kind controllers stays leader-only: the matcher holds the manager's Elected() channel (wired in via SetElected in main.go) and Broadcast drops events until it closes. This is safe because the map is always written before a broadcast is attempted, so a dropped event loses no information, and controller-runtime closes the elected channel before starting the leader-gated controllers, whose initial sync then replays every CR against the already-warm local map, subsuming any re-enqueue triggers dropped while standby. Broadcasts are only needed for flips that happen after the initial sync, and by then the replica is elected. A bonus: failover no longer has a cold-start window, since a newly elected replica's map is already correct at the moment it wins the election.

---

pkg/controller/common/watches — additions

NamespacedKind (namespaced_kind.go)

Drop-in replacement for source.Kind that injects a namespace-scope predicate:

func NamespacedKind[T client.Object](
    m *nsmatch.NamespaceMatcher,
    c cache.Cache, obj T,
    h handler.TypedEventHandler[T, reconcile.Request],
    preds ...predicate.TypedPredicate[T],
) source.SyncingSource

When m.SelectorEnabled() is false the function delegates directly to source.Kind with no overhead. When enabled it prepends a predicate that calls m.Matches(obj.GetNamespace()), so events from non-matching namespaces are filtered before reaching the handler. Used for every namespaced resource watch across all controllers.

WatchNamespaceFlips (namespace_flip.go)

Registers a source.Channel watch driven by notifier.Subscribe(). The mapper function lists all objects of the watched kind in the changed namespace and emits one reconcile.Request per object. No-ops when notifier is nil.

Takes a cache.Cache (not a client.Client) to perform that list. This matters specifically because the manager's client is the FilterClient in dynamic mode: when a namespace is being de-scoped, the FilterClient would filter out — and thus hide — the very objects this mapper needs to find in order to enqueue their cleanup reconciles. Listing straight from the cache bypasses that filtering.

NamespaceNotifier interface exposes Subscribe() and SelectorEnabled(), keeping per-kind controllers decoupled from the full NamespaceMatcher type.

WatchNamespaceFlipsMapped (namespace_flip.go)

A generalization of WatchNamespaceFlips that takes a caller-provided mapper (func(ctx, *corev1.Namespace) []reconcile.Request) instead of the default "list the kind in the changed namespace" behavior. WatchNamespaceFlips is now a thin wrapper around it. Used when the objects to re-enqueue on a namespace match-state change are not simply the ones living in that namespace — by the association controllers and by AutoOps, license, StackConfigPolicy and RemoteCluster (see the dedicated sections below).

TypedWatchNamespaceFlips has been removed; all call sites now use WatchNamespaceFlips directly.

---

All per-kind controllers (ES, Kibana, APM, Beat, Agent, …)

Every controller receives the same three changes:

1. addWatches: source.Kind → watches.NamespacedKind

All source.Kind calls for namespaced resources (primary CRDs, StatefulSets, Pods, Services, Secrets, ConfigMaps, PodDisruptionBudgets) are replaced with watches.NamespacedKind(m, ...). Cluster-scoped watches (webhook configurations, license state) keep source.Kind directly.

2. addWatches: WatchNamespaceFlips registration

Each controller registers a WatchNamespaceFlips call for its primary CRD list type (e.g. &esv1.ElasticsearchList{}). This creates the subscription to the match-state broadcaster and wires the namespace → CR re-enqueue path. The association controllers — as well as AutoOps, license, StackConfigPolicy and RemoteCluster — use the WatchNamespaceFlipsMapped variant with custom mappers instead (see the dedicated sections below).

3. Reconcile: guard at entry

if !r.NamespaceMatcher.Matches(request.Namespace) {
    r.onNamespaceOutOfScope(request.NamespacedName)
    return reconcile.Result{}, nil
}

onNamespaceOutOfScope performs the same dynamic-watch cleanup as onDelete (stop observers, remove handler keys for secrets/configmaps) without the GC step. This ensures that when a namespace loses eligibility, in-flight reconcile requests that slip through before the predicate cache is warmed are still handled cleanly.

The field carrying the notifier is named NamespaceMatcher (previously NamespaceMatchNotifier) in all reconciler structs.

The license controller also gets watches.NamespacedKind applied to its Elasticsearch and Secret watches, even though it isn't a per-kind resource controller: it watches ES clusters and the operator license Secret cluster-wide, so its Elasticsearch watch needs the same namespace-scoping predicate as everything else.

Note on the controllers that track reconciliation expectations (Elasticsearch, Logstash): onNamespaceOutOfScope does not clear expectations, unlike onDelete. The resource and its children still exist and the namespace may move back into scope before the controller cache observes recent StatefulSet/Pod changes — keeping expectations preserves the usual stale-cache safety across short namespace-label flaps. Expectations are only cleared on CR deletion or operator restart.

---

Association controllers (pkg/controller/association) — cross-namespace reference re-enqueue

Associations are the one place where a namespace match-state change must trigger reconciliation of resources living outside the changed namespace: the associated resource (e.g. a Kibana in namespace A) can reference a resource in a different namespace (e.g. an Elasticsearch in namespace B).

The scenario this handles: namespace B is de-scoped (its labels stop matching the selector) while namespace A stays in scope. Kibana A's association still points at ES B, but nothing would re-enqueue Kibana A:

• A namespace label change produces no event on the ES object itself, and even if one arrived, the NamespacedKind predicate on the dynamic referenced-resource watch now drops all events from the de-scoped namespace B.
• The default WatchNamespaceFlips mapper lists associated objects in the changed namespace — Kibana A lives in namespace A, so it is never found.
• An Established association returns no requeue, so Kibana A would keep a stale association conf pointing at a now-invisible Elasticsearch indefinitely.

The fix: addWatches registers WatchNamespaceFlipsMapped with a custom mapper (namespaceFlipRequests). On a namespace match-state change the mapper lists the associated kind cluster-wide from the cache and enqueues:

1. associated resources living in the changed namespace (the default behavior, preserved), and
2. associated resources living elsewhere that have an association of the controller's AssociationType whose AssociationRef() points into the changed namespace. AssociationRef() already resolves an empty ref namespace to the associated resource's own namespace, so the comparison is direct.

Once Kibana A is enqueued, the existing reconcile logic does the rest: the FilterClient makes ES B read as NotFound, so the association conf is removed and the status transitions to Pending. The inverse direction works the same way — when namespace B (re-)gains eligibility, Kibana A is re-enqueued and the association is re-established immediately instead of waiting for the Pending requeue backoff.

The mapper lists straight from the cache (not the FilterClient) for the same reason as WatchNamespaceFlips: associated resources referencing the changed namespace can live in any matched namespace, and resources in the namespace being de-scoped would otherwise be hidden.

This keeps a single broadcaster subscription per association controller (the mapped watch replaces, rather than complements, the default flip watch).

Known limitation: transitive references (e.g. Agent → Fleet Server → Elasticsearch, each in a different namespace) are matched on the direct reference only; a match-state change of the transitive Elasticsearch namespace does not re-enqueue the Agent.

---

Other controllers with custom flip mappers (AutoOps, license, StackConfigPolicy, RemoteCluster)

Four more controllers use WatchNamespaceFlipsMapped, each with its own mapper, because the resources a namespace flip affects are not simply the ones living in the flipped namespace. All of them list from the cache rather than the FilterClient for the same de-scoping reason as above.

• AutoOps: an AutoOpsAgentPolicy selects its Elasticsearch clusters cluster-wide by label (ResourceSelector), so a flip of any namespace can change any policy's set of matching clusters — policies in the flipped namespace must be picked up or cleaned up, and policies elsewhere must deploy agents for newly scoped clusters or clean up agents and API keys for de-scoped ones. The mapper re-enqueues all policies, mirroring the controller's Elasticsearch watch (reconcileRequestForAllAutoOpsPolicies).
• License: if the flipped namespace holds an operator license secret, the licensing outcome may change for every cluster, so all currently in-scope clusters are re-enqueued; otherwise only the clusters in the flipped namespace. The cluster-wide list deliberately goes through the filtering client — its match state is already updated when the flip fires, so it yields exactly the clusters currently in scope — while lookups inside the flipped namespace go through the cache so a de-scoped namespace's contents remain visible for the decision.
• StackConfigPolicy: re-enqueues policies in the flipped namespace plus policies in the operator namespace, which target resources cluster-wide. The operator namespace is always managed and never flips itself, so without this a scope change of a target namespace would never re-enqueue operator-namespace policies — leaving their settings unapplied on scope-in and their status counting hidden clusters on scope-out.
• RemoteCluster: re-enqueues the Elasticsearch clusters living in the flipped namespace, the counterparts their specs declare (wherever they live), and clusters elsewhere whose Spec.RemoteClusters reference a cluster in the flipped namespace. Reconcile re-evaluates all of a cluster's remote-cluster relationships in both directions (getExpectedRemoteClientsFor), so enqueueing the counterparts is enough to establish or tear down trust; the FilterClient then decides what remains visible.

---

Trial controller

addWatches now uses watches.NamespacedKind for the trial Secret watch, and Reconcile gets the same NamespaceMatcher.Matches(request.Namespace) entry guard as the per-kind controllers.

---

Webhooks (pkg/controller/common/webhook/resource_validator.go, cmd/manager/validation.go)

ResourceValidator[T] gains a namespaceMatcher *nsmatch.NamespaceMatcher field set via WithNamespaceMatcher(m).

preValidate now branches:

• If namespaceMatcher.SelectorEnabled(): skip validation when !namespaceMatcher.Matches(ns) (dynamic mode).
• Otherwise: existing static managedNamespaces check (legacy mode).

The webhook path uses Matches — the in-memory match-state map — which is valid precisely because the map is maintained on every replica (see the "High availability" section), including a non-leader pod that may be serving the admission request. The webhook receives admission requests for every namespace in the cluster: several operator instances, each watching a different set of namespaces, may be installed side by side, but only one of their webhooks is actually registered. Namespaces that do not match — including those not yet observed — are silently let through instead of denied, so an operator never rejects a request for a namespace it doesn't manage.

All RegisterResourceWebhook and RegisterWebhook call sites in validation.go are updated to pass the matcher. Webhooks in dynamic mode silently allow requests from non-matched namespaces rather than rejecting them, consistent with the TDD design note.

The webhook certificates controller (pkg/controller/webhook/webhook_certificates_controller.go) also takes the NamespaceMatcher now, applying watches.NamespacedKind to its webhook-cert Secret watch. Its watch on the Validating/MutatingWebhookConfiguration itself stays on plain source.Kind, since those are cluster-scoped resources with no namespace to filter on.

---

pkg/controller/common/license/event.go — new file

EmitEnterpriseFeatureEvent(recorder, operatorNS, msg): emits a Warning event on the operator's own Pod when an enterprise feature is used without a valid license. Pod name is resolved from os.Hostname() (Kubernetes always sets this to the pod name). A no-op if hostname cannot be determined.

---

cmd/manager/main.go and operator configuration

• New NamespaceSelectorFlag = "namespace-selector" constant (config-file only; no CLI flag, as the value is a structured metav1.LabelSelector object).
• parseNSSelector: reads the viper value, round-trips through yaml.Marshal / yaml.Unmarshal into metav1.LabelSelector, then converts via metav1.LabelSelectorAsSelector.
• Startup validation: namespaces and namespace-selector are mutually exclusive; supplying both is a fatal error.
• When dynamicNSSelector is true, DefaultNamespaces is left empty, widening the cache to cluster scope, and opts.NewClient is set to inject nsmatch.NewFilterClient so all manager-level client calls are namespace-filtered.
• nsmatch.NewNamespaceMatcher is constructed once and threaded into operator.Parameters.NamespaceMatcher.
• namespaceMatcher.SetElected(mgr.Elected()) hands the matcher the leader-election signal so Broadcast is a no-op until this replica is elected (see the "High availability" section).
• The namespace controller is the first entry in registerControllers. Registration order does not order startup (all controllers and runnables start concurrently after cache sync); the cold-start window is handled by the seeder's backfill broadcasts.

---

Telemetry (pkg/telemetry/telemetry.go) — dynamic-namespace aware

telemetry.NewReporter now takes a *nsmatch.NamespaceMatcher parameter. A new Reporter.getNamespaces() helper decides which namespaces to scan:

• If namespaceMatcher.SelectorEnabled(): calls MatchingNamespaces() for the currently-matching namespaces from the in-memory match-state map (dynamic mode) instead of using the static managedNamespaces list, since that list is empty/stale under a selector. An infallible in-memory read, so getNamespaces no longer returns an error.
• Otherwise: falls back to the existing static managedNamespaces list (legacy mode), unchanged.

Both getResourceStats and report call getNamespaces instead of reading r.managedNamespaces directly, so telemetry reporting correctly follows namespaces as they move in and out of scope at runtime, the same way reconciliation does.

---

Helm chart (deploy/eck-operator/)

values.yaml:

# namespaceSelector is a Kubernetes label selector that dynamically controls which
# namespaces the operator manages. Mutually exclusive with managedNamespaces.
# Requires createClusterScopedResources: true.
namespaceSelector: {}

configmap.yaml emits the namespace-selector: block under operator config when the value is non-empty.

---

End-to-end test (test/e2e/namespace_selector/namespace_selector_test.go)

TestNamespaceSelector:

1. Records the current operator restart count and original namespaces config.
2. Reconfigures the operator ConfigMap to use a namespace-selector matching kubernetes.io/metadata.name In [ns1] (uses the per-run label already applied by the e2e setup).
3. Waits for the operator to restart and pick up the new config.
4. Runs the standard Elasticsearch builder sequence in ns1 to verify normal reconciliation.
5. Creates an Elasticsearch CR in ns2 (not in the selector), waits 30 s, and asserts no pods were created.
6. Restores the original config in a t.Cleanup deferred block and waits for the operator to restart again.

---

License gating (Hybrid mode / Option 3)

Dynamic evaluation (namespace label changes → real-time re-enqueue) requires an enterprise license.

• At startup the initial static fetch always runs regardless of license state: the selector is evaluated once, all existing matching namespaces are seeded into the notifier, and reconciliation proceeds normally.
• Real-time event processing is enabled only when EnterpriseFeaturesEnabled returns true in the namespace controller's reconcile loop.
• Without a valid license the operator degrades gracefully: a Warning event is emitted on the operator pod, existing matched namespaces continue to be reconciled, and new/deleted namespaces are silently ignored until the license is restored.
• When a valid license is (re-)added, the namespace controller resumes real-time evaluation and the next Namespace event triggers a catch-up re-sync for affected namespaces.

Startup race and backfill

At startup there is a window between when the manager's cache is synced and when
namespaceSeedRunnable has finished evaluating all existing namespaces. During this window the
matchedNamespaces map is empty: Matches() returns false for every namespace, and the
NamespacedKind predicate drops all incoming CR events conservatively.

This means ECK could silently skip the initial reconciliation of resources in legitimately matched namespaces if their events arrive before namespaceSeedRunnable completes.

The fix is that namespaceSeedRunnable calls ObserveAndBroadcast for every existing namespace, not just ObserveNamespace. For each namespace whose selector evaluation returns isMatching = true (i.e. a state change from the zero value false), a broadcast is emitted. Every per-kind controller's source.Channel watch receives the namespace event and immediately lists and re-enqueues all CRs in that namespace. This effectively backfills any reconcile requests that were dropped during the cold-start window.

namespaceSeedRunnable is registered with mgr.Add and starts only after the manager's cache has synced. controller-runtime provides no ordering guarantee between the seeder and the reconcile loops — they start concurrently, so predicates and reconciles can briefly observe an unseeded matcher. Correctness relies on the backfill, not on ordering: any CR event dropped by a predicate before its namespace is seeded is re-enqueued by the flip watch when the seeder observes that namespace (false → true) and broadcasts it. Once seeding completes, all subsequent events see a fully seeded matchedNamespaces map.

---

Backward compatibility

• When namespaceSelector is not set the NamespaceMatcher is constructed with a nil selector. Every method is a no-op or returns true; existing behavior is unchanged.
• The namespace controller's Add function returns immediately when SelectorEnabled() is false; the controller is not registered.
• NamespacedKind degrades to source.Kind when the selector is disabled.
• WatchNamespaceFlips is a no-op when the notifier is nil.
• FilterClient is only injected when namespaceSelector is configured; all other modes use the default client.
• The static namespaces list and the new namespaceSelector are mutually exclusive and validated at startup.

---

What is not in scope

• Namespace-scoped RBAC profile (createClusterScopedResources: false): dynamic mode requires cluster-wide cache and therefore a ClusterRole. The namespace-scoped profile is not supported.
• Multi-operator coordination / conflict detection.
• Destructive cleanup (deleting CRs when a namespace loses eligibility).
• OR semantics across different label keys (the LabelSelector AND-s across keys; OR within a single key is available via In).
• Re-enqueue of associated resources on match-state changes of transitively referenced namespaces (e.g. Agent → Fleet Server → Elasticsearch); only direct association references are matched.

---

End-to-end view of how the three core entities interact across both directions: namespace events flowing in, and reconcile triggers flowing out.

sequenceDiagram
    participant K8S as Kubernetes
    participant NSC as Namespace Controller
    participant NM as NamespaceMatcher
    participant CTRL as Per-Kind Controller<br/>(ES / Kibana / Beat / …)

    note over NSC,NM: Startup — cache synced, namespaceSeedRunnable fires<br/>(runs on every replica; broadcasts are no-ops on non-leaders)
    NSC->>K8S: List all corev1.Namespace
    K8S-->>NSC: []Namespace
    loop for each existing namespace
        NSC->>NM: ObserveAndBroadcast(ctx, ns)
        NM->>NM: eval selector → update matchedNamespaces map
        alt state changed (false → true)
            NM->>CTRL: Broadcast: GenericEvent[Namespace]
            CTRL->>K8S: List CRs in namespace
            K8S-->>CTRL: []CR
            CTRL->>CTRL: enqueue reconcile.Request per CR
        end
    end

    note over K8S,CTRL: Runtime — CR event arrives (e.g. Elasticsearch updated)
    K8S->>CTRL: informer event (StatefulSet / Secret / …)
    CTRL->>NM: NamespacedKind predicate: Matches(obj.Namespace)
    NM-->>CTRL: true / false
    alt false — namespace not matched
        CTRL->>CTRL: event dropped
    else true
        CTRL->>CTRL: enqueue reconcile.Request
        CTRL->>NM: Reconcile() guard: Matches(request.Namespace)
        NM-->>CTRL: true
        CTRL->>CTRL: normal reconciliation
    end

    note over K8S,CTRL: Runtime — namespace labels change
    K8S->>NSC: Namespace Add/Update event
    NSC->>NSC: check EnterpriseFeaturesEnabled
    alt license absent / expired
        NSC->>NSC: emit Warning event on operator Pod
        NSC->>NSC: RequeueAfter 5 min
    else licensed
        NSC->>NM: ObserveAndBroadcast(ctx, ns)
        NM->>NM: eval selector → update matchedNamespaces map
        alt state unchanged
            NM-->>NSC: no broadcast
        else state changed
            NM->>CTRL: Broadcast: GenericEvent[Namespace]
            CTRL->>K8S: List CRs in namespace
            K8S-->>CTRL: []CR
            CTRL->>CTRL: enqueue reconcile.Request per CR
            CTRL->>NM: Reconcile() guard: Matches(request.Namespace)
            NM-->>CTRL: false (namespace out of scope) or true (namespace newly matched)
            alt false — namespace no longer matched
                CTRL->>CTRL: onNamespaceOutOfScope()<br/>clean up observers & dynamic watches
            else true — namespace newly matched
                CTRL->>CTRL: normal reconciliation
            end
        end
    end

Loading

[prodsecmachine]

⚠️ Snyk checks are incomplete.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
⚠️	Open Source Security	0	0	0	0	See details
⚠️	Licenses	0	0	0	0	See details

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/third-party-dependencies/main.md

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[pebrc]

[Medium] Webhook license checks can still depend on unseeded matcher state on standby replicas. The namespace-scope part now uses MatchesCachedLabels, which fixes the earlier skip-validation issue, but this checker is built from mgr.GetClient(). In dynamic mode that client is the FilterClient, and license discovery does a cluster-wide Secret list that is filtered through the in-memory NamespaceMatcher.Matches state.

Because the namespace controller/seeder are leader-election gated, non-leader replicas that still serve webhook traffic do not keep that in-memory state populated. If the valid enterprise license Secret is in a dynamically selected namespace rather than the always-managed operator namespace, a standby webhook can fail to see it and reject enterprise-gated resources even though the request namespace itself matched via cached labels.

Can the webhook license checker avoid the state-based FilterClient path here, or should we explicitly require/enforce operator licenses to be read only from the operator namespace in namespace-selector mode?

[moukoublen]

If we change the client to a wide one (e.g. cache) then the license check might fetch license secrets that it should not.

I think a proper fix for that is to enhance the license checker itself a bit. Before we move with enforcing the operator license to live in the operator namespace, let me think a bit about any possible enhancement, and I will come back.

[moukoublen]

Fixed in this 15121d2

[pkoutsovasilis]

I did review the code and also employed the 🤖, nothing major (aside the comments mentioned already) pops up, which is a good thing. I did some mild manual testing and it worked, which is another good thing, but I need to do way more. However, (maybe as a follow-up PR/thinking out loud) I do feel that the code of this PR can be simplified and become easier to grasp, because as of now monitoring the in-memory state is challenging (maybe just for me though).

In a really high-level, my thinking is:

1. namespace controller remains but changes:

• it watches Namespace events:
  • create predicate that fires only for namespaces that bear the label
  • update predicate that fires only when the selector label changes between old and new
  • delete predicate that fires only for namespaces that bear the label
• it watches to operator license Secret changes:
  • all
• in its reconcile loop it:
  • list all namespaces from the cache that match the selector
  • checks whether there is at least one valid enterprise license (valid wrt signature it can be expired) exists in the namespaces above
  • if not expired: all above namespaces are managed
  • if expired: only namespaces created before the license expiry timestamp are managed (non-destructive - existing workloads keep reconciling, new namespaces are gated off)
    • emits warning
  • if no license at all, then no namespace is managed
    • emits warning
  • it then writes the resulting namespaces (it can be empty if no license at all) to a statically named ConfigMap in the operator namespace and a requeue is scheduled for slightly after the expiration time if not already expired.
    • the above must be skipped if nothing changes content-wise

2. CR controllers

• each CR controller gets two watches:
  • a ConfigMap watch that watches the ConfigMap mentioned above; it needs to be implemented as a custom TypedEventHandler (similar to NamedWatch) rather than TypedEnqueueRequestsFromMapFunc because Update method receives the full event.TypedUpdateEvent[*corev1.ConfigMap] with both objects, allowing it to parse the JSON managed-namespace list from each and enqueue CRs only in the newly added namespaces of the new vs old; on Create it enqueues CRs in all listed namespaces (first publish / operator restart)
  • a source.Kind(&Namespace{}) watch with an update predicate that fires only when the selector label is present in the old namespace object but absent in the new one - this handles label-off descope promptly and directly, without going through the ConfigMap

• the reconcile entry gate gets two checks in sequence:
  1. read the CR's namespace from the cache, and check that it matches the label selector
  2. check that the CR's namespace is within the ConfigMap managed-set (again the configmap is read from the controller-runtime cache)

Most probably the above approach has some gaps that I missed, but IMO it gives a simplification over separation of concerns. The namespace controller is the sole responsible for capturing which namespaces are managed taking into consideration the licensing. The CRs become responsible for re-acting to that Configmap accordingly, as well as, cleaning up watches for namespaces that are no longer managed. We don't lose state on operator restarts and, last but not least, with such an approach we get a configmap that we can include in the eck-diagnostics and be able to easier debug if something isn't working. Again just thinking out loud 🙂

[moukoublen]

@pkoutsovasilis Thanks for writing this up and for testing the PR! This is a really well-thought-out design: it covers operator restarts, has explicit license semantics (the expired-vs-absent distinction with non-destructive expiry is a nice touch), diffs the CM to avoid full fan-outs, and thinks about debuggability. I like the separation-of-concerns angle, and I'd argue the current design already embodies it, so we agree on the goal; where I have doubts is the ConfigMap as the source of truth, which I think has a few real drawbacks.

The namespace controller is the sole responsible for capturing which namespaces are managed taking into consideration the licensing.

Worth noting, we already have exactly this today: the namespace controller (together with its seed runnable, which is really part of the same unit) is the sole owner/writer of the match-state map, the CR controllers only consume it (subscribe + read). So the separation of concerns doesn't hinge on the ConfigMap; the CM only changes where the state lives, not who owns it.

One thing I'll acknowledge: a ConfigMap would give all replicas the same view of the managed set, which matters for the webhook on non-leaders (though it would need some wiring for the webhooks). But with the latest change (making the namespace controller run on all replicas this is covered.

it then writes the resulting namespaces ... to a statically named ConfigMap

The big one: ConfigMaps are capped at ~1Mi. Whatever the exact number, a hard limit is a hard limit, and putting the full managed-namespace list inside a single object means the state grows with the cluster while the container for it doesn't. What happens on very large clusters, which is exactly where this feature is meant to be used? At some point the write fails and the whole mechanism stops, with no way to tune around it.

if expired: only namespaces created before the license expiry timestamp are managed

You mean the list inside the CM stays fixed after the license expires?

we get a configmap that we can include in the eck-diagnostics and be able to easier debug

This is the part I actually agree with, observability of the managed set is missing today. But I think the right tool for that is a custom metrics endpoint: exposing the matcher state (a debug endpoint that dumps the current map) from the operator itself. You get the same debuggability, eck-diagnostics can scrape it, it's read-only by construction, and there's no size limit and no second source of truth to keep in sync.

Stepping back: The two main items that would concern me are: the size cap is a cliff you can't tune away, you'd have to shard or re-encode, which is a redesign. And more structurally, the CM makes the source of truth an object that is user-mutable, only as fresh as its single writer, and dangerous in its failure default (empty/missing = nothing managed, vs. today's "keep last known state").

More minor concern: Descope has two code paths. Label-off goes through the Namespace predicate, license-expiry descope goes through the CM diff. Two paths for the same behavior = twice the edge cases.

[pkoutsovasilis]

Thanks for the detailed response @moukoublen , really appreciate the pushback!

The namespace controller (together with its seed runnable, which is really part of the same unit) is the sole owner/writer of the match-state map, the CR controllers only consume it (subscribe + read). So the separation of concerns doesn't hinge on the ConfigMap; the CM only changes where the state lives, not who owns it.

You're right that the namespace controller is already the sole owner of the match state today. But it's not just a writer - it's also a custom broadcaster. Broadcast sends to subscribers sequentially (a for loop over m.subs), and each send blocks in sendAndLogBackpressure until the channel drains or ctx is cancelled. One slow subscriber holds the namespace controller's reconcile slot and stalls delivery to every subsequent subscriber. That's home-grown flow-control code with real blocking risk.

In the ConfigMap approach, the "broadcast" disappears entirely. The namespace controller writes a Kubernetes object; every consumer reacts via a standard source.Kind informer watch. Delivery is handled by controller-runtime's workqueue machinery - decoupled, buffered per-controller, battle-tested across the ecosystem. A slow CR controller cannot affect any other controller or the namespace controller's reconcile throughput.

One thing I'll acknowledge: a ConfigMap would give all replicas the same view of the managed set, which matters for the webhook on non-leaders (though it would need some wiring for the webhooks). But with the latest change (making the namespace controller run on all replicas this is covered.

Running the namespace controller on all replicas helps with the multi-replica case, but it doesn't address the single-replica scenario - on an OOM kill or any other restart, the in-memory map is gone entirely and the seeder has to replay before the operator can serve any request. During that cold-start window the managed set is effectively empty. The ConfigMap survives the restart in etcd and is immediately available on boot with no replay and no cold-start window. So "only as fresh as its single writer" cuts both ways: the in-memory map is only as durable as the process that holds it. As for the webhook wiring, it's the same two-gate check at the reconcile entry - read the namespace from cache and check the label, then check the ConfigMap managed-set.

The big one: ConfigMaps are capped at ~1Mi. Whatever the exact number, a hard limit is a hard limit, and putting the full managed-namespace list inside a single object means the state grows with the cluster while the container for it doesn't. What happens on very large clusters, which is exactly where this feature is meant to be used?

A namespace name is at most 63 characters. A JSON-encoded list, considering the maximum name size, fits roughly 15,000 namespaces in 1MB. If you're managing 15,000+ namespaces with a single ECK operator instance, the size limit is probably the least of your concerns. IMO, framing this as a realistic constraint for the target use case might be a slight overstatement.

I think the right tool for that is a custom metrics endpoint: exposing the matcher state (a debug endpoint that dumps the current map) from the operator itself.

A debug endpoint is ephemeral: it requires the operator pod to be reachable, which has downtime on restarts, and isn't in etcd. A ConfigMap is persistent, kubectl-accessible, and survives pod restarts - which is exactly when you need to debug why something isn't being reconciled. It's also directly includable in eck-diagnostics without any additional scraping infrastructure - no job to scrape the debug endpoint, no need for port-forwarding.

the CM makes the source of truth an object that is user-mutable

The ConfigMap lives in the operator namespace, which is protected by the same RBAC that guards the operator's Secrets, license objects, and all other internal state. And even if someone did manually edit it: because the namespace controller owns the ConfigMap via owner reference, any mutation should immediately trigger a reconcile on the namespace controller, which recomputes from scratch and overwrites it back to the correct state within one reconcile cycle.

Descope has two code paths. Label-off goes through the Namespace predicate, license-expiry descope goes through the CM diff. Two paths for the same behavior = twice the edge cases.

They're genuinely different signals: label-off is unconditional and immediate, no license logic needed, so CR controllers react to it directly; license-expiry is a time-based computation that only the namespace controller can evaluate, so it flows through the ConfigMap. CR controllers only react to namespaces being added to the ConfigMap, never removed - descope is always handled by the direct Namespace watch. One source of truth, two appropriately-routed triggers.

Maintainability - perhaps the strongest argument

Stepping back: the ConfigMap approach eliminates the custom pub/sub machinery entirely - the matchedNamespaces map, the mutex, the subscriber channels, Broadcast, sendAndLogBackpressure, Subscribe, Swap, ObserveAndBroadcast, the NamespaceNotifier interface, and the namespaceSeedRunnable. That's a meaningful surface area of bespoke concurrency code, all of which needs to be understood, tested, and maintained. Replacing it with a Kubernetes object and standard informer watches means the only non-trivial logic left is the scope computation in the namespace controller - everything else falls back on primitives the whole controller-runtime ecosystem already relies on. In my book, less code, fewer moving parts, easier to reason about.

---

Anyway, this was again more of a thinking-out-loud exercise than a formal proposal - there are likely gaps I haven't thought through. If you (@moukoublen or @pebrc) want to dig into this further we can move the discussion to a dedicated issue, happy to continue there! For now I have more testing to do 🙂

[moukoublen]

@pkoutsovasilis

I'll address these here since they concern the current solution. The other points we can take elsewhere if needed, as you suggested

One slow subscriber holds the namespace controller's reconcile slot and stalls delivery to every subsequent subscriber. That's home-grown flow-control code with real blocking risk.

(Take also a look to this comment #9470 (comment))

I don't think this holds up as-is, if the concern is that consumers are unsafe, that same concern would apply equally to every Watch we already have, so we'd need to address it consistently across the board.

source.Kind:    informer callback → predicate → handler → workqueue.Add()
source.Channel: chan GenericEvent → predicate → handler → workqueue.Add()
                                                              ↓
                                              reconciler workers pull from queue

The reconciler never consumes from the channel directly. It consumes from the workqueue. And workqueue.Add() is non-blocking with unbounded internal storage plus deduplication by reconcile.Request key. So the path from channel to queue is fast regardless of how slow reconciliation is. A slow reconciler makes the queue depth grow (bounded by the number of distinct request keys, thanks to dedup), it doesn't back-pressure the channel.

Internally, source.Channel also adds its own buffer: a goroutine reads the channel and distributes events into a per-destination buffered channel (DestBufferSize, default 1024), and another goroutine drains that into the handler/workqueue. Since the drain side is just "run predicate, map to request, non-blocking enqueue," it keeps up in practice.

Running the namespace controller on all replicas helps with the multi-replica case, but it doesn't address the single-replica scenario - on an OOM kill or any other restart, the in-memory map is gone entirely and the seeder has to replay before the operator can serve any request. During that cold-start window the managed set is effectively empty. The ConfigMap survives the restart in etcd and is immediately available on boot with no replay and no cold-start window. So "only as fresh as its single writer" cuts both ways: the in-memory map is only as durable as the process that holds it. As for the webhook wiring, it's the same two-gate check at the reconcile entry - read the namespace from cache and check the label, then check the ConfigMap managed-set.

any other restart ... the seeder has to replay before the operator can serve any request.

It's not block-and-wait. Also, reconstructing the state in memory from the cache (querying the namespaces and matching the selector) is almost instant when no persistence (API write) is required.

Namespaces could be seeded initially using a direct API client in main before the manager starts. I held off on this so far because the added code didn't seem justified; the cold-start period is quite short in practice and unlikely to cause issues. The source of truth remains in the actual namespace labels and not in a second persisted entity.

Maintainability - perhaps the strongest argument

I'd argue the comparison is a bit one-sided: it only looks at the code removed, not the code the alternative would introduce. To name just one example, each controller would need a namespace watch with a predicate as well as a ConfigMap watch — meaning it would receive events for all ConfigMaps in the cluster just to filter for a single one, having code in the predicate for the ConfigMap watch (compare before and after to identify if a namespace was added/removed).

[moukoublen]

buildkite test this -f p=kind,t=TestNamespaceSelector -m s=9.4.2

[pebrc]

A few more 🤖 findings.

[pebrc]

[Medium] User-secret GC ignores namespaceSelector in dynamic mode

managedNamespaces is empty in dynamic mode (it is only populated in static mode). asyncTasks passes it directly to garbageCollectUsers (line 911), which calls association.NewUsersGarbageCollector(cfg, managedNamespaces). That constructor explicitly converts an empty list to all-namespaces (gc.go:54-55):

if len(managedNamespaces) == 0 {
    managedNamespaces = []string{AllNamespaces}
}

This means a selector-scoped operator will scan and potentially delete association user secrets across the entire cluster, including namespaces it is explicitly not configured to manage.

The fix needs care: namespaceMatcher.MatchingNamespaces() returns the in-memory set, but asyncTasks runs while the seeder goroutine may still be in progress, so the map could be incomplete at GC time. The safest option is to pass namespaceMatcher into asyncTasks and do a direct API namespace list with the configured selector at GC time, rather than relying on the in-memory map.

[moukoublen]

Same for the autoops garbadge collector I think.

Regarding the solution, if we are about to use the direct API, then perhaps pre-seeding using the direct API in main is safer?

if dynamicNSSelector {
	nsList, err := clientset.CoreV1().Namespaces().List(ctx, metav1.ListOptions{})
	if err != nil {
		log.Error(err, "Failed to list namespaces to seed the namespace matcher")
		return err
	}
	for i := range nsList.Items {
		ns := &nsList.Items[i]
		namespaceMatcher.ObserveNamespace(ns)
	}
}

(I had already tested that when exploring pre-seeding) This way we will have the state ready in the same resource (namespace matcher)

[moukoublen]

Alternatively, a receiver function like WaitForCacheSync(ctx) for nsmatcher called WaitForInitialSeeding(ctx) that unblocks when the runner is done

And call this function inside the the async task after the WaitForCacheSync(ctx)

I think this is simpler I would go with that.

[moukoublen]

Fixed in 4da2395

[moukoublen]

Closing because this version was merged: #9569