Dynamic Namespaces with restricted license gate

[moukoublen]

What this PR does

Introduces label-selector-based namespace scoping for ECK: instead of a static list of namespace names, operators can now configure a metav1.LabelSelector that the operator evaluates against namespace labels at runtime to determine which namespaces it manages.

When namespaceSelector is configured, the operator runs in dynamic mode:

• The controller-runtime cache is widened to cluster scope (no DefaultNamespaces set).
• A shared, stateless NamespaceMatcher evaluates the configured selector against a namespace's current labels, read live from the informer cache. There is no in-memory match-state to maintain, seed, or replicate: every lookup reflects the cache's present view.
• Every namespaced resource watch gets a namespace-scope predicate (watches.NamespacedKind), so events from non-matching namespaces are dropped before they reach a handler.
• Each controller additionally watches Namespace objects and reacts when a namespace flips in or out of the selector's scope (watches.WatchNamespaceScopeChange): resources are picked up or cleaned up immediately — no operator restart required.
• Reconciliation of resource controllers is wrapped by common.NewNamespacedController, which enforces the enterprise license gate and skips (with local-state cleanup) requests whose namespace has left the scope.
• A FilterClient wraps the manager's client to filter Get/List results by namespace, catching any code path that bypasses the event-driven filtering.
• Dynamic mode is an enterprise feature: while no valid enterprise license is present, the wrapped controllers skip reconciliation entirely and retry every 5 minutes (see "License gating" below).

Because matching is evaluated live from each replica's own cache, all operator replicas — leader or not — answer scope questions identically. Webhooks served by a non-leader pod behave exactly like the leader's, and failover has no warm-up window.

---

Changes by area

pkg/controller/common/nsmatch — core package

NamespaceMatcher (ns.go)

A small, stateless component: it holds the parsed labels.Selector, the set of always-managed namespaces (the operator's own namespace, plus the empty string so cluster-scoped objects always pass), and a reference to the manager's cache (wired in via SetCache after manager creation).

Method	Purpose
NewNamespaceMatcher(sel, operatorNS)	Constructor. nil selector → matcher disabled (legacy / static mode).
SelectorEnabled()	Reports whether dynamic filtering is active. Safe to call on a nil receiver.
NamespaceMatches(ns)	Evaluates the selector against the given corev1.Namespace's labels. Always-managed namespaces short-circuit to true.
NamespaceNameMatches(ctx, name)	Resolves the namespace by name from the cache, then evaluates. Returns false when the namespace cannot be read (conservative).
MatchingNamespaces(ctx)	Lists all namespaces from the cache and returns the names of those currently matching, plus the always-managed ones. Used by telemetry and by the startup garbage collection of orphaned soft-owned secrets.

No-op when disabled: with a nil selector every check returns true. Callers need no conditional logic; the matcher can be wired unconditionally.

FilterClient (filter_client.go)

A client.Client wrapper injected as the manager's client (via opts.NewClient) when namespaceSelector is configured.

• Get: when the requested key's namespace does not match, returns a synthetic NotFound error without querying the delegate — the object is invisible to callers. The GroupResource in the error is best-effort resolved from the scheme/RESTMapper so it renders like a real API-server NotFound; as with a real NotFound, obj is left untouched. Cluster-scoped objects have an empty key.Namespace, which always matches.
• List: filters out items in non-matching namespaces after the underlying List returns. Fast path: when the list is already scoped to a single namespace (client.InNamespace), one match check decides the whole result.
• All other operations (Create, Update, Patch, Delete, …) are delegated unchanged.

This is the safety net behind the event-driven filtering: reconcilers reading through the manager's client cannot see out-of-scope objects even if a request slips through.

---

pkg/controller/common/watches — additions

NamespacedKind (namespaced_kind.go)

Drop-in replacement for source.Kind that prepends a predicate calling NamespaceNameMatches(obj.GetNamespace()), so events from non-matching namespaces are filtered before reaching the handler. When the matcher is disabled it delegates directly to source.Kind with no overhead. Used for every namespaced resource watch across all controllers; cluster-scoped watches (webhook configurations) keep source.Kind.

To keep this enforced going forward, a new forbidigo lint rule (.golangci.yml) forbids direct source.Kind calls; the few deliberate uses (cluster-scoped watches, the wrapper itself, the Namespace watch) carry an explanatory //nolint:forbidigo, and test files are exempt.

The WatchPods and WatchSoftOwnedSecrets helpers now take the matcher and use NamespacedKind internally.

WatchNamespaceScopeChange (namespace_flip.go)

Registers a watch on corev1.Namespace objects (a no-op when the selector is disabled) with predicates that only let scope changes through:

• Create / Delete: passed through when the namespace matches — a new in-scope namespace needs its resources picked up; an in-scope namespace going away can leave managed resources behind.
• Update: passed through only when the match result differs between ObjectOld and ObjectNew — i.e. the label edit crossed the selector boundary. Irrelevant label churn is dropped.

The flip detection is a pure comparison of the old and new objects carried by the event itself — no stored state, hence no seeding step and no startup race to defend against.

On a flip, a caller-provided mapper (func(ctx, *corev1.Namespace) []reconcile.Request) enumerates what to re-enqueue. ReconcileObjectsInNamespace is the default mapper: it lists the controller's kind in the flipped namespace and enqueues one request per object. It lists via the cache, not the FilterClient: when a namespace is being de-scoped, the FilterClient would hide the very objects whose cleanup reconciles need to be enqueued.

---

pkg/controller/common/namespaced_controller.go — new

common.NewNamespacedController is the registration entry point for every controller whose reconciliation must be restricted to the selector's scope. When the selector is disabled it behaves exactly like common.NewController. When enabled, it:

1. Wraps the reconciler so that every reconciliation is
   • license-gated: dynamic namespace scoping is an enterprise feature, so each request first checks EnterpriseFeaturesEnabled; while disabled, a Warning event is emitted on the operator Pod and the request is requeued after 5 minutes;
   • scope-guarded: if the request's namespace no longer matches, reconciliation is skipped and OnNamespaceOutOfScope is invoked instead, letting the reconciler release any state it holds for the resource.
2. Registers the WatchNamespaceScopeChange flip watch with the mapper supplied by the controller.

The new NamespacedReconciler interface is reconcile.Reconciler plus:

// OnNamespaceOutOfScope is called instead of Reconcile when the resource's namespace no longer
// matches the namespace selector. Implementations should clean up any internal state associated
// with the resource (dynamic watches, caches, etc.) but must not touch the resource itself.
OnNamespaceOutOfScope(resource types.NamespacedName)

---

Per-kind controllers (ES, Kibana, APM, Agent, Beat, Logstash, Enterprise Search, Maps, Package Registry, ES autoscaling)

Every resource controller receives the same three changes:

1. Registration switches from common.NewController to common.NewNamespacedController, passing watches.ReconcileObjectsInNamespace for its primary CRD list type (e.g. &esv1.ElasticsearchList{}) as the flip mapper.
2. addWatches: all source.Kind calls for namespaced resources (primary CRDs, StatefulSets, Deployments, Pods, Services, Secrets, ConfigMaps, PodDisruptionBudgets) are replaced with watches.NamespacedKind.
3. OnNamespaceOutOfScope is implemented on the reconciler, releasing controller-local state — stopping observers, removing dynamic watch handler keys for secrets/configmaps — without touching the resource itself. onDelete now calls OnNamespaceOutOfScope plus the deletion-only steps (expectations removal, soft-owned secret GC).

Note on the controllers that track reconciliation expectations (Elasticsearch, Logstash): OnNamespaceOutOfScope deliberately does not clear expectations. The resource and its children still exist, and the namespace may move back into scope before the controller cache observes recent StatefulSet/Pod changes — keeping expectations preserves the usual stale-cache safety across short namespace-label flaps. Expectations are only cleared on CR deletion or operator restart.

---

Association controllers (pkg/controller/association) — cross-namespace reference re-enqueue

Associations are the one place where a namespace flip must trigger reconciliation of resources living outside the flipped namespace: the associated resource (e.g. a Kibana in namespace A) can reference a resource in a different namespace (e.g. an Elasticsearch in namespace B).

The scenario this handles: namespace B is de-scoped while namespace A stays in scope. Kibana A's association still points at ES B, but nothing would re-enqueue Kibana A:

• A namespace label change produces no event on the ES object itself, and even if one arrived, the NamespacedKind predicate on the referenced-resource watch drops events from the de-scoped namespace B.
• The default flip mapper lists associated objects in the flipped namespace — Kibana A lives in namespace A, so it is never found.
• An Established association returns no requeue, so Kibana A would keep a stale association conf pointing at a now-invisible Elasticsearch indefinitely.

The fix: the association controllers pass a custom flip mapper. On a flip it lists the associated kind cluster-wide from the cache (a new AssociatedObjListTemplate field on AssociationInfo provides the list type) and enqueues:

1. associated resources living in the flipped namespace (the default behavior, preserved), and
2. associated resources living elsewhere that have an association of the controller's AssociationType whose AssociationRef() points into the flipped namespace. AssociationRef() already resolves an empty ref namespace to the associated resource's own namespace, so the comparison is direct.

Once Kibana A is enqueued, the existing reconcile logic does the rest: the FilterClient makes ES B read as NotFound, so the association conf is removed and the status transitions to Pending. The inverse direction works the same way — when namespace B (re-)gains eligibility, Kibana A is re-enqueued and the association is re-established immediately instead of waiting for the Pending requeue backoff.

Known limitation: transitive references (e.g. Agent → Fleet Server → Elasticsearch, each in a different namespace) are matched on the direct reference only; a flip of the transitive Elasticsearch namespace does not re-enqueue the Agent.

---

Other controllers with custom flip mappers (AutoOps, StackConfigPolicy, RemoteCluster)

Three more controllers use NewNamespacedController with their own mapper, because the resources a flip affects are not simply the ones living in the flipped namespace. All of them list from the cache rather than the FilterClient for the same de-scoping reason as above.

• AutoOps: an AutoOpsAgentPolicy selects its Elasticsearch clusters cluster-wide by label (ResourceSelector), so a flip of any namespace can change any policy's set of matching clusters — policies in the flipped namespace must be picked up or cleaned up, and policies elsewhere must deploy agents for newly scoped clusters or clean up agents and API keys for de-scoped ones. The mapper re-enqueues all policies, mirroring the controller's Elasticsearch watch (reconcileRequestForAllAutoOpsPolicies).
• StackConfigPolicy: re-enqueues policies in the flipped namespace plus policies in the operator namespace, which target resources cluster-wide. The operator namespace is always managed and never flips itself, so without this a scope change of a target namespace would never re-enqueue operator-namespace policies — leaving their settings unapplied on scope-in and their status counting hidden clusters on scope-out.
• RemoteCluster: re-enqueues the Elasticsearch clusters living in the flipped namespace, the counterparts their specs declare (wherever they live), and clusters elsewhere whose Spec.RemoteClusters reference a cluster in the flipped namespace. Reconcile re-evaluates all of a cluster's remote-cluster relationships in both directions (getExpectedRemoteClientsFor), so enqueueing the counterparts is enough to establish or tear down trust; the FilterClient then decides what remains visible.

---

License and trial controllers — namespace-scoped but deliberately not license-gated

Both controllers keep common.NewController and are not wrapped by NewNamespacedController, each for its own reason:

• License controller: it must keep reconciling in the unlicensed state to remove per-cluster license secrets so clusters revert to Basic; gating it on the license would prevent exactly that. Namespace scoping is still honored: its Elasticsearch and operator-license-Secret watches go through NamespacedKind, and a flip watch re-enqueues clusters when a namespace moves in or out of scope. Its flip mapper: if the flipped namespace holds an operator license secret, the licensing outcome may change for every cluster, so all currently in-scope clusters are re-enqueued (listed through the filtering client, which at that point yields exactly the clusters in scope); otherwise only the clusters in the flipped namespace (listed via the cache, so a de-scoped namespace's contents remain visible for the decision).
• Trial controller: a trial is started exactly when no valid enterprise license exists yet, so gating reconciliation on the license would deadlock the trial bootstrap. Its Secret watch goes through NamespacedKind, and its flip mapper re-enqueues the trial-license/trial-status secrets living in the flipped namespace.

---

Webhooks (pkg/controller/common/webhook/resource_validator.go, cmd/manager/validation.go)

ResourceValidator[T] gains a namespaceMatcher field set via WithNamespaceMatcher(m). preValidate now branches:

• If SelectorEnabled(): skip validation when !NamespaceNameMatches(ctx, ns) (dynamic mode).
• Otherwise: existing static managedNamespaces check (legacy mode).

The webhook server receives admission requests for every namespace in the cluster: several operator instances, each watching a different set of namespaces, may be installed side by side, but only one of their webhooks is actually registered. Namespaces that do not match — including ones that cannot be read from the cache — are silently let through instead of denied, so an operator never rejects a request for a namespace it doesn't manage. Because the matcher reads the namespace live from the local cache, this works identically on any replica, including non-leaders serving webhook traffic.

All RegisterResourceWebhook / RegisterWebhook call sites in validation.go pass the matcher.

The webhook certificates controller (pkg/controller/webhook/webhook_certificates_controller.go) also takes the matcher now, applying watches.NamespacedKind to its webhook-cert Secret watch. Its watch on the Validating/MutatingWebhookConfiguration itself stays on plain source.Kind, since those are cluster-scoped resources with no namespace to filter on.

---

pkg/controller/common/license/event.go — new file

EmitEnterpriseFeatureEvent(recorder, operatorNS, msg): emits a Warning event on the operator's own Pod when an enterprise feature is used without a valid license. Pod name is resolved from os.Hostname() (Kubernetes always sets this to the pod name); a no-op if the hostname cannot be determined.

---

cmd/manager/main.go and operator configuration

• New NamespaceSelectorFlag = "namespace-selector" constant (config-file only; no CLI flag, as the value is a structured metav1.LabelSelector object).
• parseNSSelector: reads the viper value, round-trips through yaml.Marshal / yaml.Unmarshal into metav1.LabelSelector, then converts via metav1.LabelSelectorAsSelector.
• Startup validation: namespaces and namespace-selector are mutually exclusive; supplying both is a fatal error.
• When the selector is set, DefaultNamespaces is left empty, widening the cache to cluster scope, and opts.NewClient injects nsmatch.NewFilterClient so all manager-level client calls are namespace-filtered.
• nsmatch.NewNamespaceMatcher is constructed once, given the manager's cache via SetCache(mgr.GetCache()), and threaded into operator.Parameters.NamespaceMatcher.
• The startup garbage collection of orphaned soft-owned/user secrets resolves its namespace list via MatchingNamespaces in dynamic mode instead of the (empty) static list.

---

Telemetry (pkg/telemetry/telemetry.go)

telemetry.NewReporter takes the NamespaceMatcher. A new getNamespaces helper decides which namespaces to scan: MatchingNamespaces(ctx) in dynamic mode (the static managedNamespaces list is empty/meaningless under a selector), the existing static list otherwise. Both getResourceStats and report use it, so telemetry follows namespaces as they move in and out of scope at runtime, the same way reconciliation does.

---

Helm chart (deploy/eck-operator/)

values.yaml:

# namespaceSelector is a Kubernetes label selector that dynamically controls which
# namespaces the operator manages. Mutually exclusive with managedNamespaces.
# Requires createClusterScopedResources: true.
namespaceSelector: {}

configmap.yaml emits the namespace-selector: block under operator config when the value is non-empty. validate-chart.yaml fails rendering when namespaceSelector is combined with managedNamespaces, or with createClusterScopedResources: false.

---

End-to-end test (test/e2e/namespace_selector/namespace_selector_test.go)

TestNamespaceSelectorDynamicLabelChange (skipped unless a test enterprise license is available):

1. Installs an enterprise license, labels ns1 with eck-visible=true, and leaves ns2 unlabeled.
2. Reconfigures the operator ConfigMap to a matchLabels: {eck-visible: "true"} selector and waits for the config-change restart.
3. Verifies the standard Elasticsearch builder sequence in ns1 (normal reconciliation) and that an Elasticsearch created in ns2 gets no pods.
4. Labels ns2, then asserts the ES cluster there is reconciled to health and that the operator did not restart — the namespace was picked up dynamically.
5. t.Cleanup restores the labels and original operator config.

Shared e2e helpers were factored out into test/e2e/test/helper (UpdateOperatorConfig, GetOperatorConfigValue, OperatorRestartCount, SetNamespaceLabel / DeleteNamespaceLabel); global_ca_test.go now uses them instead of its private copies.

Unit/integration coverage: filter_client_test.go, license_controller_test.go, association controller_test.go, autoops/namespace_flip_test.go, stackconfigpolicy/namespace_flip_test.go, remotecluster/watches_test.go, and telemetry tests for the dynamic path.

---

License gating

The dynamic namespace selector is an enterprise feature, enforced per-reconciliation by the NewNamespacedController wrapper:

• While no valid enterprise license is present, every wrapped controller skips reconciliation, emits a Warning event on the operator Pod, and requeues after 5 minutes. Resources in matching namespaces are left untouched (not modified, not cleaned up) until a license appears.
• The license and trial controllers are exempt from the gate (see their section above), so a license can always be installed — or a trial started — to unblock the operator; the next requeue then resumes normal reconciliation.
• In legacy / static mode (no selector) nothing changes: the wrapper is not installed and no license is required.

---

Backward compatibility

• When namespaceSelector is not set the NamespaceMatcher is constructed with a nil selector: every check returns true, NewNamespacedController falls through to NewController, NamespacedKind degrades to source.Kind, and WatchNamespaceScopeChange is a no-op. Existing behavior is unchanged.
• FilterClient is only injected when namespaceSelector is configured; all other modes use the default client.
• The static namespaces list and the new namespaceSelector are mutually exclusive, validated both at chart-render time and at operator startup.

---

What is not in scope

• Namespace-scoped RBAC profile (createClusterScopedResources: false): dynamic mode requires a cluster-wide cache and therefore a ClusterRole. The namespace-scoped profile is not supported.
• Multi-operator coordination / conflict detection.
• Destructive cleanup (deleting CRs when a namespace loses eligibility).
• OR semantics across different label keys (the LabelSelector AND-s across keys; OR within a single key is available via In).
• Re-enqueue of associated resources on match-state changes of transitively referenced namespaces (e.g. Agent → Fleet Server → Elasticsearch); only direct association references are matched.

[snyk-io]

⚠️ Snyk checks are incomplete.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
⚠️	Open Source Security	0	0	0	0	See details
⚠️	Licenses	0	0	0	0	See details

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/third-party-dependencies/main.md

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[moukoublen]

buildkite test this -f p=kind,t=TestNamespaceSelector -m s=9.4.2

[barkbay]

Quick, AI-assisted review (Claude Code), so worth a human sanity-check, especially the non-suggestion comments. Scope: the namespace-selector / license-gate machinery.

Most items are minor and inline below. Two cross-cutting notes that don't map to a single line:

• The six namespaceFlipRequests mappers (association, autoops, license, trial, remotecluster, stackconfigpolicy) share a list-cluster-wide, extract, build-requests skeleton; a shared helper taking a per-item mapper would dedupe them while keeping the "use the cache, not the FilterClient" invariant in one place.
• Cross-namespace flip behavior (e.g. an in-scope Kibana referencing an ES whose namespace flips out) has unit coverage on the mappers but no integration/e2e; the current e2e only exercises standalone-ES scope-in. An envtest for cross-namespace plus scope-out, and one e2e for the runtime association break, would be worth adding.

Nothing here is a hard blocker.

[barkbay]

This emits the event on the operator Pod rather than on the affected resource, so kubectl describe elasticsearch <name> gives no hint why it stopped reconciling. Volume is fine (events aggregate on the single Pod), but discoverability suffers. A status condition on the resource (writable by the ungated license controller) would surface it where users look; worth considering as a follow-up, not a blocker.

[barkbay]

nit: namespaceSelector collides with the existing webhook.namespaceSelector in this same file (different scope) and is asymmetric with its sibling managedNamespaces. Consider managedNamespaceSelector to pair with managedNamespaces and disambiguate from the webhook selector. Cheap to change before it ships as public API.

[moukoublen]

Renamed in 98fb717

[pkoutsovasilis]

Did a first pass, utilised also 🤖 , need to do some manual testing but here are my comments so far

[pkoutsovasilis]

here I think we need to call also this

r.dynamicWatches.Secrets.RemoveHandlerForKey(certificates.CertificateWatchKey(agent.Namer, obj.Name))

and the same happen

[moukoublen]

Was that missing in the implementation before this pr?

[pkoutsovasilis]

no it wasn't so this is a drive-by fix, but sure it can come as a follow-up independent PR

[pkoutsovasilis]

same comment as the elastic-agent one removing the dynamic watches for http certificates

[pkoutsovasilis]

LGTM, did run the following manual test scenarios and this works as expected, nice!

#	Scenario	Result
1	ES in labeled vs unlabeled namespace	In-scope reached health=green; out-of-scope had zero status fields, no pods/secrets ever created.
2	Namespace flip OUT/IN (Kibana + ES association)	Flip out: spec change bumped generation but observedGeneration stayed stuck 30s+ (no new pod). Flip in: caught up within ~5s. OnNamespaceOutOfScope only drops watches, doesn't tear down child resources.
3	Flip IN, cross-namespace association reference	Kibana (in scope) → ES in a namespace that just came into scope: association went Pending→Established in ~8s with no event on the ES itself — proves association controller's cluster-wide "case B" flip matching.
4	StackConfigPolicy, 2 ES clusters (1 in/1 out)	Baseline resources=1 readyCount=1/1. Flip out-of-scope in → resources=2, then readyCount=2/2. Flip both out → readyCount=0/0, status.details cleared; ES pods keep running untouched.
5	Two SCPs, same ES, different weights	High-weight SCP (100, co-located, starts out of scope) vs low-weight SCP (10, operator namespace, always in scope). After flip-in, both report 1/1, but the merged <es>-es-file-settings secret shows the weight-100 value (77mb), higher weight wins.
6	AutoOpsAgentPolicy, 2 ES clusters (1 in/1 out of scope)	One AutoOpsAgentPolicy in the operator namespace targeting two ES clusters by label (cluster-wide selector, no same-namespace restriction, unlike StackConfigPolicy). Baseline: resources=1 — only the in-scope cluster matched. Flip the out-of-scope one in → resources=2 immediately, and a second AutoOps agent Deployment appears once that cluster finishes provisioning. Flip both out → resources=0 readyCount=0/0, and both AutoOps Deployments get garbage-collected (unlike ES/Kibana's own out-of-scope handling, which just stops reconciling and leaves child resources in place).
7	Enterprise license removal/recovery (deletion, not time-based expiry — functionally equivalent: both leave CurrentEnterpriseLicense returning nil)	Deleted eck-license → gated resource's observedGeneration froze, 4x Warning InvalidLicense events fired with the exact expected message. Restored the license → recovered within ~5-10s via an already-in-flight reconcile trigger (no dedicated license-secret watch, so worst case is the 5-minute RequeueAfter).

[moukoublen]

buildkite test this -f p=kind,t=TestNamespaceSelector -m s=9.4.2

[moukoublen]

buildkite test this -f p=kind,t=TestNamespaceSelector -m s=9.4.2