M11 multitenancy xds#8
Merged
Merged
Conversation
…uard.Do) First slice of M11. Adds a small, dependency-free package every network-touching built-in module will share for outbound resilience: - Breaker (closed/open/half-open) with configurable threshold, cool-down, and half-open success requirement; thread-safe; injectable clock. - RetryBudget (token bucket) with capacity + refill rate, gating retry attempts only (first attempt is always free), matching gRPC-LB intent. - Guard.Do(ctx, fn) composes both with bounded exponential backoff and an operator-supplied Retryable predicate (defaults to 'everything except ctx errors'). Sentinel errors ErrCircuitOpen and ErrRetryBudgetExceeded let callers map to module.ErrUpstream so M5's negative cache won't memoize transient outages. Non-retryable errors do not erode breaker health. 29-package suite green; standalone pkg/upstream tests cover breaker tripping, half-open transitions, budget refill+cap, retry-until-success, max-retries exhaustion, circuit-open short-circuit, ctx cancellation during backoff, and the default retry predicate.
Adds upstream.FromMap to parse a shared 'resilience' YAML block (breaker + retries + budget) from any module's raw config, with safe defaults so a bare or missing block still produces a usable Guard. Migrates the two highest-traffic outbound modules: - pkg/authz/openfga: OpenFGA Check call now runs under guard.Do; non-2xx responses count as breaker failures, retries replay the request body, ErrCircuitOpen is mapped to module.ErrUpstream so the M5 negative cache won't memoize a transient outage. New tests verify the breaker short-circuits after repeated 500s and that retries succeed against a flaky upstream. - pkg/identity/introspection: RFC 7662 POST runs under guard.Do; same error mapping; existing per-identifier LRU + singleflight stay in place (cache then breaker, in that order). All 29 packages green.
Two M11 slices, both small and additive:
**Per-tenant rate limits (pkg/ratelimit).** New token-bucket limiter keyed by Request.TenantID with separate PerTenant + Default buckets. Wired into pipeline.Engine via Options.RateLimiter; on rejection the engine returns a 429 deny *before* identifier work runs, so a misbehaving tenant cannot exhaust shared module resources (OPA hot loops, OpenFGA/introspection RPCs). AuthConfig grows a new optional rateLimit: { perTenant: {rps,burst}, default: {rps,burst} } block; absent block = no limiting. nil limiter is always-allow so the change is zero-cost for everyone who hasn't opted in. Pipeline test verifies burst+denial+tenant-isolation; ratelimit package has its own unit suite (nil limiter, disabled spec, burst-then-refill, per-tenant isolation, default-vs-per-tenant precedence, concurrent-safe).
**Outbound-caller migration finished.** pkg/clientauth.ClientCredentialsSource now wraps its /token round trip in upstream.Guard.Do; new ClientCredentialsConfig.Resilience field exposes the same GuardConfig knobs the YAML 'resilience' block exposes elsewhere. internal/cache/valkey wraps Get/Set/Delete in a Guard so a slow Valkey fails fast under a circuit breaker instead of chewing pipeline goroutines; valkey.New() pre-builds a default Guard, the registered factory honors spec.Extra.resilience for tuning.
All 30 packages green.
Completes the M11 dev-loop slice. lwauthctl now ships three subcommands genuinely useful for debugging multi-tenant configs: - 'validate --config FILE' (enhanced): compiles the AuthConfig offline through the same module registry the running pipeline uses; non-zero exit on any factory error so it's CI-friendly. Prints a one-line summary (hosts, module counts, cache/rateLimit on/off). - 'diff --from A.yaml --to B.yaml' (new): module-level structural diff. Shows added/removed/changed identifiers, authorizers, and mutators (keyed by module name), plus changes to top-level scalars (hosts, tenantId, withBody, identifierMode), the cache block, and the rateLimit block. Both files are validated before the diff so we never report 'changes' against a config that wouldn't even start. '(no changes)' when identical. - 'explain --config FILE --request req.json' (new): dry-runs a request through the configured module graph stage by stage. Prints which identifier matched (and which ones returned no_match / invalid_credential / error along the way), the authorizer's verdict and reason, and which response mutators ran on allow. The request JSON shape mirrors module.Request's user-facing fields. Exits cleanly on the first deny so the output is readable for an operator looking at why a specific request was rejected. Tests boot the actual built binary against synthetic AuthConfig YAML for happy + denial paths on each subcommand (apikey + RBAC, since they need no network). 6 new tests, all 30 packages still green.
Extends IdentityProviderSpec (header/scheme/minRefreshInterval) and adds a controller-side resolver that expands an identifier's idpRef into the equivalent jwt config before Compile. Tenant-set fields override the cluster IdP; audiences set-union and de-dupe. Reconciler now Watches IdentityProvider so cluster-wide key rotations re-enqueue the watched AuthConfig automatically.
Adds pkg/configstream.Broker, the transport-agnostic primitive that replaces ConfigMap+SIGHUP. Latest-wins conflation keeps Publish non-blocking when subscribers stall; late subscribers are primed with the current snapshot. Reconciler gains an optional Broker that it Publish()es to after each successful Compile-and-Swap. The gRPC server wrapping this broker lands in the next slice.
Adds the lightweightauth.v1.ConfigDiscovery service: a single server-streaming RPC that pushes JSON-encoded AuthConfig snapshots from a controller to subscribed lwauth pods. Server wraps pkg/configstream.Broker; primes new streams with the latest snapshot, then forwards each Publish. Client helper Stream() decodes and invokes a caller-supplied handler so embedders compose retry/Compile/Swap themselves. Bufconn round-trip tests cover initial-snapshot delivery, mid-stream updates and multi-client fan-out.
lwauthd.Run does `audit.Default() == audit.Discard` to decide whether to install a default slog sink. Discard was a SinkFunc (a function type, which is uncomparable in Go); the comparison panicked at process start with 'comparing uncomparable type audit.SinkFunc', killing the lwauth container in docker compose immediately on boot. Switch Discard to a named zero-sized struct so interface-wrapped equality is well-defined. Adds a regression test that locks in the comparable invariant.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.