feat(client): multi-db client with automatic endpoint failover - #3435
Draft
nkaradzhov wants to merge 10 commits into
Draft
feat(client): multi-db client with automatic endpoint failover#3435nkaradzhov wants to merge 10 commits into
nkaradzhov wants to merge 10 commits into
Conversation
Sketch of a multi-database client wrapper managing a homogeneous array of
underlying clients (standalone/pool/cluster/sentinel) behind one drop-in
client surface. Type mechanics only — failover/health/routing stubbed.
- @redis/client: dedicated factories (createMultiDbClient/Pool/Cluster/
Sentinel) returning { client, controller }; client typed exactly as the
base client (true drop-in), controller holds multi-db-only admin surface.
Command forwarding via prototype-walk (no runtime Proxy on hot path).
- redis meta-package: createMultiDbClient wrapper injecting default Stack
modules, mirroring createClient.
- playground.ts for manual poking.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…stubs Move MultiDbManager/MultiDbController out of index.ts, add typed stubs for config/circuit/database/failure-detector/health-check/failover-strategy/errors per the multi-db API contract, export the public surface from the package index, accept flat MultiDbConfig options in all factories, and drop the playground script (superseded by the quickstart smoke script). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…per, switch primitive - config.ts: defaults table + resolution/validation (weights in [0,1], unique ids with generated db-<n> fallback, health-check timing, >=1 db) - circuit.ts: CLOSED/OPEN/HALF_OPEN machine with clock-derived HALF_OPEN, grace-period restart on probe failure, consecutive-probe counting - database.ts: member wrapper (id/weight/circuit/role) with client lifecycle listeners feeding circuit and role; listeners are disposed only after close()/destroy() so a teardown-window 'error' emit cannot crash the process - manager.ts: atomic switch primitive with typed controller events and non-awaited old-member housekeeping; per-command result hook wired through the forwarding closures for the upcoming failure detector - controller.ts: typed event map (failover/fallback/database-unhealthy/ database-recovered/all-databases-down/error), descriptor-based getDatabases()/getActiveDatabase() - comments follow the constitution v1.0.0 principles: contract JSDoc (+@experimental) on the public surface, why-only implementation comments, cross-module constraints with file:symbol references - unit tests for circuit transitions and config resolution (no docker) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Remove spec-artifact references from comments (requirement IDs, research citations, data-model mentions) — the spec docs don't ship, so the references would dangle for external readers; each comment now carries its rationale inline. Task/story markers on unimplemented stubs stay as development scaffolding and are removed by the change that implements each stub. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- health-check.ts: default PING check, chain runner with per-probe
timeout bounding the whole chain, ALL/MAJORITY/ANY early-exit round
aggregation, round budget used as the member readiness bound
- manager.ts: initial connection flow — fan-out connect with bounded
establishment, per-member initial health check, initialAvailability
gate ('all'/'majority'/'one') with full teardown on rejection,
weight-based active selection with config-order tiebreak; runtime
addDatabase (pre-opened circuit until established, skipInitialHealthCheck
honored only here), removeDatabase (active switches to replacement
first), setWeight; repeat connect() re-probes instead of tripping
circuits of already-open members
- index.ts: per-topology MemberAdapter (create + keyless sendCommand)
wired through all four factories into manager construction
- controller.ts: addDatabase/removeDatabase/setWeight admin surface
- config.ts: probe-knob validation (numProbes, timeout, delays, empty
healthChecks), shared per-member identity resolution
- tests: probe-policy unit matrix (no docker); docker integration suite —
weighted selection, availability matrix with dead members, runtime
reconfiguration incl. skip-flag semantics, repeat connect, drop-in typing
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3 tasks
…g, pub/sub transfer - failure-detector.ts: sliding-window DefaultFailureDetector (count AND rate thresholds, 0 disables a condition, error filter counts filtered errors as traffic but not failures; injectable clock) - failover-strategy.ts: WeightBasedStrategy — highest-weight CLOSED member, member-order tiebreak - manager.ts: detection→failover wiring — per-command outcomes and member lifecycle errors feed the detector with source attribution (a stale in-flight rejection from the previous active must not trip the new one); failover procedure opens the failed circuit, switches to the strategy's pick, or gates traffic and retries selection up to maxFailoverAttempts before going permanently unavailable; commands fail fast with TemporarilyUnavailableError while searching and PermanentlyUnavailableError after exhaustion - pub/sub transfer on switch (standalone members): listeners are removed from the old member and re-subscribed on the new one, so a recovering old member cannot double-deliver - database.ts: per-topology signal mapping — sentinel per-node errors arrive via client-error and a sentinel-internal master change must not open the member circuit; cluster node errors aggregate into 'error' - tests: detector/strategy unit suites; failover integration (kill active → event + traffic continuity, in-flight rejection, pub/sub survival, all-down escalation); topology integration (cross-cluster failover, sentinel no-false-failover during its own master change, cross-sentinel-deployment failover) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- manager.ts: per-member unref'd health scheduler covering the active member too (a silently dead server never trips the organic detector); OPEN members rest through their grace period, HALF_OPEN members get recovery probes fed straight into the circuit (closing emits database-recovered), CLOSED members failing a round open with database-unhealthy — or fail over with reason 'health-check' when active; recovery probing keeps running during the all-down search so attempts can succeed; permanent unavailability stops all timers - auto-fallback loop (disabled by default): returns traffic to a strictly higher-weight healthy member, emitting 'fallback'; controller.setAutoFallback(intervalMs | false) retunes it at runtime - passive members that end or fail checks announce database-unhealthy without switching; deliberate removals stay silent - tests: recovery closes the circuit only after grace + probes (no flapping on a fast-restarting member), auto-fallback on/off/runtime toggle, passive failure reported without failover Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…alth check - lag-aware-health-check.ts: probes the RE cluster REST API's database availability endpoint with lag verification (extend_check=lag + availability_lag_tolerance_ms, default 5s tolerance) via the global fetch — no new dependency; basic auth, per-database endpoint/uid resolvers for members on different clusters, request timeout, and a requestOptions escape hatch for custom TLS dispatchers - custom failure detectors, chained health checks and custom failover strategies were already threaded through config — now exercised end-to-end and exported (LagAwareHealthCheck added to the public surface) - tests: stub-HTTP-server suite for the lag-aware check (availability, lag/auth failures, timeout, unreachable endpoint, per-database resolvers); integration cases for an error-filtering detector, a user-supplied detector driving failover (and being reset by the switch), and an all-must-pass check chain Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- manager.ts: setActiveDatabase(id) health-checks the target first — present reality overrides a stale OPEN circuit, so a verified target is closed (announced recovered) before the forced switch; the forced selection pins, suspending auto-fallback until releasePin(); any automatic switch away from the pin clears it, so a pin never traps traffic on a dead member; a successful force also rescues a client mid-search with every member down - controller.ts: setActiveDatabase/releasePin admin surface - tests: pin holds against auto-fallback until released; unhealthy target rejected; automatic failover off a dead pinned member clears the pin (proven by the fallback loop resuming); same-member force pins without a switch; force-rescue during the all-down search Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- docs/multi-db.md: quick start, selection and failover lifecycle, configuration reference with defaults, controller and events reference, custom checks incl. the lag-aware example, and the behavior contracts (in-flight rejection, eventual consistency, pub/sub loss window and per-topology transfer support, per-member client-side caching, resource overhead) - examples/multi-db-failover.js: runnable two-container demo of failover, recovery and fallback - @experimental on every public multi-db export - client-side caching regression test: no stale reads across a switch, no flush needed - topology coverage: pool failover, per-factory typing assertions, forced-pin smokes on cluster and sentinel members, sentinel fallback-after-recovery; the kill tests accept either automatic failover reason — the organic detector and the background health check legitimately race on slow-to-reject topologies Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a multi-database client to
@redis/client: one drop-in client backed by N memberdatabases (standalone/pool/cluster/sentinel). All traffic goes to one weight-selected
active member; the client fails over when it goes down. Every new export is
@experimental.Each factory returns
{ client, controller }.clienthas exactly the base client type:commands forward to the active member, lifecycle methods fan out to all members.
controllerholds the multi-db surface: topology, weights, runtime add/remove, events.Done: per-member circuit breaker, atomic switch primitive, config defaults and
validation,
connect()with initial health checks and anall/majority/oneavailability gate, weighted selection, runtime add/remove/setWeight, PING health check
with ALL/MAJORITY/ANY probe policies. Unit and docker integration tests.
Remaining: failover engine, pub/sub transfer and per-topology failure signals,
background health checks and auto-fallback, pluggable detectors and the lag-aware check,
forced failover with pinning, docs and example.
Additions only — no existing exported type or behavior changes.
🤖 Generated with Claude Code