docs: dynamic configuration reload

Signed-off-by: Shane Utt <shaneutt@linux.com>

Author: shaneutt

docs/architecture.md

@@ -305,6 +305,31 @@ Request conditions gate both `on_request` and body hooks.
 Response conditions gate only `on_response` and response
 body hooks.
 
+## Dynamic Configuration Reload
+
+Praxis swaps filter pipelines at runtime without
+restarting the server or disrupting in-flight requests.
+
+Each handler holds an `Arc<ArcSwap<FilterPipeline>>`
+instead of a plain `Arc<FilterPipeline>`. On every
+request, the handler calls `pipeline.load()` to get a
+snapshot pinned for that request's lifetime. A reload
+stores a new pipeline into the `ArcSwap`; the next
+request loads the new pointer while in-flight requests
+drain on the old one.
+
+A file watcher (`notify` crate, 500ms debounce) monitors
+the config file. On change it validates the new config,
+rebuilds all pipelines, and swaps them atomically. If
+validation fails, nothing changes. Health check tasks
+are cancelled and respawned with a fresh registry on
+each successful reload.
+
+Changes that cannot be applied dynamically (listener
+topology, protocol type, compression module, TLS toggle)
+are detected by diffing old and new configs and logged
+as warnings.
+
 ## Crate Layout
 
 ### Workspace Crates
@@ -357,7 +382,9 @@ benchmarks                      Benchmark tool and library
 
 praxis                          Binary entry point
 ├── pipelines                   Pipeline resolution from config
-└── server                      Protocol registration, startup
+├── reload                      Config reload orchestration (validate, swap, health lifecycle)
+├── server                      Protocol registration, startup
+└── watcher                     File watcher with debounce for config hot-reload
 
 praxis-core                     Configuration, errors, and server factory
 ├── config/                     YAML parsing, defaults, and validation
@@ -438,10 +465,12 @@ praxis-filter                   Filter pipeline engine
     │   │   └── json_body_field Extract JSON field, promote to header
     │   ├── security/
     │   │   ├── cors            CORS preflight handling, origin validation
+    │   │   ├── credential_injection  Per-cluster API key injection
     │   │   ├── forwarded_headers  X-Forwarded-For/Proto/Host injection
     │   │   ├── guardrails      Reject requests matching string/regex rules
     │   │   └── ip_acl          Allow/deny by source IP/CIDR
     │   ├── traffic_management/
+    │   │   ├── circuit_breaker Per-cluster circuit breaking (closed/open/half-open)
     │   │   ├── rate_limit      Token bucket rate limiting (per-IP, global)
     │   │   ├── router          Path-prefix + host routing to clusters
     │   │   ├── redirect         3xx redirect without upstream

docs/configuration.md

@@ -17,6 +17,43 @@ shutdown_timeout_secs: # Optional. Graceful drain time (default: 30).
 insecure_options:      # Optional. Dev/test overrides. See development.md.
 ```
 
+## Dynamic Configuration Reload
+
+Praxis watches the config file for changes and
+automatically reloads filter pipelines without restart
+or disruption. When the file is modified, the server
+validates the new config, rebuilds pipelines, and swaps
+them atomically. In-flight requests complete on the old
+pipeline; new requests pick up the new config.
+
+If the new config is invalid (bad YAML, unknown filter,
+validation failure), the server logs the error and
+continues serving with the old config.
+
+**Dynamically reloadable:**
+
+- Filter pipeline configuration
+- Router routes and path mappings
+- Load balancer endpoints and weights
+- Rate limit and circuit breaker settings
+- Health check configuration
+
+**Requires restart (logged as warning):**
+
+- Listener add, remove, or address rebind
+- Protocol changes (HTTP to TCP)
+- Compression module addition
+- TLS enable/disable
+
+Stateful filters (rate limiter, circuit breaker) reset
+their state on reload. Operators should expect a brief
+burst window for rate limiters and a closed circuit for
+circuit breakers immediately after reload.
+
+See [hot-reload.yaml] for an example.
+
+[hot-reload.yaml]: ../examples/configs/operations/hot-reload.yaml
+
 ## Admin
 
 `admin.address` binds a separate HTTP listener that serves
@@ -268,13 +305,15 @@ supports TCP-level filters too.
 | `timeout` | Traffic Management | HTTP |
 | `static_response` | Traffic Management | HTTP |
 | `rate_limit` | Traffic Management | HTTP |
+| `circuit_breaker` | Traffic Management | HTTP |
 | `headers` | Transformation | HTTP |
 | `request_id` | Observability | HTTP |
 | `access_log` | Observability | HTTP |
 | `tcp_access_log` | Observability | TCP |
 | `forwarded_headers` | Security | HTTP |
 | `guardrails` | Security | HTTP |
 | `ip_acl` | Security | HTTP |
+| `credential_injection` | Security | HTTP |
 | `json_body_field` | Payload Processing | HTTP |
 | `compression` | Payload Processing | HTTP |
 | `cors` | Security | HTTP |
@@ -347,12 +386,24 @@ recover. See [health-checks.yaml].
 | `timeout_ms` | integer | 2000 | Per-probe timeout in ms |
 | `healthy_threshold` | integer | 2 | Consecutive successes to mark healthy |
 | `unhealthy_threshold` | integer | 3 | Consecutive failures to mark unhealthy |
+| `passive_unhealthy_threshold` | integer | none | Consecutive upstream failures (5xx or connect error) to mark unhealthy without probes |
+| `passive_healthy_threshold` | integer | none | Consecutive upstream successes to recover a passively-marked endpoint |
 
 TCP health checks only verify a TCP connection can be
 established; `path` and `expected_status` are ignored.
 When active health checks are configured, the admin
 `/ready` endpoint reports per-cluster health counts.
 
+Passive health checking tracks upstream request
+outcomes inline. When `passive_unhealthy_threshold` is
+set, endpoints that return consecutive 5xx responses or
+connect errors are marked unhealthy without dedicated
+probe traffic. Set `passive_healthy_threshold` to
+control how many consecutive successes are required to
+recover. Passive and active checks can be used together;
+either mechanism can mark an endpoint unhealthy, and
+either can recover it.
+
 By default, health check endpoints that resolve to
 loopback or cloud metadata addresses are rejected
 (SSRF protection).
@@ -448,6 +499,36 @@ When `allow` is set, only matching IPs are permitted.
 `allow` takes precedence over `deny`. Denied requests
 receive a `403 Forbidden` response.
 
+### Credential Injection
+
+Injects per-cluster API credentials into upstream
+requests and strips client-provided credentials to
+prevent forwarding. Pair with a source discriminator
+(IP ACL, client authentication) to control which
+clients receive credential upgrades. See
+[credential-injection.yaml].
+
+```yaml
+- filter: credential_injection
+  clusters:
+    - name: openai
+      header: Authorization
+      value: "sk-example-key"
+      header_prefix: "Bearer "
+      strip_client_credential: true
+```
+
+| Field | Type | Required | Description |
+| ----- | ---- | -------- | ----------- |
+| `clusters[].name` | string | yes | Cluster to inject credentials for |
+| `clusters[].header` | string | yes | Header name to set |
+| `clusters[].value` | string | one of | Inline credential value |
+| `clusters[].env_var` | string | one of | Environment variable containing the credential |
+| `clusters[].header_prefix` | string | no | Prefix prepended to the value (e.g. `"Bearer "`) |
+| `clusters[].strip_client_credential` | bool | no | Remove client-sent value before injection (default: true) |
+
+[credential-injection.yaml]: ../examples/configs/ai/credential-injection.yaml
+
 ### TCP Access Log
 
 Structured JSON logging of TCP connections. Works on both
@@ -515,6 +596,31 @@ successful responses.
 | `rate` | float | yes | Tokens per second (must be > 0) |
 | `burst` | integer | yes | Max bucket capacity (must be >= rate) |
 
+### Circuit Breaker
+
+Per-cluster circuit breaker that prevents cascading
+failures. When consecutive upstream failures reach the
+threshold, the circuit opens and subsequent requests
+receive 503 immediately. After the recovery window, a
+single probe request is forwarded; if it succeeds the
+circuit closes. See [circuit-breaker.yaml].
+
+```yaml
+- filter: circuit_breaker
+  clusters:
+    - name: backend
+      consecutive_failures: 5
+      recovery_window_secs: 30
+```
+
+| Field | Type | Required | Description |
+| ----- | ---- | -------- | ----------- |
+| `clusters[].name` | string | yes | Cluster name to protect |
+| `clusters[].consecutive_failures` | integer | yes | Failures before opening |
+| `clusters[].recovery_window_secs` | integer | yes | Seconds before half-open probe |
+
+[circuit-breaker.yaml]: ../examples/configs/traffic-management/circuit-breaker.yaml
+
 ### Guardrails
 
 Rejects requests matching string or regex rules against

docs/features.md

@@ -30,6 +30,13 @@
 - **Active health checks** - HTTP and TCP health check
   probes with configurable thresholds; unhealthy hosts
   are automatically removed from load balancer rotation
+- **Passive health checks** - track upstream failures
+  inline; endpoints that exceed a consecutive failure
+  threshold are marked unhealthy without dedicated
+  probe traffic
+- **Circuit breaker** - per-cluster circuit breaker that
+  short-circuits requests to failing upstreams with 503,
+  then gradually recovers via a half-open probe window
 - **Redirect** - return 3xx redirects without upstream;
   supports `${path}` and `${query}` template placeholders
 - **Timeout enforcement** - 504 rejection when upstream
@@ -138,6 +145,14 @@ deployment guidance.
 
 ## Operations
 
+- **Dynamic configuration reload** - filter pipelines,
+  routes, endpoints, health checks, and rate limits
+  are swapped atomically at runtime when the config
+  file changes. In-flight requests complete on the old
+  pipeline; invalid configs are rejected and logged.
+  Changes that require a restart (listener topology,
+  TLS toggle, protocol type) are detected and logged
+  as warnings.
 - **Graceful shutdown** - configurable drain timeout
 - **Runtime tuning** - thread pool sizing and
   work-stealing toggle
@@ -182,6 +197,12 @@ rather than bolted-on external processors.
   header-based routing to provider-specific clusters.
   Uses StreamBuffer to inspect the body before upstream
   selection.
+- **Credential injection** (`credential_injection`):
+  per-cluster API key injection with client credential
+  stripping. Supports inline values and environment
+  variable sources. Pair with a source discriminator
+  (IP ACL, client auth) to control which clients get
+  credential upgrades.
 
 ### Planned
 
@@ -199,8 +220,6 @@ filter pipeline.
   with sliding window or token bucket
 - **Cost attribution**: token counting mapped to user,
   session, model, and endpoint
-- **Credential injection**: per-cluster API key
-  injection with credential stripping
 - **SSE streaming inspection**: per-event filter hooks
   for streaming responses
 - **Semantic caching**: prompt deduplication via vector

docs/filters.md

@@ -249,6 +249,9 @@ hooks have default implementations that pass through.
 - `Release` : forward accumulated StreamBuffer data to
   upstream; behaves as `Continue` in non-StreamBuffer
   contexts
+- `BodyDone` : signal that this filter has finished body
+  processing; subsequent body chunks skip this filter
+  while other filters continue normally
 
 ```rust
 FilterAction::Reject(Rejection::status(429)
@@ -470,6 +473,7 @@ A filter can have both `conditions` (request phase) and
 | `timeout` | Traffic Management | HTTP | `timeout_ms` (504 on exceed) |
 | `static_response` | Traffic Management | HTTP | `status` (required), `headers`, `body` |
 | `rate_limit` | Traffic Management | HTTP | `mode`, `rate`, `burst`; token bucket with per-IP and global modes |
+| `circuit_breaker` | Traffic Management | HTTP | `clusters[].consecutive_failures`, `.recovery_window_secs`; per-cluster circuit breaking |
 | `headers` | Transformation | HTTP | `request_add`, `response_add/set/remove` |
 | `request_id` | Observability | HTTP | Propagates/generates `X-Request-ID` |
 | `access_log` | Observability | HTTP | Structured JSON logging; optional `sample_rate` |
@@ -479,6 +483,7 @@ A filter can have both `conditions` (request phase) and
 | `forwarded_headers` | Security | HTTP | `trusted_proxies` (CIDR list) |
 | `guardrails` | Security | HTTP | Reject requests matching header/body string or regex rules |
 | `ip_acl` | Security | HTTP | `allow` / `deny` (CIDR lists); 403 on denial |
+| `credential_injection` | Security | HTTP | Per-cluster API key injection with client credential stripping |
 | `json_body_field` | Payload Processing | HTTP | Extract a JSON body field and promote to header |
 | `json_rpc` | Payload Processing | HTTP | Parse JSON-RPC 2.0 envelopes and extract method/id/kind for routing |
 | `compression` | Payload Processing | HTTP | Gzip, brotli, and zstd response compression |