fix(relay): make readiness local and stop dropping sockets on DB errors

A reconnect burst exhausted the per-pod writer pools and two feedback
loops turned that into a total outage.

Readiness evaluated shared Postgres, Redis, and deletion-catalog health,
so every replica went NotReady together and the burst had nowhere to
land. The probe was also part of the load: the deletion-catalog check
acquires the writer pool, so each pod spent writer connections against
the exhausted pool every five seconds while failing. /_readiness now
answers from local process lifecycle only — shutting_down is 503,
anything else is 200 — and the dependency evaluation moves to /_status
on the same private health listener, under a `dependencies` object
carrying the fields the readiness body used to return. No startup state
is added: the health listener binds only after the database,
migrations, Redis, and pub/sub are up, so a process that can answer has
booted.

run_registered_community_connection collapsed Ok(false) and Err into
"not active", so a writer-pool timeout in is_community_active read as
confirmed archival and dropped the socket, which reconnected and
re-checked. Only a confirmed Ok(false) cancels now; a lookup failure
admits the socket with a structured warning and defers to the periodic
revalidate_live_communities backstop. Writes are unaffected and remain
fail-closed on their own per-event fence.

Telemetry keeps its existing names: buzz_readiness_checks_total narrows
to {ready, shutting_down}, the dependency families are now sampled by
/_status, dependency gauges are dropped, and one new bounded counter,
buzz_community_admission_checks_total{outcome}, counts the admission
decision. The per-pod raw-series ceiling drops from 99 to 86.

This deletes the readiness publication machinery — the mutex, probe
generations, ProbeTicket/ProbeStart, finish_probe,
finish_public_evaluation, and a second shutdown flag duplicating
AppState::shutting_down. All of it existed to order concurrent async
dependency evaluations against shutdown. Readiness is now a single
atomic load, so the one ordering guarantee still worth keeping — a
racing shutdown must win, and never leave a draining pod advertising a
ready gauge — is a post-write re-read in record_readiness_probe rather
than a generation-fenced mutex.

Co-authored-by: Claude Code <noreply@anthropic.com>

Redis had no startup gate at all. `deadpool_redis` pools dial lazily and
PubSubManager::new only allocates channels, so "Redis pub/sub connected"
was logged against a dead port and boot ran to completion. With readiness
now answering from local lifecycle alone, such a pod bound its health
listener and advertised ready for the rest of its life. state::
verify_redis_command_path acquires one connection from the command pool
and issues PING before AppState is built, and therefore before the health
listener binds, because binding is the one-way latch that makes a pod
routable. No startup_ready flag is added for the same reason. Post-start
Redis failures are unchanged: they are dependency failures and never move
readiness. Postgres startup connection behavior is untouched.

Signed-off-by: tornquist <tornquist@squareup.com>

Author: TheSentinel454

ARCHITECTURE.md

@@ -625,7 +625,7 @@ pub enum AuthState { Pending { challenge: String }, Authenticated(AuthContext),
 | GET | `/.well-known/nostr.json` | NIP-05 identity |
 | GET | `/health` | Health check |
 | GET | `/_liveness` | Liveness probe |
-| GET | `/_readiness` | Readiness probe |
+| GET | `/_readiness` | Readiness probe — local process lifecycle only |
 | POST | `/events` | Submit a signed Nostr event over HTTP (same ingest path as WebSocket `EVENT`) |
 | POST | `/query` | Query Nostr events over HTTP with NIP-01 filters |
 | POST | `/count` | Count Nostr events over HTTP with NIP-45 filters |

crates/buzz-relay/src/main.rs

@@ -454,7 +454,13 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
         cfg.create_pool(Some(deadpool_redis::Runtime::Tokio1))
             .map_err(|e| anyhow::anyhow!("Redis pool creation failed: {e}"))?
     };
-    let redis_health_pool = redis_pool.clone(); // cheap Arc clone — shared with readiness handler
+    let redis_health_pool = redis_pool.clone(); // cheap Arc clone — shared with AppState
+                                                // One-time bootstrap gate, deliberately before AppState and therefore before
+                                                // the health listener binds. Post-start Redis failures are dependency
+                                                // failures and must never move readiness; never having connected at all is
+                                                // a broken deployment, not a blip.
+    buzz_relay::state::verify_redis_command_path(&redis_health_pool).await?;
+    info!("Redis command path connected");
     let pubsub = Arc::new(
         PubSubManager::new(&config.redis_url, redis_pool)
             .await

crates/buzz-relay/src/metrics.rs

@@ -205,6 +205,7 @@ pub fn try_install(port: u16, gauge_idle_timeout_secs: u64) -> Result<(), Metric
     metrics::set_global_recorder(recorder)
         .map_err(|_error| MetricsInstallError::RecorderConflict)?;
     describe_readiness_metrics();
+    describe_community_admission_metrics();
     describe_db_pool_metrics();
     tokio::spawn(exporter);
     Ok(())
@@ -219,24 +220,37 @@ pub fn install(port: u16, gauge_idle_timeout_secs: u64) {
         .unwrap_or_else(|error| panic!("metrics exporter must install exactly once: {error}"));
 }
 
-/// Register the frozen readiness metric descriptions with the active recorder.
+/// Register the frozen readiness and dependency-diagnostic metric descriptions.
+///
+/// The two `buzz_readiness_*` probe families describe local process lifecycle.
+/// The two dependency families keep their names for dashboard continuity but
+/// are sampled by the diagnostic `/_status` endpoint, not by the Kubernetes
+/// probe — a shared-dependency failure no longer deroutes the pod.
 pub(crate) fn describe_readiness_metrics() {
     metrics::describe_counter!(
         "buzz_readiness_checks_total",
-        "Kubernetes health-listener readiness probes by terminal bounded reason"
+        "Kubernetes health-listener readiness probes by lifecycle reason (ready, shutting_down)"
     );
     metrics::describe_counter!(
         "buzz_readiness_dependency_checks_total",
-        "Completed readiness dependency attempts by dependency and bounded outcome"
+        "Completed /_status dependency attempts by dependency and bounded outcome"
     );
     metrics::describe_histogram!(
         "buzz_readiness_check_duration_seconds",
         metrics::Unit::Seconds,
-        "Completed readiness check duration without outcome label multiplication"
+        "Completed /_status dependency check duration without outcome label multiplication"
     );
     metrics::describe_gauge!(
         "buzz_readiness_state",
-        "Latest publishable readiness state by check, where 1 is ready and 0 is not ready"
+        "Local readiness of this process, where 1 is ready and 0 is shutting down"
+    );
+}
+
+/// Register the bounded community-admission contract.
+pub(crate) fn describe_community_admission_metrics() {
+    metrics::describe_counter!(
+        "buzz_community_admission_checks_total",
+        "Durable community-active checks at socket admission by bounded outcome"
     );
 }