feat(firehose): self-test wedge defence + alive sentinel + decode-error robustness

[cloakmaster]

Why (Day-7 → Day-8 follow-up)

Two firehose observability findings surfaced during the Day-7 verification gate (PR #40's bench artifact + Day-7 close memory):

1. postgres-js LISTEN dispatch can wedge on a synchronous JSON parse failure in the user callback. We observed this live: a single malformed pg_notify payload caused the listener to receive exactly ONE event then silently drop every subsequent NOTIFY despite the TCP connection staying alive. Restart cleared it. Production triggers always emit valid row_to_json JSON so this isn't a current runtime risk — but a future migration or event type would silently kill the firehose with no automatic recovery.
2. Reconnect was silent (postgres-js library-level), not loud (our wrapper-level). When we kill the listener PID via pg_terminate_backend, postgres-js handled the LISTEN reconnect transparently and our wrapper's catch block never fired — events resumed without any firehose_listener_reconnect attempt=N sentinel reaching operator logs.

This PR adds defence-in-depth for both, scoped narrowly per the Day-8 directive.

What

1. Self-test wedge defence (Finding #1)

If selfTestIntervalMs is set (default 60s, 0 disables), the listener periodically fires its own pg_notify carrying a __listener_self_test__ event_type + fresh nonce. The listener's own onnotify handler:

• recognizes the event_type,
• filters it from the public EventEmitter (SSE clients never see it),
• updates lastSelfTestEchoNonce.

On the next heartbeat tick, if the prior nonce never echoed within selfTestIntervalMs * 1.5 tolerance, the listener throws → forces the reconnect-backoff path → resubscribes. New sentinel:

firehose_listener_self_test_failed observed_lag=Nms expected_within=Mms

This catches the wedge regardless of what postgres-js's library-level state is doing — we don't need to monkey-patch the library, we just verify our own dispatch is healthy from the inside.

2. Periodic alive sentinel (Finding #2 prep)

firehose_listener_alive heartbeats=N events=M self_tests_ok=K

Fired every aliveLogEveryHeartbeats (default 10, 0 disables) → ~5 min at default heartbeat. Gives operators a positive-confirmation signal without grepping for the absence of reconnect logs. Counters are per-cycle (reset on resubscribe).

3. Decode-error robustness (documentation only)

The existing JSON-parse catch + sentinel log already worked. Adding a comment explaining that received bytes prove dispatch is healthy at the postgres-js layer — the wedge wasn't from our parse failure but from postgres-js internal state. No behaviour change.

What's NOT in this PR

• postgres-js onnotice callback wiring. The library exposes onnotice as a construction-time option on postgres(); threading it through createDirectPostgresClient would need a small factory-options extension. Doable, but onnotice only fires for Postgres NOTICE-level messages (e.g. RAISE NOTICE), not connection-state transitions — which is what Finding ADR 0001: service-agnostic core (accepted, verified) #2 was actually asking for. The self-test mechanism above is the load-bearing defence; onnotice would be a small additional visibility layer. Filed as a follow-up if NOTICE-level visibility becomes operationally necessary.
• A workflow / cron to alert on firehose_listener_alive absence. That's an operator-dashboard concern; the sentinel logs are the building block.

Tests

3 new cases in apps/api/__tests__/firehose.listener.test.ts (existing 67 tests still pass; 70 total in apps/api):

• Self-test fires + echoes — heartbeat tick triggers self-test ping; injected echo via mock.notify is filtered from public emitter; subsequent real event passes through normally.
• Self-test wedge — ping fires but echo never arrives; tolerance window elapses; firehose_listener_self_test_failed logged; reconnect-backoff fires; factory invoked a second time (resubscribe). Validates the full wedge → reconnect → recovery path.
• aliveLogEveryHeartbeats=0 disables alive sentinel — clean-log surface for low-noise environments.

Verification

• pnpm check:core-isolation
• pnpm check:generated (8 files match)
• pnpm -r typecheck
• pnpm --filter @foxbook/api test → 70 passed + 4 skipped
• CI green
• Drafted for review — opening as draft per directive ("DO NOT merge yet — open as PR for my review").

Refs

• Day-7 close memory (project_day7_close.md): Findings Day 1 scaffold: three-language monorepo, core-isolation rule, envelope pipeline, CI #1 + ADR 0001: service-agnostic core (accepted, verified) #2 filed for Day-8.
• PR chore: 2026-04-27 firehose latency artifact + firehose-tail awk fflush fix #40 bench artifact: ops/bench-results/2026-04-27-firehose-latency.csv notes both findings.
• Listener: apps/api/src/firehose/listener.ts.