docs: add PRD-029 Operational Gateway (#1296)

* docs: add PRD-029 Operational Gateway

Asset-agnostic control signal gateway that routes non-financial
instructions from ledger state changes to external systems. Reuses
PRD-024 bidirectional MappingDefinition pattern for payload
transformation. Aligns to BIAN Operational Gateway, Service Provider
Operations, and Servicing Order service domains.

* docs: address review feedback on PRD-029

- Fix G3 goal: declarative mapping-based transformation (not Starlark)
- Fix synchronous response description to reference mapping engine
- Renumber subsections 5.x to 6.x under Section 6. Service Design
- Remove PII logging risk: mandate redaction/allow-listed fields only
- Add note clarifying RETRYING/CIRCUIT_OPEN as transient sub-states
- Fix SQL index to use DISPATCHING (not RETRYING) as persisted status

* docs: expand PRD-029 to bidirectional Operational Gateway

Expand scope to match BIAN's full Operational Gateway definition which
handles both sending and receiving non-financial messages. Key changes:

- Add Phase 5: Inbound Non-Financial Messages (8 points, 42 total)
- Add InboundMessage model, persistence, gRPC RPCs, Kafka events
- Add inbound_routes manifest section with CEL classifiers
- Add SecretStore port (EnvVarSecretStore Phase 1, Vault Phase 2)
- Add correlation field contract for callback mappings
- Add BIAN terminology mapping table (dispatch/receive action terms)
- Reference Platform Scheduler (PRD-021) for health check workers
- Note circuit breaker shared state requirement for multi-pod
- Update domain composition diagram to show bidirectional flow
- Update goals, success criteria, risks for inbound coverage

Incorporates feedback from Kagi (BIAN alignment review) and Gemini
(architectural refinements).

* docs: add open design questions, reduce implementation specificity

Replace proto definitions and SQL schemas with conceptual descriptions
— these are implementation decisions, not requirements. The PRD was
overspecified in easy areas and underspecified in hard ones.

Add Section 6: Open Design Questions covering 9 architectural
challenges that need resolution during implementation:
- Compensation after delivery (can't un-send an HTTP request)
- Timeout composition (dispatch vs ack vs SLA timeouts)
- In-flight concurrency and backpressure
- Dead letter handling (promote from Phase 4 to Phase 2)
- Duplicate callback idempotency
- Ordered vs unordered processing semantics
- Fan-out / fan-in patterns
- Circuit breaker scope in multi-pod deployments
- Secret rotation

Expand Section 8 (Manifest Apply) into declarative intent model:
- Versioned gateway config alongside instruments and sagas
- Additive vs breaking change semantics
- Route config captured at instruction creation time
- Tenants evolve integrations through manifest, not code

Fix section numbering (1-16) after inserting new sections.

* fix: address review feedback on PRD-029

- Use mapping-first language for idempotency (outbound mapping, not
  transform script)
- Clarify Starlark transformer as exceptional fallback
- Strengthen Phase 1 secret management: note K8s Secrets for env var
  injection
- Document Redis as Phase 2 infrastructure dependency for circuit
  breakers, clarify Phase 1 uses local-only breakers
- Update README description to reflect bidirectional scope
- Replace "AI-generated transform scripts" with "AI-generated mappings"

* feat: add open design questions for service composition and control logic

Add questions 6.10-6.13 exploring:
- Reference data / market data as gateway inputs (forecasting pattern)
- Starlark/CEL for event-driven control logic (conditional dispatch,
  market-driven triggers, provider failover, escalation chains)
- Platform Scheduler integration for periodic gateway operations
- Two control surfaces (React + MCP) and operational tooling needs

* feat: resolve open design questions 6.10-6.13 via Six Hats analysis

Convert open questions to resolved design decisions after
White/Black/Yellow/Green/Blue Hat analysis:

- 6.10: Config in manifest (payment_rails precedent), Prometheus
  for telemetry. Ref data and market data integration rejected.
- 6.11: Gateway is dispatch engine, not decision engine. Add
  fallback_connection_id for single-hop failover. Sagas own all
  control logic. Cross-saga rules explicitly scoped outside gateway.
- 6.12: WorkerLifecycle for all sub-minute ops. CronScheduler
  reserved for future tenant-defined scheduled instructions only.
- 6.13: Standard gRPC RPCs, 3 MCP tools max in Phase 1, PRD-025
  for real-time events. Connection CRUD via existing manifest tools.

Also add routing intelligence to non-goals and fallback_connection_id
to the manifest YAML example.

* feat: add statelessness constraint and Stripe migration path

Add Section 4.9 (Statelessness and Horizontal Scaling):
- Hard constraint: any pod handles any request for any tenant
- Dispatch pod != callback pod (async requires shared state)
- All state in CockroachDB or Redis, not in-process
- FOR UPDATE SKIP LOCKED for dispatch worker scaling
- Lesson from existing Stripe per-pod circuit breaker issue

Expand Stripe migration path (Section 13):
- Current architecture mapped to gateway equivalents
- What moves (HTTP calls, webhooks) vs what stays (domain logic)
- 6-step migration plan
- Validates the gateway abstraction: if it can't subsume Stripe,
  the design is wrong

* feat: add runtime-configurable provider integration section (4.10)

Make explicit that the gateway requires zero compiled code per
provider. New integrations are defined entirely at runtime through
manifest config + MappingDefinitions — same principle as Starlark
sagas and CEL validations.

Compiled code grows with protocol diversity (HTTP, gRPC, MQTT),
not provider count. Adding 100 HTTP/JSON providers requires zero
compiled code.

* docs: pin BIAN Release 14.0 and document topic naming rationale

Add BIAN v14 version pin to Section 2 (unchanged from v13). Document
deliberate divergence from BIAN AsyncAPI channel naming in Section 7.4
— Meridian's dot-notation encodes service ownership, event semantics,
and schema version.

---------

Co-authored-by: Ben Coombs <bjcoombs@users.noreply.github.com>

Author: bjcoombs