feat: safety hardening + exception hierarchy + replay harness with CI gate

[ArielB1980]

Summary

Comprehensive safety hardening of the live trading stack, structured exception handling across all critical paths, and a production-faithful event-driven replay backtest harness with CI integration.

Capital Safety (P0)

• DRY_RUN at transport boundary: KrakenClient.place_futures_order() refuses real orders when dry_run=True — no silent simulation
• Global order rate limiter: 60 orders/min, 10 orders/10s in ExecutionGateway — catches runaway loops, recursion bugs
• max_loss_per_trade_usd: Risk manager rejects trades where stop distance x size exceeds configurable max dollar loss
• Deadman switch: Heartbeat file at runtime/heartbeat.txt updated each tick — external watchdog detects hung loops, DNS hangs, event loop starvation

Exception Hierarchy (Tier 1)

• Replaced all bare except Exception: pass in kill switch, live trading, and safety integration with structured handling:
  • OperationalError -> retry/backoff (bounded)
  • InvariantError -> halt (fail-fast)
  • DataError -> log + skip
  • Unknown -> log + crash (systemd restarts)
• Narrowed circuit breaker classification: whitelist of networky errors only (no more string-matching "500")
• Routed stop self-heal + ShockGuard through ExecutionGateway (single choke point, consistent WAL/breaker/logging)

Replay Backtest Harness

• ReplayKrakenClient: Drop-in simulated Kraken exchange with:
  • Stop entered_book lifecycle with vol/depth-dependent delay + seeded jitter
  • Maker/taker via mid-crossing at placement (not fixed 80/20)
  • reduceOnly caps at flat; non-reduce can flip with two logical fills
  • Order rejections: min size, reduceOnly conflict, insufficient margin
  • Layer 1 visibility quirk toggle (hide_entered_book_from_open_orders)
  • Per-symbol funding rate curves with vol-spike multiplier
  • Deterministic seeded jitter on fills, delays, slippage (--seed N)
  • Per-API-call latency model (50-200ms, seeded)
• FaultInjector: Scripted API timeouts, rate limits, data errors, AttributeError at specific timestamps
• 6 episodes: Normal market, high-vol spike, liquidity drought, API outage, restart/split-brain, bug injection
• Safety-first pass/fail: Episodes fail on invariant violations, kill switch, rate limiter trips, breaker opens (context-dependent)
• make replay, make replay-sweep (seeds 1-5)

CI Gate

• .github/workflows/replay-gate.yml triggers on PRs touching execution/, risk/, safety/, kraken_client.py, live/, circuit_breaker.py, replay_harness/
• Matrix: unit tests -> replay across 3 jitter seeds (42, 1, 7)
• Artifacts uploaded per seed (14-day retention)

Docs and Cleanup

• Archived 28 obsolete docs and 20 legacy scripts to docs/archive/ and scripts/archive/
• Updated FORAI.md with lessons learned
• New operational tools: pre_flight_check, sync_positions, recover_sl_order_ids, check_tp_coverage, monitor_trade_execution

Test plan

• 500/500 unit tests passing (includes 49 replay harness tests)
• No linter errors on changed files
• No secrets in commit
• make smoke with .env.local on local machine (requires credentials)
• make replay passes all 6 episodes
• CI replay gate passes on this PR (3 seeds)
• After merge: make deploy -> verify systemd restart + heartbeat file appears

Made with Cursor