[Roadmap] Financial correctness hardening: live gates, backtest engines, shadow-account chain

[he-yufeng]

I spent this week going through the money paths end to end: live mandate gates, backtest engines, data loaders, and the shadow-account chain. Three read-only audit passes, then I re-verified every finding below line by line against current main. Filing this as the tracking roadmap. I will send the fixes as a series of small test-backed PRs, one per finding, each referencing this issue.

Verified correct, please do not "fix" these

• Daily engine timing: signals are shift(1) per symbol calendar and fill at the next bar open; sizing uses open equity; optimizers read strictly past data. No lookahead in the daily path.
• BaseEngine accounting: capital and equity stay consistent across open / add / reduce / close in both directions, with proportional margin and commission release.
• A-share single engine: T+1, no shorting, limit-band checks use the historical base price only, stamp duty sell-side only, commission floor correct.
• strict perpetual path: funding sign, settlement dedup, maintenance-margin brackets, adverse-extrema liquidation. Good shape.
• quantlib (XIRR, TWR, VaR conventions, waterfalls) and backtest/metrics.py: zero/negative prices and empty series are deliberately handled.
• Slippage direction, tick-grid rounding toward the adverse side (KR/VN), tax single vs double-sided rules (CN/HK/IN/KR/VN), PIT fundamentals at daily granularity.
• Live idempotency: deterministic client order ids, no-retry writes, cancel audit. expires_at handling is consistently UTC and fail-closed.
• scheduled_research: cron evaluation (DST gaps, fold), interval advance, backoff cap, verdict previous embedding. No numeric hazards.

Phase 0: live gate fail-opens (real money at stake, small diffs)

1. Alpaca short positions read as long in exposure math. agent/src/trading/connectors/alpaca/sdk.py:849 maps positions with quantity: qty (Alpaca's qty is always positive; direction lives in side), but _position_signed_market_value (agent/src/live/enforcement.py:407) takes the quantity branch first and never reaches the side branch. The gate's own tests use signed quantities for shorts (agent/tests/test_mandate_enforcement.py:381), so an Alpaca short books as positive exposure. Selling more then lowers the computed exposure toward zero, so max_total_exposure_usd / max_leverage effectively stop binding, and a buy-to-cover reads as adding long. Scenario: $50k short plus another $50k sell computes post-trade exposure near 0 while the real book doubles.

2. OKX and Futu position reads degrade API errors into "empty book, status ok". connectors/okx/sdk.py:658 returns [] on any non-zero business code; connectors/futu/sdk.py:1071 unwraps errors to None then []. _safe_read only fails closed on exceptions, so a transient API error makes the gate evaluate the mandate against an empty book and let the order through.

3. Kill-switch flatten records broker error envelopes as success. runtime/flatten.py:194 appends any non-exception response to cancelled_order_ids and audits accepted; the submit path (src/tools/mcp.py:520) returns {"status": "error"} envelopes instead of raising. A shaky broker connection during a halt writes a compliant-looking audit trail while resting orders stay live, exactly the scenario the sweeper exists to prevent.

4. HALT survives restarts but the flatten latch does not. HALT is a file sentinel (live/halt.py:46); LiveRunner.__init__ resets _flatten_fired = False every boot (runtime/runner.py:391). After a restart with flatten orders still working, the next tick replays the whole sweep: a cancel pass plus a fresh market order per position, which can flip the account from long to net short.

5. MARKET triggers never check market hours. runtime/runner.py:51 imports due_now and never calls it (zero call sites); _jobs_from_triggers (runner.py:775) downgrades a market trigger to a plain 60s interval job. Weekend ticks submit orders that queue into the Monday open.

Phase 1: price-data adjustment caliber

6. Yahoo/yfinance chains deliver raw OHLC while every other source is adjusted. loaders/yahoo_client.py:224 reads only indicators.quote[0] and never adjclose; yfinance_loader.py:114 sets auto_adjust=False and drops Adj Close. Yahoo is the chain head for us_equity and several other markets (loaders/registry.py:141), while tencent/eastmoney/baostock/akshare/tushare all return qfq. AAPL's 4:1 split reads as a real -75% day, NVDA's 10:1 as -90%, and every ex-dividend gap books as a fake loss. Two runs over the same market can mix adjusted and raw sources, so results are not comparable across fallbacks. Fix direction: read adjclose (and request events=div,splits) on the Yahoo paths, or stamp an adjustment provenance and refuse to mix calibers within one chain.

7. Partial batch success stops the fallback chain. src/market_data.py:227 breaks on any non-empty data_map; loaders omit failed symbols instead of raising (see the yahoo_loader docstring), so one 404 out of five symbols leaves the rest in _unresolved with no per-symbol retry down the chain. Only the Canadian venue alias gets a rescue.

Phase 2: backtest engine realism

8. Non-strict perpetual funding settles once per day; the documented model is every 8 hours. engines/_market_hooks.py:257 dedups by (symbol, date, hour) and daily bars land on one slot, so the drag is 0.01% x 365 (~3.65%/yr) where crypto.py's own docstring says three settlements per day (~10.95%/yr). Funding-neutral strategies are wrong by construction, and spot-proxy shorts collect funding that does not exist.

9. Non-strict liquidation checks the close only, and exempts leverage <= 1. _market_hooks.py:300 marks to bar close instead of the adverse extremum the strict path uses, and pos.leverage <= 1.0 returns False outright. A 1x short survives a 2x adverse move in simulation while equity goes negative.

10. Composite engine drops market rules. engines/composite.py:199 delegates can_execute to stateless sub-engines: India's T+1 reads the sub-engine's (empty) positions, and every limit band reads _close_arr, which only the running engine ever sets. In composite runs, India allows same-day round trips and A-share limit-up opens are fillable.

11. Options engine has no signal-to-execution shift. engines/options_portfolio.py:248 prices same-day spot/IV and :328 executes instructions keyed to that same date, so a signal written the natural way fills at the close it was computed from: roughly a full day of underlying move times delta of phantom edge. The HV backfill at :45 also leaks the 30th bar's volatility into bars 1..29.

12. Options engine has no margin or buying-power constraint. options_portfolio.py:352 credits short premium without margin, and cash can go arbitrarily negative, so naked-selling strategies produce the classic free-money curve.

Phase 3: shadow-account chain

13. shadow_total_pnl is always 0. shadow_account/backtester.py:476 reads total_return_abs / total_pnl, and the engine never writes either key (backtest/metrics.py:605 returns final_value, total_return, ...). The test stub (agent/tests/test_shadow_account.py:354) hand-writes exactly that nonexistent key, which is why CI stays green. Every delta-attribution report currently shows Shadow PnL = 0 with five distorted buckets.

14. real_pnl sums raw amounts across currencies. backtester.py:539 adds every roundtrip's pnl with no market grouping, while the futu parser explicitly supports mixed HK+US journals. HKD amounts get summed with USD and then compared against a single-currency shadow pool.

15. FIFO pairing ignores corporate actions entirely. tools/trade_journal_tool.py:34 pairs raw qty/price; a split between buy and sell fabricates a large fake loss and silently drops the leftover shares. Cash dividends never appear in fills, so high-yield names are systematically understated.

16. Short sales parse but are inexpressible in FIFO. The parser accepts sell-short / buy-to-cover tokens, but the queue only models buy-first: the short sale vanishes silently, and the later cover enters as a phantom long lot that poisons subsequent matches.

17. Attribution buckets double count. backtester.py:546-559: the early/late conditions are subsets of not-within-rule, so one trade lands in both noise and early/late, and explained sums those twice. The waterfall shows mutually inconsistent buckets.

Phase 4: mandate gate hardening

18. Limit price never enters notional math. _normalize_notional (sdk_order_gate.py:525) prices quantity x current quote; a buy limit at 2x the market passes a cap sized for the quote while being fillable at twice the authorized amount.

19. Mandate commit widening checks go silent on non-numeric and list fields. mandate/commit.py:265 only compares when both sides are numeric; {"leverage": "10"} against a ceiling of 2 slips through and commits as 10.0, and the instruments list has no comparison branch at all.

Minor, batched into the phase PRs that touch the same files

tencent 500-bar silent truncation on long windows; long halts valued at cost after the ffill limit; extractor US price features always NaN (missing .US suffix); non-strict crypto closes at maker fee; shadow report cache keyed by shadow_id only; MT5 silently clamps oversize volume; eToro limit order without limit_price sends an untriggered MIT; futu K-line adjustment caliber undocumented; fundamentals PIT leaks into intraday bars below 1D; bare 6+ letter codes fall into a_share rules.

How this lands

One PR per finding, small and test-backed, in the phase order above; Phase 0 first because those gates guard real accounts. I am taking the list unless someone claims an item here first. I will keep this issue updated as the PRs merge.

[warren618]

Phase 0 items 1 and 2 are merged — I verified both against main before merging (reproduced the unsigned-short exposure math and the ok-empty envelope on a rejected read independently).

One note on #1209: the == "short" check only fired on the TAP JSON path — alpaca-py's Position.side is a (str, Enum) member whose str() is "PositionSide.SHORT", so the direct-SDK path (the one the gate actually guards) still booked shorts unsigned. I pushed a small commit to your branch normalizing via .value with an enum-typed regression test before merging. Worth carrying into the remaining connector findings: envelope fixes need a test in both shapes, SDK object and HTTP JSON.

The roadmap is excellent — the "verified correct, do not fix" section especially. Your plan and cadence (one small test-backed PR per finding, phase order) work for me, and the list is yours as proposed. Two priorities from my side: findings 3–5 to close out Phase 0, and finding 6 (Yahoo delivering raw OHLC at the chain head) pulled forward — mixed adjustment calibers across fallbacks poison every us_equity backtest, so I'd take that one ahead of the rest of Phase 1. The independent reviews on #1203/#1204 were spot on too — appreciated.

[he-yufeng]

Progress so far:

• Item 1 (Alpaca unsigned shorts): merged in fix(alpaca): sign position quantity by side for the mandate gate #1209, including warren618's enum normalization on the direct-SDK path.
• Item 2 (OKX/Futu ok-empty envelopes): merged in fix(live): fail closed when broker position reads return an API error #1212.
• Item 3 (halt sweep records error envelopes as success): PR fix(live): fail closed on broker error envelopes during the halt sweep #1232. Both the cancel pass and the flatten pass now fail closed on the adapter's error envelope, with the envelope kept on the audit record.
• Item 13 (shadow PnL always 0): merged in fix(shadow): derive shadow PnL from runner metrics, fail closed when unknown #1217, PnL now derives from runner metrics with an explicit unknown state.

Next up is item 4 (the flatten latch does not survive restarts) unless someone takes it first.

[he-yufeng]

Item 4 (the flatten latch does not survive restarts) is up in #1233. The latch is persisted next to the per-broker HALT sentinel, bound to the halt episode (tripped_at, mtime fallback for hand-touched sentinels). Clearing HALT and re-tripping re-arms the sweep, and a crashed sweep is never retried, so the no-retry rule holds exactly as before. 8 new tests including the two restart scenarios.

Item 5 (MARKET triggers never check market hours) is next.