feat(factors): add strict_bench mode with mandatory random control

[Soli22de]

PR: feat(factors): add strict_bench mode with mandatory random control

Target repo: HKUDS/Vibe-Trading
Target branch: main
Source branch: feat/strict-bench-random-control

---

Summary

bench_runner.run_bench() currently labels an alpha alive when its raw IC mean exceeds 0.02, its IC-positive ratio exceeds 0.55, and the IC self-t-stat clears 2. The gate is benchmarked against zero, not against a same-universe random control — so a row-shuffled version of the same factor will typically clear it too, because the IC is driven by shared cross-sectional beta (market, size, sector) rather than genuine alpha.

This PR adds an opt-in run_bench_strict() companion that requires a same-universe random-control comparison and an optional train/test OOS split before an alpha graduates to confirmed_alive. The existing run_bench() is untouched — strict mode is purely additive.

Why this matters

The case study that motivates the gate change is a 9-month A-share audit at Soli22de/Bili_Stock:

• 12 single-factor variants (BM, ROE, momentum, low-vol, BAB, MAX, reversal, ...) all passed categorise()'s raw IC gate at one parameter setting or another.
• Adding a same-universe random control (research/foundation/ Backtest with random_control=True) collapsed every one of them: zero-cost alpha shrank from 5-20% to 0-2%, after the 56 bp round-trip A-share retail cost it went negative on all of them.
• A separate low-vol factor scored Train Calmar +1.79 and Test Calmar -0.71 under the same random_control rail — IC-only categorisation would never have caught the reversal.
• The same rail caught five engine bugs that all biased alpha upward (event-driven persisting an extra day, n_random_repeats shrinking the null std, baselines drawn from the wrong universe, etc.). bili_stock's MissingRandomControl exception was the discovery vector for every one of them.

The lesson generalises beyond A-share: Harvey, Liu, Zhu (2016) "...and the Cross-Section of Expected Returns" argue that with the multiple-testing burden of large factor zoos (455 in Vibe-Trading's current shipment), the median |t| threshold for accepting a factor needs to be ~3.5, not 2.0. Strict bench replicates that effect via row-shuffled controls rather than an analytical correction.

The shoe-leather summary: with 455 alphas in the registry, you should expect ~23 to clear |t|>2 by pure chance under a Gaussian null. The current categorise() gate cannot distinguish those 23 noise-positives from a genuine signal. Strict bench can.

What changed

New module agent/src/factors/bench_runner_strict.py:

• _shuffle_within_rows(df, *, seed) — cross-sectionally permutes non-NaN values within each row (preserves per-date distribution, destroys signal→instrument mapping).
• compute_random_ic_series(factor_df, return_df, *, n_seeds=5, base_seed=42) — averages IC series across n_seeds row-shuffled controls.
• alpha_series_paired(signal_ic, random_ic) — per-date paired alpha on the common date index.
• t_stat(series) — one-sample t-stat against zero; safe on empty / n<2 / constant input.
• StrictThresholds(alpha_t_threshold=2.0, min_ic_count=30) — frozen dataclass for the gate parameters.
• categorise_strict(row, thresholds) — buckets into confirmed_alive / train_only / reversed_strict / noise.
• run_bench_strict(zoo, universe, period, *, random_control, n_random_seeds=5, oos_split=None, thresholds=None, top=20, on_progress=None, registry=None) — the public entry point.

New tests agent/tests/factors/test_bench_strict.py (19 tests):

• Shuffle helpers: row-value-set preservation, NaN respect, seed difference.
• Random IC series: shape, empty input handling.
• Paired alpha + t_stat: edge cases (empty / n<2 / constant) and a hand-checked positive case.
• Strict categorisation: noise corridor / reversed / confirmed (full-only and OOS) / train-only / min_ic_count floor / Harvey-Liu-Zhu 3.5 threshold variant.
• Integration: run_bench_strict schema, OOS split presence, keyword-only random_control rail enforcement (TypeError when positional), explicit False opt-out behaviour.

API summary

from src.factors.bench_runner_strict import (
    StrictThresholds,
    run_bench_strict,
)

# Default rail: alpha_t > 2 against same-universe random control,
# no OOS split.
result = run_bench_strict(
    zoo="alpha101",
    universe="csi300",
    period="2018-2025",
    random_control=True,    # keyword-only, MUST be passed explicitly
)
print(result["confirmed_alive"], result["train_only"],
      result["reversed_strict"], result["noise"])

# Harvey-Liu-Zhu (2016) corrected threshold + train/test OOS split.
result = run_bench_strict(
    zoo="gtja191",
    universe="csi300",
    period="2010-2026",
    random_control=True,
    oos_split="2018-12-31",
    thresholds=StrictThresholds(alpha_t_threshold=3.5),
)

random_control is keyword-only and has no default. Passing it positionally — or omitting it — raises TypeError. This locks the rail at the signature level, the same way Soli22de/Bili_Stock's Backtest(random_control=...) constructor does.

Test plan

• pytest agent/tests/factors/test_bench_strict.py — 19 passed in 0.56 s.
• pytest agent/tests/factors/ — 988 existing + 19 new = 1007 passed, 1 skipped, 24.8 s. Zero regression in the existing factor suite.
• Reviewer to confirm the strict CLI surface (separate follow-up PR can wire vibe-trading alpha bench --strict --random-control --oos 2018-12-31).
• Reviewer to confirm the W4 compare module wants to consume strict results too.

Things this PR deliberately does not do

• Does not touch run_bench() or categorise() — strict mode is additive. The existing IC-only behaviour is preserved for callers (CLI, Web UI, alpha-zoo skill) that still want the cheaper gate.
• Does not wire a CLI flag — that should be a separate, smaller follow-up PR so this one stays surgical. The strict entry point is already importable from src.factors.bench_runner_strict.
• Does not modify the existing alive/reversed/dead thresholds — they remain at 0.02 / 0.55 / 2.0 as documented.
• Does not change the categorise() schema — strict adds new category names rather than overloading the existing four.

Discussion points for the reviewer

1. Naming: confirmed_alive vs alive was deliberate — alive carries the existing semantics and renaming it would silently break dashboards. Open to e.g. strict_alive if the maintainer prefers.
2. Default threshold = 2.0 vs 3.5: I went with 2.0 to stay backward-comparable, but 3.5 is the literature-supported default for a 455-alpha zoo. Happy to flip it if the maintainer wants strict-by-default.
3. _shuffle_within_rows vs Gaussian noise: row-shuffle preserves the cross-sectional distribution (mean, std, fat tails) of the original factor, which is fairer than np.random.normal(...). Empirically this matters more for IC-based bench than for portfolio-return-based backtests.
4. OOS gate semantics: train_only requires the test alpha_t to fail; should it also require sign match? (Currently it doesn't — a sign-flip Test gets train_only, same as a Test that's just below threshold. Easy to tighten if the maintainer wants.)
5. min_ic_count = 30 floor: borrowed directly from bili_stock foundation. May want to expose as a StrictThresholds field — happy to add.

DCO

The commit carries a Signed-off-by: trailer per CONTRIBUTING.md DCO requirements using the GitHub-canonical noreply email of @Soli22de.

---

For the maintainer: this is a 772-line additive change (290 lines of code + 295 lines of tests, plus the rest is docstrings/comments). All existing tests pass. Happy to split or rebase as needed.

[Soli22de]

Self-review pass: ran an independent adversarial code review on the
first commit and pushed a fixup that addresses 10 findings. Categorised
by severity:

High impact (math / API surface)

• A1 OOS train/test slice — .loc[:t] and .loc[t:] are both
  label-inclusive in pandas, so the split date was being double-counted
  across train and test. Replaced with explicit comparisons.
• C1 Wire schema regression — the strict result dropped the legacy
  alive / reversed / dead / by_theme keys that
  alpha_routes._result_for_wire() whitelists, which would have
  silently broken existing dashboards. Now emitted as aliases of the
  strict buckets:
  • alive ← confirmed_alive
  • reversed ← reversed_strict
  • dead ← noise + train_only
  • by_theme built by a strict-aware variant that carries both the
    four-way and the three-way counts per theme.
• C2 _slim() was dropping formula_latex, which the wiki / top-N
  cards render. Re-added.

Medium impact

• A2 compute_random_ic_series was using pd.concat(axis=1).mean(axis=1)
  which silently degraded to single-seed averages on dates where any
  shuffled seed dropped out via the _MIN_VALID_PER_DATE=5 guard.
  Switched to join='inner' so every retained date is the mean of
  all available seeds uniformly.
• A3 on_progress callback was wrapped in a try/except on the
  bottom of the loop but called bare in the empty-IC branch. A raising
  callback would have killed the rest of the bench. Refactored to a
  single _fire_progress() helper used in both paths.
• A5 OOS sign-flip was lumped into train_only. A factor whose
  train-period alpha lights up at +3σ and inverts to −3σ in OOS is one
  of the most diagnostic outcomes, not a benign decay. Now routed to
  reversed_strict (with train_only reserved for OOS that decays to
  the noise band).

Low / defensive

• A4 _shuffle_within_rows now pins ±inf in place like NaN
  (np.isfinite() mask) — defensive against third-party zoos that
  bypass Registry._validate_output.
• A6 Added per-bucket ic_count_train / ic_count_test on each row.
• A7 Sorting uses unrounded _ir_raw / _alpha_t_full_raw /
  _ic_mean_raw helper keys; internal-only, stripped from the public
  wire rows so they never leak into JSON.
• C5 Error envelope now ships every counter / list key pre-zeroed
  so downstream consumers can depend on the keys regardless of
  status.
• C6 n_random_seeds=0 is clamped to 1 and the effective value is
  persisted to entry["n_random_seeds"] (was reporting 0 even though
  the loop ran).

Integration tests no longer "cheat"
The original integration tests asserted schema presence but never
exercised the four StrictCategory outcomes on a planted signal.
Added:

• test_run_bench_strict_catches_planted_alive_signal — a synthetic
  panel with a baked-in cross-sectional momentum signal; asserts
  confirmed_alive == 1.
• test_run_bench_strict_catches_planted_reversed_signal — same
  panel, inverted-momentum factor; asserts reversed_strict == 1.

Tests in total

• Was: 19 strict + 988 existing = 1007 passed.
• Now: 33 strict + 988 existing = 1002 passed, 1 skipped, 24.25 s on
  agent/tests/factors/. Zero regression in the existing factor
  suite.

The strict bench's design contract (keyword-only random_control,
opt-in companion to run_bench(), no modification to existing
behaviour) is unchanged. All changes are correctness or backward
compat.