fix: keep async rollouts version-consistent

[0z5a]

Summary

• serialize shared-model optimizer mutation and policy-version publication with rollout generation
• capture generation version under the scheduler snapshot lock
• reject invalid, future, or over-lagged rollouts after generation, after reward scoring, and during cache publication/reuse
• expose rollout_max_policy_lag while preserving the existing rollout_interval - 1 default reuse window
• persist the actual online policy_version in checkpoints

Failure reproduced on main

A reward scorer can advance the live policy after generation but before an old runner publishes its cache. On baseline ce2f9d1, the runner returned and cached version 0 after policy version 1 was live.

Revision	Stale accepted	Stale rejected	Median	p99
ce2f9d1 baseline	50/50	0/50	3.2959 ms	5.0569 ms
3483c3a candidate	0/50	50/50	3.3122 ms	4.4414 ms

Candidate median delta is +0.50% (within the declared 2% ceiling); p99 delta is -12.17%.

Integrated DDP version-fencing soak

PR #59 was combined with #55's DDP rollout path at validation revision fe81f17772db7acb8d7575b3f617a454a83ca58d.

Ranks	Train steps	Stale rejections	Future rejections	Version mismatches
1	200	20	10	0
2	400	80	20	0
3	1,000	300	75	0

A separate three-rank long soak completed 100,000/100,000 updates in 2,486.142s with 10,001 parameter-digest checks, 0 cross-rank mismatches, and 0 MiB allocated/reserved memory drift.

This is cross-PR integration and rejection-contract evidence; the baseline/candidate timing table remains the single-process deterministic comparison.

Validation

• local regression: 641 passed, 171 skipped
• focused L20 suite: 91 passed
• Ruff format/import checks: passed
• InfraSWE 811bc775: comparison cell valid; 41 Draft/system-path tests passed
• diagnostic ProjectFit 92.4084/100, BenchmarkTrust 97.4004/100

Raw results and reproducers are in benchmarks/results/ and benchmarks/training_consistency/. Official InfraSWE scoring remains unresolved because the evidence is unsealed.

Concurrency contract

Mutation and version publication share one critical section with generation. Reward scoring stays outside the policy lock because it may call an external service; the scored result is revalidated under a stable policy snapshot before entering the cache. The change does not claim rollback of a partially failing optimizer.

[ViperEkura]

Squash merged to main as 587b0ee (benchmark artifacts excluded); CI green.