[FEATURE] RSIBench-Data: meta-evaluation of pipeline research stage — checkpoint preservation rule + four trajectory diagnostic patterns (78% regression-after-peak signal)

[Lexus2016]

Source

• Paper: https://arxiv.org/abs/2607.25886 (Lingxiao Du et al., Evolvent AI / NUS, Jul 28, 2026)
• Code: https://github.com/evolvent-ai/RSIBench-Data

Problem

The evolution pipeline does not benchmark its own data-centric research capability. The research stage is itself a data-centric researcher -- it diagnoses capability gaps (via web search), proposes improvements (findings), and iterates from feedback (prior reports). There is no controlled evaluation of whether this process exhibits the patterns that distinguish strong runs from weak ones.

Key Finding from the Paper

Agents improve from feedback in 58% of settings but regress after their peak 78% of the time. This is the same overfitting signal RSEA documented at the agent-state level (33-point train-test gap) and GSME documented at the harness-edit level (phantom wins ~60% of the time). RSIBench-Data extends it to the data-strategy level.

Four Trajectory Patterns That Distinguish Stronger Runs

1. Accurate hypotheses -- the agent correctly diagnoses which capability gap to target
2. Validation-grounded supervision -- the agent uses evaluation feedback to ground its data strategy, not just to score it
3. Behavior-aligned data -- the training data the agent selects actually targets the diagnosed gap
4. Preservation of strong checkpoints -- the agent does not discard a good checkpoint in favor of a later, worse one (direct analogue of RSEA strict-improvement-only rule)

Proposed Feature

Adopt RSIBench-Data as a meta-evaluation instrument for the pipeline research stage:

1. Evaluate whether the pipeline research stage exhibits the four strong-run patterns
2. Implement a checkpoint-preservation rule -- preserve the best-performing configuration (skills, prompts, tool definitions) as a checkpoint and do not overwrite it with a later, worse revision (the frozen best pattern from RSEA, now motivated at the data-strategy level)
3. Use the four trajectory patterns as diagnostic criteria for pipeline runs: did it form accurate hypotheses? Was its supervision validation-grounded? Was the data behavior-aligned? Did it preserve strong checkpoints?

Impact: Medium-High | Effort: Low-Medium