(eval) Support simple evaluation

[Randomizez]

• support AIME25, GPQADiamond, HMMT25, IFBench, MMLUPro
• add evaluation results of step-3.5-flash-sft
• bump version of torch to 2.10; vllm to 0.17.1

[brian14708]

Review Summary

Overall this is a well-structured eval framework addition. A few items to address before merge:

Breaking Changes (please confirm scope)

1. VLLMDeployConfig.get_sampling_params() removed along with temperature, top_p, top_k, eos_ids, stop_strings, vllm_sampling_params fields. Any downstream code calling these will break. Please grep internal repos to confirm no other consumers.

2. vllm_client.completion() return type changed from CompletionResponse to dict[str, Any]. Same concern.

Code Issues

3. HMMT25/benchmark.py: Remove the redundant pass statement.

4. IFBenchBenchmark._load_records() duplicates the parent JsonlChatBenchmark shuffle/downsample logic (same seed, same slicing). If the parent ever changes strategy, these will silently diverge. Consider calling super()._load_records() or extracting the shared logic.

5. SimpleChatGeneratable context budget silent truncation: When remaining_context < sampling_params.max_tokens, resolved_max_tokens is silently clamped. For eval correctness this should at least emit a logger.warning so users know their generation budget was reduced.

6. _is_correct signature inconsistency across AIME25/GPQA/HMMT: They take (result, answer) but _build_metric.is_success_fn expects Callable[[Generated], bool]. The lambda adapter works but is redundant — consider having _is_correct pull gold answer from result.case.benchmark.context internally.

Infrastructure Concerns

7. vllm_router: TCPConnector(limit=0, limit_per_host=0) + all timeouts disabled. Fine for eval, but if the router is shared with non-eval workloads, unlimited connections risk fd exhaustion. Please confirm scope or add a config knob.

8. pyproject.toml: transformers upper bound removed entirely. Major version bumps could break tokenizer APIs — consider keeping <6.0 or similar.

Nits

• GenerationController: If a callback calls submit_with_callback, it will deadlock on _state_lock since _callback_loop already holds it. Worth documenting this constraint.
• Worker exception wrapping (RuntimeError(f"{type(e).__name__}: {e}")) loses the original traceback — consider including traceback.format_exc() in the message.

[Randomizez]

make sense, transformers has to be < 5.0 for correct apply_chat_template()

[brian14708]

LGTM