[BUG] OPD rollout aborts in _compute_zero_std_metrics: TypeError: type dict doesn't define __round__ method #3265

Summary

A pure on-policy distillation run (--use-opd --opd-type sglang) aborts inside the rollout,
before the first optimizer step:

File "miles/ray/rollout/metrics.py", line 250, in _compute_zero_std_metrics
    interesting_rewards = [str(round(g[0].get_reward_value(args), 1)) for g in interesting_sample_groups]
TypeError: type dict doesn't define __round__ method

Full traceback (rollout log, job exited non-zero):

Traceback (most recent call last):
  File ".../miles/ray/rollout/rollout_executor.py", line 126, in _get_rollout_data
    log_rollout_data(rollout_id, self.args, data, metrics, time.time() - start_time)
  File ".../miles/ray/rollout/metrics.py", line 68, in log_rollout_data
    log_dict |= dict_add_prefix(_compute_metrics_from_samples(args, samples), "rollout/")
  File ".../miles/ray/rollout/metrics.py", line 250, in _compute_zero_std_metrics
    interesting_rewards = [str(round(g[0].get_reward_value(args), 1)) for g in interesting_sample_groups]
TypeError: type dict doesn't define __round__ method

Reproduction

Command (abridged): Qwen3.5-0.8B student, an external SGLang Qwen3.5-27B teacher, 2 prompts/step,
greedy decoding, 64 response tokens, --advantage-estimator grpo, --train-backend fsdp:

python train.py \
  --rollout-batch-size 2 --n-samples-per-prompt 1 --global-batch-size 2 \
  --use-opd --opd-type sglang --opd-kl-coef 1.0 --opd-log-prob-top-k 0 \
  --custom-rm-path miles.rollout.on_policy_distillation.reward_func \
  --custom-reward-post-process-path miles.rollout.on_policy_distillation.post_process_rewards \
  --rm-url http://127.0.0.1:31000/generate

Minimal repro that does not need GPUs, just the metrics function:

from types import SimpleNamespace

from miles.ray.rollout.metrics import _compute_zero_std_metrics


class FakeSample:
    group_index = 0

    def get_reward_value(self, args):
        # what miles.rollout.on_policy_distillation.reward_func leaves in Sample.reward
        return {"text": "42", "output_ids": [1, 2, 3]}


args = SimpleNamespace(advantage_estimator="grpo", reward_key=None)
print(_compute_zero_std_metrics(args, [FakeSample()]))
# TypeError: type dict doesn't define __round__ method

Root cause

reward_func returns the teacher's raw response dict, which is stored in Sample.reward.
rollout_executor._get_rollout_data calls log_rollout_data(...) before
convert_samples_to_train_data(..., custom_reward_post_process_func=...), and it is the latter
that turns the response into the scalar used for advantages. So at the time
_compute_zero_std_metrics runs, get_reward_value() legitimately returns a dict.

_is_zero_std only compares rewards for equality, which works on dicts, so every group looks
"interesting" and the whole batch flows into the round() comprehension.

The module already accepts non-scalar rewards one function earlier — _compute_training_sample_metrics
guards with if isinstance(raw_reward, Number) — so the same guard is missing here.

Suggested fix

Bucket non-numeric rewards by type name instead of calling round() on them, keeping the
zero_std/count_* counters complete and leaving the all_zero/all_one percentages untouched.
Patch: #3266

Related (not fixed here)

_compute_passrate_from_samples (same file, line ~309) has the same precondition but fails
silently instead of crashing: with --log-passrate, n_samples_per_prompt > 1 and non-scalar
rewards, np.array(flat_rewards) becomes an object array and rewards_of_group == 1 is
elementwise False, so passrate/pass@k is reported as 0.0 rather than being skipped.

Environment

• Miles 327eefa0fbb8bb0a391dcac3ec5f5813d85ef935; still reproducible at ab48452ea (current main)
• SGLang 0.5.11.dev6342+ga8e5c632f
• AMD MI325X (gfx942), ROCm 7.0, --train-backend fsdp