Skip to content

fix(lm_eval): don't build continuous metrics with a null bound#206

Open
borgr wants to merge 1 commit into
evaleval:mainfrom
borgr:fix/lm-eval-unbounded-metric-bounds
Open

fix(lm_eval): don't build continuous metrics with a null bound#206
borgr wants to merge 1 commit into
evaleval:mainfrom
borgr:fix/lm-eval-unbounded-metric-bounds

Conversation

@borgr

@borgr borgr commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

The bug

The lm_eval converter crashes on results containing ter — or any metric absent from its 15-entry bounds table.

KNOWN_METRIC_BOUNDS declares 'ter': (0.0, None), and for any metric not in the table KNOWN_METRIC_BOUNDS.get(...) returns None. In adapter.py the bounds were then used to build a continuous MetricConfig unconditionally:

bounds = KNOWN_METRIC_BOUNDS.get(metric_name)
min_score = bounds[0] if bounds else None
max_score = bounds[1] if bounds else None
metric_config = MetricConfig(score_type=ScoreType.continuous, min_score=min_score, max_score=max_score, ...)

continuous requires numeric min_score/max_score (enforced by validate_score_type_requirements), so with a None bound the MetricConfig(...) constructor raises score_type 'continuous' requires max_score — the whole conversion fails. EEE has no unbounded score_type, and float('inf') is not a fix (pydantic serializes it to null, which fails the same validator on read-back, and Infinity is non-standard JSON).

The fix

  • ter(0.0, 100.0): TER (sacrebleu translation edit rate) is reported on a 0–100 scale, like bleu in the same table. It can exceed 100 in pathological cases, but 0–100 is the reported range and EEE does not range-check scores, so a finite nominal bound is correct here.
  • Skip metrics with no finite bound (e.g. perplexity, word_perplexity) with a warning, instead of constructing an invalid continuous config — since EEE cannot represent an unbounded metric.
  • Drop a task whose every metric was skipped, rather than emit a contentless EvaluationLog.
  • Regression test (test_unbounded_metrics_do_not_produce_invalid_records): ter yields a valid 0–100 continuous record; a genuinely-unbounded metric is skipped; a perplexity-only task is dropped.

uv run pytest tests → 207 passed, 20 skipped. ruff check clean.

Note: the deeper gap is that EEE has no score_type for genuinely unbounded metrics (elo, perplexity, dollars) — continuous mandates finite bounds. This PR stops the crash within the current schema; a first-class unbounded representation would be a separate schema discussion.

KNOWN_METRIC_BOUNDS declared ter as (0.0, None), and any metric absent from
the table fell through to min/max = None. The adapter then built a continuous
MetricConfig unconditionally, and the score-type validator raises
"score_type 'continuous' requires max_score" at construction — so converting
any lm-eval results containing ter (or an unlisted metric) crashed the
conversion rather than producing output.

- ter: use a finite 0-100 bound (the sacrebleu reporting scale, matching bleu)
  instead of None.
- Skip a metric with no finite bound (e.g. perplexity) with a warning instead
  of constructing an invalid continuous config; EEE has no unbounded score_type.
- Drop a task whose every metric was skipped rather than emit a contentless log.
- Regression test covering ter + a skipped unbounded metric.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant