fix: represent unbounded metric bounds as the JSON string "Infinity"#207
Open
borgr wants to merge 1 commit into
Open
fix: represent unbounded metric bounds as the JSON string "Infinity"#207borgr wants to merge 1 commit into
borgr wants to merge 1 commit into
Conversation
An unbounded continuous bound (PSNR, perplexity, TER, ...) is float('inf')/-inf,
but pydantic's default serialization wrote it to JSON null -- which then failed
the "continuous requires min/max_score" validator. inf could be read but not
written, so unbounded metrics could only be dropped, faked, or made invalid.
Core fix (every_eval_ever/eval_types.py):
- Field serializer on MetricConfig.min_score/max_score emits inf/-inf as the JSON
string "Infinity"/"-Infinity": valid RFC-8259 JSON (parses in JS/Go/jq/orjson,
not just Python) that pydantic reads back to float('inf').
- Chosen over model config `ser_json_inf_nan` because that only affects
model_dump_json(); the codebase also writes via json.dump(model_dump(mode='json'))
(cli.py, instance writers), which would emit the bare, INVALID Infinity token. The
field serializer (when_used='json') covers BOTH paths; mode='python' keeps the
native float. It also targets exactly the two bound fields -- no blast radius.
- The field stays typed float|None, so "convertible to a number" is enforced by the
type; the schema pins the string form to exactly {"Infinity","-Infinity"}.
Semantics kept strict on purpose:
- null stays INVALID for a continuous metric: missing != unbounded, so null cannot
masquerade as inf; unbounded uses inf, genuinely-missing bounds stay invalid.
- NaN bounds are rejected (never meaningful; NaN != NaN breaks comparisons).
Apply the convention in-repo so nothing is left emitting null-for-unbounded
(converters/lm_eval): 'ter' bound changes from (0.0, None) -> (0.0, inf), and a
metric with no entry in KNOWN_METRIC_BOUNDS is now skipped with a warning rather
than building an invalid null-bounded continuous config. Supersedes evaleval#206.
Regen-safe: post_codegen re-applies the validator + serializer to MetricConfig.
Tests: tests/test_inf_serialization.py and a new lm_eval case.
Follow-ups (out of this repo / parallel, not required for correctness here):
- eval-card-registry (separate repo): reclassify the ~9 canonical metrics whose
max_score:null means "unbounded" (elo, cost-per-task, latency-*, rank, ...) to
inf; leave the genuinely-vague ones (score, *.mean placeholders) null.
- EEE_datastore: re-run/patch affected records in parallel.
borgr
added a commit
to borgr/eval-card-registry
that referenced
this pull request
Jul 21, 2026
EEE now distinguishes inf (unbounded) from null (missing) -- see evaleval/every_eval_ever#207. Reclassify the canonical metrics whose null max_score meant "unbounded": - unbounded above (finite min, open top) -> max_score: .inf: cost-per-task, total-cost, avg-attempts, rank, mean-response-time, p95-latency, mean-latency, latency-stddev - unbounded both ways -> (-.inf, .inf): elo (ratings have no fixed bounds) Left as null on purpose (genuinely undefined/vague, not "unbounded"): score, mean-score, and the *.mean placeholders. .inf/-.inf parse to float('inf')/-inf via yaml.safe_load and store as float64; seed tests pass. No JSON path serializes these bounds (they are float columns), so no inf->null regression.
borgr
added a commit
that referenced
this pull request
Jul 21, 2026
Convert PwC PostgreSQL dumps from the HF bucket nielsr/paperswithcode-backups
into EEE aggregate EvaluationLogs (one log per model; source_type=documentation).
Metric bounds/direction resolve against a vendored snapshot of the
eval-card-registry canonical metrics, with a three-tier policy:
- resolved -> canonical bounds/id; scores rescaled to the canonical scale;
unbounded (null) bounds emitted as +/-inf, serialized as the
JSON string "Infinity"/"-Infinity" (depends on every_eval_ever#207)
- unresolved -> FAIL CLOSED by default (never ship un-vetted bounds); an
--allow-unresolved opt-out emits observed-range bounds
Read with pgdumplib (declared as the 'paperswithcode' extra; pure-python, no
PostgreSQL server). Includes offline fixture tests, a README, and a snapshot
regenerator. Stacked on every_eval_ever#207 for the "Infinity" bound
serialization; rebase onto main once #207 merges.
borgr
added a commit
that referenced
this pull request
Jul 21, 2026
Convert PwC PostgreSQL dumps from the HF bucket nielsr/paperswithcode-backups
into EEE aggregate EvaluationLogs (one log per model; source_type=documentation).
Metric bounds/direction resolve against a vendored snapshot of the
eval-card-registry canonical metrics, with a three-tier policy:
- resolved -> canonical bounds/id; scores rescaled to the canonical scale;
unbounded (null) bounds emitted as +/-inf, serialized as the
JSON string "Infinity"/"-Infinity" (depends on every_eval_ever#207)
- unresolved -> FAIL CLOSED by default (never ship un-vetted bounds); an
--allow-unresolved opt-out emits observed-range bounds
Read with pgdumplib (declared as the 'paperswithcode' extra; pure-python, no
PostgreSQL server). Includes offline fixture tests, a README, and a snapshot
regenerator. Stacked on every_eval_ever#207 for the "Infinity" bound
serialization; rebase onto main once #207 merges.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
An unbounded continuous bound (PSNR, perplexity, TER, …) is
float('inf')/-inf, but pydantic's default serialization wrote it to JSONnull— which then failed the existingcontinuous requires min/max_scorevalidator. Soinfcould be read but not written: unbounded metrics could only be dropped, faked, or made invalid.Fix
Core — a field serializer on
MetricConfig.min_score/max_scoreemitsinf/-infas the JSON string"Infinity"/"-Infinity":float('inf');ser_json_inf_nanbecause that only affectsmodel_dump_json()— the codebase also writes viajson.dump(model_dump(mode='json'))(cli.py, instance writers), which would emit the bare, invalidInfinitytoken.field_serializer(when_used='json')covers both paths;mode='python'keeps the native float;float | None, so "convertible to a number" is enforced by the type; the schema pins the string form to exactly{"Infinity","-Infinity"}.Semantics (kept strict on purpose)
nullstays invalid for a continuous metric: missing ≠ unbounded, sonullcannot masquerade asinf; unbounded usesinf, genuinely-missing bounds stay invalid.NaNbounds are rejected (never meaningful;NaN != NaNbreaks comparisons).Convention applied in-repo (
converters/lm_eval) so nothing is left emitting null-for-unbounded:terbound(0.0, None)→(0.0, inf), and a metric with no entry inKNOWN_METRIC_BOUNDSis now skipped with a warning instead of building an invalid null-bounded continuous config. Supersedes #206.Regen-safe:
post_codegen.pyre-applies the validator + serializer toMetricConfig.Tests
tests/test_inf_serialization.py(both dump paths incl. themode='json'regression,mode='python'float, strict-JSON validity, float typing/round-trip, null-still-invalid, NaN-rejected) + a newlm_evalcase. Full suite green (207 passed / 20 skipped), ruff clean.Portability
"Infinity"/"-Infinity"is valid JSON everywhere. A statically-typed consumer (Gofloat64) needs a small custom unmarshaller for thenumber | "Infinity"union — the schemaenumdocuments exactly what to expect.Follow-ups (separate — not required for correctness here)
eval-card-registry(separate repo): reclassify the ~9 canonical metrics whosemax_score: nullmeans unbounded (elo,cost-per-task,latency-*,rank, …) toinf; leave the genuinely-vague ones (score,*.meanplaceholders)null.EEE_datastore: re-run / patch affected records in parallel.