Add Papers with Code adapter (utils/paperswithcode)#209
Open
borgr wants to merge 2 commits into
Open
Conversation
An unbounded continuous bound (PSNR, perplexity, TER, ...) is float('inf')/-inf,
but pydantic's default serialization wrote it to JSON null -- which then failed
the "continuous requires min/max_score" validator. inf could be read but not
written, so unbounded metrics could only be dropped, faked, or made invalid.
Core fix (every_eval_ever/eval_types.py):
- Field serializer on MetricConfig.min_score/max_score emits inf/-inf as the JSON
string "Infinity"/"-Infinity": valid RFC-8259 JSON (parses in JS/Go/jq/orjson,
not just Python) that pydantic reads back to float('inf').
- Chosen over model config `ser_json_inf_nan` because that only affects
model_dump_json(); the codebase also writes via json.dump(model_dump(mode='json'))
(cli.py, instance writers), which would emit the bare, INVALID Infinity token. The
field serializer (when_used='json') covers BOTH paths; mode='python' keeps the
native float. It also targets exactly the two bound fields -- no blast radius.
- The field stays typed float|None, so "convertible to a number" is enforced by the
type; the schema pins the string form to exactly {"Infinity","-Infinity"}.
Semantics kept strict on purpose:
- null stays INVALID for a continuous metric: missing != unbounded, so null cannot
masquerade as inf; unbounded uses inf, genuinely-missing bounds stay invalid.
- NaN bounds are rejected (never meaningful; NaN != NaN breaks comparisons).
Apply the convention in-repo so nothing is left emitting null-for-unbounded
(converters/lm_eval): 'ter' bound changes from (0.0, None) -> (0.0, inf), and a
metric with no entry in KNOWN_METRIC_BOUNDS is now skipped with a warning rather
than building an invalid null-bounded continuous config. Supersedes #206.
Regen-safe: post_codegen re-applies the validator + serializer to MetricConfig.
Tests: tests/test_inf_serialization.py and a new lm_eval case.
Follow-ups (out of this repo / parallel, not required for correctness here):
- eval-card-registry (separate repo): reclassify the ~9 canonical metrics whose
max_score:null means "unbounded" (elo, cost-per-task, latency-*, rank, ...) to
inf; leave the genuinely-vague ones (score, *.mean placeholders) null.
- EEE_datastore: re-run/patch affected records in parallel.
Convert PwC PostgreSQL dumps from the HF bucket nielsr/paperswithcode-backups
into EEE aggregate EvaluationLogs (one log per model; source_type=documentation).
Metric bounds/direction resolve against a vendored snapshot of the
eval-card-registry canonical metrics, with a three-tier policy:
- resolved -> canonical bounds/id; scores rescaled to the canonical scale;
unbounded (null) bounds emitted as +/-inf, serialized as the
JSON string "Infinity"/"-Infinity" (depends on every_eval_ever#207)
- unresolved -> FAIL CLOSED by default (never ship un-vetted bounds); an
--allow-unresolved opt-out emits observed-range bounds
Read with pgdumplib (declared as the 'paperswithcode' extra; pure-python, no
PostgreSQL server). Includes offline fixture tests, a README, and a snapshot
regenerator. Stacked on every_eval_ever#207 for the "Infinity" bound
serialization; rebase onto main once #207 merges.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Converts Papers with Code results — nightly PostgreSQL dumps in the HF bucket
nielsr/paperswithcode-backups— into EEE aggregateEvaluationLogs (one log per model;source_type=documentation). Read withpgdumplib(pure-python; no PostgreSQL server).This branch is stacked on #207 (unbounded bounds →
"Infinity"serialization). The first commit here (18452751) is #207's — review only the adapter commit. Once #207 merges I'll rebase ontomainand that commit drops out. If #207 changes, the unbounded-emit path is a single resolver choke-point to adjust.Design
--allow-unresolvedopt-out emits observed-range bounds (flagged).±inf→"Infinity"(per fix: represent unbounded metric bounds as the JSON string "Infinity" #207).Verification
--allrun: 3,010 logs / 12,043 results, sample validates, 0 scores out of bounds.Related / follow-ups
Needs the
paperswithcodeextra (pip install pgdumplib).