Skip to content

Reconcile model-registry metrics with published leaderboards #3

Description

@unes07

Problem

Metrics recorded in the model registry differ substantially from values in the published leaderboards, making model selection and reported performance difficult to audit.

Proposed work

  • Trace each registry metric and leaderboard row to its exact dataset, split, preprocessing, target, and artifact.
  • Identify whether differences come from stale reports, different protocols, leakage, metric aggregation, or artifact mismatch.
  • Establish one authoritative evaluation output and documented selection policy.
  • Regenerate inconsistent reports, registry entries, and artifacts as needed.

Acceptance criteria

  • Every published model metric is reproducible from the training pipeline.
  • Registry and leaderboard values agree for the same evaluation protocol.
  • Selection criteria are documented per crop and target.
  • Zero-valued targets use appropriate metrics; unstable MAPE values are not presented without context.
  • A regression test prevents report, registry, and artifact metadata from diverging.

Depends on #2.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions