Skip to content

[evals] Add diff and patch PPL eval sets #5095

@dlwh

Description

@dlwh

🤖 Part of #5005.

Description

Add PPL/gap eval sets for diffs, patches, and repo history. Raw source files do not cover the syntax and distribution agents produce when editing code.

Initial sources:

Break out slices for raw git diff, commit-message-plus-diff, PR-review-plus-diff, and issue-to-patch contexts.

Definition of Done

  • Add raw eval builders for at least two diff/patch sources.
  • Mask provenance fields that should not be predicted.
  • Produce separate metrics for patch text versus surrounding issue/review text.
  • Add docs for train/eval leakage checks.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions