Skip to content

test: cover HumanEval evaluator details - #2622

Open
motodriver wants to merge 1 commit into
open-compass:mainfrom
motodriver:test/humaneval-evaluator-details
Open

test: cover HumanEval evaluator details#2622
motodriver wants to merge 1 commit into
open-compass:mainfrom
motodriver:test/humaneval-evaluator-details

Conversation

@motodriver

Copy link
Copy Markdown

Motivation

HumanEval and HumanEvalPlus expose per-sample details, but the evaluator output contract was not covered by regression tests. This is related to #723.

Changes

  • mock HumanEval result files and verify per-sample is_correct, prompts, and pass@k scores
  • mock EvalPlus results and verify base/plus statuses, composed solutions, and pass@1
  • avoid executing generated code or downloading models in tests

Verification

  • focused isolated unittest run: 8 passed
  • python3 -m py_compile tests/datasets/test_humaneval.py
  • git diff --check

The full local import path requires optional OpenCompass runtime dependencies that are not installed in this environment; CI will run the test in the project environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants