Skip to content

Add an example script to run lighteval based evaluation#83

Merged
dongwang218 merged 1 commit intomainfrom
dong_lighteval
Sep 4, 2025
Merged

Add an example script to run lighteval based evaluation#83
dongwang218 merged 1 commit intomainfrom
dong_lighteval

Conversation

@dongwang218
Copy link
Contributor

Why ?

add an example to showcase integration with lighteval

How ?

via litellm and openai comptabile api

Test plan

python scripts/light_eval_example.py  --cleanup True --max_samples 10
██| 1/1 [00:00<00:00, 521.87ba/s]
|        Task        |Version|    Metric     |Value|   |Stderr|
|--------------------|-------|---------------|----:|---|-----:|
|all                 |       |pass@k_with_k&n|  0.4|±  |0.1633|
|lighteval:math_500:0|       |pass@k_with_k&n|  0.4|±  |0.1633|

Running command: scancel 10940707 10940709
cluster lighteval_test shutdown

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 3, 2025
@dongwang218 dongwang218 merged commit 5b32c2e into main Sep 4, 2025
8 checks passed
@dongwang218 dongwang218 deleted the dong_lighteval branch September 4, 2025 02:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants