Summary. A fresh pip install lighteval today resolves inspect-ai to the latest release (the dependency is declared as inspect-ai>=0.3.140 with no upper bound), and lighteval eval then fails at startup for any model/task with a pydantic ValidationError: current inspect_ai.model.GenerateConfig rejects unknown fields.
Repro:
pip install lighteval # resolves inspect-ai 0.3.255 today
lighteval eval mockllm/model "lighteval|gsm8k|0" --max-samples 4 --log-dir out
ValidationError: 1 validation error for GenerateConfig
Value error, Unknown GenerateConfig field(s): frequence_penalty, log_probs, response_format.
Root cause (lighteval/main_inspect.py): the config is built with three field names current inspect-ai doesn't accept — frequence_penalty (lines 270/477 — note this one is a typo; inspect-ai spells it frequency_penalty), log_probs (line 483; inspect-ai: logprobs), and response_format (line 489; no longer a GenerateConfig field).
Verified workaround until a fix lands: pip install "inspect-ai==0.3.140" — with that pin the same command runs to completion and writes results.
Suggested fix: rename the two fields, route response_format through extra_body or drop it, and consider an upper bound on inspect-ai so releases can't break installs retroactively. Happy to open a small PR for the renames if useful.
Environment: Python 3.12.13, lighteval 0.13.0, inspect-ai 0.3.255 (broken) / 0.3.140 (works), macOS.
Summary. A fresh
pip install lightevaltoday resolvesinspect-aito the latest release (the dependency is declared asinspect-ai>=0.3.140with no upper bound), andlighteval evalthen fails at startup for any model/task with a pydanticValidationError: currentinspect_ai.model.GenerateConfigrejects unknown fields.Repro:
Root cause (
lighteval/main_inspect.py): the config is built with three field names current inspect-ai doesn't accept —frequence_penalty(lines 270/477 — note this one is a typo; inspect-ai spells itfrequency_penalty),log_probs(line 483; inspect-ai:logprobs), andresponse_format(line 489; no longer aGenerateConfigfield).Verified workaround until a fix lands:
pip install "inspect-ai==0.3.140"— with that pin the same command runs to completion and writes results.Suggested fix: rename the two fields, route
response_formatthroughextra_bodyor drop it, and consider an upper bound oninspect-aiso releases can't break installs retroactively. Happy to open a small PR for the renames if useful.Environment: Python 3.12.13, lighteval 0.13.0, inspect-ai 0.3.255 (broken) / 0.3.140 (works), macOS.