Skip to content

lighteval eval crashes on fresh install: GenerateConfig rejects frequence_penalty / log_probs / response_format with current inspect-ai #1327

Description

@sk8ordie84

Summary. A fresh pip install lighteval today resolves inspect-ai to the latest release (the dependency is declared as inspect-ai>=0.3.140 with no upper bound), and lighteval eval then fails at startup for any model/task with a pydantic ValidationError: current inspect_ai.model.GenerateConfig rejects unknown fields.

Repro:

pip install lighteval          # resolves inspect-ai 0.3.255 today
lighteval eval mockllm/model "lighteval|gsm8k|0" --max-samples 4 --log-dir out
ValidationError: 1 validation error for GenerateConfig
  Value error, Unknown GenerateConfig field(s): frequence_penalty, log_probs, response_format.

Root cause (lighteval/main_inspect.py): the config is built with three field names current inspect-ai doesn't accept — frequence_penalty (lines 270/477 — note this one is a typo; inspect-ai spells it frequency_penalty), log_probs (line 483; inspect-ai: logprobs), and response_format (line 489; no longer a GenerateConfig field).

Verified workaround until a fix lands: pip install "inspect-ai==0.3.140" — with that pin the same command runs to completion and writes results.

Suggested fix: rename the two fields, route response_format through extra_body or drop it, and consider an upper bound on inspect-ai so releases can't break installs retroactively. Happy to open a small PR for the renames if useful.

Environment: Python 3.12.13, lighteval 0.13.0, inspect-ai 0.3.255 (broken) / 0.3.140 (works), macOS.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions