[BUG] LLM judge silently scores API error strings: litellm backend sends max_tokens as a list, provider returns 400 #1296

Describe the bug

When a judge is constructed with max_tokens set, the litellm backend sends max_tokens as a one-element list instead of an integer. Any spec-compliant OpenAI-compatible server rejects the request with 400 Bad Request. lighteval then retries 3x, gives up, and returns the string as the judge's response. That string is passed straight into process_judge_response() and scored as if it were a real judgment. No crash, no warning, just corrupted judge scores.

Root cause — src/lighteval/metrics/utils/llm_as_judge.py, in JudgeLM.__call_litellm (line 341 on main, line 331 in the released 0.13.0):

if max_new_tokens is not None:
    kwargs["max_tokens"] = (max_new_tokens,)   # trailing comma -> 1-tuple

Serialized to JSON, that becomes "max_tokens": [64].

Note:

1. This only fires when max_tokens is explicitly set on the judge.
2. Only the litellm backend is affected.

To Reproduce

Against any OpenAI-compatible server. I used vLLM:

vllm serve Qwen/Qwen3.6-27B-FP8 \
    --api-key sk-fake

import os

VLLM = "http://localhost:8000/v1"
os.environ["OPENAI_BASE_URL"] = VLLM
os.environ["OPENAI_API_KEY"] = "sk-fake"

import litellm
from lighteval.metrics.utils.llm_as_judge import JudgeLM

# print what lighteval hands to litellm
real_completion = litellm.completion
def spy(**kwargs):
    print("max_tokens passed to litellm:", repr(kwargs["max_tokens"]))
    return real_completion(**kwargs)
litellm.completion = spy

judge = JudgeLM(
    model="openai/Qwen/Qwen3.6-27B-FP8",
    templates=lambda question, answer, options=None, gold=None, **kw: [
        {"role": "user", "content": question}
    ],
    process_judge_response=lambda r: r,
    judge_backend="litellm",
    max_tokens=64,
    backend_options={"caching": False},
)

_, _, responses = judge.evaluate_answer_batch(
    questions=["Explain step by step why 2 + 2 = 4."],
    answers=["4"], options=[None], golds=["4"],
)
print("judge response:", responses[0])

# control: identical call, max_tokens as the int it was meant to be
ok = real_completion(
    model="openai/Qwen/Qwen3.6-27B-FP8",
    messages=[{"role": "user", "content": "Explain step by step why 2 + 2 = 4."}],
    max_tokens=64,
)
print("control ok:", ok.choices[0].message.content[:40])

Output:

max_tokens passed to litellm: (64,)
judge response: ERROR: Failed to get response from the API.
control ok: ...

vLLM's access log — same server, same model, same prompt; the only difference is the tuple:

POST /v1/chat/completions HTTP/1.1" 400 Bad Request   <- judge, attempt 1
POST /v1/chat/completions HTTP/1.1" 400 Bad Request   <- judge, attempt 2  (3x retry)
POST /v1/chat/completions HTTP/1.1" 400 Bad Request   <- judge, attempt 3
POST /v1/chat/completions HTTP/1.1" 200 OK            <- control (max_tokens=64 as int)

I also captured the raw request body by putting a recording server behind a real litellm proxy. Neither the litellm SDK nor the proxy validates or coerces the value and the malformed value goes to the provider

{
  "messages": [{"role": "user", "content": "..."}],
  "model": "dummy-model",
  "max_tokens": [512],
  "n": 1
}

Expected behavior

max_tokens should be sent as an integer, and the request should succeed:

kwargs["max_tokens"] = max_new_tokens

With that one-line change, the same script returns 200 OK from vLLM and the judge produces a real judgment instead of an error string.

Version info

• OS: Linux
• lighteval: 0.13.0 (from PyPI) — also present on current main (64f4f5ae)
• Python 3.12, litellm 1.92.0
• Backend: litellm

Happy to open a PR with the fix.