Per-language and per-subject evaluation of recent LLMs on MMLU-ProX-Lite (588 parallel questions × 14 subjects), in 10 of its 29 languages, run with lm-evaluation-harness.
This is not a leaderboard. The core comparisons are across languages and across subjects within each model:
- Language gap: how much each model's accuracy drops in each language relative to its own English accuracy, measured on the same (parallel) questions.
- Subject profile: which subjects each model is weak in, overall and per language.
- Exploratory: language × subject combinations, and whether a low score comes from wrong answers or from not following the answer format in that language.
Models are deliberately not ranked against each other: reasoning effort differs by model (the developer's default for most of them), so differences between models mix model quality with reasoning budget.
Language gap vs. English (percentage points). Each model against its own English score on the same questions. * marks a 95% paired-bootstrap confidence interval that excludes zero; the intervals are in results/summary.md.
| Model | zh | ja | ko | pt | de | id | ru | ar | sw |
|---|---|---|---|---|---|---|---|---|---|
| Claude Haiku 5.5 | -1.4 | -1.0 | -0.5 | -0.7 | -0.3 | -0.2 | -0.2 | -2.2 | -5.4* |
| GPT-6 Luna | -2.6* | -1.7 | -1.9 | -0.7 | -1.5 | -2.0 | -2.4* | -1.9 | -13.6* |
| Gemini 3.8 Flash | -1.4 | -1.5 | -2.7* | -1.5* | -1.4 | -1.4 | -0.9 | -0.7 | -2.7* |
| Claude Sonnet 5.5 † | -4.1* | -3.6* | -3.6* | -0.5 | -2.4* | -1.4 | -1.7 | -3.1* | -5.3* |
| GPT-6.1 Sol † | -2.4* | -2.6* | -2.7* | -1.9* | -1.7* | -1.4 | -0.9 | -1.4 | -14.6* |
| DeepSeek V4.1 Flash | -3.1* | -2.4 | -3.7* | +0.2 | -2.2 | -2.7* | +0.7 | -2.2 | -3.6* |
| GLM-5.3 † | -2.6* | -1.9 | -0.7 | -0.5 | -1.5 | +0.3 | -2.2* | -1.2 | -5.1* |
| GLM-5.3-Flash | -3.4* | -3.7* | -2.6* | -1.0 | -2.6* | -1.5 | -1.7 | -3.6* | -7.0* |
| Qwen 3.8 27B | -3.1* | -3.4* | -2.7* | -3.6* | -1.9 | -2.2 | -2.4* | -2.7* | -8.5* |
| Gemma 4 31B | -2.7* | -3.1* | -2.2 | -2.0 | -2.7* | -1.2 | -0.9 | -2.4* | -6.3* |
Strict accuracy (%) by language, the scores the gaps above are computed from.
| Model | en | zh | ja | ko | pt | de | id | ru | ar | sw |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Haiku 5.5 | 83.8 | 82.5 | 82.8 | 83.3 | 83.2 | 83.5 | 83.7 | 83.7 | 81.6 | 78.4 |
| GPT-6 Luna | 85.0 | 82.5 | 83.3 | 83.2 | 84.4 | 83.5 | 83.0 | 82.7 | 83.2 | 71.4 |
| Gemini 3.8 Flash | 90.5 | 89.1 | 88.9 | 87.8 | 88.9 | 89.1 | 89.1 | 89.6 | 89.8 | 87.8 |
| Claude Sonnet 5.5 † | 89.3 | 85.2 | 85.7 | 85.7 | 88.8 | 86.9 | 87.9 | 87.6 | 86.2 | 84.0 |
| GPT-6.1 Sol † | 90.1 | 87.8 | 87.6 | 87.4 | 88.3 | 88.4 | 88.8 | 89.3 | 88.8 | 75.5 |
| DeepSeek V4.1 Flash | 86.9 | 83.8 | 84.5 | 83.2 | 87.1 | 84.7 | 84.2 | 87.6 | 84.7 | 83.3 |
| GLM-5.3 † | 86.1 | 83.5 | 84.2 | 85.4 | 85.5 | 84.5 | 86.4 | 83.8 | 84.9 | 81.0 |
| GLM-5.3-Flash | 86.4 | 83.0 | 82.7 | 83.8 | 85.4 | 83.8 | 84.9 | 84.7 | 82.8 | 79.4 |
| Qwen 3.8 27B | 85.4 | 82.3 | 82.0 | 82.7 | 81.8 | 83.5 | 83.2 | 83.0 | 82.7 | 76.9 |
| Gemma 4 31B | 86.2 | 83.5 | 83.2 | 84.0 | 84.2 | 83.5 | 85.0 | 85.4 | 83.8 | 79.9 |
Strict accuracy (%) by subject, pooled over the 10 languages. The last row is the number of questions per subject in each language; small subjects are noisy.
| Model | biology | business | chemistry | computer science | economics | engineering | health | history | law | math | other | philosophy | physics | psychology |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Haiku 5.5 | 90.0 | 93.0 | 86.1 | 73.0 | 90.2 | 76.9 | 82.9 | 57.9 | 54.2 | 96.2 | 74.8 | 82.8 | 88.9 | 86.2 |
| GPT-6 Luna | 87.2 | 95.5 | 87.0 | 74.5 | 87.4 | 70.4 | 86.3 | 67.9 | 58.8 | 83.7 | 76.3 | 92.4 | 89.1 | 88.5 |
| Gemini 3.8 Flash | 88.1 | 96.8 | 92.1 | 79.5 | 94.8 | 90.4 | 85.7 | 75.3 | 77.3 | 95.0 | 80.2 | 91.2 | 93.2 | 90.8 |
| Claude Sonnet 5.5 † | 88.6 | 94.2 | 90.4 | 80.5 | 95.0 | 84.8 | 83.1 | 72.6 | 67.1 | 94.3 | 78.9 | 92.4 | 90.8 | 88.8 |
| GPT-6.1 Sol † | 86.4 | 95.8 | 92.3 | 80.5 | 92.4 | 80.2 | 87.4 | 81.6 | 75.8 | 86.2 | 84.8 | 91.6 | 90.8 | 90.5 |
| DeepSeek V4.1 Flash | 90.0 | 96.2 | 88.0 | 77.5 | 90.2 | 82.3 | 86.3 | 66.8 | 57.7 | 92.8 | 79.6 | 89.2 | 91.2 | 87.0 |
| GLM-5.3 † | 88.3 | 94.5 | 88.4 | 73.0 | 90.7 | 80.6 | 83.4 | 73.2 | 58.5 | 93.4 | 77.2 | 89.2 | 91.4 | 86.5 |
| GLM-5.3-Flash | 87.8 | 94.5 | 89.6 | 70.0 | 92.9 | 76.9 | 83.4 | 69.5 | 53.3 | 95.4 | 73.7 | 89.6 | 90.9 | 85.8 |
| Qwen 3.8 27B | 88.6 | 94.8 | 88.2 | 67.5 | 90.7 | 79.2 | 85.7 | 65.3 | 53.8 | 92.8 | 67.8 | 80.4 | 90.5 | 84.8 |
| Gemma 4 31B | 85.3 | 93.5 | 90.7 | 73.0 | 90.7 | 78.8 | 81.4 | 72.6 | 54.6 | 94.1 | 73.9 | 86.8 | 92.0 | 89.5 |
| Questions per language | 36 | 40 | 56 | 20 | 42 | 48 | 35 | 19 | 48 | 68 | 46 | 25 | 65 | 40 |
Language × subject cells that differ from English (exploratory). A cell is listed when its gap from the same subject in English survives Benjamini–Hochberg correction (q < 0.05) across that model's 126 cells, using exact McNemar tests. Gaps are in percentage points; every cell is in results/summary/language_subject_gap.csv.
| Model | Cells (language · subject, gap) |
|---|---|
| Claude Haiku 5.5 | none |
| GPT-6 Luna | sw · math -94.1 |
| Gemini 3.8 Flash | none |
| Claude Sonnet 5.5 † | none |
| GPT-6.1 Sol † | sw · math -95.6 |
| DeepSeek V4.1 Flash | none |
| GLM-5.3 † | none |
| GLM-5.3-Flash | none |
| Qwen 3.8 27B | none |
| Gemma 4 31B | none |
Lenient re-scoring: percentage points gained over strict (method under Methodology). Large gains would mean answers given in a format the strict regex misses.
| Model | en | zh | ja | ko | pt | de | id | ru | ar | sw |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Haiku 5.5 | +1.4 | +0.2 | +0.2 | +0.0 | +0.0 | +0.0 | +0.0 | +0.0 | +0.2 | +0.0 |
| GPT-6 Luna | +0.5 | +0.5 | +1.0 | +1.0 | +0.2 | +0.0 | +0.9 | +1.4 | +0.7 | +0.3 |
| Gemini 3.8 Flash | +0.0 | +0.0 | +0.0 | +0.2 | +0.0 | +0.0 | +0.0 | +0.0 | +0.0 | +0.0 |
| Claude Sonnet 5.5 † | +0.2 | +0.2 | +0.0 | +0.7 | +0.0 | +0.0 | +0.0 | +0.0 | +0.2 | +0.0 |
| GPT-6.1 Sol † | +0.0 | +0.0 | +0.0 | +0.2 | +0.0 | +0.0 | +0.0 | +0.0 | +0.0 | +0.0 |
| DeepSeek V4.1 Flash | +0.2 | +0.0 | +0.0 | +0.3 | +0.0 | +0.2 | +1.0 | +0.0 | +0.0 | +0.0 |
| GLM-5.3 † | +0.2 | +0.0 | +0.0 | +0.0 | +0.0 | +0.0 | +0.0 | +1.0 | +0.0 | +0.0 |
| GLM-5.3-Flash | +0.0 | +1.2 | +0.2 | +0.7 | +0.0 | +0.0 | +0.2 | +0.0 | +0.2 | +0.2 |
| Qwen 3.8 27B | +0.0 | +0.0 | +0.0 | +0.0 | +0.0 | +0.0 | +0.2 | +0.0 | +0.0 | +0.0 |
| Gemma 4 31B | +0.0 | +0.0 | +0.0 | +0.0 | +0.0 | +0.0 | +0.0 | +0.0 | +0.2 | +0.2 |
What the lenient re-scoring shows. The Decisions API graded all 8,980 strict-wrong answers of the 10 models. In 93% of them it read the same letter the strict regex had extracted, so they are genuinely wrong answers. In 5% the response gave no final answer; 298 of these are empty responses (below). Only 93 answers turned out to be correct, 70 of them answers the strict regex could not extract at all, and no model gains more than 1.4 points in any language. Format-following failures therefore explain almost none of the language gaps above.
Empty responses. For GPT-6 Luna and GPT-6.1 Sol, Azure's content filter blocked all 68
Swahili math questions (finish_reason: content_filter, no output), and they count as wrong. This
block is most of their large Swahili gaps, and it is the −94.1 and −95.6 point Swahili math cells in
the language × subject table. Without math, their Swahili gaps are −3.1 and −4.0 points, close to
their other languages. The other 162 empty responses are spread across languages and subjects:
GLM-5.3 93, GLM-5.3-Flash 55, DeepSeek V4.1 Flash 5 and Qwen 3.8 27B 2, all but two of which ran into
the 64,000-token cap while still reasoning (finish_reason: length), and Gemini 3.8 Flash 7,
stopped by its safety filter (SAFETY).
Confidence intervals, answer-extraction failures, 10-language averages, lenient accuracies and
per-language subject accuracies are in results/summary.md; every lenient
grading decision is in results/lenient/. Raw per-question outputs (prompts,
full responses, extracted answers) and per-question token usage (config token-usage) are on
Hugging Face.
How many tokens each model spent per question, by language, recovered after the run from the providers' records (how). Counts are in each model's own tokenizer, so compare languages within a model, not counts across models.
Output tokens (reasoning + answer) relative to English. en: median tokens per English question. Other columns: total tokens in that language ÷ total on the same questions in English, so 1.30 means 30% more. Matched questions only.
| Model | en (tokens) | zh × | ja × | ko × | pt × | de × | id × | ru × | ar × | sw × |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Haiku 5.5 | 459 | 1.35 | 1.50 | 1.58 | 1.44 | 1.74 | 1.59 | 1.44 | 1.81 | 1.68 |
| GPT-6 Luna | 186 | 1.02 | 1.39 | 1.12 | 1.09 | 1.06 | 1.10 | 1.10 | 1.10 | 1.21 |
| Gemini 3.8 Flash | 474 | 1.23 | 1.27 | 1.18 | 1.16 | 1.22 | 1.22 | 1.16 | 1.27 | 1.32 |
| Claude Sonnet 5.5 † | 196 | 1.19 | 1.36 | 1.48 | 1.25 | 1.45 | 1.37 | 1.27 | 1.53 | 1.62 |
| GPT-6.1 Sol † | 98 | 1.26 | 1.64 | 1.32 | 1.18 | 1.23 | 1.24 | 1.23 | 1.14 | 1.27 |
| DeepSeek V4.1 Flash | 359 | 1.32 | 1.55 | 1.55 | 1.26 | 1.31 | 1.41 | 1.28 | 1.29 | 1.73 |
| GLM-5.3 † | 544 | 1.31 | 1.24 | 1.09 | 1.22 | 1.39 | 1.32 | 1.01 | 1.17 | 1.49 |
| GLM-5.3-Flash | 448 | 1.20 | 1.38 | 1.18 | 0.90 | 1.14 | 1.18 | 0.96 | 1.13 | 1.18 |
| Qwen 3.8 27B | 686 | 1.09 | 1.62 | 1.65 | 1.30 | 1.18 | 1.14 | 1.37 | 1.30 | 1.84 |
| Gemma 4 31B | 1,376 | 1.02 | 0.95 | 0.91 | 0.90 | 0.93 | 0.87 | 0.96 | 0.88 | 0.97 |
Reasoning tokens relative to English, same layout. Reported by LLM Gateway; estimated for the Neuralwatt models. Claude Sonnet 5.5 and GPT-6.1 Sol answer most questions without reasoning (English median 0), so their ratios rest on the questions where they did reason.
| Model | en (tokens) | zh × | ja × | ko × | pt × | de × | id × | ru × | ar × | sw × |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Haiku 5.5 | 64 | 1.05 | 1.08 | 1.05 | 1.12 | 1.24 | 1.10 | 1.11 | 1.19 | 1.18 |
| GPT-6 Luna | 106 | 1.01 | 1.25 | 1.09 | 1.06 | 1.02 | 1.07 | 1.09 | 1.08 | 1.21 |
| Gemini 3.8 Flash | 309 | 1.26 | 1.26 | 1.16 | 1.15 | 1.24 | 1.23 | 1.14 | 1.27 | 1.31 |
| Claude Sonnet 5.5 † | 0 | 1.76 | 2.02 | 2.01 | 1.71 | 1.86 | 1.62 | 1.65 | 2.34 | 2.15 |
| GPT-6.1 Sol † | 0 | 1.13 | 1.19 | 1.11 | 1.10 | 1.09 | 1.07 | 1.12 | 1.05 | 1.25 |
| DeepSeek V4.1 Flash | 240 | 1.34 | 1.55 | 1.58 | 1.28 | 1.31 | 1.43 | 1.29 | 1.31 | 1.79 |
| GLM-5.3 † | 351 | 1.31 | 1.21 | 1.06 | 1.21 | 1.39 | 1.30 | 0.99 | 1.15 | 1.48 |
| GLM-5.3-Flash | 267 | 1.21 | 1.35 | 1.14 | 0.87 | 1.12 | 1.16 | 0.94 | 1.11 | 1.16 |
| Qwen 3.8 27B | 459 | 1.09 | 1.63 | 1.69 | 1.31 | 1.17 | 1.13 | 1.38 | 1.30 | 1.84 |
| Gemma 4 31B | 920 | 1.00 | 0.93 | 0.88 | 0.87 | 0.90 | 0.83 | 0.92 | 0.84 | 0.93 |
- Most models spend more tokens on the same questions outside English, up to about 1.8× in Swahili; Gemma 4 31B is the main exception. Part of this is tokenization (some scripts take more tokens) and the language a model reasons in, so the ratios are not a measure of effort alone.
- Claude Sonnet 5.5 reasoned on 14% of questions and GPT-6.1 Sol on 44%, at their default efforts; every other model reasoned on nearly all.
- Every model spent more reasoning tokens, on average, on questions it got wrong.
Per-language medians, right/wrong splits and token-cap counts are in
results/recovered-usage/summary.md.
lm-eval keeps only each answer's text. Neuralwatt's usage API and LLM Gateway's request logs (read through a dashboard login) still held each request's token counts, finish reason, duration and cost, but no prompt or answer text. Each record was tied to its question by two facts: lm-eval sends a model's requests in a fixed order (language, subject, question), and the record's prompt token count must equal the question's. That count is exact for the Neuralwatt models (their own tokenizers and chat templates) and the GPT models (o200k + 10). For Claude and Gemini it is known only for the questions of a short re-run, which anchor an alignment by order. The re-run (2026-10-13) asked every model a random 1/10 of the questions with the main run's settings and logged each API's usage report; it was stopped partway, is not a benchmark, and only its prompt token counts and request IDs are kept.
- Neuralwatt models: 5,877–5,880 of 5,880 questions matched; 1,305 of 1,307 re-run requests, whose question is known from their request ID, matched the right one. Neuralwatt does not report reasoning tokens; they are estimated as output − answer − a fixed overhead, within ±1 token on re-run requests.
- GPT-6 Luna, GPT-6.1 Sol: every question matched except the 68 blocked Swahili math questions; reasoning tokens as reported.
- Claude Haiku 5.5, Claude Sonnet 5.5, Gemini 3.8 Flash: 5,772–5,825 matched; 88–94% of anchors held out of the alignment tie to the right question, and the misses are swaps between neighbouring questions, so per-language figures hold. Gemini's anchors stop before Russian, Arabic and Swahili.
Per-question rows and the providers' records are in
results/recovered-usage/.
Full settings with sources are in configs/models/.
† Larger mid-tier or flagship models included as references; the others are lightweight models.
Closed models, through LLM Gateway (DevPass plan), which picks the provider for each request; it cannot be pinned on this plan. Served by: the provider of the answers kept, from the gateway's logs.
| Model | Reasoning effort | API model ID | Served by | Max tokens |
|---|---|---|---|---|
| Claude Haiku 5.5 | medium |
claude-haiku-5-5 |
AWS Bedrock (99%), Anthropic | 64,000 |
| GPT-6 Luna | medium |
gpt-6-luna |
Azure (99%), OpenAI | 64,000 |
| Gemini 3.8 Flash | medium |
gemini-3.8-flash |
Vertex AI | 65,536 |
| Claude Sonnet 5.5 † | high |
claude-sonnet-5-5 |
AWS Bedrock (99%), Anthropic | 64,000 |
| GPT-6.1 Sol † | medium |
gpt-6.1-sol |
Azure | 64,000 |
Open-weight models, served by Neuralwatt on its own GPUs. Neuralwatt publishes the exact Hugging Face checkpoint it serves, including quantization.
| Model | Reasoning effort | Neuralwatt model ID | Hugging Face checkpoint | Max tokens |
|---|---|---|---|---|
| DeepSeek V4.1 Flash | high |
deepseek-v4.1-flash-flex |
deepseek-ai/DeepSeek-V4.1-Flash | 64,000 |
| GLM-5.3 † | max |
glm-5.3-flex, then glm-5.3 |
zai-org/GLM-5.3 (FP8-native) | 64,000 |
| GLM-5.3-Flash | max |
glm-5.3-flash-flex |
zai-org/GLM-5.3-Flash | 64,000 |
| Qwen 3.8 27B | xhigh |
qwen-3.8-27b-flex, then qwen-3.8-27b |
Qwen/Qwen3.8-27B-FP8 (FP8) | 64,000 |
| Gemma 4 31B | max (thinking on) |
gemma-4-31b |
nvidia/Gemma-4-31B-IT-NVFP4 (4-bit NVFP4) | 16,384 |
IDs ending in -flex use Neuralwatt's Flex tier: the same checkpoint at a lower price, with
requests that may queue. Gemma 4 31B has no Flex tier and ran on the standard tier. GLM-5.3 and
Qwen 3.8 27B moved from Flex to the standard tier partway through their runs (see
Changes during the study). Quantization can hurt non-Latin-script
languages more than English (Marchisio et al., 2024), so
results for quantized checkpoints describe that checkpoint, not the full-precision model.
The full protocol is in configs/protocol.yaml and was fixed before any main-run results were seen.
- Harness: lm-evaluation-harness
0.4.13, tasksmmlu_prox_lite_{lang}unchanged, backendlocal-chat-completions(OpenAI-compatible chat API) for every model,--apply_chat_template --fewshot_as_multiturn false. - Data:
li-lab/MMLU-ProX-Liteat revisione82aafb, test split (588 questions per language). - Languages:
en,zh,ja,ko,pt,de,id,ru,ar,sw. - Prompt: the task's 5-shot chain-of-thought prompt: a system message with the task description and one user message with 5 worked examples from the same subject plus the test question.
- Changes to the task's generation defaults (same for every model):
- temperature
0.0→1.0: every model's developer default or recommendation, and several APIs reject or ignore other values; - stop strings → none: chat models end their turn by themselves, and a stop string such as
Q:inside the reasoning would cut it off; max_gen_toks2048→ the per-model value above, since reasoning tokens count toward it.
- temperature
- Reasoning effort: each developer's official API default; otherwise the model card's recommendation; otherwise the host's default. Exception: small open models (~30B: Gemma 4 31B, Qwen 3.8 27B) use their highest effort.
- Sampling: each developer's recommended values (see the model configs).
- Scoring: the official score is lm-eval's
exact_matchwith the task's per-language answer regex (custom-extract); unextractable answers count as wrong. A lenient re-scoring separates format failures from wrong answers: a strict-wrong response counts as correct when the OpenAI Decisions API (gpt-6-luna), shown only the response and the answer options, reads the response's own final answer as the gold option. The grader never sees the question, so it reads the answer rather than solving the question. - Statistics: language gaps are paired comparisons on the same questions (paired bootstrap CIs). Language × subject cells use Benjamini–Hochberg correction.
- 2026-10-10, before any main run: Kimi K3 was dropped. At its default
maxeffort it would have cost more than all other open-weight models combined, and it is flagship-class rather than lightweight. - 2026-10-10: GPT-6.1 Sol and Claude Sonnet 5.5 were added as reference models. Both decisions were about cost, not results.
- 2026-10-12: the lenient re-scoring was changed from the rule-based version in protocol v1
(strict, then any language's answer phrase, then the last standalone
(A)–(J)) to an LLM reading of the response's final answer with the OpenAI Decisions API. The change was made after strict results for 8 models had been seen. The strict score, which is the official one, is unaffected. - 2026-10-12: GLM-5.3 and Qwen 3.8 27B moved from Neuralwatt's Flex tier to its standard tier,
which serves the same checkpoint, to finish sooner. GLM-5.3 generated about twice as fast there;
Qwen 3.8 27B at about the same speed. GLM-5.3 moved after 2,936 answers (60 of them from an
earlier, brief standard-tier attempt) and Qwen 3.8 27B after 1,387. The standard tier cuts
responses that take over 100 s, so these requests were streamed through
scripts/stream_relay.py. Prompts, settings and scoring are unchanged. - 2026-10-12: the language × subject analysis showed that Azure's content filter had blocked all 68 Swahili math questions for GPT-6 Luna and GPT-6.1 Sol. Those 136 questions were sent again, after the results had been seen, and were all blocked again. The results are unchanged: the blocked questions count as wrong.
- Lite subjects are unbalanced (per language: 19 history … 68 math questions). Subject-level 95% CIs can be as wide as ±22 points; treat single-cell differences as exploratory.
- lm-eval logs only the final response text, not the reasoning or the finish reason, so a truncated response looks like a wrong answer. Max tokens are set generously to keep truncation rare. Token counts and finish reasons were recovered afterwards from the providers' records; the reasoning text was not.
- Averages are over the 10 evaluated languages and are not comparable to 29-language averages reported elsewhere.
- API models change behind stable names; result files carry their run timestamp.
- Interrupted runs resumed from lm-eval's request cache, and failed requests were sent again. lm-eval caches an empty response only when a run finishes, so a resumed run re-sent the questions that had come back empty. GLM-5.3 and Qwen 3.8 27B, interrupted most often, got the most such second tries.
configs/protocol.yaml evaluation protocol v1
configs/models/ one file per model: ID, reasoning effort, sampling, max tokens, sources
scripts/run_eval.py turns a model config into an `lm-eval run` command and runs it
scripts/stream_relay.py streams requests for API routes that cut long non-streaming responses
analysis/analyze.py builds results/summary.md and the tables in this README
analysis/lenient.py lenient re-scoring with the OpenAI Decisions API
analysis/usage_recovery/ recovers the main run's token usage from the providers' records
results/lenient/ every lenient grading decision, one JSON line per strict-wrong answer
results/recovered-usage/ recovered token usage: summary, one row per question, matching checks,
and the providers' records it was built from (records/)
results/ lm-eval results files per model, and summary tables (committed)
runs/ raw lm-eval outputs (git-ignored, published to Hugging Face)
hf/ Hugging Face dataset card for the raw outputs
uv sync
cp .env.example .env # LLM_GATEWAY_API_KEY, NEURALWATT_API_KEY, HF_TOKEN (write access)
# 14 requests (English, 1 question per subject) to check the API accepts every parameter
uv run scripts/run_eval.py configs/models/qwen-3-8-27b.yaml --smoke
# main run; rerunning the same command resumes from the request cache.
# Copies results_*.json into results/<model>/ when it finishes.
uv run scripts/run_eval.py configs/models/qwen-3-8-27b.yaml < /dev/null
# lenient re-scoring of strict-wrong answers with the OpenAI Decisions API
# (needs OPENAI_API_KEY in the environment; already-graded answers are skipped)
uv run python analysis/lenient.py
# rebuild results/summary.md and the README results tables from all finished runs
uv run python analysis/analyze.py
# token usage tables, from the committed provider records in results/recovered-usage/records/
# (each script in analysis/usage_recovery/ says how its records were fetched)
uv run python analysis/usage_recovery/match.py
uv run python analysis/usage_recovery/summarize.pyRaw outputs are uploaded afterwards with the Hugging Face CLI
(hf upload <repo> runs/<model>/main/<model_id> <model_id> --repo-type dataset).
--dry-run prints the exact lm-eval run command without sending requests.
To cite this evaluation, use Cite this repository on GitHub (generated from
CITATION.cff) or:
@misc{3ae3ae2026mmluproxlite,
author = {3ae3ae},
title = {{MMLU-ProX-Lite}: Where Do Recent Lightweight {LLMs} Break Down?},
year = {2026},
howpublished = {\url{https://github.com/3ae3ae/mmlu-prox-lite-evals}},
note = {Raw outputs: \url{https://huggingface.co/datasets/3ae3ae3ae/mmlu-prox-lite-evals}}
}Please also cite the benchmark and the evaluation harness:
@article{xuan2025mmluprox,
title={Mmlu-prox: A multilingual benchmark for advanced large language model evaluation},
author={Weihao Xuan and Rui Yang and Heli Qi and Qingcheng Zeng and Yunze Xiao and Aosong Feng and Dairui Liu and Yun Xing and Junjue Wang and Fan Gao and Jinghui Lu and Yuang Jiang and Huitao Li and Xin Li and Kunyu Yu and Ruihai Dong and Shangding Gu and Yuekang Li and Xiaofei Xie and Felix Juefei-Xu and Foutse Khomh and Osamu Yoshie and Qingyu Chen and Douglas Teodoro and Nan Liu and Randy Goebel and Lei Ma and Edison Marrese-Taylor and Shijian Lu and Yusuke Iwasawa and Yutaka Matsuo and Irene Li},
journal={arXiv preprint arXiv:2503.10497},
year={2025}
}
@misc{eval-harness,
author = {Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and Reynolds, Laria and Schoelkopf, Hailey and Skowron, Aviya and Sutawika, Lintang and Tang, Eric and Thite, Anish and Wang, Ben and Wang, Kevin and Zou, Andy},
title = {The Language Model Evaluation Harness},
month = 07,
year = 2024,
publisher = {Zenodo},
version = {v0.4.3},
doi = {10.5281/zenodo.12608602},
url = {https://zenodo.org/records/12608602}
}Code and results: MIT.
Third-party material:
- MMLU-ProX (questions included in the logged samples): MIT License, as stated on its dataset card. It is built on MMLU-Pro (MIT).
- lm-evaluation-harness: MIT License; used as an unmodified dependency, not redistributed here.
- Model outputs in the logs were generated by the models listed above and are shared for evaluation analysis.