Skip to content

Commit 77c5ad0

Browse files
0bserver07claude
andcommitted
fix(providers): register glm-5.2 — the model the published numbers came from
A bench-matrix run naming glm-5.2 failed with 'Unknown provider' and wrote an error cell: only glm-4.6 and glm-5.1 were in the catalog, so the model behind this repo's flagship scorecard could not be re-run through its own CLI. Found while re-running mbpp-plus under the newly-wired plus harness. The fallback to glm-5.1 CONFOUNDED the result — grading changed and the model changed at once, so the 99.7% -> 85.4% drop could not be attributed. Registering 5.2 makes the re-run a one-variable measurement against the published number. Verified served by the same z.ai Anthropic-compat endpoint as 5.1; bare id and the [1m] long-context tag both answer. Cost row mirrors 5.1 and matches PRICING, so the catalog-vs-cost.py parity gate stays green (glm-5.2 remains a declared PRICING_PLACEHOLDER pending Zhipu's public sheet). Gates: 1784 passed / 25 skipped, ruff clean, mypy 731 files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 56545f8 commit 77c5ad0

2 files changed

Lines changed: 24 additions & 0 deletions

File tree

CHANGELOG.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,17 @@ commit receipts.
99

1010
### Fixed
1111

12+
- **`glm-5.2` was missing from the provider catalog** — the model this repo
13+
publishes its flagship scorecard with. A `bench-matrix --model glm-5.2` run
14+
failed with `Unknown provider: 'glm-5.2'` and recorded an **error cell**, so
15+
the model behind the published numbers could not be re-run through the CLI at
16+
all; only `glm-4.6` and `glm-5.1` were registered. Found while re-running the
17+
mbpp-plus column: the fallback to `glm-5.1` silently confounded the
18+
comparison, because the grading change and a model change would have moved
19+
the number together. Verified served by the same z.ai Anthropic-compat
20+
endpoint as 5.1 (bare id and the `[1m]` long-context tag both answer). Rates
21+
mirror 5.1 and match `PRICING`, which keeps glm-5.2 a declared placeholder.
22+
1223
- **`numpy` was declared in no extra, so the plus contract could not run.**
1324
The EvalPlus expanded harnesses (`mbpp-plus`, `humaneval-plus`) `import numpy`
1425
inside the graded program, so under the repo's own documented

chimera/providers/catalog.py

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,19 @@ def resolve_api_key(self) -> str | None:
136136
base_url="https://api.z.ai/api/anthropic",
137137
api_key_env="ANTHROPIC_AUTH_TOKEN",
138138
context_window=200_000, cost=(2.0, 8.0)),
139+
# glm-5.2 — the model this repo actually publishes its flagship scorecard
140+
# with, and which was MISSING from the catalog until 2026-07-29: a
141+
# bench-matrix run naming it failed with "Unknown provider: 'glm-5.2'" and
142+
# recorded an error cell, so the model behind the published numbers could
143+
# not be re-run through the CLI. Verified served by the same z.ai
144+
# Anthropic-compat endpoint as 5.1 (bare id and the `[1m]` long-context
145+
# tag both answer). Rates mirror glm-5.1 pending Zhipu's public sheet —
146+
# PRICING/PRICING_PLACEHOLDERS in cost.py is the source of truth and keeps
147+
# glm-5.2 as a declared placeholder, so this row must not diverge from it.
148+
ModelConfig("glm-5.2", "anthropic",
149+
base_url="https://api.z.ai/api/anthropic",
150+
api_key_env="ANTHROPIC_AUTH_TOKEN",
151+
context_window=200_000, cost=(2.0, 8.0)),
139152
# GPT-OSS (OpenAI open-weights). Distributed via Ollama; both 20B
140153
# and 120B run locally on adequately-sized hardware. $0/$0 because
141154
# local serve is free.

0 commit comments

Comments
 (0)