The harness tests Sandchest and AssemblyAI on identical audio with explicit language selection and automatic detection. It reports recognition, language identification, word metadata, manual timestamp accuracy, failures and latency separately. A model declaring a language is not evidence of quality parity.
All recordings, references, credentials, provider responses and run artifacts stay under the ignored .sandchest/benchmarks directory. Console progress contains sample identifiers and aggregate results, never transcript text or credentials. This tooling does not start services, change Cap, migrate a shared database or deploy anything.
- The API inventory is imported from
lib/api/contract.ts: 99 base languages plus four regional aliases. Related languages are not substituted to make coverage look complete. - Human reference text produces WER and CER. Existing provider transcripts produce provider disagreement, not ground-truth WER. Synthetic references are a third, separate category.
- Unicode normalization preserves Indic combining marks. Character error is the primary metric for Chinese, Japanese, Thai, Lao, Khmer and Burmese; both metrics remain available. Numbers are not rewritten using the expected answer. These are harness-specific scores, not directly interchangeable with publications using different normalization.
- A completed transcript with bad word metadata or a replacement character retains its actual text score. CER counts replacement characters as errors; encoding defects remain separately visible. A real failed transcription contributes deletion errors. An unsubmitted or interrupted task remains unscored and visibly incomplete.
- Manual word annotations are required for timestamp accuracy. Automatically aligned or provider-generated timestamps are not gold. Reports include matching coverage and, where possible, comparisons on the same reference words across providers. No numerical timestamp-accuracy claim is made for text-only corpora.
- Development, validation and holdout samples are separate. Duplicate audio/source IDs and cross-split speaker overlap, where speaker IDs exist, are rejected. The adaptive selector only accepts development results and never reads holdout references for tuning. Split checks cover the supplied manifest; the operator must include or separately exclude protected samples from earlier corpora.
- Native HTTP latency and full upload→submit→poll latency are labeled separately. Resumed jobs have no fresh latency measurement. Serial latency is not concurrency, capacity or production-tail evidence.
Use scripts/evaluation/script-audit.ts when providers may return different writing systems, including recordings without known language or trusted reference text. This is an additive diagnostic: it does not rerun models, change existing scores, transliterate answers, or infer which transcript is correct.
bun --no-env-file scripts/evaluation/script-audit.ts \
--manifest .sandchest/benchmarks/my-study/script-pairs.json \
--out .sandchest/benchmarks/my-study/script-audit.jsonThe private manifest contains version: 1 and pairs. Each pair has a unique safe id, a language claim or null, and left/right sides. A completed side is { "state": "completed", "path": "PRIVATE_JSON_PATH", "sha256": "FILE_SHA256", "textPointer": "/response/text" }. The pointer follows JSON Pointer escaping; use /rawRecognition/text for the standalone MMS receipt. Failed, interrupted or unrun sides contain only their corresponding state (failed, interrupted, not_run) and remain in the planned denominator. Audio pairing is the caller's responsibility; this tool verifies saved file hashes, not audio identity.
Output contains Unicode letter-script counts and no transcript text. NFC preserves writing systems; combining marks are counted separately. A dominant script requires at least sixteen letters and an 80% share. Unknown scripts, short text and mixed scripts remain visible without inventing a language. Different dominant scripts are review warnings, never an accuracy failure or a beta pass. Existing metrics and failure receipts remain unchanged. Input paths must remain inside private benchmark storage; hashes, text sizes, total work and unique IDs are checked, and output is write-once. CLI errors do not echo potentially sensitive JSON/schema fragments.
The first actual use audits forty saved Turkmen-development pairs (twenty each against prior Sandchest and AssemblyAI outputs). It flags twenty MMS/AssemblyAI and eight MMS/prior-Sandchest writing-system differences. No model/provider request is added. These counts describe the scripts of saved outputs, not the language actually spoken or human recognition accuracy.
For canonical mono 16kHz PCM16 corpora, run a language-independent audit against an existing report:
bun --no-env-file scripts/evaluation/timing-audit.ts \
--run .sandchest/benchmarks/my-study/run \
--report .sandchest/benchmarks/my-study/report.json \
--out .sandchest/benchmarks/my-study/timing-audit.jsonIt verifies the plan/corpus identities, each report's canonical result hash, and the exact audio bytes. It then finds maximal runs of digital zero lasting at least 200ms (configurable with --minimum-silence-ms, 100–10000ms), and reports words wholly inside or spanning those runs, plus invalid interval indices. Quiet nonzero audio is not treated as silence. It never changes transcripts, requires no model/provider call, and writes no transcript text into its output. Failed, missing and historically reinterpreted results remain explicitly unaudited. Output records audit code, PCM reader and harness hashes; existing results are never overwritten.
These are review warnings, not human timestamp accuracy or an automatic quality pass. A speaker can pause within a word. Counts for maximal natural digital-zero runs can differ from an audit of a specific inserted pause inside a longer zero run; preserve both definitions and their original reports. Noncanonical audio must first be converted through the corpus collection workflow, with its own immutable identity. Five focused tests extend the existing evaluation suite, including changed-result rejection, missing-task rejection, quiet audio, interval boundaries and failure accounting.
Use decoded frame counts when constructing exact crop tests. An integer durationMs can be rounded metadata: 67,988 samples at 16kHz last 4249.25ms, so requesting a 4249ms crop correctly removes four samples. Equal text and millisecond timestamps do not imply that confidence must be bit-for-bit identical for those different waveforms.
Create repeatable controls with the shared fixture tool:
bun --no-env-file scripts/evaluation/crop-fixture.ts \
--input PRIVATE_SOURCE.wav --sha256 SOURCE_SHA256 \
--out NEW_PRIVATE_DIRECTORY --mode to-end --prefix-ms 1001to-end prepends exact digital silence and emits only audio_start_from; leaving audio_end_at unset preserves every source frame, including fractional-millisecond tails. bounded-end adds a suffix (default 1500ms) and emits both endpoints, but requires a source frame count divisible by 16. It rejects an inexact bound instead of truncating or padding the source. Inputs must be canonical mono 16kHz PCM16. The tool verifies source/selected PCM identity, binds audio and tool hashes in receipts, preserves a 3 GiB disk reserve, and never overwrites an output directory. Use the generated options.json when submitting the derived audio.
Compare complete words/confidence after removing the requested timestamp offset; do not loosen confidence checks to hide a changed waveform. Six automated tests cover exact tails, bounds, malformed PCM, private paths and immutable receipts. A real API check on two reused Haitian development recordings passed eight jobs and eight same-ID retries across both crop modes and explicit/automatic recognition: all word fields, text and shifted boundaries were exact, with eight unique usage events and no extra billing. These are mechanical controls, not new independent quality samples or human word-time annotations. Private receipts: exact-crop-controls1; isolated app checks: exact-crop-app-checks1.
The earlier Haitian recovery keeps its original completed, four-sample-shorter crop as an extra probe and finishes the remaining checks with an exact whole-millisecond source; retries and actual job/usage totals remain reported separately.
Also distinguish genuine sample differences from floating-point representation: multiplying API seconds by 1000 can produce 4086.9999999999995 for integer 4087ms. A comparison may canonicalize only that negligible arithmetic noise after checking the expected integer duration; it must not accept a real millisecond shift or rewrite historical results.
Use an immutable JSON corpus with version, id, description and samples. Each sample contains:
| Field | Content |
|---|---|
id, language, split |
Safe identifier, exact API language code, development/validation/holdout |
expectedSpeech |
Whether speech should be present; include silence/noise controls separately |
source |
Dataset, pinned revision, source ID, provenance, optional speaker ID |
audio |
Private local path, SHA256, bytes, measured duration in milliseconds |
reference |
Optional private JSON path and SHA256, kind human/provider/synthetic, timing manual/forced-alignment/provider/none |
Reference JSON is { "text": "...", "words": [{ "text": "...", "start": 0, "end": 200 }] }; omit words for text-only references. Word times are milliseconds. Store assets privately before adding them to the manifest. Never embed an answer in provider prompts.
Google FLEURS has exact mappings for 82 of the 99 base languages. The pinned revision is 70bb2e84b976b7e960aa89f1c648e09c59f894dd. It provides published reference text for read speech, not manual word times or Cap-domain evidence. Training-data overlap with either model is unknown.
The missing languages are sq ba eu br fo ht haw la mg nn sa si su tt bo tk yi. These need other corpora or reviewed real recordings. no maps to Bokmål; it does not validate nn (Nynorsk). Regional English and Swiss German need their own accent/dialect evidence.
The fast collector uses the dataset row API, a deterministic 100-row window, and unique source sentences. It validates each audio URL against the pinned revision, language config and row, then stores exact source hashes and canonical mono 16 kHz PCM16 WAVs. This is an efficient development screen, not a representative language benchmark. Failures stay in coverage.json; unavailable languages are not silently counted as supported.
bun --no-env-file run benchmark:collect-languages \
--input .sandchest/benchmarks/my-study/collection-request.json \
--out .sandchest/benchmarks/my-study/corpusThe request schema in scripts/evaluation/collect-fleurs-viewer.ts requires a private, checksummed copy of the pinned Hub metadata, selected languages, a seed, clips per language, duration bounds, download budget and the FFmpeg executable. It fingerprints the collector and FFmpeg, preserves a 3 GiB disk reserve and never overwrites source assets. Publisher asset URLs can expire; retain downloaded audio rather than treating URLs as permanent corpus storage.
scripts/evaluation/collect-fleurs-archive.py is a smaller fallback for viewer failures. It verifies the published TSV against the pinned Git blob, streams only a bounded archive prefix, ignores archive paths, and writes checksummed clips into private directories. It explicitly records that the full archive hash was not verified. It rejects invalid audio formats before model selection and records the rejected count. This collector needs NumPy and SoundFile; it does not load a model.
For an independent expansion, its request accepts excludeCorpora, an array of { "path": "private/corpus.json", "sha256": "<actual SHA256>" } identities. It verifies each manifest and excludes previous source IDs and audio hashes. For development samples it also verifies and hashes normalized reference text, preserving combining marks; validation/holdout reference files are never opened. It deduplicates the newly selected audio and reference text as well. This normalization is only for conservative duplicate exclusion, not recognition scoring. The existing 20-clip, 64 MiB archive-prefix and 90-second read limits still apply; exhausted budgets remain visible failures.
PYTHONDONTWRITEBYTECODE=1 workers/inference/.venv/bin/python \
scripts/evaluation/collect-fleurs-archive.py \
.sandchest/benchmarks/my-study/archive-request.json \
.sandchest/benchmarks/my-study/archive-corpus
PYTHONDONTWRITEBYTECODE=1 workers/inference/.venv/bin/python \
-m unittest discover -s tests -p test_fleurs_archive_collector.pyThe archive request can also select datasetSplit: "test" instead of the default published "validation", independently of evaluationSplit: "development" | "validation" | "holdout". Use published test data only after freezing a policy, never to adapt development selection. The receipt records both roles; cached results from another split are rejected. The existing corpus-collector suite passed 15 offline tests after this split change, including real mocked dev/test archive requests and wrong-split cache rejection.
The offline tests cover combining marks, changed manifest/reference hashes, protected-reference non-access, invalid requests, and a complete synthetic archive selection with source/text/audio exclusions. Keep failed viewer collections and their receipts; a successful archive fallback is a separate collection, not a reason to overwrite the failed attempt.
scripts/evaluation/collect-fleurs.ts is the full-file alternative when the viewer fails. It downloads a pinned Parquet shard, verifies its complete SHA256, extracts deterministic samples and retires only its own verified temporary shard after durable clip verification. Interrupted downloads retain their prefixes. This optional corpus preparation path requires Python with pyarrow==25.0.1, NumPy and SoundFile; inference remains Rust. Configure the Python executable and, if needed, SANDCHEST_EVAL_PYTHONPATH. Never collect a public test split for adaptive development.
scripts/evaluation/collect-openslr-ranges.py adds public Basque, Sinhala, Sundanese and Tibetan recordings. It uses HTTP byte ranges with a strong ETag, checks ZIP CRCs and selected-byte hashes, and never extracts archive paths. The complete archive checksum is not verified. Source recordings and reference text remain private alongside canonical PCM audio.
PYTHONDONTWRITEBYTECODE=1 workers/inference/.venv/bin/python \
scripts/evaluation/collect-openslr-ranges.py \
--out .sandchest/benchmarks/my-study/openslr \
--languages eu si su bo --clips 4 --seed my-study-v1
PYTHONDONTWRITEBYTECODE=1 workers/inference/.venv/bin/python \
-m unittest discover -s tests -p test_openslr_collector.pyThis optional collector requires httpx, SoundFile and FFmpeg; model serving remains Rust. It limits transfer bytes, member sizes, elapsed download time and disk use, selects distinct speakers and reference sentences, and records excluded source rows. Tibetan source IDs are checked against the publisher's audio map. Unknown Sundanese annotations and missing Tibetan references are excluded before any model runs. Published evaluation-directory paths are excluded; unsplit source data is used only for local development, with no official evaluation split claimed.
Completed languages have individual sample manifests and range receipts. A failed collection does not produce a complete top-level corpus. Never overwrite an interrupted directory; preserve its source snapshot and recover completed files only after checking source/reference/audio hashes. The initial interrupted Basque/Sinhala collection was independently recovered by reproducing its PCM conversion byte-for-byte.
scripts/evaluation/collect-omnilingual-viewer.py collects Hawaiian (haw→haw_Latn), Eastern Yiddish (yi→ydd_Hebr), Gheg Albanian (sq→aln_Latn) and Plateau Malagasy (mg→plt_Latn) from the Meta Omnilingual ASR Corpus, pinned to 8648ba8946377697b427ae952076e49fc0e5e44d. It reads publisher train rows, validates exact revision/configuration/row asset URLs, samples across publisher speaker IDs and deduplicates references/audio. Tagged unclear/nonverbal references and partial/truncated viewer rows cannot silently become clean reference evidence. Language/script metadata, decoded audio duration, decoder limits, download budget and disk reserve are checked.
The request contains id, checksummed private metadataPath/metadataSha256, languages (haw, yi, sq, mg), seed, samplesPerLanguage (1–20), maximumDownloadBytes and ffmpeg. Optional fields are excludeCorpora (the same frozen source/audio/development-text exclusions as the archive collector), evaluationSplit (development by default, or validation/holdout), and maximumClipsPerSpeaker (1–20, default two). Exclusions never open protected validation/holdout reference text. A local validation label does not turn publisher training data into model-independent evaluation data.
Run it with --input PRIVATE_REQUEST.json --out NEW_PRIVATE_DIRECTORY; the offline tests are in tests/test_omnilingual_collector.py. Sampling uses one clip per speaker ID before adding another, up to the explicit cap. Receipts retain the actual per-speaker counts and variety scope. More clips from the same speaker add reference words, not independent-speaker evidence. The initial Hawaiian/Yiddish collection obtained four speaker IDs per language; the later Albanian/Malagasy collection has 20 clips each but only two/five publisher speaker IDs.
These are published reference transcripts, not independently reviewed Cap recordings or manual word boundaries. Gheg does not establish Tosk/all Albanian dialects; Plateau Malagasy and Eastern Yiddish likewise do not establish every variety of their base language. This corpus is a known training source for Meta's Omnilingual model: do not use it to claim independent quality for that model. Training overlap with Whisper and AssemblyAI is unknown.
scripts/evaluation/collect-mlsuperb-viewer.py reads bounded, checksummed viewer windows from ESPnet ML-SUPERB, revision 5a0634ecdbffdc99d8b55eb2b2486f3deb89691c. Exact mappings are ba→bak, br→bre, nn→nno, tt→tat; Bokmål/Norwegian are not substituted for Nynorsk. Published dev rows become local development data.
Its private request contains id, metadataPath/metadataSha256, rowWindows (one {language,path,sha256} per language), languages, seed, samplesPerLanguage (1–20), minimumDurationMs/maximumDurationMs (4–30 seconds), maximumDownloadBytes, ffmpeg, optional excludeCorpora, and optional evaluationSplit (development, validation or holdout). Pin metadata and exact revision/config/split/row asset URLs before downloading audio; do not assume viewer filtering or source-ID sorting is reliable. Collection verifies every row independently.
PYTHONDONTWRITEBYTECODE=1 workers/inference/.venv/bin/python \
scripts/evaluation/collect-mlsuperb-viewer.py \
--input .sandchest/benchmarks/my-study/mlsuperb-request.json \
--out .sandchest/benchmarks/my-study/mlsuperb-corpus
PYTHONDONTWRITEBYTECODE=1 workers/inference/.venv/bin/python \
-m unittest discover -s tests -p test_mlsuperb_collector.pyThe collector removes only the exact publisher [language] prefix, excludes unknown/unclear tags, verifies mono 16kHz decoding and byte/duration limits, and deduplicates audio and reference text. It reuses the existing bounded transfer and exclusion helpers; protected reference files are not opened. It never overwrites a run or counts a partial collection as complete. Eight offline tests cover actual conversion, identity/URL tampering, protected references, invalid inputs, duplicates, incomplete receipts and validation/holdout roles.
Freeze the model and routing policy before collecting validation data, and supply prior development corpora through excludeCorpora. Source IDs, normalized development text and audio hashes are excluded without opening protected references. evaluationSplit changes only the local study role: the publisher split and pinned asset URLs remain dev, and both roles are recorded explicitly. It does not establish model-training or speaker independence.
These are published Common Voice-derived benchmark transcripts without speaker IDs or manual word times. Training overlap and speaker independence are unknown. The first collection has 20 clips per language, all only 4–8 seconds long. Longer derived controls must be reported separately from original recordings: concatenating five clips with known pauses is not five new speakers, natural long-form evidence or a guarantee of sufficient spoken duration for language detection. Keep exact source-PCM offsets and never supply reference text to inference.
scripts/evaluation/collect-faroese-viewer.py collects human read-speech references from Ravnursson, pinned to 03665210706cf1f49b068b3c1188942484964b05, configuration ravnursson_asr. It verifies one to eight checksummed row windows of at most 100 rows each. Exact revision, publisher split, row index and audio path must match before downloading.
The private request contains id, metadataPath/metadataSha256, rowWindows (each {path,sha256}), seed, samplesPerLanguage (1–20), minimumDurationMs/maximumDurationMs (4–30 seconds), maximumDownloadBytes and ffmpeg. Optional fields are datasetSplit (train by default), evaluationSplit (development by default), maximumClipsPerSpeaker (two by default) and excludeCorpora. Publisher test data cannot be used for adaptive development.
PYTHONDONTWRITEBYTECODE=1 workers/inference/.venv/bin/python \
scripts/evaluation/collect-faroese-viewer.py \
--input .sandchest/benchmarks/my-study/faroese-request.json \
--out .sandchest/benchmarks/my-study/faroese-corpusSampling visits each conservative speaker-prefix group before adding a second clip. Publisher speaker_id values contain sessions, so only the first component defines the group; these are not independently verified people. Validation/holdout collection additionally excludes supplied earlier speaker groups without opening protected reference text. Source IDs, normalized development text and audio hashes are deduplicated. The collector rejects changed/truncated rows, wrong audio formats, excessive transfers and overwritten outputs, preserving partial receipts on failure. It reuses existing bounded network, PCM and exclusion helpers.
Seven offline tests in tests/test_faroese_collector.py cover actual conversion, source tampering, speaker/session grouping, protected-reference non-access, duplicate content and incomplete collection. The complete collector suite now has 36 passing tests. The first real corpus contains 20 recordings across 13 groups, with no manual word times or known training-data independence.
The Faroese study raises cross-study reference coverage to 95/99, leaving ht, la, sa, tk unmeasured. Its high recognition errors and zero correct auto labels show why collecting a language is not a quality pass. Dedicated native candidates remain separate from full serving/API results.
scripts/evaluation/collect-latin-csv.py collects Vox Classica / Latin-Audio at revision 6268020076fce4943a2dae04b45b67d6c94518b3. It verifies the complete 3,695,773-byte publisher metadata.csv against Git blob f55693a39ce943448a7b08972b047e41f8b1e479 and SHA256 5d9d2f8bfdb5cfa9ca7cf79213e3c94e5fb925fc908a22e1012319ea5d0b5e0c. Audio paths are constructed from that verified CSV and pinned revision; HTTPS redirects are restricted to Hugging Face domains. Selected audio hashes are recorded locally, not claimed as independently published checksums.
The private request requires id, metadataPath, metadataSha256, seed, samplesPerLanguage (1–20), minimumDurationMs/maximumDurationMs (4–30 seconds), maximumDownloadBytes and ffmpeg. Optional fields are evaluationSplit, maximumClipsPerSpeaker and excludeCorpora, with the same protected-reference exclusion behavior as the other collectors. Publisher data is train; a local validation/holdout label does not establish independence from model training.
PYTHONDONTWRITEBYTECODE=1 workers/inference/.venv/bin/python \
scripts/evaluation/collect-latin-csv.py \
--input .sandchest/benchmarks/my-study/latin-request.json \
--out .sandchest/benchmarks/my-study/latin-corpus
PYTHONDONTWRITEBYTECODE=1 workers/inference/.venv/bin/python \
-m unittest discover -s tests -p test_latin_collector.pyThe collector visits different publisher speaker IDs before adding further clips, excludes missing speaker IDs and unclear annotations, preserves published spelling/diacritics, and checks actual decoded duration and PCM uniqueness. It bounds transfers, redirects, decoder time and disk use. Failed or duplicate-only collections retain receipts without producing a complete corpus. Eight offline tests cover these behaviors, real PCM conversion, source tampering, trusted redirects and protected-reference non-access.
The first collection contains 20 clips across 20 publisher speaker IDs, totaling 3.66 minutes. These are not independently verified people. The publisher describes human read speech with potentially automatic macronization and curator-reviewed sentence cuts initially located using Whisper. This provides reference text, not manual word boundaries. No Latin recognition/automatic-detection benchmark or quality pass is claimed from collection alone.
scripts/evaluation/collect-pinned-catalog.py separates source discovery from audio collection. Prepare a private catalog from reviewed publisher metadata, then use the same importer for additional languages instead of writing another downloader. It accepts pinned Hugging Face repository files or dataset-viewer audio rows; it does not execute dataset code.
The catalog has version: 1, language, dataset, a 40-character revision, datasetSplit, referenceKind, a candid provenance description, checksummed metadata files, and records. Each record has sourceId, a revision-pinned url, and the exact published text; speakerId, durationSeconds and a published sourceSha256 are optional. Keep synthetic, human-reference and provider-derived catalogs separate. The importer verifies identity and bytes, not the truth of a publisher's transcription claim. The importer reads only the literal language list from lib/api/contract.ts, without executing JavaScript or loading application credentials. It accepts all four regional aliases and rejects unsupported codes before downloads; contract refactors fail closed. The normal corpus validator remains authoritative. Generated identifiers, source IDs, dataset names and full provenance text must fit that corpus schema, including UTF-16 length accounting.
The request has id, catalogPath, catalogSha256, seed, samplesPerLanguage (1–20), maximumDownloadBytes (up to 256 MiB), and the FFmpeg executable. Optional fields are evaluationSplit, maximumClipsPerSpeaker, excludeCorpora and candidateSha256. Validation/holdout collection requires a frozen candidate hash. Publisher test/holdout splits cannot feed development selection. Existing protected reference files are never opened for duplicate-text filtering, and known protected speakers are excluded in both directions: a later development collection cannot reuse a validation/holdout speaker, even under another language label in the same dataset. Unknown speakers stay unknown rather than inflating the speaker count.
PYTHONDONTWRITEBYTECODE=1 workers/inference/.venv/bin/python \
scripts/evaluation/collect-pinned-catalog.py \
--input .sandchest/benchmarks/my-study/catalog-request.json \
--out .sandchest/benchmarks/my-study/catalog-corpus
bun --no-env-file scripts/evaluate-multilingual.ts validate \
--corpus .sandchest/benchmarks/my-study/catalog-corpus/corpus.jsonCollection is bounded to 80 attempted records, 600 seconds, 4–30 second mono inputs, restricted HTTPS redirects and a 3 GiB disk reserve. Audio is decoded once with libsndfile and resampled/quantized to PCM16 by FFmpeg. Duration comes from decoded frames, not an MP3 container estimate; publisher durations rounded to hundredths are checked within half a hundredth plus one source sample. No manual gain or trimming is applied. Source bytes, decoded-float/PCM hashes, decoder versions and FFmpeg hash are recorded. Both providers must receive the same resulting PCM. Failed collections keep their receipts and never produce a complete corpus manifest. Twelve offline tests cover these controls, including actual MP3 conversion, rounding boundaries, regional aliases, metadata bounds and protected-speaker exclusion during future development.
Initial catalogs target Haitian Creole and Sanskrit. The Haitian release has only one published speaker ID despite broader language-variety claims. Sanskrit selection uses only scholar and recitation_corpus training rows, excluding tts_aug, automatically graded flywheel and the protected in_the_wild_test split. These limitations must remain in every resulting benchmark report. Collecting either corpus does not establish recognition parity.
The same pinned-catalog importer now collects a second Haitian source, CMU Haitian Creole Speech at revision 4569351079203f9188abf62f8c7a2371930df35f: 20 development clips, 14 conservative filename-prefix groups, 3.50 minutes. The source-specific step produces only reviewed metadata/catalogs; downloading, conversion, bounds, deduplication and split protections stay in the shared importer.
Do not assume a field named speaker_id identifies a person. This mirror repeats its utterance ID there. Preserve the original identity in a sidecar, group conservatively by the reviewed filename prefix, and explicitly label those groups as unverified. Exclude unknown filename patterns instead of manufacturing speaker IDs. Use literal text, not a normalized or ASR-generated rewrite. Partial metadata availability must remain in corpus provenance: a bounded fallback is not a random whole-repository sample.
For a new model, compare against both the current worker and AssemblyAI on the same PCM. Preserve actual baseline responses and raw receipts; cached responses have cached latency and do not count as new submissions. Report distinct source cohorts separately, retain failures, and freeze the candidate before collecting unused validation. Candidate-hash checks and prior-source/text/audio/speaker exclusions are already part of the shared importer. The private CMU metadata attempts retain upstream failures and bookkeeping corrections instead of overwriting evidence.
Create a private plan request with these fields:
{
"id": "language-screen-v1",
"corpusPath": ".sandchest/benchmarks/my-study/corpus/corpus.json",
"phase": "development",
"providers": [
{
"id": "sandchest",
"protocol": "transcript-api",
"baseUrl": "http://127.0.0.1:3189/v2",
"credentialEnv": "SANDCHEST_EVAL_API_KEY",
"authorization": "raw",
"modelLabel": "frozen local candidate",
"options": { "punctuate": true, "format_text": true, "disfluencies": true }
},
{
"id": "assemblyai",
"protocol": "transcript-api",
"baseUrl": "https://api.assemblyai.com/v2",
"credentialEnv": "ASSEMBLYAI_API_KEY",
"authorization": "raw",
"modelLabel": "universal-3-5-pro with universal-2 fallback",
"options": { "speech_models": ["universal-3-5-pro", "universal-2"], "punctuate": true, "format_text": true, "disfluencies": true }
}
],
"modes": ["explicit", "auto"],
"samplesPerLanguage": 2,
"seed": "language-screen-v1",
"maximumRequests": 800,
"maximumAudioMs": 14400000,
"requestTimeoutMs": 180000,
"pollIntervalMs": 250
}Optional languages and sampleIds select a subset. maximumRequests limits transcription tasks, not HTTP polling operations; the audio budget includes each provider/mode attempt. The planner refuses a budget overrun instead of silently shrinking the set. Provider artifactSha256 can identify a pinned local binary, but the harness does not independently attest a remote server from that field. Preserve launch/build/model receipts as well. Returned model identities are retained in reports.
bun --no-env-file run benchmark:multilingual validate --corpus .sandchest/benchmarks/my-study/corpus/corpus.json
bun --no-env-file run benchmark:multilingual plan --input .sandchest/benchmarks/my-study/request.json --out .sandchest/benchmarks/my-study/plan.json
bun --no-env-file run benchmark:multilingual run --input .sandchest/benchmarks/my-study/plan.json --run .sandchest/benchmarks/my-study/run
bun --no-env-file run benchmark:multilingual report --run .sandchest/benchmarks/my-study/run --out .sandchest/benchmarks/my-study/report.json
bun --no-env-file run benchmark:multilingual next --corpus .sandchest/benchmarks/my-study/corpus/corpus.json --runs .sandchest/benchmarks/my-study/run --limit 50 --out .sandchest/benchmarks/my-study/next.jsonSupply credentials only through the named environment variables. The AssemblyAI variable is restricted to its official API origin. Native-only profiles use native-http, an explicit loopback base URL, and a bearer token. Redirects are rejected. Webhooks and external audio_url overrides are not allowed in benchmark profiles.
The runner checks corpus bytes, selected audio/references, scoring code, runtime and dependency lock before calling a provider, and keeps a private copy of the harness source used for each run. Each POST has a durable intent recorded before the request. A known transcript ID resumes through GET polling; an ambiguous POST is never blindly replayed. Keep the same plan, run directory and harness to resume. If code changes, preserve the old source snapshot and unresolved receipts; a new plan is a new experiment, not permission to duplicate an unresolved remote job.
The next command makes a proposal only. It balances new language coverage, favors failures/language-routing/word-metadata problems and uses unseen development audio. Samples already planned in another supplied run stay reserved, including ambiguous or unfinished jobs. It never changes a deployed model or submits requests automatically. A proposal is not a quality gate.
Before building any prior report, next validates every supplied plan as development-only and verifies its corpus bytes and task graph. A later holdout/validation entry therefore cannot expose protected references through an earlier scoring call. Prior report/task identities stay bound to the supplied plan and corpus. Reservation uses the recording audio hash and dataset/language/source identity, rather than a global sample ID: renamed duplicate recordings remain reserved, while unrelated recordings that reuse a local ID remain eligible. Unattempted and interrupted tasks stay reserved too. This protects against known identities in the supplied history; it does not prove unseen speakers, recoded-audio uniqueness or model-training independence. Three new regressions cover protected-reference non-access, changed corpus bytes and cross-corpus identity collisions; the local app suite passes228 tests with7conditional skips.
scripts/evaluation/readiness.ts combines existing, checksummed plans and reports into one offline inventory for all 99 base languages and four exact aliases. It does not run models, read audio/reference bodies or raw transcript receipts, or select a new candidate. Development, validation and holdout remain separate; holdout results must never drive the adaptive development selector.
bun --no-env-file scripts/evaluation/readiness.ts \
--input .sandchest/benchmarks/my-study/readiness-request.json \
--input-sha256 ACTUAL_REQUEST_FILE_SHA256 \
--out .sandchest/benchmarks/my-study/readiness.jsonThe version-1 request contains:
| Field | Required evidence |
|---|---|
candidate.artifactSha256 |
Exact candidate binary hash declared in every selected candidate plan |
candidate.modelLabels |
Explicit, unique allowlist of matching plan model labels |
candidate.responseModels |
Allowed model names in completed candidate responses |
policy.requireCurrentReportHarness |
Whether historical report code is insufficient for the inventory gate |
policy.minimumHumanPairedSamplesPerMode |
Positive sample minimum, or null to report the policy as missing |
policy.maximumHumanPrimaryErrorRate |
Maximum WER/CER rate, or null when no acceptance policy has been set |
policy.maximumHumanPrimaryErrorRateDeltaVsAssemblyAI |
Maximum candidate-minus-provider rate, or null |
sources |
Checksummed plan/report paths and selected provider roles |
Each source needs planPath, planFileSha256, reportPath and reportFileSha256, plus candidateProviderId, assemblyAIProviderId, or both. Omit an unused role rather than setting it to null. This permits a new candidate-only run to pair with the AssemblyAI results from an older paired run without accidentally counting that older Sandchest binary as current. File hashes are hashes of the exact stored bytes, not reformatted JSON.
Pairs must share exact effective request options, audio/reference hashes, reference kind, language and mode, with equal scoring denominators. Changed plans, missing tasks, repeated audio for the same role/mode/split, known cross-split source/audio/speaker overlap, inconsistent edit counts and unapproved candidate identities are rejected. Human, synthetic and provider-reference evidence stay separate: provider references produce disagreement metrics, not ground-truth WER. Failures remain visible; pending tasks remain unscored. Serial latency is split by native/full API scope and declared local/remote endpoint.
A count or observed error-rate gate is not statistical parity. Hashes attest to supplied bytes, not independent execution or score correctness. Runtime attestation, representative Cap-domain coverage, independent word-timing acceptance and production capacity remain unverified by this tool. Its overall result therefore stays allLanguageBetaReadiness: "not_established"; an absent policy is never silently treated as approval. Inspect failed/missing development cells to prioritize the next bounded study, freeze the resulting candidate, then evaluate untouched validation or holdout data without tuning on it.
The actual metadata-only replay in readiness-haitian1 inventories 280 tasks: 100 for one pinned Haitian candidate and 180 AssemblyAI tasks, including cached comparator-only Latin/Sanskrit evidence. It reconstructs all 100 exact Haitian pairs across development and validation without claiming current candidate results for the other languages. Distinct source cohorts must still be reported separately when their populations differ. The inventory has 20 focused tests; the complete isolated app checkpoint in readiness-app-checks1 passes 220 tests, seven conditional model skips, typecheck, lint, 56 collector tests and five language-detector preparation tests. No shared service, provider request, model call or deployment is part of that check.
Language coverage alone is insufficient. Require representative independent accuracy comparisons, explicit and auto language behavior, word metadata and timestamp checks, and the exact Cap API/queue/persistence flow. Include accents, code switching, long recordings, poor audio and no-speech controls. Keep unresolved API options, language gaps, latency/capacity limits and fallback behavior explicit. A few good public clips do not establish all-language parity or authorize a production rollout.
Current retained benchmarks are summarized in BENCHMARK_STATUS.md. The new collection and runs are development evidence until separately reviewed; no production settings have changed.
- Collected 320 recordings across 80 newly targeted languages: 304 FLEURS clips and 16 OpenSLR clips. Six languages with earlier studies remain separate; 13 base languages still need source coverage, alongside regional-alias evidence.
- Completed 672 real API tasks on 168 distinct recordings, with identical audio/options for each provider. Sandchest completed 301/336, AssemblyAI 335/336. Correct automatic labels were 96/168 and 107/168. Two or four recordings per language expose gaps but cannot establish quality parity.
- All 336 local jobs had one attempt and exactly 301 unique usage events. Failed jobs were not billed. Every owned API/native process was stopped and checked absent. No shared database, Cap source or production service changed.
- Completed responses with encoding or word-metadata defects: Sandchest 0, AssemblyAI 20. These corpora have no manual word boundaries, so they do not prove acoustic timestamp accuracy.
- Scoring v2 retains actual completed text with replacement characters and separately flags defects. Original receipts and source snapshots remain intact.
screen1/report-v2.jsonandscreen2/report-v2.jsonsupersede earlier derived reports; the other reports already use v2. Re-scoring made no provider requests. - A private decoded-text repetition check recovered 12 of 26 baseline failures on the same 304 FLEURS audio/mode tasks: 290 completed, no word-structure defects, all 278 old completions retained, 277 with identical text and word fields. Auto correctness rose from 92 to 93/152. Several recovered transcripts still have high error; this is not an all-language quality or full-API reliability improvement claim. It is not promoted.
- Full Whisper large-v3 was tested on 36 clips across nine languages. It was slower, completed only 24/36 and had one replacement-character defect. Tamil improved from 42 to 19 word errors on 79 reference words, warranting more independent Tamil examples. No global model switch was made.
- The beam-width-10 experiment was rejected as invalid configuration: the pinned SDK supports at most eight decoders. It is not an accuracy result. Real non-English preflight now precedes private decoder runs.
- App checks: 189 tests passed, 7 conditional skips, typecheck and lint passed. The new collector has six passing offline tests, including changed/malformed ranges, transfer/deadline bounds, corrupt CRCs, source-format exclusions and Tibetan mapping. Private candidate builds passed 185 native tests plus their focused C++ tests.
See MULTILINGUAL_BENCHMARKS.md for all 99 language rows and explicit gaps. Private source manifests, per-case results, frozen code, comparison and process-exit receipts remain in .sandchest/benchmarks/multilingual-20260828. Candidate integration still needs manual timing, Cap-domain and fault regressions. No serving default, deployment or production traffic was changed.
- That historical baseline reached 904 real API tasks on 276 distinct recordings in 82 languages. The five-language expansion added 100 fresh recordings with prior source/audio/reference-text exclusions; Hawaiian and Eastern Yiddish added four speakers each. Six earlier language studies remain separate; 11 base languages lacked reference coverage at that checkpoint; the later studies below reduce this to five.
- Baseline completion is 357/452 versus AssemblyAI 451/452. Correct automatic labels are 96/176 versus 110/176. The expansion intentionally targets weaknesses, so these aggregate rates do not estimate production traffic reliability. There are 0 versus 42 completed responses with encoding/word-metadata defects.
- All 452 local jobs had one attempt and 357 unique usage events. Failed jobs were not billed. Every owned native/API process was stopped and checked absent; no shared DB, Cap source or production setting changed.
- A private Rust MMS ASR prototype passed five numerical comparisons against the reference implementation. It then returned nonempty text on all 100 fresh clips in Amharic, Assamese, Gujarati, Tamil and Telugu, with lower surface-text WER than the current model and AssemblyAI on this set. The largest provider gaps involve wrong-script output. These remain small development groups, not all-language quality guarantees.
- Full Whisper large-v3 completed all 20 fresh Tamil clips, scoring 133/299 word errors versus current Turbo 192/299, MMS 121/299 and AssemblyAI 135/299. Its median native HTTP time was 3.74 seconds; MMS raw-model timing excludes timestamps/API processing, so those timings are not equivalent service paths. Neither model is promoted.
- The current app source passed 189 tests, 7 conditional skips, typecheck and lint from an isolated credential-free snapshot. The final collector changes passed 14 offline tests, including a parent-path regression that failed before the fix. No SDK/deployment/native-serving rebuild is claimed for these collector-only changes.
The candidates still need word timing, formatting, automatic routing, long-audio and API integration, followed by Cap-domain and fault regressions. Detailed scores and every language row are in MULTILINGUAL_BENCHMARKS.md.
The private native adapter now adds a second verification layer to the reusable API harness. Keep these stages separate: a candidate can have good text but bad word times, and a structurally valid completed response can still have poor recognition.
- Freeze the model, binary, reference corpus, scorer and options before inference. Compare timing methods on the same recognition output; never supply gold text to ASR. Report matched-word coverage and use the same common words for timestamp comparisons.
- Replay short recordings exactly after adding windowing. Test repeated CTC labels across processing boundaries, final partial frames, finite scores and bounded allocations. Preserve text, confidence and word metadata through alignment.
- Add constructed long recordings with recorded source offsets and multiple boundary positions. Preserve float PCM when resampling manual-timestamp audio. Label composites as derived development regressions, not new speakers or natural long-form quality evidence. Compare WER and CER with separate-clip recognition and audit words spanning inserted pauses.
- Test digital silence, tones, invalid inputs, deadlines and cancellation. Record the stage reached; a timer firing during audio decoding does not prove inference cancellation. A partial run can supply diagnostic scores, but cannot pass the complete reliability gate. Confirm exact recovery and owned-process termination.
- Only then test the candidate through the existing upload/queue/API harness and actual Cap examples. Keep auto detection, punctuation, formatting and language-level coverage as explicit gates. Neither a successful candidate build nor declared model support changes the serving default.
The final studies completed116 timing cases and130 window/fault-study scoring cases, with186 library and11 focused native tests passing. All116 short outputs stayed exact. Digital-silence and cancellation fixes pass. That initial long-audio candidate regressed in four of five weak languages. The speech-partitioning follow-up below substantially reduces the regression, but manual timestamp evidence remains Spanish-only and full language-quality parity is still unproved. No additional AssemblyAI calls or production changes occurred. Full details and private receipt names are in MULTILINGUAL_BENCHMARKS.md.
The native candidate now partitions long audio at detected pauses while retaining every sample. The chosen private512ms policy passes a full130-case replay, including all116 exact short outputs, seven constructed long inputs, no-speech controls and eight failure/cancellation controls. Full long repeats verify that VAD state recovers after interruption. The observed words merged across inserted pauses are gone; reference scores and alignment coverage improve substantially, with remaining per-language regressions reported rather than hidden.
A separate natural-audio stress reuses three Cap recordings totaling18.31minutes. Both native variants complete and recover exactly. Provider-text agreement is reported separately from human-reference WER, and explicit use of cached language labels is not an auto-detection test. When a study requests word-only agreement, call the existing words and editDistance functions directly; do not calculate an unused character metric or weaken its size limit. All original attempts and scoring corrections remain auditable.
Latest receipts are mms-segmented-gap16-full, mms-cap-natural1 and mms-segmentation-checkpoint1.json. The512ms choice is the next integration candidate, not a deployed/default model or a claim of complete API/language parity. No additional provider calls, Cap service access or production changes occurred.
The five-language MMS route is now an opt-in native implementation, not only a prototype. The exact measured binary was tested through the isolated application API on100 explicit and100 unrestricted-auto jobs. Keep these as repeated development measurements; do not add them to independent sample counts. The detailed results and subsequent guarded-detector improvement are in MULTILINGUAL_BENCHMARKS.md.
For future candidate integrations, verify all build configurations, exact retained-route outputs with the feature both off and on, and native→API word/text equivalence. Keep automatic recognition separate from response language metadata: hints or fallback labels may change the reported label while recognition remains conditioned on the acoustic detector. A successful job or expected response label is not evidence that the correct recognizer ran. Use unrestricted auto measurements and private native diagnostics to audit actual routing.
Exercise crop offsets, format flags, literal custom spelling, silence, physical audio channels, invalid input, real deadlines and subsequent exact recovery. Assertion expectations must preserve the actual contract: punctuation=false removes Unicode edge punctuation/symbols; an invalid range can persist duration0 with no content or usage; successful stereo usage measures channel-duration while public duration remains the physical recording duration. Failed jobs never accrue usage. Keep failed harness runs and corrected checks separately, without changing application behavior to fit a mistaken test assumption.
The initial integration has100/100 explicit completions but only36/100 correct unrestricted language choices. The additional opt-in detector raises that to98/100 with100 completions on the same development set. Do not enable broad beta traffic from reused development results. Manual timestamp annotations and broader language/Cap validation are still required. The later frozen validation closes only the five-language unseen-to-study check; the latest source-coverage gap is five base languages.
Freeze the exact model revision, label order, aliases, numerical recipe, audio prefix and crop policy before inference. Preserve extra classes and raw confidence. Compare the native implementation against reference outputs and intermediate tensors before judging its language accuracy. The current native detector matches450 reference predictions; declared99-code coverage must not be counted as99 measured ASR quality results.
Use retained languages as negative routing controls, not only examples from the intended new routes. The current study includes229 additional retained controls and50Cap recordings; this caught Hindi/English-to-Gujarati errors that a weak-language-only evaluation would miss. Keep all these exposures in the development exclusion inventory. A published validation split that has been used to choose a threshold is not a fresh holdout. Never expose transcript text or manual boundaries to the detector or recognizer.
Then use the existing full API runner on explicit and unrestricted-auto modes, preserving source/model hashes, exact requests and returned model identities. Audit hinted/fallback response metadata separately from the acoustic recognizer. A fallback label does not bypass a requested confidence threshold. Verify new audio caching with crops and channel isolation, and repeat recovery after actual native deadlines.
Audit failure categories explicitly: no-spoken-audio and invalid-range failures preserve the existing0 duration; native inference and invalid-media failures retain null. All remain unbilled with no transcript content. Successful stereo usage is physical duration times channel count. Preserve failed test attempts and perform corrected read-back audits against immutable responses rather than changing application behavior to satisfy an incorrect expectation.
For reproducible assets, workers/inference-rs/tools/prepare_vox_lid.py converts only pinned local checkpoints in an isolated preparation environment. It rejects changed inputs, unexpected output bytes and existing destinations. Five focused offline tests and a real byte-identical conversion accompany the native/API measurements. Python is used for preparation and reference checks, never serving inference.
The guarded detector now has a fresh 400-job comparison on 100 previously unused published FLEURS test recordings (local validation), after source/text/canonical-PCM exclusions. Both providers completed every job. Auto labels are 97/100 versus 60/100; per-language recognition, wrong-script diagnostics, metadata defects and physical-duration strata remain separate. See the current validation results. Do not reuse these outcomes to tune the frozen policy and then continue calling them unseen validation.
The ML-SUPERB collector and reusable public plan/run/report commands completed 320 original-clip and 64 derived-control API tasks. Keep the two cohorts separate. All original-clip jobs completed; one derived Bashkir job failed after three configured queue attempts. The read-back audit confirms 191 completed jobs/usage events and no failed-job charge. Generic audit code must distinguish a client's forbidden blind POST replay from the application's bounded inference retries: a legitimate transient failure need not have exactly one queue attempt.
That initial screen had0/80correct local automatic labels. The subsequent ba/br/tt API integration improves explicit recognition and raises auto labels to15/80; Tatar/Nynorsk automatic selection remains weak. Future adaptive selection should prioritize these observed weaknesses and unmeasured development samples, never the fresh five-language validation references. At that checkpoint the Albanian/Malagasy study brought cross-study reference coverage to94/99, with five base codes unmeasured:fo/ht/la/sa/tk. The later Faroese study above raises coverage to95/99, without establishing quality parity. Source metadata leads remain in missing-language-source-scout4/remaining-language-leads.json; that older ledger predates the two new collections. No production setting or serving default changed.
The private four-language MMS experiment follows the same separation of concerns: pinned base/adapter identities, exact native-versus-CPU numerical comparisons, then96 fixed recognition inputs and invalid-input/exact-recovery controls. All 96 recognition inputs complete, but Nynorsk regresses on the longer derived cohort and Breton remains weak. Keep a language-specific decision; do not replace every recognizer because a pooled score improves. No new provider submissions or serving changes are involved.
The initial prototype exports CTC frames, not verified word timestamps. A successful math/recognition screen still needs automatic routing, acoustic alignment, retained-language/Cap regressions and an unseen evaluation before integration. Its 90–106ms native model timings are not the full API measurements. The original API's Bashkir failure and all comparative scores remain preserved beside the candidate results.
Recognition, normalization, word alignment and automatic detection are separate capabilities. The first ba/br/tt integration demonstrated why: the recognizer passed its standalone study and retained-route tests, but the aligner rejected its language identifiers when real API jobs arrived. Startup now validates the recognition-to-alignment mapping, with surface-script unit coverage, and three real new-route smoke requests precede the bulk run. Preserve stopped batches and their queued/failed jobs; do not recycle their runtime directory or resubmit ambiguous requests.
The repaired candidate passes192fresh local API jobs,386exact native regressions,104API controls including 50 Cap recordings,104idempotent replays and two real deadline/recovery checks. The historical auditor needed a read-back correction for zero-duration no-spoken-language errors; responses and application behavior were unchanged. The audit must follow the frozen application's failure categories, not assume every failure has null duration or every request is a new billable job.
Keep the detector handoff list independent from the ASR adapter list. The unchanged0.8confidence gate would incorrectly route five Bashkir clips to Tatar, so automatic Tatar handoffs remain disabled despite explicit Tatar's recognition gain. The selected ba/br handoffs show no false handoff on815development/control inputs, including derived and reused data; this is not a production error-rate estimate.
The initial 20 Gheg Albanian and 20 Plateau Malagasy recordings illustrated another distinction: collection/reference coverage is not a quality pass. All Albanian jobs completed, but word error remained high; seven Malagasy jobs failed in that baseline. A separate40-input detector screen identifies both labels correctly, which identified recognition/routing as the next experiment, now measured below. Keep the two/five publisher-speaker counts and dialect limitations visible. Those historical failures remain in their original scoring, and no model or threshold is adapted from protected validation/holdout references.
The subsequent native candidate improves Malagasy but regresses Albanian, so only Malagasy is integrated. Its numerical gate compares four CPU/native probes; the full development replay retains all 40 Albanian responses and completes all 40 Malagasy requests. Recognition, auto language selection, word structure and latency remain separate metrics. See the current results.
The reuse of the public collector and plan/run/report harness then supplies a frozen 20-recording validation, with a maximum of five clips per speaker and explicit source/text/audio/PCM exclusions. All four speakers also appear in development, and publisher train becomes local validation; neither fact is hidden by calling recordings unused. Both providers complete all 40 requests. The model and confidence threshold stay unchanged through collection, inference and scoring. Do not feed those protected references into later adaptive candidate selection.
Runtime verification adds 389 retained native comparisons, 112 API controls/replays and two real deadline recoveries. Before starting any future run, verify the collector's actual CLI (--input/--out here), claim the exact isolated runtime namespace, and confirm its database/output paths are absent. Preserve preflight failures separately; zero-data or zero-inference attempts are not completed experiments. Keep the tested binary, immutable source snapshot, model identities, raw responses and source-promotion receipt together. Once source is promoted, validate it against that immutable snapshot rather than treating the old mutable source baseline as current.
The corpus still has high absolute word error and no manual word boundaries. This closes a measured Malagasy recognition/routing gap, not every language's quality, timestamp or API-parity gate. That checkpoint had 94/99 coverage; the later Faroese collection raises cross-study coverage to95/99. The later Haitian Creole, Latin and Sanskrit API comparisons raise cross-study reference coverage to98/99, with substantial quality gaps still visible; Turkmen still needs verified reference access.
The current study uses the collector's validation role on 40 unused Bashkir/Tatar recordings after freezing model weights, routing policy and candidate code. Expired public audio signatures were refreshed only after verifying that the pinned source rows remained identical. Prior source IDs, normalized development text, audio hashes and canonical PCM were excluded. Publisher dev and local validation roles remain explicit; no speaker or model-training independence is claimed.
All 160 paired API tasks complete. Explicit/auto WER is 10.14%/32.61% for Bashkir and 24.03%/46.51% for Tatar, versus 98.55%/104.35% and 96.90%/106.98% for AssemblyAI on the same references. Correct automatic labels are 14/20 and 9/20 locally, versus 0/20 for each remotely. The policy was not retuned on these results. Recognition, label accuracy, structural word metadata and manual timestamp accuracy remain separate measures.
The first integration passed 517 native comparisons and 192 local development API jobs. The final candidate changes only diagnostic timing aggregation from that build; it passes 132 API compatibility cases/replays, two real deadline recoveries and the fresh validation. Interrupted test attempts remain preserved: one asserted the wrong diagnostic nesting, another compared fractional seconds to integer milliseconds with exact binary-float equality. Recovery audits retain all actual jobs and identify the one additional completed stereo probe rather than hiding it. All 29 offline collector tests pass. Final isolated application checks pass 189 tests with 7 conditional skips, typecheck, lint and 5 detector-preparation tests.
The code is integrated as an opt-in, with no default or deployment change. Detailed metrics and limitations are in the current study. After the later Faroese collection, four base languages remain unmeasured. Many measured languages still need larger comparisons, stronger recognition and manual timing evidence before an all-language beta claim.
The sixteen-language expansion uses the existing bounded archive collector to obtain20additional development recordings per language, excluding all64prior examples by source/text/audio/canonical PCM. The public planner freezes all320examples and identical options before inference. The real baseline has640API jobs; the candidate adds320local API jobs without resubmitting the already captured provider baseline. Audits preserve90baseline failures as unbilled outcomes and reconcile every successful duration/usage event.
Private analysis adds a reproducible paired-recording bootstrap:5,000resamples, fixed seed20260829, and the ratio of summed errors to summed reference units within each language. It streams immutable model-file hashes rather than loading multi-gigabyte buffers. Native recognition timing, full API latency, human WER/CER, metadata defects and automatic language detection remain separate. The full-worker/API comparison verifies exact raw word metadata, including speaker/channel fields; the scorer's normalized word representation alone cannot prove that.
Automatic routing is selected separately. The primary detector is checked against320new inputs plus815retained development/control predictions; related-language and provider-labelled controls are identified as such. A measured false handoff cannot be hidden by an aggregate recognition gain. Kannada/Uzbek require a second acoustic opinion after observed Malayalam/Tatar confusions; both raw thresholds remain0.8. The fixed confirmation screen accepts39/40new qualifying positives and rejects both retained false positives. Full serving verification must follow; classifier eligibility is not API parity.
Do not use lexical scores to approve multilingual punctuation, number formatting, fillers or manual word timestamps. The new CTC/ICU word-grouping tests check token/byte reconstruction and actual frame boundaries across windows; no synthetic test or provider timestamp is human timing gold. See the completed explicit API study for current measurements and pending gates.