Fix/hf offline resilient loading#526
Conversation
wilke0818
commented
Jun 30, 2026
- HF models create 429 errors when loading too many, especially when parallelizing across nodes and calling many models sequentially
- Fixes base logic for this
- Should extend across all models/HF spaces if not already done
…#510) * Add analyze_audio.py: multi-task audio analysis with cache + LS export Single-command developer script that runs senselab's full task suite (diarization, AST, YAMNet, multi-backend features incl. squim quality metrics, ASR, speaker embeddings) on an input audio file with and without speech enhancement. Multiple models per task; AST and YAMNet each driven at their native temporal resolution. Output layout under artifacts/analyze_audio/<stem>_<timestamp>/: - summary.json + per-(pass × task × model) JSON files - labelstudio_tasks.json + labelstudio_config.xml for direct LS import (one task per audio variant; one timeline track per analyzer × model) Content-addressable cache under artifacts/analyze_audio_cache/ keyed by sha256(audio_signature, task, model, params, wrapper_hash, senselab_ver, schema_ver). Re-running with identical inputs replays prior outcomes without invoking models. Per-model failures are captured in their own JSON without aborting the run. Forced-alignment auto-pairing for timestamp-less ASR models, the multilingual aligner, NeMo Canary-Qwen, and Alibaba Qwen3-ASR backends land in follow-up commits on this branch (see specs/20260506-154425-audio-analysis-asr-extensions/tasks.md). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add uroman to nlp extra for forced-alignment ja/zh romanization uroman (MIT, pure-Python, ~1.3.1.1) is needed by the upcoming MMS forced-alignment backend to romanize Japanese and Chinese transcripts before MMS's Roman-character tokenizer sees them. Lives in the optional [nlp] extra so default `uv sync` is unaffected. The MMS aligner imports uroman lazily, only when the language is ja or zh — users who never align in those languages don't need the extra. Verified default `uv sync` does not pull uroman; `uv sync --extra nlp` does. uv.lock updated to record the new package; existing transitive deps are unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Foundational: cache schema v2 + ASR dispatch table for new backends scripts/analyze_audio.py: - Bump _CACHE_SCHEMA_VERSION from 1 to 2. Comment documents that v2 introduces alignment as a separate cache entry from the ASR call that produced its input transcript. v1 entries are inert and can be removed. src/senselab/audio/tasks/speech_to_text/api.py: - Add three new module-level prefix tables for upcoming backends: _CANARY_PREFIXES = ("nvidia/canary-",) # NeMo SALM via subprocess venv _QWEN_ASR_PREFIXES = ("Qwen/Qwen3-ASR",) # qwen-asr wrapper via subprocess venv _TIMESTAMP_LESS_HF_MODELS = ("ibm-granite/granite-speech-",) - Add stub dispatch branches for Canary and Qwen-ASR that raise NotImplementedError pointing at the follow-up tasks (T025-T027 for Canary, T030-T032 for Qwen). Lands the routing structure now so the modules can be filled in independently without re-touching api.py. - For models matching _TIMESTAMP_LESS_HF_MODELS, default return_timestamps=False so the HF pipeline does not raise. Callers can still override explicitly. This unblocks IBM Granite Speech 3.3 through the existing huggingface.py path (the analyze_audio script then auto-aligns via the upcoming MMS forced-alignment backend). Dispatch order: NeMo (existing) -> Canary (stub) -> Qwen-ASR (stub) -> HF pipeline (default with timestamp-less knowledge). The four prefix groups are disjoint. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Script: separable ASR/alignment cache + auto-align CLI flags + smoke tests Three-phase update to scripts/analyze_audio.py landing US1+US2+US3 script-side scope: US1 (one-command E2E): no behavior change; smoke tests in src/tests/scripts/analyze_audio_test.py exercise argparse defaults, audio_signature stability, the cache-key composition contract, the LS label-collection helper, and the serializer. 12 tests, no model loads, runs in <1 s on default install. US2 (re-run without re-computation): - Add transcript_signature(text), align_cache_key(...), and run_alignment_cached(...) — independent caching for the alignment step that mirrors run_task_cached but keys on (audio_signature, transcript_sha, language, aligner_model, aligner_params, wrapper, senselab_ver). - Failed alignments are NOT stored, so a future fix to the aligner / senselab upgrade triggers a fresh attempt; the parent ASR cache is unaffected per FR-024 / FR-025 / FR-026. US3 (hierarchical LS export): - Three-case LS-export branch for ASR: (a) ASR with native timestamps -> per-segment regions from the ASR result; (b) ASR text-only with a successful alignment outcome attached -> per-segment regions from alignment; (c) ASR text-only with no/failed alignment -> single full-audio TextArea region (graceful degradation per FR-025). - Add _asr_has_timestamps() helper used by the LS export and (eventually) by the auto-align stage to short-circuit cases where alignment is unnecessary. CLI surface (per contracts/cli.md, wired to the upcoming auto-align stage that lands with US6): - --no-align-asr (toggle off auto-align) - --aligner-model (default facebook/mms-1b-all) - --asr-language (force a specific language for the aligner) - --qwen-asr-no-timestamps (skip Qwen's bundled forced-aligner) - 'alignment' added to --skip choices. Default --asr-models updated to include ibm-granite/granite-speech-3.3-8b back into the comparison set; senselab's HF-pipeline path now defaults return_timestamps=False for that model id (committed in da04a29), and the upcoming MMS auto-align stage will add per-segment timestamps. The auto-align stage itself is not yet wired (T021 lands with the MMS backend in US6). Until then, --no-align-asr / --aligner-model / --asr-language are accepted but unused. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * forced_alignment: MMS multilingual aligner backend (1100+ languages) Adds Meta's MMS (Massively Multilingual Speech) as an opt-in aligner backend for senselab.audio.tasks.forced_alignment, covering 1100+ languages via per-language adapters on a single base model. Selected via the new ``align_transcriptions(..., aligner_model=...)`` keyword; the legacy per-language wav2vec2 path stays the default for existing callers (no behavior change for current code). constants.py: - MMS_MODEL_ID = "facebook/mms-1b-all" (CC-BY-NC 4.0 weights; override the model id for commercial-friendly alternatives). - ISO_1_TO_3 map for adapter selection (en, fr, de, es, pt, it, ja, zh, plus the 19 other languages already in DEFAULT_ALIGN_MODELS_HF, which MMS now also covers). forced_alignment.py: - _romanize_for_mms(text): lazy uroman import (optional dep, [nlp] extra) for ja/zh transcripts. Without uroman installed, raises an actionable ImportError pointing at `uv sync --extra nlp`. - _load_mms_aligner(iso3, device, cache): one-time load of the MMS base model + selected per-language adapter via processor.tokenizer .set_target_lang(iso3) + model.load_adapter(iso3). Cached by (model_id, iso3) so multiple calls on the same audio reuse weights. - align_transcriptions: new ``aligner_model`` kwarg. When set to MMS_MODEL_ID, dispatches to the MMS path; ja/zh transcripts go through _romanize_for_mms before tokenization. Uses model_type= "huggingface" internally — the wav2vec2 forward pass and CTC trellis/backtrack code is identical, so no changes to _get_prediction_matrix / _align_transcription / _assign_timestamps. Output shape unchanged: List[List[ScriptLine | None]]. Existing 15 forced_alignment tests still pass (run via uv run pytest src/tests/audio/tasks/forced_alignment_test.py). speech_to_text/huggingface.py: - _get_hf_asr_pipeline and transcribe_audios_with_transformers now accept ``return_timestamps: Union[str, bool, None]`` (was Optional[str]). HuggingFace's automatic-speech-recognition pipeline supports False to disable timestamps; the existing senselab dispatcher (api.py, committed earlier) defaults this to False for known-timestamp-less HF models like ibm-granite/granite-speech-3.3-8b. Public docstring updated to call out this case. This commit lands T016-T020 from the spec-kit task plan. Next: T021 wires the analyze_audio script's auto-align stage to call align_transcriptions(..., aligner_model=facebook/mms-1b-all) on every timestamp-less ASR output. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * analyze_audio: wire MMS auto-align stage; HF return_timestamps=False fix scripts/analyze_audio.py: - Wire the auto-align stage in run_pass(): after the ASR family runs, iterate every successful ASR ModelRun. Skip when --no-align-asr is set, when 'alignment' is in --skip, or when the ASR result already has native timestamps (Whisper-style start/end or word chunks). Otherwise call senselab.audio.tasks.forced_alignment.align_transcriptions via run_alignment_cached(), with the resolved language (from --asr-language, default "en"), the chosen aligner_model (default facebook/mms-1b-all), and an alignment-specific provenance block (transcript_sha, language, parent_asr_cache_key) per FR-026. - Imports forced_alignment.align_transcriptions and the Language / ScriptLine types from senselab.utils.data_structures. - Add _extract_transcript_text() helper that handles both ScriptLine objects (post-fresh-ASR) and the dict shape (post-cache-hit) so the auto-align step works after either kind of ASR outcome. - Update _asr_has_timestamps() to handle dict-shape ScriptLines too. - Refresh the script docstring's ASR section: Granite is now in defaults (auto-aligned), removing the older "intentionally NOT in defaults" note that the spec clarification overrode. Add a paragraph describing the auto-align stage and the --no-align-asr toggle. src/senselab/audio/tasks/speech_to_text/huggingface.py: - Fix _get_hf_asr_pipeline to OMIT return_timestamps from the kwargs passed to transformers.pipeline() when the caller asked for False. HF's AutomaticSpeechRecognitionPipeline behaves differently per model family: CTC raises on False (only "char"/"word" allowed), Whisper accepts False, and other generative seq2seq (Granite Speech) raises on True/word/segment. Omitting the kwarg lets each pipeline default sanely; omitting only happens when the caller explicitly passed False (timestamps were not asked for). Other values (str/None/True) are passed through unchanged. src/tests/audio/tasks/forced_alignment/mms_test.py: - New skipif-guarded smoke tests for the MMS aligner. Skipif checks whether facebook/mms-1b-all is in the local HF cache (~1.6 GB download otherwise; default CI install does not pull it). When available: verify (a) English alignment produces well-shaped ScriptLines via the real-speech fixture, (b) Japanese alignment exercises the uroman romanization path, (c) an unknown language code raises a clear ValueError pointing at the ISO_1_TO_3 registry. Assertions are SHAPE-ONLY — synthetic transcripts on real audio do not align meaningfully but the API contract is still verified. src/tests/audio/tasks/speech_to_text/huggingface_no_timestamps_test.py: - New skipif-guarded smoke test for return_timestamps=False. Uses facebook/wav2vec2-base-960h (already cached locally for the existing English aligner path) rather than Granite Speech (multi-GB). Verifies: when return_timestamps=False, transcribe_audios_with_ transformers returns text-only ScriptLines without raising the HF pipeline's CTC-rejects-False ValueError. Existing 11 senselab speech_to_text_test.py tests still pass; existing 15 forced_alignment_test.py tests still pass. This commit lands T021-T023. T024 (end-to-end Granite + Whisper validation through the analyze_audio script with real model downloads) is deferred to a follow-up session — needs ~16 GB Granite Speech download plus ~1.6 GB MMS download. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * speech_to_text: NVIDIA Canary-Qwen 2.5B subprocess-venv backend Adds CanaryQwenASR which loads SALM (Speech-Augmented Language Model) from nemo.collections.speechlm2.models in an isolated nemo-canary-qwen venv (kept separate from the existing nemo-diarization venv to avoid destabilizing Sortformer/Conformer-CTC paths). The chat-style prompt format using model.audio_locator_tag matches the published model card. Canary-Qwen is text-only (no native timestamps); the analyze_audio script's auto-align stage (US6) now adds per-segment timing via the multilingual MMS forced-aligner downstream when this model is used. Wires the api.py dispatch (replaces the NotImplementedError stub) and adds a shape-only smoke test that auto-skips when the venv has not been provisioned (~5 GB first-run install). Marks T025-T028 done in specs tasks.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * speech_to_text: Alibaba Qwen3-ASR subprocess-venv backend Adds QwenASR which loads Qwen3-ASR via Alibaba's qwen-asr Python wrapper (Qwen3ASRModel.from_pretrained) inside an isolated qwen-asr venv, optionally pairing with the bundled Qwen3-ForcedAligner-0.6B companion to produce per-word / per-CJK-char timestamps. Wires api.py dispatch (replaces the NotImplementedError stub) and threads --qwen-asr-no-timestamps from analyze_audio.py through to the ASR call so the script's MMS auto-align stage can take over when the bundled aligner is undesirable. Adds shape-only smoke tests that auto-skip until the qwen-asr venv has been provisioned. Marks T030-T033 done in specs tasks.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * polish: lint cleanup + mark Phase 9 tasks done - Add docstring to src/tests/scripts/__init__.py (D104). - Mark T035-T037 done in specs tasks.md after running: * full ruff check + mypy on changed paths (clean) * full senselab pytest suite excluding scripts/ (421 passed, 25 skipped, no regressions; voice_cloning_test ERROR in bulk run is a known order-dependent flake — passes in isolation, unrelated to forced_alignment / speech_to_text changes). T035 (CLAUDE.md update) is applied to the local file; CLAUDE.md is not yet under version control in this repo so the section addition lives in the working tree only — user decides whether to track it. T038 (final E2E validation against a tutorial WAV) intentionally deferred: requires the nemo-canary-qwen and qwen-asr venvs (~5-15 GB combined) and the MMS aligner (~1.6 GB) to be provisioned locally. The skipif-guarded smoke tests ride the venvs whenever the user provisions them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * e2e validation: fix bugs surfaced by full-pipeline run Validates T012/T015/T015a/T024/T029/T034/T038 end-to-end. Bugs found and fixed during the validation cycle: - canary_qwen.py worker: SALM.generate expects prompts as list[list[dict]] (a batch of conversations), not list[dict]. Fixed. Canary-Qwen now succeeds in ~70s on a 38s clip. - forced_alignment.py: senselab's Language normalizes ISO-639-1 ('en') to ISO-639-3 ('eng'). The MMS aligner now accepts either form. - script_line.py from_dict: three pre-existing latent bugs surfaced by MMS-aligner output: * d["chunks"][0] accessed without checking non-empty (IndexError) * d["timestamps"][0] accessed without checking length >= 2 (IndexError) * recursive construction of empty subsegments with text="" failing the "at least text or speaker" pydantic validator Empty subsegments are now filtered before recursive construction. - analyze_audio.py LS export: pre-existing bug where getattr(seg, "start") returned None for cache-restored dicts, silently dropping all regions on cache-hit runs. Added _seg_attr helper that handles both Pydantic objects and JSON dicts. Also unwrap align_transcriptions' List[List[ScriptLine]] to inner segment list before passing to _asr_to_ls. - mms_test: align test expectation with senselab.Language's stricter ISO validation (Language("xx") raises before reaching MMS). Validation results in artifacts/analyze_audio_e2e_validation.md (not committed). Marks T012/T015/T015a/T024/T029/T034/T038 done. The two known gaps documented in the validation note: YAMNet (corrupt local tfhub cache, environmental) and Granite Speech 3.3 (HF pipeline contract mismatch — needs a dedicated subprocess-venv backend in a follow-up). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * speech_to_text: IBM Granite Speech 3.3 in-process backend Adds GraniteSpeechASR which loads ``GraniteSpeechProcessor`` and ``AutoModelForSpeechSeq2Seq`` directly in-process — no subprocess venv needed because transformers, torch, torchaudio, and peft all live in senselab's core install (peft is required for Granite's LoRA adapter; loading without it produces gibberish). Why a dedicated backend rather than the HF ASR pipeline path: Granite uses ``GraniteSpeechProcessor(text, audio, device, **kwargs)`` and a ``GraniteSpeechFeatureExtractor`` whose ``__call__(audios, device)`` signature does not accept the ``{array, sampling_rate}`` audio dict that ``AutomaticSpeechRecognitionPipeline`` passes in. Routing through the pipeline failed at the feature-extractor boundary in earlier E2E runs. Worker subtleties surfaced by the E2E: - ``return_tensors="pt"`` is mandatory; without it Granite's processor returns ``input_ids`` / ``attention_mask`` as nested Python lists, which the underlying ``generate()`` then tries to call ``.shape`` on. - The feature extractor expects a 1-D audio tensor; senselab's ``Audio.waveform`` is shape ``[channels, T]``, so we squeeze after downmix. - bfloat16 on accelerators (CUDA + MPS), float32 on CPU. CPU bf16 runs but is ~40x slower than fp32 because PyTorch's CPU kernels lack a native bf16 SIMD path. MPS bf16 was verified against torch 2.11. Adds ``peft>=0.13`` to senselab's core deps. Wires api.py dispatch (replaces the timestamp-less HF route), drops Granite from ``_TIMESTAMP_LESS_HF_MODELS``, and adds a shape-only smoke test that auto-skips when the model weights are not in the local HF cache (~16 GB). Routed automatically by ``transcribe_audios`` when the model id starts with ``ibm-granite/granite-speech-``. Granite is text-only; the analyze_audio script's auto-align stage adds per-segment timestamps via the multilingual MMS forced-aligner downstream. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * PR review: type annotations, MMS aligner cache, ruff format Addresses pre-commit failures and Gemini's review feedback on PR #510: - Test files: add ``model: HFModel`` annotations on canary_qwen_test, qwen_test, granite_test, huggingface_no_timestamps_test (mypy 1.20.1 is stricter than 1.18.x and infers Self<HFModel> when the literal HFModel(...) call is on the RHS — explicit annotation pins the type). - ``forced_alignment.align_transcriptions``: hoist the processor/model cache from a per-call local dict to a module-level ``_ALIGNER_CACHE``. The 1B-parameter MMS base weights (and any per-language wav2vec2 variants) now load once per process instead of once per call — fixes a perf cliff Gemini flagged in the review. - ruff format: pick up the auto-applied formatting in scripts/generate_model_registry.py + scripts/profile_model_tiers.py (no functional changes; ruff 0.15.11 vs older local), plus a small re-flow in script_line.py from_dict and granite.py. Did NOT apply Gemini's two other suggestions: - Drop ``device=`` arg in Granite ``processor(...)`` call: verified Granite's ``GraniteSpeechProcessor.__call__`` accepts ``device`` explicitly in its signature; works on MPS in run 15. - Drop ``input_len`` slicing on generate output: empirically verified via standalone MPS run that with slicing the transcript decodes correctly (``"so once upon a time there were four little rabbits..."``); without slicing it would prepend the prompt. Granite is an LLM-style multimodal generator, not a true seq2seq. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * analyze_audio: per-window features (1s/0.5s default), parquet output Replaces the single global summary that the script was emitting for opensmile / parselmouth / torchaudio-squim with proper time-series output. The rest of the analysis (diarization, AST, YAMNet, ASR, alignment, embeddings) is already temporal — features should be too. Routing: - opensmile uses ``LowLevelDescriptors`` (native ~10 ms frame grid). Run once; one parquet row per opensmile frame. - parselmouth + torchaudio-squim are summary statistics by design; the senselab wrapper only exposes the aggregate form, so these run in an external sliding window driven by the new ``--features-win-length`` (default 1.0 s) and ``--features-hop-length`` (default 0.5 s) flags. One parquet row per window. Each backend writes its own parquet sidecar (``<pass>/features/<backend>.parquet``) — different columns and different time grids don't share a schema; per-backend files keep each grid clean and let downstream readers join on ``start``/``end`` if they want a unified view. The cache outcome metadata stays in JSON for inspection parity with the other tasks (``features.json`` now points to the parquet directory). Verified end-to-end on a 38.4 s clip (run 16, both passes): - opensmile.parquet: (3837, 27), 668 KB, 0–38.411 s - parselmouth.parquet: (75, 43), 49 KB, 0–38 s - torchaudio_squim.parquet: (75, 6), 6.5 KB, 0–38 s Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * PR review: pre-commit fixes (codespell + pretty-format-toml) - codespell: add ``vie`` (Vietnamese ISO-639-3 code in ISO_1_TO_3 registry, not the English noun) to ignore-words-list. - pretty-format-toml: re-flow [tool.codespell] block so the ``vie`` addition lands on a properly-formatted line. No code changes; all green pre-commit locally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* analyze_audio: foundational comparator framework + CLI flags
Lays the foundation for the comparison & uncertainty stage. Adds the
new CLI flags, cache key composition, ComparisonRow column lists,
parquet writer with comparator_provenance metadata, ComparisonGrid
dataclass + iter_buckets, _token_overlaps_window helper, uncertainty
aggregator (min/mean/harmonic_mean/disagreement_weighted), LS XML
extension hooks for comparator tracks, and disagreements.json
index builder.
This commit alone produces no reviewer-visible output — the
user-story phases (US1–US4) plug differencer logic into this
plumbing in subsequent commits.
Constitution checks:
- §VIII (No Hardcoded Parameters): every comparator threshold and
grid resolution is a CLI flag; defaults documented.
- §IV (CI Must Stay Green): existing 12 script tests still pass;
11 new comparator-foundational tests added (23 total, all green).
- §VII (Simplicity First): comparator code lives inline in
analyze_audio.py; no new senselab task module needed.
Adds g2p-en to the [nlp] extra (small wheel; needed by the future
US3 ASR↔PPG comparator). NLTK averaged_perceptron_tagger_eng
auto-downloads on first use.
Bumps cache schema 2 → 3 to reflect the new comparator key shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* analyze_audio: US1 raw-vs-enhanced comparator + tests
Adds the per-task differencer registry (_diff_diarization,
_diff_classification, _diff_asr) and the top-level
compare_raw_vs_enhanced driver. Wires it into main() so a default
two-pass run now emits parquet sidecars under
<run_dir>/comparisons/raw_vs_enhanced/<task>/<model>.parquet plus a
disagreements.json index ranking the top-N flagged regions.
LS bundle additions: one Labels track per (task, model) actually
compared, named pass_pair__compare__<task>__<model> with the fixed
enum {agree, disagree, incomparable, one_sided}; ASR-vs-ASR pairs
also get a sibling TextArea track carrying WER + both transcripts.
Track names are appended via build_labelstudio_config(... ,
comparator_tracks=tracks); a run with --skip comparisons produces
bit-identical output to current main (FR-005, SC-005 anchored by
test_comparator_skip_no_op_preserves_ls_bundle_shape).
Cache key composition follows the comparison_cache_key helper from
Phase 2, using the upstream-task cache_keys so re-running an
upstream task cleanly invalidates downstream comparisons.
Tests added (29 total now in src/tests/scripts/, all green):
- test_raw_vs_enhanced_diarization_diff covers boundary_shift
- test_raw_vs_enhanced_asr_text_diff covers per-window WER + a/b text
- test_raw_vs_enhanced_handles_failed_pass covers the incomparable
edge case from FR-010 / US1 acceptance scenario 3
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* analyze_audio: US2 within-stream comparator + tests
Adds compare_within_stream that, for each task with two or more
successful models in a pass, iterates itertools.combinations and
emits one parquet per ordered model pair under
<run_dir>/<pass>/comparisons/<task>/<a>_vs_<b>.parquet (US2 layout
in contracts/comparison-row.parquet.md). For ASR pairs the WER row
schema is reused; for diarization the speaks_a/speaks_b shape; for
classification (AST vs YAMNet) the top1/score/entropy shape, with
the existing scene_agreement.json semantics preserved as a strict
subset.
LS bundle: one Labels track per (pass, task, model_pair) with the
fixed enum {agree, disagree, incomparable, one_sided}; ASR pairs
also emit a sibling TextArea track carrying WER + both transcripts.
ASR-vs-ASR honors --asr-reference-model (default Whisper turbo) so
the soft-reference side is consistent across runs.
Tests added (29 total now, all green):
- test_within_stream_asr_pair_emits_wer covers the WER per-bucket path
- test_within_stream_single_model_no_op covers the graceful no-op
when a task has only one successful model
- test_within_stream_classification_superset_of_scene_agreement
covers the AST-vs-YAMNet matching-grid path
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* analyze_audio: US3 cross-stream comparator + tests
Adds compare_cross_stream and three sub-comparators:
- _diff_asr_vs_diarization: per cross-stream-grid bucket, sets
asr_says_speech via _token_overlaps_window (Q4 clarification:
any transcript token whose timestamp range overlaps the bucket
counts) and diar_says_speech via diarization-segment overlap.
- _diff_classification_vs_diarization: per bucket, picks the
AST/YAMNet window whose center is closest to the bucket center
and asks whether its top-1 label is in the configurable
--speech-presence-labels allowlist (default: AudioSet 'Speech'
subtree); compares to diarization the same way.
- _diff_asr_vs_ppg: per bucket, derives ARPAbet phonemes from the
ASR transcript via g2p_en, argmaxes the PPG output frames, and
reports a continuous phoneme_per (jiwer.wer) plus a boolean
phoneme_disagreement flag at >= --phoneme-disagreement-threshold
(default 0.50; Q5 two-tier clarification).
PPG path gracefully degrades when no PPG result is available —
records "PPG backend not provisioned" in disagreements.json's
incomparable_reasons rather than crashing (FR-010, US3 acceptance
scenario 4).
LS bundle: one Labels track per (pass, stream_pair, model_pair)
with the fixed enum {agree, disagree, incomparable, one_sided}.
Tests added (37 total now in src/tests/scripts/, all green):
- test_cross_stream_asr_speech_in_silent_window
- test_cross_stream_classification_speech_allowlist
- test_cross_stream_ppg_unavailable_degrades_gracefully
- test_cross_stream_phoneme_per_two_tier (continuous PER + boolean flag)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* analyze_audio: US4 per-region uncertainty + ranked disagreements.json
Adds per-region confidence harvesters and threads combined_uncertainty
through every comparator emitted by the previous phases:
- _whisper_chunk_confidence converts Whisper's per-chunk
avg_logprob -> exp(avg_logprob) confidence in [0, 1] and surfaces
no_speech_prob alongside.
- _whisper_bucket_confidence averages overlapping chunks within a
comparison-grid bucket (FR-007: arithmetic mean of native events
inside one bucket).
- _enrich_diff_asr_with_confidence + _enrich_diff_classification_with_confidence
populate confidence_a/b and combined_uncertainty in-place on the
rows produced by the existing differencers.
- _models_without_native_confidence walks the summary and lists the
pyannote/Sortformer/Granite/Canary-Qwen/Qwen3-ASR backends — these
contribute to comparisons but lack a native scalar in v1, per
research.md §2.
The disagreements.json index now ranks regions by combined_uncertainty
descending (NaN-last). The aggregator is the configurable
--uncertainty-aggregator (default min — most-doubtful model wins;
mean / harmonic_mean / disagreement_weighted available). Per FR-007
the missing-confidence model list rides on the index for
transparency.
Tests added (38 total now in src/tests/scripts/, all green):
- test_uncertainty_whisper_avg_logprob_extracted
- test_uncertainty_aggregator_min_default_via_compare (end-to-end:
build two ASR results with different avg_logprob, verify the
parquet's combined_uncertainty matches 1 - min(...))
- test_uncertainty_missing_signal_dropped_not_zeroed
- test_disagreements_index_top_n_and_ordering
- test_models_without_native_confidence_lists_known_null
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* analyze_audio: aligned timeline plot
Adds build_aligned_timeline_plot which reads a finished run dir and
emits <run_dir>/timeline.png — a stacked, time-aligned figure
showing diarization speaker bars per (pass, model), ASR token
spans per (pass, model), AST/YAMNet scene-window strips colored
by top-1 score, and one comparator row per parquet with
disagreement bars colored by combined_uncertainty (Reds; deeper red
= higher uncertainty).
Hooked into main() at the very end of a run (after disagreements.json
is written) so every default invocation now produces the figure
alongside the existing parquets / JSON. Best-effort: a plot failure
is logged to stderr and the rest of the outputs are unaffected.
Plot row layout, top-down:
- One row per (pass, diarization_model) — speaker-colored bars.
- One row per (pass, asr_model) — token spans (filled bars from
chunk timestamps, or a single span for text-only ASR after
auto-align).
- One row per (pass, scene-classifier) — score-shaded windows.
- One row per comparator parquet — disagreement bars; comparator
agreement is rendered as the empty grey background.
Constitution checks:
- §VII (Simplicity First): one new function, no new module; uses
matplotlib (already pulled in by pyannote-audio).
- §VIII (No Hardcoded Parameters): figure size scales with
duration; no hardcoded paths beyond the conventional run_dir.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* polish: mark Phase 7 tasks done
T052-T058 marked complete in tasks.md after full E2E validation.
E2E (run twin-1_20260509-032631):
- All 13 tasks succeed across both raw_16k + enhanced_16k passes
- 38 comparator parquets emitted (within_stream / cross_stream /
raw_vs_enhanced)
- disagreements.json: 16,513 rows, 10,133 disagreements, top-100
- timeline.png: aligned multi-row plot rendered
Senselab full suite: 399 passed, 1 failed (the same voice_cloning
order-flake from PR #510; passes in isolation; unrelated to
comparator). SC-005 (no regression in existing senselab outputs)
holds.
Smoke tests: 38 passed, 7 skipped.
Known v1 gap documented in artifacts/compare_uncertainty_e2e_validation.md:
the senselab HuggingFaceASR wrapper does not currently surface
Whisper's avg_logprob, so the disagreements.json ranking has only
NaN combined_uncertainty values across the whole run. The harvester
is correct — it has nothing to read. Follow-up PR will opt the HF
pipeline into per-chunk avg_logprob.
T059 (PR open) intentionally left unchecked — pending a fresh push.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* polish: mark T059 (PR opened) done
PR #512 opened against alpha:
#512
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* PR review: pre-commit fix (mypy 1.20 stricter type inference)
CI's mypy 1.20.1 flagged two var-annotated diagnostics that the
local mypy 1.18.x missed:
- test_comparator_skip_no_op_preserves_ls_bundle_shape: annotate
``summary: dict[str, Any]`` so the empty-dict literal doesn't
collapse to ``dict[str, dict[str, dict[Never, Never]]]``.
- test_comparison_cache_key_changes_with_inputs: convert the
``base = dict(...)`` factory call to a literal ``{...}`` with
an explicit ``dict[str, Any]`` annotation so the inferred value
type accepts the heterogeneous values (str / list / dict).
Pull the unpacked-params expression out into a local so mypy
doesn't try to type-check ``{**base["params"], ...}`` against
the wrong base inference.
No code changes; all pre-commit hooks pass locally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* PR #512 review: address Gemini findings
Applies the 5 actionable Gemini comments on the comparator stage:
- HIGH (line 1526): compare_raw_vs_enhanced now also runs for AST and
YAMNet — they're top-level summary keys (single-model tasks) rather
than under by_model. Their parquet rows carry real time spans driven
by the per-task win/hop and combined_uncertainty derived from the
top-1 score via _enrich_diff_classification_with_confidence.
- MEDIUM (line 725): _enrich_diff_asr_with_confidence now passes the
per-row WER into _aggregate_uncertainty as ``mismatch_severity`` so
the disagreement_weighted aggregator actually scales with the
textual disagreement (matches FR-003 intent; otherwise it
collapsed to ``mean``).
- MEDIUM (line 991/995): build_aligned_timeline_plot now reads AST
and YAMNet win/hop from ``summary["scene_window"]`` rather than
hardcoded 10.24/0.96 constants. CLI overrides surface in the plot
correctly.
- MEDIUM (line 1316): _diff_classification now uses ``win_length_a``
+ new ``hop_length_a`` to compute real per-row start/end timestamps
rather than integer placeholders. Both call sites
(raw_vs_enhanced AST/YAMNet and within_stream AST-vs-YAMNet) pass
the explicit hop.
- MEDIUM (line 2234): _whisper_bucket_confidence now falls back to
the segment-level avg_logprob when the parent ScriptLine has no
chunks — covers post-aligned text-only ASR backends like Granite
/ Canary-Qwen if they ever surface a per-segment scalar in a
future senselab release.
Also tightens the matplotlib colormap usage to mypy 1.20-compliant
``matplotlib.colormaps.get_cmap`` form (was ``plt.cm.<name>`` which
mypy now flags as attr-defined).
Did NOT change: NLTK runtime download (line 1914). The download
is a one-time cmudict + averaged_perceptron_tagger_eng fetch
(small files); air-gapped envs can pre-populate ~/nltk_data via the
existing ``T001`` setup task before running. Will revisit if a
follow-up requires fully offline operation.
All 38 script smoke tests still pass; mypy + ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* audio_analysis workflow: speech-aware speaker clustering + 3-axis uncertainty
The audio_analysis workflow lands as a standalone module under
src/senselab/audio/workflows/audio_analysis/ with the analyze_audio script
as a thin wrapper.
Highlights from this iteration:
- Speech-aware embedding clustering: a YAMNet → AST → openSMILE-loudness
fallback mask is built per embedding window; non-speech windows are tagged
NOISE and skip clustering, so silent / music / vehicle stretches no longer
inflate the speaker count.
- Spectral clustering on a precomputed cosine-similarity affinity is now
the default, with k-means as the per-k fallback. New CLI flag
--clustering-algorithm exposes the choice.
- Min-cluster-size guard (max(5, 10% of valid windows)) rejects partitions
where a 2-3 window outlier is propping up the silhouette score.
- Cross-model speaker unification (clustering.py + plot.py) processes the
synthetic embedding source first to seed the centroid pool, then freezes
the pool — so a noisy 3-segment pyannote / sortformer label can't spawn a
unified third "speaker" beyond what the embedding source validated.
Validation: twin-1.wav unified-speaker count drops from 3 → 2; English
two-speaker conversation stays at 2 (no regression). 57/57 workflow tests
pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* PR review: pre-commit fixes (mypy annotations + test naming + toml format)
- global_summary.py: explicit `list[float]` annotations on stoi/pesq/sisdr accumulators
- test_aggregate.py: `dict[str, dict[str, Any]]` annotations on votes literals
(mypy 1.20 narrows mixed-value literals to dict[str, object])
- Rename test_*.py → *_test.py to match repo convention
- pyproject.toml: pretty-format-toml auto-fix
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* PR review: address 6 findings + same-speaker post-merge step
Review findings addressed:
- HIGH: ``min_cluster_size`` floor was 5 windows but ``min_windows_for_clustering``
is 4 — for passes with 4-9 valid windows every k≥2 partition got rejected
silently and we fell to single-cluster regime with no diagnostic. Lowered
floor to ``max(2, round(0.10 * n))`` so the gate is reachable; record a
``failures`` entry when all partitions are rejected by min-size.
- HIGH: ``compute_uncertainty_axes`` mutates the caller's ``passes`` dict
(injects the synthetic embedding diar source). Removed the "pure function"
claim from the module docstring and documented the mutation explicitly.
- MEDIUM: ``_speech_window_mask`` is a YAMNet veto, not a fallback ladder.
Documented the veto explicitly (YAMNet decides; AST only when YAMNet
missing; loudness only when both missing) and the known YAMNet-confusion
failure modes (child voices flagged as "Music") plus the mitigation.
- MEDIUM: extracted ``assign_unified_clusters_with_seed_phase`` shared
helper in clustering.py; both ``cluster_speaker_labels_by_embedding`` and
the plot's ``_cluster_speakers_by_embedding`` now share the seed-then-
freeze logic.
- LOW: replaced the schema-progression block comment + hardcoded
``"schema_version": 1`` parquet literal with a reference to the
``_CACHE_SCHEMA_VERSION`` constant so they can never drift.
- LOW: surfaced the spectral→k-means fallback through the ``failures`` dict
(per-k entries) and recorded which algorithm actually produced the
partition in the cluster output.
New: same-speaker post-merge step. Spectral / k-means at k≥2 sometimes
splits one speaker's recording into sub-clusters when prosody / distance
varies. After silhouette-driven k* selection, the closest centroid pair is
iteratively merged while cosine similarity ≥ ``merge_threshold`` (default
0.55). The threshold keeps similar-voiced but distinct speakers (same
gender + age + family resemblance ~cos_sim 0.3) separate while folding in
clear-cut same-speaker prosodic outliers (typical adult VoxCeleb same-
speaker cos_sim 0.6+). No target speaker count is assumed anywhere.
New: single-utterance / quiet-environment path. When 1+ valid speech
windows exist but the count is below ``min_windows_for_clustering``, the
clusterer returns ``n_speakers=1`` with all valid windows tagged ``S0``
instead of skipping. This is the "single speaker, just one word in an
otherwise quiet recording" case the prior fall-through silently dropped.
Validation: both ``twin-1.wav`` (2 distinct child speakers, one speaking
briefly) and the English two-speaker conversation still report
``n_speakers=2``; merge step does not fire on either (centroid cos_sim
0.31 and 0.15 respectively, both below 0.55). 57/57 workflow tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* unified speaker clustering: group-aware seed phase + cross-pass threshold
Two-pass speaker-axis fix surfaced by ``audio_48khz_mono_16bits.wav`` (a 5-second
clip where 4 distinct speakers each say ``This is <Name>``):
The synthetic embedding source's per-pass clusterer correctly found 4 distinct
speakers in both raw_16k and enhanced_16k. The plot's cross-pass alignment
``_cluster_speakers_by_embedding`` then collapsed them to 2 because it ran
``assign_unified_clusters_with_seed_phase`` with a single 0.5 cosine threshold
that didn't distinguish "within-pass distinct speakers (validated by silhouette
+ min-size + post-merge)" from "cross-pass same-speaker match". For short
similar-phonetic utterances, different speakers can sit at cos_sim > 0.5 and
got merged.
Fix: ``assign_unified_clusters_with_seed_phase`` is now group-aware:
- ``seed_groups: list[list[(key, mean_emb)]]`` — each inner list comes from a
single per-source clustering. Within a group, every label gets its own
distinct centroid (no within-group merging — those clusters were already
validated upstream).
- Across groups, items merge when centroid cosine ≥ ``cross_group_threshold``
(default 0.75 — same-speaker cos_sim across raw/enh is typically 0.85+,
different speakers within a pass sit around 0.30-0.50).
- ``other_items`` (pyannote / sortformer) still snap to the seed pool at the
legacy ``cosine_threshold`` (0.5) and never spawn new clusters.
Both callers updated:
- ``cluster_speaker_labels_by_embedding`` (per-pass): one seed group =
embedding source's labels for this pass.
- Plot's ``_cluster_speakers_by_embedding`` (cross-pass): one seed group
per (pass, embedding_source) so raw and enhanced cluster sets stay
intact and align speaker-to-speaker across passes at 0.75.
Validation (cache-hit reruns after the fix):
- audio_48khz_mono_16bits (4 distinct speakers, brief utterances): legend
S0–S3, 4 colors visible per embedding-source row.
- twin-1 (2 distinct child speakers): legend S0/S1, no collapse.
- english_conversation_higgs (2 adult speakers): legend S0/S1, no regression.
The per-pass speaker clustering is unchanged; only the cross-pass alignment
in the timeline plot + the per-pass cross-model unification in clustering.py
were affected. Embedding clustering remains independent for raw and enhanced
passes — same-speaker matching across them happens only when cos_sim crosses
the 0.75 cross-group bar.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix Ruff and mypy issues * Suggested follow-ups for PR #511 continuous SER fix - Auto-resample audio to the processor's expected sampling rate to avoid silent correctness bugs when callers pass non-16 kHz input. - Coerce id2label keys to int (HF stores them as strings in JSON) and fall back to class_{i} so the raw-config path doesn't KeyError. - Memoize the dynamically built Wav2Vec2 emotion model class so isinstance checks stay stable across calls. - Drop redundant `AutoConfig as _AC` alias and unused `Tuple` import. - Strengthen the continuous-SER test with score-range and regression assertions that would catch the random-head bug this PR fixes. * Pad labels when shorter than scores in continuous SER Address Gemini review on PR #513: the previous slice-only logic could produce a labels list shorter than the model output if num_labels in the config doesn't match the actual head size. Pad with class_{i} so AudioClassificationResult always sees matching lengths. * Drop regression-guard assertion that depends on fixture content The 'any(abs(s - 0.33) > 0.05 for s in scores)' check would fail if the test fixture happened to produce audeering outputs all in [0.28, 0.38]. Keep only the label-set and [0, 1] range checks, which are robust to any fixture content. * Fix in-process loading for Wav2Vec2ForSpeechClassification SER models Three follow-on issues surfaced when running the new continuous-SER backend end-to-end against audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim: 1. AutoConfig fallback raised TypeError because config.json's own "model_type" key collided with the positional arg in AutoConfig.for_model("wav2vec2", ...). Pop it before splat. 2. The same checkpoints ship vocab_size: null, which trips huggingface_hub>=1.0 strict-dataclass validators when Wav2Vec2Config is instantiated. The regression head doesn't use vocab_size, so drop the key and let the dataclass default apply. 3. _Wav2Vec2EmotionModel called self.init_weights(), but transformers>=5.0's from_pretrained finalizer reads self.all_tied_weights_keys, which is populated by post_init. Switch to post_init() so loading succeeds. Additionally swap Wav2Vec2Processor.from_pretrained for Wav2Vec2FeatureExtractor.from_pretrained, since the processor path re-invokes AutoConfig.from_pretrained for the tokenizer (re-tripping the vocab_size validator) and regression inference doesn't need a tokenizer. Mypy cleanups along the way: cast config to Wav2Vec2Config at the Wav2Vec2Model construction site, and route feature_extractor.sampling_rate through getattr to drop the attr-defined warning on Wav2Vec2Processor. https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66 * Add speechbrain_loading_cwd context manager (smoke test) * Fix discrete-SER ehcalabres + redirect SpeechBrain inner-lobe save_paths Wires the audeering Wav2Vec2ForSpeechClassification fix from PR #513 into the discrete-SER path so the ehcalabres RAVDESS checkpoint stops emitting ~1/8-uniform softmax outputs: * Generalize the custom emotion classifier head to support both final-layer attribute names actually shipped in the wild: ``classifier.out_proj.*`` (audeering MSP-Dim) and ``classifier.output.*`` (ehcalabres). Build the matching nn.Module per layout and cache by name. * Detect the head layout cheaply via the checkpoint's safetensors index or bin shard manifest (no full weight download); fall back to a small registry for single-bin checkpoints like ehcalabres. * Dispatcher now routes any Wav2Vec2 emotion checkpoint with a 2-layer dense+linear head through ``_classify_wav2vec2_speech_cls_ser``, applying softmax for discrete and leaving raw logits for continuous. * Discrete SER test asserts max(scores) > 0.2 to detect random-head regressions, plus that scores form a softmax distribution. Wraps every SpeechBrain ``from_hparams`` call in ``speechbrain_loading_cwd(savedir)`` so CWD-relative ``save_path`` lobes inside hyperparams.yaml (e.g. ``save_path: wav2vec2_checkpoints`` on ``speechbrain/emotion-recognition-wav2vec2-IEMOCAP``) resolve under ``{HF_HOME}/senselab_cache/speechbrain/<safe_key>/`` instead of dropping 700+MB into the user's working tree. Verified end-to-end against two HAPPY recordings: ehcalabres now returns ``happy: 0.49`` (vs. ~0.13 uniform before), and CWD stays clean after a full SER pipeline run. https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66 * Narrow except Exception to (StrictDataclassError, ValueError, TypeError, KeyError) Addresses gemini-code-assist's review on PR #511 about broad-except. The actual exception observed in the audeering MSP-Dim case is huggingface_hub.errors.StrictDataclassError (parent of StrictDataclassFieldValidationError, raised on vocab_size: null with huggingface_hub>=1.0). Standard parsing failures fall under ValueError / TypeError / KeyError. OSError is intentionally excluded — that means the network or filesystem is broken, and the fallback path needs the same network hop, so it can't recover. The tuple is constructed with a try/except ImportError to stay compatible with huggingface_hub<1.0 (where StrictDataclassError doesn't exist). Verified by re-running test_speech_emotion_recognition_continuous against audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim — the StrictDataclassFieldValidationError is correctly caught by the narrowed tuple and the raw-config-dict fallback runs as before. https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66 * Loud-fail guard for random/mismatched Wav2Vec2 emotion heads + remediation plan Phase 1 of specs/20260509-204500-ser-silent-failure-guards/plan.md. The custom Wav2Vec2 emotion-head path in `_classify_wav2vec2_speech_cls_ser` trusts the dispatcher peek to have correctly matched the checkpoint's head layout. When that assumption is wrong (new encoder family, unrecognized final-layer attribute name, or single-bin checkpoint not in the registry) transformers silently random-initializes the head and inference returns ~uniform softmax — the exact bug PR #511 is fixing, just one routing branch upstream. Convert that to a loud `RuntimeError`. After `from_pretrained` returns we inspect `loading_info` for any `classifier.*` entry in `missing_keys` (weights absent, would be randomly initialized) **or** `mismatched_keys` (weights present but wrong shape — equally a silent-random failure that the original missing-keys check would miss; flagged by architect review). We also assert the final layer's `out_features` equals `config.num_labels` / `len(id2label)`, catching the case where head and config disagree on the number of classes. Adds a parametrised unit test that mocks `EmotionModel.from_pretrained` to return each broken-load shape and asserts the diagnostic raises with a matching message. Also commits the full remediation plan (specs/.../plan.md) enumerating Phases 2-7 (standard-pipeline missing-key warning, encoder-family generalization for HuBERT/WavLM, head-registry richening, problem_type based discrete/continuous detection, resampling in the standard pipeline, diagnostics CLI). Plan was self-audited and reviewed by the Plan agent before this commit; its recommendations are folded in (mismatched_keys coverage promoted into Phase 1, dynamic HeadSpec cut from Phase 4 as premature, encoder-attribute-name registry promoted from a Phase 3 footnote to a load-bearing task). https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66 * Phases 2-7: encoder-family registry, problem_type, doc + diagnostics CLI Implements the remaining phases from specs/20260509-204500-ser-silent-failure-guards/plan.md, layered on top of Phase 1's loud-fail guard: - Phase 4 (head registry): introduce ``_HeadEntry`` dataclass holding ``final_layer`` / ``activation`` / ``dropout_field``; promote ``_KNOWN_WAV2VEC2_EMOTION_HEADS`` (str→str) to ``_KNOWN_HEAD_LAYOUTS`` (str→_HeadEntry). Generalize ``_make_wav2vec2_regression_head_class`` to ``_make_regression_head_class(head: _HeadEntry)`` with a small ``_ACTIVATIONS`` table. Cache key now includes activation/dropout_field so future entries don't collide. - Phase 3 (encoder-family generalization): ``_BASE_REGISTRY`` maps ``config.model_type`` → ``(PreTrainedModel base name, encoder attr name)`` for wav2vec2/hubert/wavlm/wav2vec2-bert. Resolved lazily so imports only fire when the custom-head path runs. ``_emotion_head_kind`` now returns ``(model_type, _HeadEntry)`` tuples; ``_make_emotion_model_class`` threads the right encoder class and attribute name. The Phase-1 guard catches mis-registered families: any wrong attr-name will surface as "every classifier.* key missing" with a clear ``RuntimeError``. - Phase 5 (problem_type detection): ``_resolve_apply_softmax`` prefers ``config.problem_type`` over the keyword-based label heuristic. A regression head with non-AVD axes (e.g. ``["energy","pleasantness"]``) no longer gets softmax misapplied. Falls back to ``_get_ser_type`` for legacy checkpoints that omit ``problem_type``. - Phase 2 (standard-pipeline missing-key warning): ``_get_hf_audio_classification_pipeline`` now loads the model explicitly via ``AutoModelForAudioClassification.from_pretrained(..., output_loading_info=True)`` and threads the loaded instance into ``pipeline(model=..., feature_extractor=...)``. ``_check_head_loaded_cleanly`` warns on suspect missing/mismatched head keys (``classifier.``/``head.``/``score.``/``out_proj.``/``projector.``). ``SENSELAB_STRICT_HEAD_LOAD=1`` promotes the warning to a hard error. - Phase 6 (resampling in standard pipeline): verified already implemented at ``classify_audios_with_transformers`` lines 141-142; no code change required. Plan updated to reflect the discovery. - Phase 7 (docs + diagnostics CLI): - ``doc.md`` enumerates the three dispatch paths (custom in-process head / subprocess venv / standard pipeline) and points at the CLI. - New ``python -m senselab.audio.tasks.classification.speech_emotion_recognition --probe <repo_id>`` reports model_type, architectures, dispatch decision, head source, and emits advisories. Pure config / manifest inspection — no weights loaded. Backwards-compat shims kept in place: - ``_make_wav2vec2_emotion_model_class(final_layer)`` (delegates to ``_make_emotion_model_class("wav2vec2", _HeadEntry(final_layer=...))``) - ``_wav2vec2_emotion_head_kind`` (returns just the final-layer name from the new ``_emotion_head_kind`` tuple). - ``final_layer=`` kwarg on ``_classify_wav2vec2_speech_cls_ser`` short-circuits to ``_HeadEntry(final_layer=...)``. Verified locally: all 8 classification tests pass (1 GPU-skip), audeering happy path passes, ehcalabres CLI probe correctly identifies its head from ``_KNOWN_HEAD_LAYOUTS`` registry, audeering CLI probe peeks the manifest, superb correctly routes to the standard pipeline with an advisory. Test coverage gaps (deferred per plan): - Phase 2 unit test for ``_check_head_loaded_cleanly`` (covered end-to-end by audeering/ehcalabres tests for now). - Phase 5 unit test for ``_resolve_apply_softmax`` (no fixture model with problem_type=regression + confounding labels). - Phase 4 numerical-regression test on existing checkpoints. https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66 * Reviewer-pass cleanups: explicit encoder cls names, CLI null-config note Two small fixes from the second-pass architect review: 1. Make ``_BASE_REGISTRY`` a 3-tuple ``(base_pretrained_cls_name, encoder_model_cls_name, encoder_attr_name)`` instead of deriving the encoder class name via ``base_name.replace("PreTrainedModel", "Model")``. The string-mutation worked for current Wav2Vec2/HuBERT/WavLM/W2V-BERT but would silently break the day a family ships an encoder with a different naming convention. Explicit > clever. Updated both indexers (``_emotion_head_kind`` in api.py and the probe CLI's status print) to read attr name from index ``[2]``. 2. Probe CLI: when ``AutoConfig.from_pretrained`` fails (e.g. audeering's ``vocab_size: null`` tripping the strict validator), make the diagnostic note explicit that the dispatcher's raw-config-dict fallback handles this — the routing decision and advisories that follow still work correctly because ``getattr(None, ...)`` short- circuits the ``config.model_type`` checks. Verified: existing guard tests pass; ``uv run python -m senselab.audio.tasks.classification.speech_emotion_recognition --probe audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim`` still produces a correct routing report end-to-end. https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66 * Annotate model type in probe CLI to satisfy mypy --extra-checks CI pre-commit hook runs mypy with --extra-checks which flagged the inferred type for ``model = HFModel(...)`` in __main__.py. Add the explicit annotation. cpu-tests already pass; this is the only blocker. https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66 * Strict-reviewer second pass: 5 silent-failure surfaces fixed + Phase 2 unit tests The strict reviewer identified 5 critical defects against the user's "zero tolerance for silent failures" criterion. All addressed in this commit: D1. ``wav2vec2-bert`` removed from ``_BASE_REGISTRY``. The family consumes log-mel ``input_features`` (via ``SeamlessM4TFeatureExtractor``) rather than raw ``input_values``. The loader hardcodes the latter and accesses ``inputs["input_values"]`` so any wav2vec2-bert checkpoint would have produced ``KeyError`` at best, silent garbage at worst. Add a per-family preprocessing adapter before re-introducing. D2. Phase-1 guard now covers encoder-key misses. Previous filter was ``classifier.*`` only, which meant a wrong ``_BASE_REGISTRY`` entry (wrong encoder attribute name) would leave every encoder weight missing yet pass the guard — silent garbage output. The guard now treats any missing key under either ``classifier.`` or ``{encoder_attr}.`` as suspect, with a small whitelist of HF post_init buffers (``masked_spec_embed``). D3. ``_resolve_apply_softmax`` peeks ``architectures`` on ``problem_type``-absent checkpoints. Any model declaring ``Wav2Vec2ForSpeechClassification`` (audeering signature) is treated as regression. Reduces the silent-corruption window for regression heads with English-emotion-word labels. D4. ``_peek_head_final_layer`` raises when ``classifier.dense.weight`` is present but no recognized final-layer attribute matches. Previously returned None and let the standard pipeline silently random-initialize the head. The error message lists the actual classifier keys so a maintainer can extend ``_FINAL_LAYER_NAMES`` or ``_KNOWN_HEAD_LAYOUTS``. D5. ``SENSELAB_STRICT_HEAD_LOAD`` default inverted to ``=1`` (raise). Warn-only meant downstream consumers like ``classify_audios`` windowed segmentation kept aggregating corrupted scores. Set ``SENSELAB_STRICT_HEAD_LOAD=0`` to demote to a warning if you intentionally want a partial-head load (e.g. fine-tuning warm-start). T203. Three unit tests for ``_check_head_loaded_cleanly``: parametrised raise on suspect missing/mismatched-shape keys (default strict), warn-only mode under ``SENSELAB_STRICT_HEAD_LOAD=0``, and silent pass on clean loads (encoder-buffer misses don't trip). Plan checkboxes T203 / T304 / T502 promoted from deferred-shortcut to actually-shipped. Audeering happy-path test still passes; mypy clean with ``--extra-checks``. https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66 * Reviewer round 3: align user-facing docs with strict-by-default behavior Reviewer round 3 verified D1-D5 + T203 fixes are functionally correct, but flagged three pieces of user-facing text that still contradict the new behavior — themselves a silent-assumption violation per the user's "no silent failures and assumptions beneath those" criterion: * ``doc.md`` listed ``wav2vec2-bert`` as a supported encoder for the custom-head path (it isn't; D1 dropped it from ``_BASE_REGISTRY``) and described ``SENSELAB_STRICT_HEAD_LOAD=1`` as a way to "promote the warning to a RuntimeError" — but ``=1`` is now the default (D5). A user who reads the doc, doesn't set the env var, and expects warn-mode would be surprised by a hard fail. * ``__main__.py`` probe CLI advisory had the same inverted message about the env var. * ``api.py:503`` dispatcher comment + ``api.py:144`` encoder-attribute comment still mentioned ``wav2vec2-bert``. This commit: 1. Removes ``wav2vec2-bert`` from the doc's supported-encoder list, adds an explicit "intentionally not supported" sentence with the ``SeamlessM4TFeatureExtractor`` reason, so users know why and what it would take to add. 2. Inverts the env-var description in both the doc and the CLI to match the strict-by-default semantics. 3. Drops stale ``wav2vec2-bert`` mentions from the api.py comments. No code-path or behavior change. Verified clean: ruff, ruff format, mypy --extra-checks all pass. https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66 * Plan cleanup: drop T203/T502 'legacy text for traceability' duplicates When the strict reviewer's second pass forced T203 and T502 from deferred-shortcut to actually-shipped, I appended the new ``[x]`` text and kept the original ``[ ]`` lines tagged "legacy text, kept for traceability". On re-read those duplicates just made the plan look like there were unfinished items when there weren't. Drop the duplicates; the ``[x]`` entries are sufficient. Remaining ``[ ]`` items after cleanup: T305 — attention-mask shim for batched HuBERT/WavLM (don't optimize for unknowns) T403 — numerical regression test against fixed scores (Phase-1 guard + bounded-score covers) T602 — sampling-rate test (Phase 6 was a no-change phase, end-to-end coverage exists) All three are honest deferrals, not shortcuts. https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66 * PR #511 follow-up: AutoConfig memo (per-key locked) + SpeechBrain CWD lock + registry smoke tests (#515) Stacks on top of PR #511 to address @wilke0818's review. Changes: 1. **CWD race fix**: `speechbrain_loading_cwd` now serializes via a module-level `threading.Lock` so concurrent threads can't race on the process-global `os.chdir`. Docstring documents the trade-off (use processes, not threads, for parallel SpeechBrain init). 2. **AutoConfig deduplication**: introduces `_load_config_cached(model)` returning `(typed_config_or_None, raw_dict_or_None)`, memoized per-(path, revision). The four call sites (`_get_ser_type`, `_emotion_head_kind`, `_resolve_apply_softmax`, `_classify_wav2vec2_speech_cls_ser`) now share one network round-trip instead of four. Type annotated with `PretrainedConfig`. 3. **Per-key locking** (not a single global lock): mirrors the strategy used by `dependencies.ensure_hf_model` (`_HeartbeatLock` per `(repo_id, revision)`). Two threads loading the *same* model dedupe to one round-trip; two threads loading *different* models proceed in parallel. In-memory `threading.Lock` per key here, since the memo is per-process (cross-process is already handled by HF_HOME cache and `ensure_hf_model`). 4. **`_BASE_REGISTRY` smoke tests**: three parametrised unit tests verify `_resolve_base("wav2vec2"|"hubert"|"wavlm")` resolve to real transformers classes — catches typos at test-collection time without downloading model weights. 5. **Test isolation fix**: autouse fixture clears the module-level caches (`_config_memo`, `_config_memo_locks`, `_wav2vec2_emotion_models`) between tests. Previously the parametrised `test_wav2vec2_speech_cls_ser_raises_on_random_head` and `_raises_on_shape_mismatch` would silently serve a cached MagicMock from a prior test's monkeypatch instead of re-invoking their own; tests only passed because the mocks happened to agree. Verified locally: ruff, format, mypy --extra-checks clean. 18 fast tests pass including 4 new ones. CI: ✅ cpu-tests, pre-commit, build-and-deploy all green on `be77c5d`. https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66 --------- Co-authored-by: Satrajit Ghosh <satrajit.ghosh@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
New module ``senselab.utils.cuda_probe`` provides: - ``detect_host_cuda()`` — parses ``CUDA Version: X.Y`` from ``nvidia-smi``'s default header, falls back to ``release X.Y`` from ``nvcc --version``, then ``HostCuda(version=None)`` when neither is available. Each probe has a 5s subprocess timeout. No driver→CUDA lookup table to maintain — both probes report the CUDA version directly. - ``pick_torch_index(host_cuda, env_override=None)`` — picks the highest static-map entry whose CUDA is ``<=`` host's, from ``[cu128, cu126, cu124, cu121]``; falls back to ``cpu`` index. When ``env_override`` is set + non-empty, returns that URL verbatim with ``tag="override"``. - ``SenselabCudaCompatibilityError`` — wraps wheel-not-found install failures with host CUDA + attempted index + failing packages and a one-line recommended action (downgrade CUDA / set ``SENSELAB_TORCH_INDEX_URL=https://download.pytorch.org/whl/cpu`` / wait for upstream wheels). Two frozen dataclasses (``HostCuda``, ``TorchIndex``) carry the probe and picker results so they're individually testable and the resolved index URL is auditable in the subprocess-venv marker file. Tests: ``src/tests/utils/cuda_probe_test.py`` — 15 unit tests, all ``subprocess.run`` calls mocked, no shell-out on CI. Coverage: both probe paths, both probe failures, all 5 index map entries, env override short-circuit + empty-string-treated-as-unset, error message contents. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
``ensure_venv`` now resolves a PyTorch wheel index for the host before
running ``uv pip install`` and routes the install through that index. On
hosts with system CUDA newer than the PyTorch default-wheels CUDA (e.g.
CUDA 12.9 against PyTorch's ``cu128`` default), this prevents ``torch``
and ``torchaudio`` from resolving to mismatched CUDA toolchains and
breaking their ABI contract at import time.
Resolution order inside the lock:
1. ``env_override = os.getenv("SENSELAB_TORCH_INDEX_URL") or None`` (empty
string treated as unset).
2. If ``env_override`` is set: skip the host CUDA probe — the override
wins regardless of host CUDA.
3. Otherwise: ``host_cuda = detect_host_cuda()`` followed by
``pick_torch_index(host_cuda, env_override=env_override)``.
The marker file gains a ``torch_index`` field; absence of that field in
an existing marker counts as a mismatch and triggers a rebuild — so
users upgrading through this fix get one automatic rebuild on first run.
No explicit schema-version field; the marker is internal cache state,
not a published contract.
Install argv now prefixes ``--index-url <torch_index.url>
--extra-index-url https://pypi.org/simple ...`` — the PyTorch index for
torch/torchaudio resolution, PyPI as the extra index so non-torch
packages (nemo_toolkit, qwen-asr, etc.) still resolve.
Failure handling: ``uv pip install`` failures pass through
``_classify_uv_failure`` to distinguish "no matching distribution" /
"could not find a version" errors (wrap into
``SenselabCudaCompatibilityError`` with named host/index/packages/action)
from unrelated failures (re-raise the original ``CalledProcessError``).
Either way the half-built venv directory is wiped before the raise so
the next run starts clean — addresses the recovery-from-broken-state
case.
Added in-source warnings to ``canary_qwen.py``, ``nemo.py``, and
``qwen.py`` near each ``*_REQUIREMENTS`` list: developers editing the
torch pins should not bypass ``ensure_venv`` with a per-backend install
shellout, since the CUDA routing is what guarantees the toolchain match.
Tests: ``src/tests/utils/subprocess_venv_test.py`` — 14 integration
tests, ``subprocess.run`` mocked. Coverage: marker-mismatch rebuild
paths (pre-fix-shape marker, matching index, different index), install
argv routing, env override short-circuit + empty-string-unset, wheel
not-found wrapping, unrelated-error pass-through, ``_classify_uv_failure``
unit cases, parameterized regression test walking all three real backend
requirement lists.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
README "System Requirements" section now mentions: - Subprocess-venv backends auto-route their ``torch``/``torchaudio`` install through the matching PyTorch wheel index based on host CUDA; no manual configuration needed. - ``SENSELAB_TORCH_INDEX_URL`` operator override for internal mirrors, unsupported CUDA versions, or forced CPU fallback. - Unsupported-CUDA failures surface as ``SenselabCudaCompatibilityError`` with named host CUDA, attempted index, and recommended action. Spec docs land under ``specs/20260512-204619-fix-canary-cuda-conflict/``: spec.md (3 prioritized user stories), plan.md (Constitution check + 6 research decisions), research.md, data-model.md, contracts/, quickstart.md (validation recipes for happy path, CPU fallback, negative path, regression, recovery), tasks.md (28 tasks across 6 phases), and checklists/requirements.md (passed twice). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Local-reviewer findings on the cuda_probe + subprocess_venv commits:
- MEDIUM (subprocess_venv): on the env-override path, host_cuda was
hardcoded to ``HostCuda(version=None)`` so the wrapped
``SenselabCudaCompatibilityError`` reported "Host CUDA: none" even on
real CUDA hosts using internal mirrors. Run ``detect_host_cuda()``
unconditionally now; the override still bypasses the static index map
but the probe result is preserved for diagnostic surface.
- MEDIUM (subprocess_venv): the failing-package extraction regex was
pulling EVERY backticked token from uv's stderr, so unrelated hints
like ``\`uv cache clean\`\`\` could leak into the
``SenselabCudaCompatibilityError`` message as fake "failing packages".
Anchored the capture to the "no matching distribution" / "could not
find a version" phrases so only the actual failing requirement is
surfaced.
- LOW (subprocess_venv): ``uv venv`` failure (e.g. requested Python
version unavailable) re-raised without wiping a partially-populated
venv directory. Mirror the install-failure cleanup so both failure
paths leave a clean cache state.
- LOW (cuda_probe): ``_PYTORCH_INDEX_MAP`` ordering ("highest CUDA
first") was asserted by comment only. Wrapped the literal in
``sorted(..., reverse=True)`` so a future contributor appending an
entry in the wrong position can't silently break routing.
- LOW (subprocess_venv): on the wrap-into-compat-error path, the
pre-classification ``logger.error`` was duplicating the wrapped
exception's already-formatted message and dumping unbounded uv
stderr. Demoted to ``logger.debug``; the pass-through path still logs
at error level since that's the only signal the user gets for
non-wheel failures.
Test updates: ``test_env_override_short_circuits_probe_*`` renamed and
reworked to assert the probe DOES run (not skips) on the override path
— the contract is the URL routing, not the probe-skip side effect.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pre-fix-time local pre-commit pass was reading a stale mypy cache. Once invalidated (CI / fresh clone), five errors surfaced: - src/tests/utils/cuda_probe_test.py:44,46,67,69 — `_runner(*args: object)` was too generic for the indexed access `args[0]` (mypy "Value of type 'object' is not indexable"). Tightened to the actual ``subprocess.run`` positional signature `_runner(args: list[str], **kwargs: object)` — matches reality and unblocks indexing. - src/tests/utils/subprocess_venv_test.py:80 — `_SubprocessRecorder.hook` was typed `Optional[object]`, which mypy refuses to call. Tightened to `Optional[Callable[[list[list[str]]], subprocess.CompletedProcess]]` — the type the hook actually implements. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two occurrences in research.md flagged by codespell on CI. Same fix shape — the hyphenated forms are non-standard per the project's codespell allowlist. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Distinguish ``text=""`` (model was called, returned no transcript — a
real signal) from ``text=None`` (field absent / model never called).
The earlier validator used falsiness (``not values.get("text")``),
treating both the same and rejecting any ``ScriptLine`` whose text was
empty even when its speaker was also unset.
In practice this only bit ASR backends honest enough to emit empty
output on unintelligible / silent audio. The trigger we observed: IBM
Granite-Speech (``ibm-granite/granite-speech-3.3-8b``) on heavily-
enhanced short clips (1.4–3.4s after speech enhancement) — it
correctly decoded "no speech" and emitted ``{"text": ""}``, which the
``ScriptLine`` validator then refused with ``"At least text or speaker
must be provided."``. Less-honest ASR backends (Whisper hallucinates
filler on silence, the LLM-augmented Canary-Qwen / Qwen3-ASR pass the
prompt's implicit "say something" pressure through and almost always
emit text) silently slipped past the same validator by producing a
non-empty string. The vulnerability lives in every backend that
constructs ``ScriptLine(text=...)`` directly — Granite was just the
only one that surfaced it in this dataset.
There was also a subtle inconsistency: a whitespace-only input like
``text=" "`` survived the validator (``" "`` is truthy) and was then
stripped to ``""`` by the field validator, while an explicit ``""``
was rejected up front. The new presence-based rule (``text is None
and speaker is None``) treats both consistently.
Negative-timestamp validation and the whitespace-stripping field
validator are unchanged.
Tests in ``src/tests/utils/data_structures/script_line_test.py``
cover the Granite-style empty-transcript path, the whitespace-normalize
case, both-fields-absent rejection, speaker-only construction, and
preservation of existing happy-path behavior (timestamps, stripping,
``from_dict``).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…equirements PR #516 routes the install through ``--index-url <cuda> --extra-index-url https://pypi.org/simple`` so non-torch packages can still resolve from PyPI. But uv treats ``--extra-index-url`` as having higher priority than ``--index-url`` (opposite of pip), so PyPI wins for every package — torch and torchaudio included. On hosts where PyPI ships those with mismatched ``+cu`` local-version tags (currently ``torch==X+cu129`` vs ``torchaudio==X`` with no tag, internally cu128), the resulting venv reproduces the same ABI mismatch this PR is meant to fix: RuntimeError: Detected that PyTorch and TorchAudio were compiled with different CUDA versions. PyTorch has CUDA version 12.9 whereas TorchAudio has CUDA version 12.8. Reproduced on MIT ORCD with the env override set (``SENSELAB_TORCH_INDEX_URL=https://download.pytorch.org/whl/cu128``) — override URL was honored in the marker but uv still resolved both wheels from PyPI. So the override path was non-functional too. Fix: split the install into two stages and decide whether Stage 1 fires from the caller's ``requirements`` itself. Stage 1 — ``uv pip install --index-url <chosen>`` listing every ``torch`` / ``torchaudio`` spec found in ``requirements``, with NO ``--extra-index-url``. The chosen CUDA index is unambiguously primary; both wheels and their ``nvidia-cuda-runtime-cu12`` transitives come from the same toolchain. Multiple constraints for the same package (``["torch>=2.8", "torch<2.9"]``) are all forwarded so uv combines them at resolve time. Stage 2 — ``uv pip install <rest of requirements> safetensors numpy`` with no index flags. Backend-pinned torch / torchaudio specs are filtered out so uv can't be tempted to re-resolve them from PyPI against the matched ``+cu128`` wheels installed in Stage 1. Drop the unconditional ``torchaudio`` IPC append, in two ways: 1. ``_torch_install_specs`` no longer pads its return with bare ``torch`` / ``torchaudio`` names. If neither is in ``requirements``, it returns ``[]``. 2. ``ensure_venv`` treats ``_torch_install_specs(requirements) == []`` as the trigger to skip the probe entirely and run a single torch-free install pass against default PyPI. No ``nvidia-smi`` shellout, no ``torchaudio`` force-appended. This addresses @satra's review concern that the previous default forced ``torchaudio`` into every subprocess venv, even backends that don't use it. With the change, ``yamnet`` and ``continuous-ser`` — both of which read audio via ``soundfile`` in their worker scripts and never import torchaudio — get ~200 MB leaner venvs. Backends that genuinely need ``torch`` / ``torchaudio`` (including via a transitive dep) MUST pin them explicitly so Stage 1 routes them through the matched CUDA index; this is also why ``qwen.py``'s ``_REQUIREMENTS`` now names them directly even though ``qwen-asr==0.0.6`` would otherwise pull them transitively — relying on the transitive would let Stage 2's PyPI resolution split the local-version tags. The 7 backends that already pin ``torchaudio`` explicitly (canary, nemo, ppgs, sparc, coqui, s3prl) are unaffected. The earlier ``requires_torch=False`` parameter from a previous round of this review (an explicit opt-out flag) is removed in favor of the auto-detection — it's structurally equivalent and the explicit flag becomes redundant noise. Behavioral guarantees preserved from PR #516: marker schema (extended, not changed), cache-hit fast path, env override (``SENSELAB_TORCH_INDEX_URL``), ``SenselabCudaCompatibilityError`` wrapping on wheel-not-found errors (now scoped to Stage 1 where it semantically belongs), pass-through of unrelated ``CalledProcessError``, half-built-venv cleanup on failure, host-CUDA probe diagnostic surface, cross-backend routing. Tests: 34 in total covering (1) Stage 1 names only ``--index-url`` with torch + torchaudio pinned per requirements; (2) Stage 2 carries no index flags + IPC deps; (3) Stage 2 filters torch / torchaudio specs from caller's requirements; (4) multiple constraints for the same package all forward through Stage 1 verbatim; (5) requirements without torch / torchaudio skip the probe and Stage 1 entirely; (6) yamnet- style requirements get NO torchaudio in their install argv; (7) switching a venv name from torch-free to torch-using correctly invalidates the cache and rebuilds. The cross-backend regression test asserts both that Stage 1 routes through the chosen index AND that Stage 2 contains no torch / torchaudio specs at all. Review feedback addressed: - @satra (PR #518 review): drop the unconditional ``torchaudio`` IPC append; auto-detect torch routing from the caller's ``requirements`` instead of forcing every venv to pay for it. Adds explicit ``torch`` + ``torchaudio`` to qwen's ``_REQUIREMENTS`` so its transitive-dep resolution still flows through the CUDA index. - gemini-code-assist #1: filter torch / torchaudio specs from Stage 2. - gemini-code-assist #2: list-based ``_torch_install_specs`` keeps every matching constraint instead of clobbering through a dict. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
subprocess_venv: two-stage install to fix torch/torchaudio CUDA-tag s…
…conflict subprocess_venv: route torch/torchaudio via CUDA-aware PyTorch index
…opt-in Hosts on Python versions newer than spaCy ships wheels for (3.14 today) silently fell back to the no-detector path in ``pii.py`` and produced ``contains_pii: false`` reports across every clip — a false-negative on PII-bearing audio for any host that can't install Presidio in-process. Move detection into an isolated Python-3.13 subprocess venv so the host can stay on whatever recent Python it likes without losing PII checks. Detection content ----------------- Two detectors run inside the new ``pii-detection`` venv and their spans are merged before cross-ASR-model corroboration runs in the host: - **Microsoft Presidio Analyzer** — regex + spaCy-NER recognizers for emails, phone numbers, SSNs, credit cards, IPs, dates, locations. - **GLiNER PII** (``nvidia/gliner-pii`` by default, configurable) — a 570M-param transformer fine-tuned on ~100k synthetic PII/PHI records. Catches medical / health entities Presidio doesn't natively cover, plus the long tail of unstructured personal-context references that zero-shot well from the labels we pass. GLiNER's lowercase labels are normalized to Presidio's uppercase scheme inside the worker so the existing ``(category, text.lower())`` dedupe key in ``pii.py`` treats both detectors' hits on the same entity as the same finding. Per-detector opt-in ------------------- ``detect_pii_in_pass`` and ``detect_pii_via_subprocess`` gain a ``detectors`` argument symmetrical to the project's ASR-model selection pattern: - ``detectors=None`` (default) → both Presidio and GLiNER run. - ``detectors=["presidio"]`` → skip GLiNER's model load (~5 s saved). - ``detectors=["gliner"]`` → skip Presidio. - ``detectors=[]`` → explicit-disable; no subprocess spawned, no venv build attempted, report carries ``failures["pii_disabled"]`` so an auditor can distinguish "didn't run on purpose" from "ran and found nothing" and "subprocess crashed". ``gliner_model``, ``gliner_labels``, and ``gliner_threshold`` are also exposed on ``detect_pii_in_pass`` so callers can swap in a different GLiNER variant (e.g. ``urchade/gliner_multi-v2.1`` for multilingual coverage) or restrict the label set without touching the worker. Architectural changes --------------------- - New module ``pii_subprocess.py`` — venv constants (``_PII_VENV = "pii-detection"``, Python 3.13, presidio + spacy + en_core_web_lg wheel pinned via direct URL + gliner + torch), GLiNER -> Presidio category map, worker script, and the ``detect_pii_via_subprocess`` dispatch function. ``torch`` is named explicitly in ``_PII_REQUIREMENTS`` so ``ensure_venv``'s auto-detect (post-#516) routes it through the matched CUDA index — GLiNER's transformer wants matched ``torch`` / nvidia-cuda-runtime wheels. ``torchaudio`` is intentionally omitted (no audio decoding). - ``pii.py`` simplified — removed ``_load_presidio_analyzer``, ``_load_spacy_nlp``, ``_scan_with_presidio``, ``_scan_with_spacy``, and the in-process tier-fallback chain. Dispatch now goes to the subprocess; everything else (transcript concatenation, ``PiiSpan`` materialization, cross-ASR-model corroboration, ``PiiPassReport`` construction) stays in-process. - Subprocess-only — no in-process fallback. Matches the canary/nemo/qwen subprocess-venv pattern for hermetic deployment. A host that can't build the venv at all gets a single ``failures["pii_subprocess"]`` entry, not a confusing partial-fail state. Worker model loads are paid per subprocess invocation. For a typical ``analyze_audio.py`` run (138 ``detect_pii_in_pass`` calls per subject) that's ~15 min of model-load overhead per subject; batching all of a subject's transcripts into a single subprocess invocation is a worthwhile follow-up but isn't required for correctness. Tests ----- ``src/tests/audio/workflows/audio_analysis/pii_subprocess_test.py``, 10 tests covering: default-both-detectors plumbing, single-detector selection, empty-list short-circuit (no venv build, no subprocess spawn), unknown-detector ``ValueError``, dedupe with order preservation, default-arg pass-through (threshold, GLiNER model id, labels, full GLiNER -> Presidio category map), caller overrides for the GLiNER settings, response field pass-through. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The PII axis previously surfaced only ``contains_pii ? 1.0 : 0.0`` to ``global_summary``, throwing away every per-span detector score that the worker produced. Worse, when the subprocess crashed or no detector loaded, the report fell to ``0.0`` — indistinguishable from "we checked, found nothing." That's the wrong default; it tells a reviewer "this clip is clean" when the truth is "we don't know." Add a continuous ``detection_confidence`` field on ``PiiPassReport``, combined from three signals per unique ``(category, normalized_text)`` finding: - **per-span detector score** — Presidio analyzer score or GLiNER prediction probability, ``max()`` within the finding's group. - **cross-detector agreement** — Presidio AND GLiNER both flagged the same finding (factor 1.0) vs only one detector (0.5). The strongest "real entity vs hallucination?" signal at this layer, mirroring the model-voting pattern presence/identity already use. - **cross-ASR-model agreement** — fraction of available ASR transcripts that contained the finding. Spans only one ASR transcribed and no sibling ASR confirms are the prototypical hallucination case. Then ``max()`` across findings, matching how the transcript and single-speaker axes combine their internal sub-signals. Deliberately NO category-severity weighting. In pediatric / clinical voice data the categories nominally most "severe" (US_SSN, CREDIT_CARD, MEDICAL_RECORD_NUMBER) have near-zero true-positive rate — children don't speak SSNs aloud — and the synthetic-PII-trained detectors gladly flag any 9-digit ASR hallucination as SSN. Weighting them up would inflate exactly the findings a reviewer should de-prioritize. Heuristic enhancements (category weights, find-count severity) are deferred as later goals; this commit just plumbs through what the detectors and ASRs actually report. ``None`` propagation -------------------- ``detection_confidence`` is ``None`` when no detector ran: - subprocess raised at dispatch (CalledProcessError / RuntimeError) - both detectors failed to load inside the subprocess venv - caller passed ``detectors=[]`` (explicit-disable short-circuit) ``0.0`` is reserved for "detectors ran, found nothing" — the confident-negative case. ``global_summary.compute_pass_global_summary`` propagates the same distinction into ``no_pii_uncertainty`` so the ``combined_uncertainty`` consumer can tell "didn't actually check" apart from "checked clean." No silent demotion to 0. Tests ----- ``src/tests/audio/workflows/audio_analysis/pii_test.py``, 12 tests: - ``_compute_detection_confidence`` unit cases: empty spans -> 0.0, single detector / single ASR -> 0.5 x score, both detectors -> 1.0 factor, cross-ASR scaling by fraction, max across independent findings (not sum), whitespace-only text filtered, missing per- span score collapses to 0. - ``detect_pii_in_pass`` integration with mocked ``detect_pii_via_subprocess``: happy path populates the field, ``detectors_used=[]`` -> None, subprocess raise -> None, detectors-ran-no-spans -> 0.0, ``detectors=[]`` short-circuit -> None plus ``failures["pii_disabled"]``. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Empirical finding from a diagnostic on ``john.doe@example.com``:
nvidia/gliner-pii exhibits competing-claim interference. When the
prompt includes overlapping labels that can both plausibly cover the
same span — e.g. ``person`` + ``first_name`` + ``last_name`` plus
``email`` + ``email_address`` — the model commits the span to one
label and silently drops it from consideration under the other,
*even when the other would have been the right answer*. The
diagnostic at ``temp/diag_gliner_email.py`` walks this through
explicitly at threshold=0.0:
Labels: [person, first_name, last_name, email, email_address, ...]
→ John = first_name (1.0), Doe = last_name (1.0),
john.doe@example.com = no email span at any score above 0.0
Labels: [name, address, date, phone_number, email, ssn, ...] (flat)
→ John, Doe = name (1.0), john.doe@example.com = email (1.0)
Same model. Same threshold. Same input. The only difference was
label-set granularity. Email goes missing under the over-decomposed
set; both name and email surface cleanly under the flat HIPAA set.
This is the same set b2aiprep PR #256 ships in its
``hipaa_labels.json``. Aligning on it has the secondary benefit
that the two projects' PII outputs now share a vocabulary, which
matters for downstream consumers that aggregate across both.
Changes
-------
- ``_DEFAULT_GLINER_LABELS`` replaced with the HIPAA Safe Harbor 18
identifiers (45 CFR §164.514(b)(2)) plus two clinical-voice
extensions (``health_condition``, ``medication``) that don't
overlap with the HIPAA labels.
- Long block comment above the constant captures the empirical
finding so future contributors don't walk back into the trap
(i.e. "this would catch more if I added first_name and
last_name..."). Names the diagnostic, names the symptom, names
the fix.
- ``gliner_labels`` docstring on ``detect_pii_via_subprocess``
points at the warning so anyone customizing the label set hits
it before re-introducing overlap.
The category map (``_GLINER_TO_PRESIDIO_CATEGORY``) already handles
every new HIPAA label that has a Presidio analogue (``name``,
``email``, ``date``, ``phone_number``, ``address``,
``social_security_number``, ``ip_address``,
``medical_record_number``). New labels without Presidio analogues
(``fax_number``, ``health_plan_number``, ``account_number``,
``license_number``, ``vehicle_identifier``, ``device_identifier``,
``url``, ``biometric_identifier``, ``photographic_image``,
``unique_identifier``) fall through to ``str.upper()`` and stand on
their own — Presidio doesn't emit competing categories for them so
there's nothing to align with. Entries for the now-removed labels
(``first_name``, ``last_name``, ``street_address``, ``city``,
``date_of_birth``, ``patient_id``, etc.) stay in the map for
backward-compat — a caller can still pass them explicitly via the
``gliner_labels`` override.
Tests
-----
22 tests in pii_subprocess_test.py + pii_test.py still pass — they
check the request shape and detection_confidence math, not the
content of the default label list, so the swap is transparent at
the dispatch layer. The behavioral verification is in the real
end-to-end run via temp/test_pii_subprocess.py + temp/diag_gliner_email.py.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous commit on this branch (a2f384c) shipped the HIPAA Safe Harbor 18 identifiers PLUS two clinical-voice-specific extensions (``health_condition``, ``medication``) tacked on the end. Running the smoke test against ``temp/test_pii_subprocess.py`` showed Step 3 (detectors=["gliner"]) still didn't catch ``john.doe@example.com`` even with the new flat HIPAA-aligned base. Cross-referenced against the threshold=0.0 diagnostic (``temp/diag_gliner_email.py``): the bare HIPAA-18 set catches the email at GLiNER score 1.0. Adding ``health_condition`` and ``medication`` on top is enough to push the email's score below the 0.5 threshold and make it disappear from the report, even though those two labels are not semantically competing with ``email`` at all. The mechanism is unclear — possibly a label-list length effect on the model's projection space, possibly an artefact of nvidia/ gliner-pii's training data — but the symptom is reproducible. Two-detector mode (the default) is robust to this: Presidio catches the email at score 1.0 and feeds it through ``pii.py``'s cross- detector dedup, so the report still surfaces the EMAIL_ADDRESS span. The user-visible regression is confined to ``detectors=["gliner"]`` mode and to the contribution of the email to ``detection_confidence`` when both detectors run (cross-detector agreement collapses from 1.0 to 0.5 for that one span). For the b2ai voice-data workflow the cost of losing the clinical-extension coverage was lower than the cost of weakening the email signal — so this commit removes the extensions and matches b2aiprep #256's HIPAA-18 list verbatim. The empirical finding is captured as a second ``DO NOT extend this list`` warning above the constant, so a future contributor doesn't re-add ``health_condition`` / ``medication`` / similar adjacent labels without first re-running the diagnostic against the email- detection case. Callers who actually want the clinical-extension coverage (or any other labels beyond the HIPAA-18) can still pass them via ``gliner_labels=[...]`` on the dispatch function — they just take on the responsibility of verifying their specific additions don't trip the interference on their corpus. Module docstrings on both pii.py and pii_subprocess.py updated to reflect that GLiNER's coverage is now strictly HIPAA-aligned — no longer the broader "medical / health entities and unstructured personal-context references" framing the previous commit used. 22 tests in pii_subprocess_test.py + pii_test.py still pass; they key off ``_DEFAULT_GLINER_LABELS`` as a literal, not its content. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
#2 (pii.py:159) — replace literal `2` in detector-agreement divisor with `len(_KNOWN_DETECTORS)`. Today they're equal, but the symbolic form removes the silent-miscalculation trap if a third detector is ever added to `_KNOWN_DETECTORS`. #3 (pii_subprocess.py:327) — drop the `or None` fallback when passing `entities` to `AnalyzerEngine.analyze`. Presidio interprets `entities=None` as "scan for ALL entity types"; `[] or None` collapses to `None`, so a caller passing `entities=[]` (explicit "scan for nothing") got the opposite of intent. The dispatcher already substitutes the default full list when the caller-side `presidio_entities` is None, so the worker never needs to pass None itself — empty list now passes straight through to Presidio. #4 (pii_subprocess.py:334) — change the post-scan category-filter guard from `if presidio_entities` to `if presidio_entities is not None`. Same falsy-empty-list trap as #3: with the old guard, an explicit empty list silently skipped the filter so any Presidio result would pass through. With `is not None`, the empty-list path filters everything out, matching the "scan for nothing" semantics. The comment is updated to clarify this is a belt-and-suspenders check on Presidio's output rather than a category fallback. Gemini's review comment #1 (the torch version pin) is being rejected in a separate PR reply — torch 2.8 has been on PyPI for months (latest is 2.12.0), the pin is intentional for NeMo trunk SALM compatibility, and Gemini's training data predates the release. Tests: src/tests/audio/workflows/audio_analysis/{pii,pii_subprocess}_test.py 22/22 still pass; the empty-list path was already covered by the detector-agreement tests with `len(_KNOWN_DETECTORS)` in scope. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
#1 (high) — `_is_meaningful` filter in `from_dict` still used truthy checks (`bool(c.get("text"))`), which silently dropped `{"text": ""}` child chunks even though the root validator now accepts them. That defeats the PR's whole point for payloads that pass through `from_dict` (subprocess IPC, saved JSON, MMS-aligner outputs that go through the dict shape). Switched to `is not None` so the filter only drops children whose text AND speaker are wholly absent / None — the same invariant the root validator enforces. Comment block updated to explain why empty-text chunks must survive and what the filter is now guarding against. #2 (medium) — added two tests that would have caught the inconsistency above: - test_from_dict_preserves_empty_text_in_nested_chunks: payload with `chunks=[{"text": ""}]` survives `from_dict` and the empty chunk is preserved. - test_from_dict_still_drops_chunks_missing_both_text_and_speaker: the original `_is_meaningful` purpose (filtering placeholder children that would trip the root validator) is preserved — only its over-aggressive truthiness behavior is loosened. Tests: 13/13 pass (was 11; +2 new). Pre-commit clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Feat/pii subprocess venv
ScriptLine: accept empty text as "model produced no transcript" signal
Under many parallel jobs (e.g. analyze_audio across SLURM nodes), every from_pretrained/pipeline call makes a network revision check against the HF Hub even when the model is already cached. Concurrent jobs collectively trip HTTP 429 rate limits, which surface as hard ASR/alignment task failures (a recent batch showed ~3k Qwen3-ASR failures, all 429 on huggingface.co/api/models/...). Add in-library resilience — no external env vars required: - hf_offline_loading(): ensure a snapshot is cached once across processes/nodes (existing cross-node heartbeat lock + retry), then load from the local cache only (HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE) so no network revision check fires. Restores env afterward (even on error). - load_hf_resilient(): loader-agnostic offline-load + retry-on-429 wrapper. - hf_subprocess_env(): subprocess analogue; sets the offline flags in the child env only when all referenced models are cached (supports workers that load >1 model, e.g. Qwen3-ASR + its companion forced aligner). Apply at the ASR + forced-alignment load sites that were 429-ing: Whisper/HF ASR pipeline, Granite, MMS aligner + per-language wav2vec2 (in-process); Qwen, Canary, NeMo (subprocess workers). Add unit tests for the new helpers (no network). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive audio analysis workflow, a three-axis uncertainty framework, multilingual MMS forced-alignment, and support for new ASR backends (Canary-Qwen, Qwen3-ASR, and Granite Speech) with robust CUDA-aware virtual environment provisioning and PII detection. The review feedback highlights several critical bugs and improvement opportunities, including a timestamp ratio calculation bug for stereo audio in forced_alignment.py, potential process crashes in cuda_probe.py due to narrow exception handling, a potential AttributeError in script_line.py, and opportunities to reduce code duplication and remove unused helper functions in aggregate.py.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| # multiplies the ratio by N and inflates timestamps into the millions of | ||
| # seconds, which is what produced the spurious chunk start/end values | ||
| # observed in earlier Granite/Canary alignment outputs. | ||
| ratio = duration * waveform_segment.size(0) / (trellis.size(0) - 1) |
There was a problem hiding this comment.
The calculation of ratio multiplies duration by waveform_segment.size(0). Since waveform_segment is shaped (channels, samples), size(0) represents the number of channels (which is 1 for mono, but 2 for stereo). If the input audio is stereo, this will double the ratio and result in incorrect timestamps. It is safer and cleaner to calculate the ratio directly as duration / (trellis.size(0) - 1) without multiplying by the channel count.
| ratio = duration * waveform_segment.size(0) / (trellis.size(0) - 1) | |
| ratio = duration / (trellis.size(0) - 1) |
| f"Attempted index: {self.attempted_index.url}\n" | ||
| f"Failing packages: {self.failing_packages}\n" | ||
| "Action: downgrade CUDA, set SENSELAB_TORCH_INDEX_URL=https://download.pytorch.org/whl/cpu " | ||
| "for CPU fallback, or wait for upstream wheels." |
There was a problem hiding this comment.
Catching only FileNotFoundError when running nvidia-smi can lead to unhandled exceptions (such as PermissionError or other system-level OSError variants) which would crash the process. It is safer to catch OSError to handle all process execution failures gracefully.
except (OSError, subprocess.TimeoutExpired):| text=True, | ||
| timeout=_PROBE_TIMEOUT_S, | ||
| ) | ||
| except (FileNotFoundError, subprocess.TimeoutExpired): |
There was a problem hiding this comment.
Catching only FileNotFoundError when running nvcc can lead to unhandled exceptions (such as PermissionError or other system-level OSError variants) which would crash the process. It is safer to catch OSError to handle all process execution failures gracefully.
| except (FileNotFoundError, subprocess.TimeoutExpired): | |
| except (OSError, subprocess.TimeoutExpired): |
| def _is_meaningful(c: Dict[str, Any]) -> bool: | ||
| return c.get("text") is not None or c.get("speaker") is not None |
There was a problem hiding this comment.
To prevent potential AttributeError if a chunk element c is None, add a defensive null check before calling .get() on it.
| def _is_meaningful(c: Dict[str, Any]) -> bool: | |
| return c.get("text") is not None or c.get("speaker") is not None | |
| def _is_meaningful(c: Dict[str, Any]) -> bool: | |
| return c is not None and (c.get("text") is not None or c.get("speaker") is not None) |
| def aggregate_presence(votes: dict[str, dict[str, Any]]) -> float | None: | ||
| """Calibrated "is voice present?" uncertainty in ``[0, 1]``. | ||
|
|
||
| The presence question is binary, but the goal is *not* to measure | ||
| disagreement among voters (Shannon entropy did that) — it's to tell the | ||
| user whether voice is present and how strongly the evidence supports | ||
| that conclusion. A 6-of-8 split should read as "moderate evidence for | ||
| voice", not "high uncertainty". | ||
|
|
||
| Algorithm: | ||
|
|
||
| 1. For each voter, derive a per-voter probability of voice: | ||
|
|
||
| - With ``native_confidence`` ``c`` and ``speaks=True``: ``p = c`` | ||
| (the model says "voice with confidence c"). | ||
| - With ``native_confidence`` ``c`` and ``speaks=False``: ``p = 1 - c`` | ||
| (the model says "silence with confidence c", so voice prob is ``1-c``). | ||
| - Without ``native_confidence``: ``p = 1.0`` if ``speaks=True``, else | ||
| ``0.0`` (binary vote — the model is fully committed either way). | ||
|
|
||
| 2. ``p_voice = mean(per-voter probabilities)`` — equal weight per voter. | ||
| Models with a ``native_confidence`` that aren't sure pull ``p_voice`` | ||
| toward 0.5 even when their boolean ``speaks`` flips one way; binary-only | ||
| voters get equal weight to confidence-bearing ones. | ||
|
|
||
| 3. Uncertainty = ``1 − |2 · p_voice − 1|``: 0 when all evidence agrees | ||
| (either way), 1 at a perfect 50/50 split. Whether voice is more likely | ||
| present or absent is recoverable from ``p_voice`` itself; this metric | ||
| only grades how decisive the evidence is. | ||
| """ | ||
| p_voice_per_voter: list[float] = [] | ||
|
|
||
| for v in votes.values(): | ||
| if not isinstance(v, dict): | ||
| continue | ||
| if "speaks" not in v: | ||
| continue | ||
| speak_val = v.get("speaks") | ||
| if speak_val is None: | ||
| continue | ||
| raw_nc = v.get("native_confidence") | ||
| nc: float | None | ||
| if raw_nc is None: | ||
| nc = None | ||
| else: | ||
| try: | ||
| nc = max(0.0, min(1.0, float(raw_nc))) | ||
| except (TypeError, ValueError): | ||
| nc = None | ||
| # ASR hallucination flag: the voter said "speaks" but the model's own | ||
| # silence head says otherwise — treat as a vote AGAINST voice instead | ||
| # of for it (reflecting that the transcript came from a hallucination, | ||
| # not actual audio content). | ||
| if v.get("hallucinated"): | ||
| p_voter = 0.1 # near-zero with a hint of caution rather than full 0 | ||
| elif nc is None: | ||
| p_voter = 1.0 if speak_val else 0.0 | ||
| else: | ||
| p_voter = nc if speak_val else (1.0 - nc) | ||
| p_voice_per_voter.append(p_voter) | ||
|
|
||
| if not p_voice_per_voter: | ||
| return None | ||
| p_voice = sum(p_voice_per_voter) / len(p_voice_per_voter) | ||
| return max(0.0, min(1.0, 1.0 - abs(2.0 * p_voice - 1.0))) |
There was a problem hiding this comment.
The loop in aggregate_presence is identical to the one in presence_p_voice. To reduce code duplication and improve maintainability, aggregate_presence can directly call presence_p_voice and apply the final transformation on the result.
def aggregate_presence(votes: dict[str, dict[str, Any]]) -> float | None:
p_voice = presence_p_voice(votes)
if p_voice is None:
return None
return max(0.0, min(1.0, 1.0 - abs(2.0 * p_voice - 1.0)))| def _normalize_transcript_for_wer(text: str) -> str: | ||
| """Lowercase, strip non-word punctuation, collapse whitespace.""" | ||
| if not text: | ||
| return "" | ||
| cleaned = _PUNCTUATION_PATTERN.sub(" ", text.lower()) | ||
| return _WHITESPACE_PATTERN.sub(" ", cleaned).strip() |