Skip to content

Fix/hf offline resilient loading#526

Closed
wilke0818 wants to merge 22 commits into
mainfrom
fix/hf-offline-resilient-loading
Closed

Fix/hf offline resilient loading#526
wilke0818 wants to merge 22 commits into
mainfrom
fix/hf-offline-resilient-loading

Conversation

@wilke0818

Copy link
Copy Markdown
Collaborator
  • HF models create 429 errors when loading too many, especially when parallelizing across nodes and calling many models sequentially
  • Fixes base logic for this
  • Should extend across all models/HF spaces if not already done

satra and others added 22 commits May 8, 2026 17:30
…#510)

* Add analyze_audio.py: multi-task audio analysis with cache + LS export

Single-command developer script that runs senselab's full task suite
(diarization, AST, YAMNet, multi-backend features incl. squim quality
metrics, ASR, speaker embeddings) on an input audio file with and
without speech enhancement. Multiple models per task; AST and YAMNet
each driven at their native temporal resolution.

Output layout under artifacts/analyze_audio/<stem>_<timestamp>/:
- summary.json + per-(pass × task × model) JSON files
- labelstudio_tasks.json + labelstudio_config.xml for direct LS import
  (one task per audio variant; one timeline track per analyzer × model)

Content-addressable cache under artifacts/analyze_audio_cache/ keyed by
sha256(audio_signature, task, model, params, wrapper_hash, senselab_ver,
schema_ver). Re-running with identical inputs replays prior outcomes
without invoking models. Per-model failures are captured in their own
JSON without aborting the run.

Forced-alignment auto-pairing for timestamp-less ASR models, the
multilingual aligner, NeMo Canary-Qwen, and Alibaba Qwen3-ASR backends
land in follow-up commits on this branch (see
specs/20260506-154425-audio-analysis-asr-extensions/tasks.md).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Add uroman to nlp extra for forced-alignment ja/zh romanization

uroman (MIT, pure-Python, ~1.3.1.1) is needed by the upcoming MMS
forced-alignment backend to romanize Japanese and Chinese transcripts
before MMS's Roman-character tokenizer sees them.

Lives in the optional [nlp] extra so default `uv sync` is unaffected.
The MMS aligner imports uroman lazily, only when the language is ja or
zh — users who never align in those languages don't need the extra.

Verified default `uv sync` does not pull uroman; `uv sync --extra nlp`
does. uv.lock updated to record the new package; existing transitive
deps are unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Foundational: cache schema v2 + ASR dispatch table for new backends

scripts/analyze_audio.py:
- Bump _CACHE_SCHEMA_VERSION from 1 to 2. Comment documents that v2
  introduces alignment as a separate cache entry from the ASR call that
  produced its input transcript. v1 entries are inert and can be removed.

src/senselab/audio/tasks/speech_to_text/api.py:
- Add three new module-level prefix tables for upcoming backends:
    _CANARY_PREFIXES = ("nvidia/canary-",)         # NeMo SALM via subprocess venv
    _QWEN_ASR_PREFIXES = ("Qwen/Qwen3-ASR",)        # qwen-asr wrapper via subprocess venv
    _TIMESTAMP_LESS_HF_MODELS = ("ibm-granite/granite-speech-",)
- Add stub dispatch branches for Canary and Qwen-ASR that raise
  NotImplementedError pointing at the follow-up tasks (T025-T027 for
  Canary, T030-T032 for Qwen). Lands the routing structure now so the
  modules can be filled in independently without re-touching api.py.
- For models matching _TIMESTAMP_LESS_HF_MODELS, default
  return_timestamps=False so the HF pipeline does not raise. Callers
  can still override explicitly. This unblocks IBM Granite Speech 3.3
  through the existing huggingface.py path (the analyze_audio script
  then auto-aligns via the upcoming MMS forced-alignment backend).

Dispatch order: NeMo (existing) -> Canary (stub) -> Qwen-ASR (stub) ->
HF pipeline (default with timestamp-less knowledge). The four prefix
groups are disjoint.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Script: separable ASR/alignment cache + auto-align CLI flags + smoke tests

Three-phase update to scripts/analyze_audio.py landing US1+US2+US3
script-side scope:

US1 (one-command E2E): no behavior change; smoke tests in
src/tests/scripts/analyze_audio_test.py exercise argparse defaults,
audio_signature stability, the cache-key composition contract, the LS
label-collection helper, and the serializer. 12 tests, no model loads,
runs in <1 s on default install.

US2 (re-run without re-computation):
- Add transcript_signature(text), align_cache_key(...), and
  run_alignment_cached(...) — independent caching for the alignment
  step that mirrors run_task_cached but keys on (audio_signature,
  transcript_sha, language, aligner_model, aligner_params, wrapper,
  senselab_ver).
- Failed alignments are NOT stored, so a future fix to the aligner /
  senselab upgrade triggers a fresh attempt; the parent ASR cache is
  unaffected per FR-024 / FR-025 / FR-026.

US3 (hierarchical LS export):
- Three-case LS-export branch for ASR: (a) ASR with native timestamps
  -> per-segment regions from the ASR result; (b) ASR text-only with a
  successful alignment outcome attached -> per-segment regions from
  alignment; (c) ASR text-only with no/failed alignment -> single
  full-audio TextArea region (graceful degradation per FR-025).
- Add _asr_has_timestamps() helper used by the LS export and
  (eventually) by the auto-align stage to short-circuit cases where
  alignment is unnecessary.

CLI surface (per contracts/cli.md, wired to the upcoming auto-align
stage that lands with US6):
- --no-align-asr (toggle off auto-align)
- --aligner-model (default facebook/mms-1b-all)
- --asr-language (force a specific language for the aligner)
- --qwen-asr-no-timestamps (skip Qwen's bundled forced-aligner)
- 'alignment' added to --skip choices.

Default --asr-models updated to include ibm-granite/granite-speech-3.3-8b
back into the comparison set; senselab's HF-pipeline path now defaults
return_timestamps=False for that model id (committed in da04a29), and
the upcoming MMS auto-align stage will add per-segment timestamps.

The auto-align stage itself is not yet wired (T021 lands with the MMS
backend in US6). Until then, --no-align-asr / --aligner-model /
--asr-language are accepted but unused.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* forced_alignment: MMS multilingual aligner backend (1100+ languages)

Adds Meta's MMS (Massively Multilingual Speech) as an opt-in aligner
backend for senselab.audio.tasks.forced_alignment, covering 1100+
languages via per-language adapters on a single base model. Selected
via the new ``align_transcriptions(..., aligner_model=...)`` keyword;
the legacy per-language wav2vec2 path stays the default for existing
callers (no behavior change for current code).

constants.py:
- MMS_MODEL_ID = "facebook/mms-1b-all" (CC-BY-NC 4.0 weights; override
  the model id for commercial-friendly alternatives).
- ISO_1_TO_3 map for adapter selection (en, fr, de, es, pt, it, ja, zh,
  plus the 19 other languages already in DEFAULT_ALIGN_MODELS_HF, which
  MMS now also covers).

forced_alignment.py:
- _romanize_for_mms(text): lazy uroman import (optional dep, [nlp]
  extra) for ja/zh transcripts. Without uroman installed, raises an
  actionable ImportError pointing at `uv sync --extra nlp`.
- _load_mms_aligner(iso3, device, cache): one-time load of the MMS
  base model + selected per-language adapter via processor.tokenizer
  .set_target_lang(iso3) + model.load_adapter(iso3). Cached by
  (model_id, iso3) so multiple calls on the same audio reuse weights.
- align_transcriptions: new ``aligner_model`` kwarg. When set to
  MMS_MODEL_ID, dispatches to the MMS path; ja/zh transcripts go
  through _romanize_for_mms before tokenization. Uses model_type=
  "huggingface" internally — the wav2vec2 forward pass and CTC
  trellis/backtrack code is identical, so no changes to
  _get_prediction_matrix / _align_transcription / _assign_timestamps.

Output shape unchanged: List[List[ScriptLine | None]]. Existing 15
forced_alignment tests still pass (run via uv run pytest
src/tests/audio/tasks/forced_alignment_test.py).

speech_to_text/huggingface.py:
- _get_hf_asr_pipeline and transcribe_audios_with_transformers now
  accept ``return_timestamps: Union[str, bool, None]`` (was Optional[str]).
  HuggingFace's automatic-speech-recognition pipeline supports False
  to disable timestamps; the existing senselab dispatcher (api.py,
  committed earlier) defaults this to False for known-timestamp-less
  HF models like ibm-granite/granite-speech-3.3-8b. Public docstring
  updated to call out this case.

This commit lands T016-T020 from the spec-kit task plan. Next: T021
wires the analyze_audio script's auto-align stage to call
align_transcriptions(..., aligner_model=facebook/mms-1b-all) on every
timestamp-less ASR output.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* analyze_audio: wire MMS auto-align stage; HF return_timestamps=False fix

scripts/analyze_audio.py:
- Wire the auto-align stage in run_pass(): after the ASR family runs,
  iterate every successful ASR ModelRun. Skip when --no-align-asr is
  set, when 'alignment' is in --skip, or when the ASR result already
  has native timestamps (Whisper-style start/end or word chunks).
  Otherwise call senselab.audio.tasks.forced_alignment.align_transcriptions
  via run_alignment_cached(), with the resolved language (from
  --asr-language, default "en"), the chosen aligner_model (default
  facebook/mms-1b-all), and an alignment-specific provenance block
  (transcript_sha, language, parent_asr_cache_key) per FR-026.
- Imports forced_alignment.align_transcriptions and the Language /
  ScriptLine types from senselab.utils.data_structures.
- Add _extract_transcript_text() helper that handles both ScriptLine
  objects (post-fresh-ASR) and the dict shape (post-cache-hit) so the
  auto-align step works after either kind of ASR outcome.
- Update _asr_has_timestamps() to handle dict-shape ScriptLines too.
- Refresh the script docstring's ASR section: Granite is now in
  defaults (auto-aligned), removing the older "intentionally NOT in
  defaults" note that the spec clarification overrode. Add a paragraph
  describing the auto-align stage and the --no-align-asr toggle.

src/senselab/audio/tasks/speech_to_text/huggingface.py:
- Fix _get_hf_asr_pipeline to OMIT return_timestamps from the kwargs
  passed to transformers.pipeline() when the caller asked for False.
  HF's AutomaticSpeechRecognitionPipeline behaves differently per
  model family: CTC raises on False (only "char"/"word" allowed),
  Whisper accepts False, and other generative seq2seq (Granite Speech)
  raises on True/word/segment. Omitting the kwarg lets each pipeline
  default sanely; omitting only happens when the caller explicitly
  passed False (timestamps were not asked for). Other values
  (str/None/True) are passed through unchanged.

src/tests/audio/tasks/forced_alignment/mms_test.py:
- New skipif-guarded smoke tests for the MMS aligner. Skipif checks
  whether facebook/mms-1b-all is in the local HF cache (~1.6 GB
  download otherwise; default CI install does not pull it). When
  available: verify (a) English alignment produces well-shaped
  ScriptLines via the real-speech fixture, (b) Japanese alignment
  exercises the uroman romanization path, (c) an unknown language
  code raises a clear ValueError pointing at the ISO_1_TO_3 registry.
  Assertions are SHAPE-ONLY — synthetic transcripts on real audio do
  not align meaningfully but the API contract is still verified.

src/tests/audio/tasks/speech_to_text/huggingface_no_timestamps_test.py:
- New skipif-guarded smoke test for return_timestamps=False. Uses
  facebook/wav2vec2-base-960h (already cached locally for the existing
  English aligner path) rather than Granite Speech (multi-GB).
  Verifies: when return_timestamps=False, transcribe_audios_with_
  transformers returns text-only ScriptLines without raising the HF
  pipeline's CTC-rejects-False ValueError.

Existing 11 senselab speech_to_text_test.py tests still pass; existing
15 forced_alignment_test.py tests still pass.

This commit lands T021-T023. T024 (end-to-end Granite + Whisper
validation through the analyze_audio script with real model downloads)
is deferred to a follow-up session — needs ~16 GB Granite Speech
download plus ~1.6 GB MMS download.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* speech_to_text: NVIDIA Canary-Qwen 2.5B subprocess-venv backend

Adds CanaryQwenASR which loads SALM (Speech-Augmented Language Model)
from nemo.collections.speechlm2.models in an isolated nemo-canary-qwen
venv (kept separate from the existing nemo-diarization venv to avoid
destabilizing Sortformer/Conformer-CTC paths). The chat-style prompt
format using model.audio_locator_tag matches the published model card.

Canary-Qwen is text-only (no native timestamps); the analyze_audio
script's auto-align stage (US6) now adds per-segment timing via the
multilingual MMS forced-aligner downstream when this model is used.

Wires the api.py dispatch (replaces the NotImplementedError stub) and
adds a shape-only smoke test that auto-skips when the venv has not
been provisioned (~5 GB first-run install).

Marks T025-T028 done in specs tasks.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* speech_to_text: Alibaba Qwen3-ASR subprocess-venv backend

Adds QwenASR which loads Qwen3-ASR via Alibaba's qwen-asr Python wrapper
(Qwen3ASRModel.from_pretrained) inside an isolated qwen-asr venv,
optionally pairing with the bundled Qwen3-ForcedAligner-0.6B companion
to produce per-word / per-CJK-char timestamps.

Wires api.py dispatch (replaces the NotImplementedError stub) and
threads --qwen-asr-no-timestamps from analyze_audio.py through to the
ASR call so the script's MMS auto-align stage can take over when the
bundled aligner is undesirable. Adds shape-only smoke tests that
auto-skip until the qwen-asr venv has been provisioned.

Marks T030-T033 done in specs tasks.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* polish: lint cleanup + mark Phase 9 tasks done

- Add docstring to src/tests/scripts/__init__.py (D104).
- Mark T035-T037 done in specs tasks.md after running:
  * full ruff check + mypy on changed paths (clean)
  * full senselab pytest suite excluding scripts/ (421 passed,
    25 skipped, no regressions; voice_cloning_test ERROR in
    bulk run is a known order-dependent flake — passes in isolation,
    unrelated to forced_alignment / speech_to_text changes).

T035 (CLAUDE.md update) is applied to the local file; CLAUDE.md is
not yet under version control in this repo so the section addition
lives in the working tree only — user decides whether to track it.

T038 (final E2E validation against a tutorial WAV) intentionally
deferred: requires the nemo-canary-qwen and qwen-asr venvs (~5-15 GB
combined) and the MMS aligner (~1.6 GB) to be provisioned locally.
The skipif-guarded smoke tests ride the venvs whenever the user
provisions them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* e2e validation: fix bugs surfaced by full-pipeline run

Validates T012/T015/T015a/T024/T029/T034/T038 end-to-end. Bugs found
and fixed during the validation cycle:

- canary_qwen.py worker: SALM.generate expects prompts as
  list[list[dict]] (a batch of conversations), not list[dict]. Fixed.
  Canary-Qwen now succeeds in ~70s on a 38s clip.

- forced_alignment.py: senselab's Language normalizes ISO-639-1 ('en')
  to ISO-639-3 ('eng'). The MMS aligner now accepts either form.

- script_line.py from_dict: three pre-existing latent bugs surfaced by
  MMS-aligner output:
    * d["chunks"][0] accessed without checking non-empty (IndexError)
    * d["timestamps"][0] accessed without checking length >= 2
      (IndexError)
    * recursive construction of empty subsegments with text="" failing
      the "at least text or speaker" pydantic validator
  Empty subsegments are now filtered before recursive construction.

- analyze_audio.py LS export: pre-existing bug where
  getattr(seg, "start") returned None for cache-restored dicts,
  silently dropping all regions on cache-hit runs. Added _seg_attr
  helper that handles both Pydantic objects and JSON dicts. Also
  unwrap align_transcriptions' List[List[ScriptLine]] to inner segment
  list before passing to _asr_to_ls.

- mms_test: align test expectation with senselab.Language's stricter
  ISO validation (Language("xx") raises before reaching MMS).

Validation results in artifacts/analyze_audio_e2e_validation.md (not
committed). Marks T012/T015/T015a/T024/T029/T034/T038 done. The two
known gaps documented in the validation note: YAMNet (corrupt local
tfhub cache, environmental) and Granite Speech 3.3 (HF pipeline
contract mismatch — needs a dedicated subprocess-venv backend in a
follow-up).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* speech_to_text: IBM Granite Speech 3.3 in-process backend

Adds GraniteSpeechASR which loads ``GraniteSpeechProcessor`` and
``AutoModelForSpeechSeq2Seq`` directly in-process — no subprocess venv
needed because transformers, torch, torchaudio, and peft all live in
senselab's core install (peft is required for Granite's LoRA adapter;
loading without it produces gibberish).

Why a dedicated backend rather than the HF ASR pipeline path: Granite
uses ``GraniteSpeechProcessor(text, audio, device, **kwargs)`` and a
``GraniteSpeechFeatureExtractor`` whose ``__call__(audios, device)``
signature does not accept the ``{array, sampling_rate}`` audio dict
that ``AutomaticSpeechRecognitionPipeline`` passes in. Routing through
the pipeline failed at the feature-extractor boundary in earlier E2E
runs.

Worker subtleties surfaced by the E2E:
- ``return_tensors="pt"`` is mandatory; without it Granite's processor
  returns ``input_ids`` / ``attention_mask`` as nested Python lists,
  which the underlying ``generate()`` then tries to call ``.shape`` on.
- The feature extractor expects a 1-D audio tensor; senselab's
  ``Audio.waveform`` is shape ``[channels, T]``, so we squeeze after
  downmix.
- bfloat16 on accelerators (CUDA + MPS), float32 on CPU. CPU bf16 runs
  but is ~40x slower than fp32 because PyTorch's CPU kernels lack a
  native bf16 SIMD path. MPS bf16 was verified against torch 2.11.

Adds ``peft>=0.13`` to senselab's core deps. Wires api.py dispatch
(replaces the timestamp-less HF route), drops Granite from
``_TIMESTAMP_LESS_HF_MODELS``, and adds a shape-only smoke test that
auto-skips when the model weights are not in the local HF cache
(~16 GB).

Routed automatically by ``transcribe_audios`` when the model id starts
with ``ibm-granite/granite-speech-``. Granite is text-only; the
analyze_audio script's auto-align stage adds per-segment timestamps
via the multilingual MMS forced-aligner downstream.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* PR review: type annotations, MMS aligner cache, ruff format

Addresses pre-commit failures and Gemini's review feedback on PR #510:

- Test files: add ``model: HFModel`` annotations on canary_qwen_test,
  qwen_test, granite_test, huggingface_no_timestamps_test (mypy 1.20.1
  is stricter than 1.18.x and infers Self<HFModel> when the literal
  HFModel(...) call is on the RHS — explicit annotation pins the type).

- ``forced_alignment.align_transcriptions``: hoist the
  processor/model cache from a per-call local dict to a module-level
  ``_ALIGNER_CACHE``. The 1B-parameter MMS base weights (and any
  per-language wav2vec2 variants) now load once per process instead of
  once per call — fixes a perf cliff Gemini flagged in the review.

- ruff format: pick up the auto-applied formatting in
  scripts/generate_model_registry.py + scripts/profile_model_tiers.py
  (no functional changes; ruff 0.15.11 vs older local), plus a small
  re-flow in script_line.py from_dict and granite.py.

Did NOT apply Gemini's two other suggestions:
- Drop ``device=`` arg in Granite ``processor(...)`` call: verified
  Granite's ``GraniteSpeechProcessor.__call__`` accepts ``device``
  explicitly in its signature; works on MPS in run 15.
- Drop ``input_len`` slicing on generate output: empirically verified
  via standalone MPS run that with slicing the transcript decodes
  correctly (``"so once upon a time there were four little rabbits..."``);
  without slicing it would prepend the prompt. Granite is an LLM-style
  multimodal generator, not a true seq2seq.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* analyze_audio: per-window features (1s/0.5s default), parquet output

Replaces the single global summary that the script was emitting for
opensmile / parselmouth / torchaudio-squim with proper time-series
output. The rest of the analysis (diarization, AST, YAMNet, ASR,
alignment, embeddings) is already temporal — features should be too.

Routing:
- opensmile uses ``LowLevelDescriptors`` (native ~10 ms frame grid).
  Run once; one parquet row per opensmile frame.
- parselmouth + torchaudio-squim are summary statistics by design;
  the senselab wrapper only exposes the aggregate form, so these run
  in an external sliding window driven by the new
  ``--features-win-length`` (default 1.0 s) and
  ``--features-hop-length`` (default 0.5 s) flags. One parquet row
  per window.

Each backend writes its own parquet sidecar
(``<pass>/features/<backend>.parquet``) — different columns and
different time grids don't share a schema; per-backend files keep
each grid clean and let downstream readers join on
``start``/``end`` if they want a unified view. The cache outcome
metadata stays in JSON for inspection parity with the other tasks
(``features.json`` now points to the parquet directory).

Verified end-to-end on a 38.4 s clip (run 16, both passes):
- opensmile.parquet: (3837, 27), 668 KB, 0–38.411 s
- parselmouth.parquet: (75, 43), 49 KB, 0–38 s
- torchaudio_squim.parquet: (75, 6), 6.5 KB, 0–38 s

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* PR review: pre-commit fixes (codespell + pretty-format-toml)

- codespell: add ``vie`` (Vietnamese ISO-639-3 code in ISO_1_TO_3
  registry, not the English noun) to ignore-words-list.
- pretty-format-toml: re-flow [tool.codespell] block so the ``vie``
  addition lands on a properly-formatted line.

No code changes; all green pre-commit locally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* analyze_audio: foundational comparator framework + CLI flags

Lays the foundation for the comparison & uncertainty stage. Adds the
new CLI flags, cache key composition, ComparisonRow column lists,
parquet writer with comparator_provenance metadata, ComparisonGrid
dataclass + iter_buckets, _token_overlaps_window helper, uncertainty
aggregator (min/mean/harmonic_mean/disagreement_weighted), LS XML
extension hooks for comparator tracks, and disagreements.json
index builder.

This commit alone produces no reviewer-visible output — the
user-story phases (US1–US4) plug differencer logic into this
plumbing in subsequent commits.

Constitution checks:
- §VIII (No Hardcoded Parameters): every comparator threshold and
  grid resolution is a CLI flag; defaults documented.
- §IV (CI Must Stay Green): existing 12 script tests still pass;
  11 new comparator-foundational tests added (23 total, all green).
- §VII (Simplicity First): comparator code lives inline in
  analyze_audio.py; no new senselab task module needed.

Adds g2p-en to the [nlp] extra (small wheel; needed by the future
US3 ASR↔PPG comparator). NLTK averaged_perceptron_tagger_eng
auto-downloads on first use.

Bumps cache schema 2 → 3 to reflect the new comparator key shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* analyze_audio: US1 raw-vs-enhanced comparator + tests

Adds the per-task differencer registry (_diff_diarization,
_diff_classification, _diff_asr) and the top-level
compare_raw_vs_enhanced driver. Wires it into main() so a default
two-pass run now emits parquet sidecars under
<run_dir>/comparisons/raw_vs_enhanced/<task>/<model>.parquet plus a
disagreements.json index ranking the top-N flagged regions.

LS bundle additions: one Labels track per (task, model) actually
compared, named pass_pair__compare__<task>__<model> with the fixed
enum {agree, disagree, incomparable, one_sided}; ASR-vs-ASR pairs
also get a sibling TextArea track carrying WER + both transcripts.
Track names are appended via build_labelstudio_config(... ,
comparator_tracks=tracks); a run with --skip comparisons produces
bit-identical output to current main (FR-005, SC-005 anchored by
test_comparator_skip_no_op_preserves_ls_bundle_shape).

Cache key composition follows the comparison_cache_key helper from
Phase 2, using the upstream-task cache_keys so re-running an
upstream task cleanly invalidates downstream comparisons.

Tests added (29 total now in src/tests/scripts/, all green):
- test_raw_vs_enhanced_diarization_diff covers boundary_shift
- test_raw_vs_enhanced_asr_text_diff covers per-window WER + a/b text
- test_raw_vs_enhanced_handles_failed_pass covers the incomparable
  edge case from FR-010 / US1 acceptance scenario 3

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* analyze_audio: US2 within-stream comparator + tests

Adds compare_within_stream that, for each task with two or more
successful models in a pass, iterates itertools.combinations and
emits one parquet per ordered model pair under
<run_dir>/<pass>/comparisons/<task>/<a>_vs_<b>.parquet (US2 layout
in contracts/comparison-row.parquet.md). For ASR pairs the WER row
schema is reused; for diarization the speaks_a/speaks_b shape; for
classification (AST vs YAMNet) the top1/score/entropy shape, with
the existing scene_agreement.json semantics preserved as a strict
subset.

LS bundle: one Labels track per (pass, task, model_pair) with the
fixed enum {agree, disagree, incomparable, one_sided}; ASR pairs
also emit a sibling TextArea track carrying WER + both transcripts.

ASR-vs-ASR honors --asr-reference-model (default Whisper turbo) so
the soft-reference side is consistent across runs.

Tests added (29 total now, all green):
- test_within_stream_asr_pair_emits_wer covers the WER per-bucket path
- test_within_stream_single_model_no_op covers the graceful no-op
  when a task has only one successful model
- test_within_stream_classification_superset_of_scene_agreement
  covers the AST-vs-YAMNet matching-grid path

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* analyze_audio: US3 cross-stream comparator + tests

Adds compare_cross_stream and three sub-comparators:

- _diff_asr_vs_diarization: per cross-stream-grid bucket, sets
  asr_says_speech via _token_overlaps_window (Q4 clarification:
  any transcript token whose timestamp range overlaps the bucket
  counts) and diar_says_speech via diarization-segment overlap.
- _diff_classification_vs_diarization: per bucket, picks the
  AST/YAMNet window whose center is closest to the bucket center
  and asks whether its top-1 label is in the configurable
  --speech-presence-labels allowlist (default: AudioSet 'Speech'
  subtree); compares to diarization the same way.
- _diff_asr_vs_ppg: per bucket, derives ARPAbet phonemes from the
  ASR transcript via g2p_en, argmaxes the PPG output frames, and
  reports a continuous phoneme_per (jiwer.wer) plus a boolean
  phoneme_disagreement flag at >= --phoneme-disagreement-threshold
  (default 0.50; Q5 two-tier clarification).

PPG path gracefully degrades when no PPG result is available —
records "PPG backend not provisioned" in disagreements.json's
incomparable_reasons rather than crashing (FR-010, US3 acceptance
scenario 4).

LS bundle: one Labels track per (pass, stream_pair, model_pair)
with the fixed enum {agree, disagree, incomparable, one_sided}.

Tests added (37 total now in src/tests/scripts/, all green):
- test_cross_stream_asr_speech_in_silent_window
- test_cross_stream_classification_speech_allowlist
- test_cross_stream_ppg_unavailable_degrades_gracefully
- test_cross_stream_phoneme_per_two_tier (continuous PER + boolean flag)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* analyze_audio: US4 per-region uncertainty + ranked disagreements.json

Adds per-region confidence harvesters and threads combined_uncertainty
through every comparator emitted by the previous phases:

- _whisper_chunk_confidence converts Whisper's per-chunk
  avg_logprob -> exp(avg_logprob) confidence in [0, 1] and surfaces
  no_speech_prob alongside.
- _whisper_bucket_confidence averages overlapping chunks within a
  comparison-grid bucket (FR-007: arithmetic mean of native events
  inside one bucket).
- _enrich_diff_asr_with_confidence + _enrich_diff_classification_with_confidence
  populate confidence_a/b and combined_uncertainty in-place on the
  rows produced by the existing differencers.
- _models_without_native_confidence walks the summary and lists the
  pyannote/Sortformer/Granite/Canary-Qwen/Qwen3-ASR backends — these
  contribute to comparisons but lack a native scalar in v1, per
  research.md §2.

The disagreements.json index now ranks regions by combined_uncertainty
descending (NaN-last). The aggregator is the configurable
--uncertainty-aggregator (default min — most-doubtful model wins;
mean / harmonic_mean / disagreement_weighted available). Per FR-007
the missing-confidence model list rides on the index for
transparency.

Tests added (38 total now in src/tests/scripts/, all green):
- test_uncertainty_whisper_avg_logprob_extracted
- test_uncertainty_aggregator_min_default_via_compare (end-to-end:
  build two ASR results with different avg_logprob, verify the
  parquet's combined_uncertainty matches 1 - min(...))
- test_uncertainty_missing_signal_dropped_not_zeroed
- test_disagreements_index_top_n_and_ordering
- test_models_without_native_confidence_lists_known_null

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* analyze_audio: aligned timeline plot

Adds build_aligned_timeline_plot which reads a finished run dir and
emits <run_dir>/timeline.png — a stacked, time-aligned figure
showing diarization speaker bars per (pass, model), ASR token
spans per (pass, model), AST/YAMNet scene-window strips colored
by top-1 score, and one comparator row per parquet with
disagreement bars colored by combined_uncertainty (Reds; deeper red
= higher uncertainty).

Hooked into main() at the very end of a run (after disagreements.json
is written) so every default invocation now produces the figure
alongside the existing parquets / JSON. Best-effort: a plot failure
is logged to stderr and the rest of the outputs are unaffected.

Plot row layout, top-down:
- One row per (pass, diarization_model) — speaker-colored bars.
- One row per (pass, asr_model) — token spans (filled bars from
  chunk timestamps, or a single span for text-only ASR after
  auto-align).
- One row per (pass, scene-classifier) — score-shaded windows.
- One row per comparator parquet — disagreement bars; comparator
  agreement is rendered as the empty grey background.

Constitution checks:
- §VII (Simplicity First): one new function, no new module; uses
  matplotlib (already pulled in by pyannote-audio).
- §VIII (No Hardcoded Parameters): figure size scales with
  duration; no hardcoded paths beyond the conventional run_dir.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* polish: mark Phase 7 tasks done

T052-T058 marked complete in tasks.md after full E2E validation.

E2E (run twin-1_20260509-032631):
- All 13 tasks succeed across both raw_16k + enhanced_16k passes
- 38 comparator parquets emitted (within_stream / cross_stream /
  raw_vs_enhanced)
- disagreements.json: 16,513 rows, 10,133 disagreements, top-100
- timeline.png: aligned multi-row plot rendered

Senselab full suite: 399 passed, 1 failed (the same voice_cloning
order-flake from PR #510; passes in isolation; unrelated to
comparator). SC-005 (no regression in existing senselab outputs)
holds.

Smoke tests: 38 passed, 7 skipped.

Known v1 gap documented in artifacts/compare_uncertainty_e2e_validation.md:
the senselab HuggingFaceASR wrapper does not currently surface
Whisper's avg_logprob, so the disagreements.json ranking has only
NaN combined_uncertainty values across the whole run. The harvester
is correct — it has nothing to read. Follow-up PR will opt the HF
pipeline into per-chunk avg_logprob.

T059 (PR open) intentionally left unchecked — pending a fresh push.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* polish: mark T059 (PR opened) done

PR #512 opened against alpha:
#512

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* PR review: pre-commit fix (mypy 1.20 stricter type inference)

CI's mypy 1.20.1 flagged two var-annotated diagnostics that the
local mypy 1.18.x missed:

- test_comparator_skip_no_op_preserves_ls_bundle_shape: annotate
  ``summary: dict[str, Any]`` so the empty-dict literal doesn't
  collapse to ``dict[str, dict[str, dict[Never, Never]]]``.
- test_comparison_cache_key_changes_with_inputs: convert the
  ``base = dict(...)`` factory call to a literal ``{...}`` with
  an explicit ``dict[str, Any]`` annotation so the inferred value
  type accepts the heterogeneous values (str / list / dict).
  Pull the unpacked-params expression out into a local so mypy
  doesn't try to type-check ``{**base["params"], ...}`` against
  the wrong base inference.

No code changes; all pre-commit hooks pass locally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* PR #512 review: address Gemini findings

Applies the 5 actionable Gemini comments on the comparator stage:

- HIGH (line 1526): compare_raw_vs_enhanced now also runs for AST and
  YAMNet — they're top-level summary keys (single-model tasks) rather
  than under by_model. Their parquet rows carry real time spans driven
  by the per-task win/hop and combined_uncertainty derived from the
  top-1 score via _enrich_diff_classification_with_confidence.
- MEDIUM (line 725): _enrich_diff_asr_with_confidence now passes the
  per-row WER into _aggregate_uncertainty as ``mismatch_severity`` so
  the disagreement_weighted aggregator actually scales with the
  textual disagreement (matches FR-003 intent; otherwise it
  collapsed to ``mean``).
- MEDIUM (line 991/995): build_aligned_timeline_plot now reads AST
  and YAMNet win/hop from ``summary["scene_window"]`` rather than
  hardcoded 10.24/0.96 constants. CLI overrides surface in the plot
  correctly.
- MEDIUM (line 1316): _diff_classification now uses ``win_length_a``
  + new ``hop_length_a`` to compute real per-row start/end timestamps
  rather than integer placeholders. Both call sites
  (raw_vs_enhanced AST/YAMNet and within_stream AST-vs-YAMNet) pass
  the explicit hop.
- MEDIUM (line 2234): _whisper_bucket_confidence now falls back to
  the segment-level avg_logprob when the parent ScriptLine has no
  chunks — covers post-aligned text-only ASR backends like Granite
  / Canary-Qwen if they ever surface a per-segment scalar in a
  future senselab release.

Also tightens the matplotlib colormap usage to mypy 1.20-compliant
``matplotlib.colormaps.get_cmap`` form (was ``plt.cm.<name>`` which
mypy now flags as attr-defined).

Did NOT change: NLTK runtime download (line 1914). The download
is a one-time cmudict + averaged_perceptron_tagger_eng fetch
(small files); air-gapped envs can pre-populate ~/nltk_data via the
existing ``T001`` setup task before running. Will revisit if a
follow-up requires fully offline operation.

All 38 script smoke tests still pass; mypy + ruff clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* audio_analysis workflow: speech-aware speaker clustering + 3-axis uncertainty

The audio_analysis workflow lands as a standalone module under
src/senselab/audio/workflows/audio_analysis/ with the analyze_audio script
as a thin wrapper.

Highlights from this iteration:

- Speech-aware embedding clustering: a YAMNet → AST → openSMILE-loudness
  fallback mask is built per embedding window; non-speech windows are tagged
  NOISE and skip clustering, so silent / music / vehicle stretches no longer
  inflate the speaker count.
- Spectral clustering on a precomputed cosine-similarity affinity is now
  the default, with k-means as the per-k fallback. New CLI flag
  --clustering-algorithm exposes the choice.
- Min-cluster-size guard (max(5, 10% of valid windows)) rejects partitions
  where a 2-3 window outlier is propping up the silhouette score.
- Cross-model speaker unification (clustering.py + plot.py) processes the
  synthetic embedding source first to seed the centroid pool, then freezes
  the pool — so a noisy 3-segment pyannote / sortformer label can't spawn a
  unified third "speaker" beyond what the embedding source validated.

Validation: twin-1.wav unified-speaker count drops from 3 → 2; English
two-speaker conversation stays at 2 (no regression). 57/57 workflow tests
pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* PR review: pre-commit fixes (mypy annotations + test naming + toml format)

- global_summary.py: explicit `list[float]` annotations on stoi/pesq/sisdr accumulators
- test_aggregate.py: `dict[str, dict[str, Any]]` annotations on votes literals
  (mypy 1.20 narrows mixed-value literals to dict[str, object])
- Rename test_*.py → *_test.py to match repo convention
- pyproject.toml: pretty-format-toml auto-fix

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* PR review: address 6 findings + same-speaker post-merge step

Review findings addressed:

- HIGH: ``min_cluster_size`` floor was 5 windows but ``min_windows_for_clustering``
  is 4 — for passes with 4-9 valid windows every k≥2 partition got rejected
  silently and we fell to single-cluster regime with no diagnostic. Lowered
  floor to ``max(2, round(0.10 * n))`` so the gate is reachable; record a
  ``failures`` entry when all partitions are rejected by min-size.
- HIGH: ``compute_uncertainty_axes`` mutates the caller's ``passes`` dict
  (injects the synthetic embedding diar source). Removed the "pure function"
  claim from the module docstring and documented the mutation explicitly.
- MEDIUM: ``_speech_window_mask`` is a YAMNet veto, not a fallback ladder.
  Documented the veto explicitly (YAMNet decides; AST only when YAMNet
  missing; loudness only when both missing) and the known YAMNet-confusion
  failure modes (child voices flagged as "Music") plus the mitigation.
- MEDIUM: extracted ``assign_unified_clusters_with_seed_phase`` shared
  helper in clustering.py; both ``cluster_speaker_labels_by_embedding`` and
  the plot's ``_cluster_speakers_by_embedding`` now share the seed-then-
  freeze logic.
- LOW: replaced the schema-progression block comment + hardcoded
  ``"schema_version": 1`` parquet literal with a reference to the
  ``_CACHE_SCHEMA_VERSION`` constant so they can never drift.
- LOW: surfaced the spectral→k-means fallback through the ``failures`` dict
  (per-k entries) and recorded which algorithm actually produced the
  partition in the cluster output.

New: same-speaker post-merge step. Spectral / k-means at k≥2 sometimes
splits one speaker's recording into sub-clusters when prosody / distance
varies. After silhouette-driven k* selection, the closest centroid pair is
iteratively merged while cosine similarity ≥ ``merge_threshold`` (default
0.55). The threshold keeps similar-voiced but distinct speakers (same
gender + age + family resemblance ~cos_sim 0.3) separate while folding in
clear-cut same-speaker prosodic outliers (typical adult VoxCeleb same-
speaker cos_sim 0.6+). No target speaker count is assumed anywhere.

New: single-utterance / quiet-environment path. When 1+ valid speech
windows exist but the count is below ``min_windows_for_clustering``, the
clusterer returns ``n_speakers=1`` with all valid windows tagged ``S0``
instead of skipping. This is the "single speaker, just one word in an
otherwise quiet recording" case the prior fall-through silently dropped.

Validation: both ``twin-1.wav`` (2 distinct child speakers, one speaking
briefly) and the English two-speaker conversation still report
``n_speakers=2``; merge step does not fire on either (centroid cos_sim
0.31 and 0.15 respectively, both below 0.55). 57/57 workflow tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* unified speaker clustering: group-aware seed phase + cross-pass threshold

Two-pass speaker-axis fix surfaced by ``audio_48khz_mono_16bits.wav`` (a 5-second
clip where 4 distinct speakers each say ``This is <Name>``):

The synthetic embedding source's per-pass clusterer correctly found 4 distinct
speakers in both raw_16k and enhanced_16k. The plot's cross-pass alignment
``_cluster_speakers_by_embedding`` then collapsed them to 2 because it ran
``assign_unified_clusters_with_seed_phase`` with a single 0.5 cosine threshold
that didn't distinguish "within-pass distinct speakers (validated by silhouette
+ min-size + post-merge)" from "cross-pass same-speaker match". For short
similar-phonetic utterances, different speakers can sit at cos_sim > 0.5 and
got merged.

Fix: ``assign_unified_clusters_with_seed_phase`` is now group-aware:

- ``seed_groups: list[list[(key, mean_emb)]]`` — each inner list comes from a
  single per-source clustering. Within a group, every label gets its own
  distinct centroid (no within-group merging — those clusters were already
  validated upstream).
- Across groups, items merge when centroid cosine ≥ ``cross_group_threshold``
  (default 0.75 — same-speaker cos_sim across raw/enh is typically 0.85+,
  different speakers within a pass sit around 0.30-0.50).
- ``other_items`` (pyannote / sortformer) still snap to the seed pool at the
  legacy ``cosine_threshold`` (0.5) and never spawn new clusters.

Both callers updated:

- ``cluster_speaker_labels_by_embedding`` (per-pass): one seed group =
  embedding source's labels for this pass.
- Plot's ``_cluster_speakers_by_embedding`` (cross-pass): one seed group
  per (pass, embedding_source) so raw and enhanced cluster sets stay
  intact and align speaker-to-speaker across passes at 0.75.

Validation (cache-hit reruns after the fix):

- audio_48khz_mono_16bits (4 distinct speakers, brief utterances): legend
  S0–S3, 4 colors visible per embedding-source row.
- twin-1 (2 distinct child speakers): legend S0/S1, no collapse.
- english_conversation_higgs (2 adult speakers): legend S0/S1, no regression.

The per-pass speaker clustering is unchanged; only the cross-pass alignment
in the timeline plot + the per-pass cross-model unification in clustering.py
were affected. Embedding clustering remains independent for raw and enhanced
passes — same-speaker matching across them happens only when cos_sim crosses
the 0.75 cross-group bar.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix Ruff and mypy issues

* Suggested follow-ups for PR #511 continuous SER fix

- Auto-resample audio to the processor's expected sampling rate to
  avoid silent correctness bugs when callers pass non-16 kHz input.
- Coerce id2label keys to int (HF stores them as strings in JSON) and
  fall back to class_{i} so the raw-config path doesn't KeyError.
- Memoize the dynamically built Wav2Vec2 emotion model class so
  isinstance checks stay stable across calls.
- Drop redundant `AutoConfig as _AC` alias and unused `Tuple` import.
- Strengthen the continuous-SER test with score-range and regression
  assertions that would catch the random-head bug this PR fixes.

* Pad labels when shorter than scores in continuous SER

Address Gemini review on PR #513: the previous slice-only logic could
produce a labels list shorter than the model output if num_labels in
the config doesn't match the actual head size. Pad with class_{i} so
AudioClassificationResult always sees matching lengths.

* Drop regression-guard assertion that depends on fixture content

The 'any(abs(s - 0.33) > 0.05 for s in scores)' check would fail if
the test fixture happened to produce audeering outputs all in
[0.28, 0.38]. Keep only the label-set and [0, 1] range checks, which
are robust to any fixture content.

* Fix in-process loading for Wav2Vec2ForSpeechClassification SER models

Three follow-on issues surfaced when running the new continuous-SER backend
end-to-end against audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim:

1. AutoConfig fallback raised TypeError because config.json's own
   "model_type" key collided with the positional arg in
   AutoConfig.for_model("wav2vec2", ...). Pop it before splat.

2. The same checkpoints ship vocab_size: null, which trips
   huggingface_hub>=1.0 strict-dataclass validators when Wav2Vec2Config is
   instantiated. The regression head doesn't use vocab_size, so drop the
   key and let the dataclass default apply.

3. _Wav2Vec2EmotionModel called self.init_weights(), but
   transformers>=5.0's from_pretrained finalizer reads
   self.all_tied_weights_keys, which is populated by post_init. Switch to
   post_init() so loading succeeds.

Additionally swap Wav2Vec2Processor.from_pretrained for
Wav2Vec2FeatureExtractor.from_pretrained, since the processor path
re-invokes AutoConfig.from_pretrained for the tokenizer (re-tripping the
vocab_size validator) and regression inference doesn't need a tokenizer.

Mypy cleanups along the way: cast config to Wav2Vec2Config at the
Wav2Vec2Model construction site, and route feature_extractor.sampling_rate
through getattr to drop the attr-defined warning on Wav2Vec2Processor.

https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66

* Add speechbrain_loading_cwd context manager (smoke test)

* Fix discrete-SER ehcalabres + redirect SpeechBrain inner-lobe save_paths

Wires the audeering Wav2Vec2ForSpeechClassification fix from PR #513 into
the discrete-SER path so the ehcalabres RAVDESS checkpoint stops emitting
~1/8-uniform softmax outputs:

* Generalize the custom emotion classifier head to support both final-layer
  attribute names actually shipped in the wild: ``classifier.out_proj.*``
  (audeering MSP-Dim) and ``classifier.output.*`` (ehcalabres). Build the
  matching nn.Module per layout and cache by name.

* Detect the head layout cheaply via the checkpoint's safetensors index or
  bin shard manifest (no full weight download); fall back to a small
  registry for single-bin checkpoints like ehcalabres.

* Dispatcher now routes any Wav2Vec2 emotion checkpoint with a 2-layer
  dense+linear head through ``_classify_wav2vec2_speech_cls_ser``,
  applying softmax for discrete and leaving raw logits for continuous.

* Discrete SER test asserts max(scores) > 0.2 to detect random-head
  regressions, plus that scores form a softmax distribution.

Wraps every SpeechBrain ``from_hparams`` call in
``speechbrain_loading_cwd(savedir)`` so CWD-relative ``save_path`` lobes
inside hyperparams.yaml (e.g. ``save_path: wav2vec2_checkpoints`` on
``speechbrain/emotion-recognition-wav2vec2-IEMOCAP``) resolve under
``{HF_HOME}/senselab_cache/speechbrain/<safe_key>/`` instead of dropping
700+MB into the user's working tree.

Verified end-to-end against two HAPPY recordings: ehcalabres now returns
``happy: 0.49`` (vs. ~0.13 uniform before), and CWD stays clean after a
full SER pipeline run.

https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66

* Narrow except Exception to (StrictDataclassError, ValueError, TypeError, KeyError)

Addresses gemini-code-assist's review on PR #511 about broad-except.
The actual exception observed in the audeering MSP-Dim case is
huggingface_hub.errors.StrictDataclassError (parent of
StrictDataclassFieldValidationError, raised on vocab_size: null with
huggingface_hub>=1.0). Standard parsing failures fall under ValueError /
TypeError / KeyError. OSError is intentionally excluded — that means the
network or filesystem is broken, and the fallback path needs the same
network hop, so it can't recover.

The tuple is constructed with a try/except ImportError to stay compatible
with huggingface_hub<1.0 (where StrictDataclassError doesn't exist).

Verified by re-running test_speech_emotion_recognition_continuous against
audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim — the
StrictDataclassFieldValidationError is correctly caught by the narrowed
tuple and the raw-config-dict fallback runs as before.

https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66

* Loud-fail guard for random/mismatched Wav2Vec2 emotion heads + remediation plan

Phase 1 of specs/20260509-204500-ser-silent-failure-guards/plan.md.

The custom Wav2Vec2 emotion-head path in `_classify_wav2vec2_speech_cls_ser`
trusts the dispatcher peek to have correctly matched the checkpoint's head
layout. When that assumption is wrong (new encoder family, unrecognized
final-layer attribute name, or single-bin checkpoint not in the registry)
transformers silently random-initializes the head and inference returns
~uniform softmax — the exact bug PR #511 is fixing, just one routing branch
upstream.

Convert that to a loud `RuntimeError`. After `from_pretrained` returns we
inspect `loading_info` for any `classifier.*` entry in `missing_keys`
(weights absent, would be randomly initialized) **or** `mismatched_keys`
(weights present but wrong shape — equally a silent-random failure that
the original missing-keys check would miss; flagged by architect review).
We also assert the final layer's `out_features` equals
`config.num_labels` / `len(id2label)`, catching the case where head and
config disagree on the number of classes.

Adds a parametrised unit test that mocks `EmotionModel.from_pretrained`
to return each broken-load shape and asserts the diagnostic raises with
a matching message.

Also commits the full remediation plan (specs/.../plan.md) enumerating
Phases 2-7 (standard-pipeline missing-key warning, encoder-family
generalization for HuBERT/WavLM, head-registry richening, problem_type
based discrete/continuous detection, resampling in the standard pipeline,
diagnostics CLI). Plan was self-audited and reviewed by the Plan agent
before this commit; its recommendations are folded in (mismatched_keys
coverage promoted into Phase 1, dynamic HeadSpec cut from Phase 4 as
premature, encoder-attribute-name registry promoted from a Phase 3
footnote to a load-bearing task).

https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66

* Phases 2-7: encoder-family registry, problem_type, doc + diagnostics CLI

Implements the remaining phases from
specs/20260509-204500-ser-silent-failure-guards/plan.md, layered on top
of Phase 1's loud-fail guard:

- Phase 4 (head registry): introduce ``_HeadEntry`` dataclass holding
  ``final_layer`` / ``activation`` / ``dropout_field``; promote
  ``_KNOWN_WAV2VEC2_EMOTION_HEADS`` (str→str) to ``_KNOWN_HEAD_LAYOUTS``
  (str→_HeadEntry). Generalize ``_make_wav2vec2_regression_head_class``
  to ``_make_regression_head_class(head: _HeadEntry)`` with a small
  ``_ACTIVATIONS`` table. Cache key now includes activation/dropout_field
  so future entries don't collide.

- Phase 3 (encoder-family generalization): ``_BASE_REGISTRY`` maps
  ``config.model_type`` → ``(PreTrainedModel base name, encoder attr
  name)`` for wav2vec2/hubert/wavlm/wav2vec2-bert. Resolved lazily so
  imports only fire when the custom-head path runs. ``_emotion_head_kind``
  now returns ``(model_type, _HeadEntry)`` tuples; ``_make_emotion_model_class``
  threads the right encoder class and attribute name. The Phase-1 guard
  catches mis-registered families: any wrong attr-name will surface as
  "every classifier.* key missing" with a clear ``RuntimeError``.

- Phase 5 (problem_type detection): ``_resolve_apply_softmax`` prefers
  ``config.problem_type`` over the keyword-based label heuristic. A
  regression head with non-AVD axes (e.g. ``["energy","pleasantness"]``)
  no longer gets softmax misapplied. Falls back to ``_get_ser_type`` for
  legacy checkpoints that omit ``problem_type``.

- Phase 2 (standard-pipeline missing-key warning):
  ``_get_hf_audio_classification_pipeline`` now loads the model
  explicitly via ``AutoModelForAudioClassification.from_pretrained(...,
  output_loading_info=True)`` and threads the loaded instance into
  ``pipeline(model=..., feature_extractor=...)``. ``_check_head_loaded_cleanly``
  warns on suspect missing/mismatched head keys
  (``classifier.``/``head.``/``score.``/``out_proj.``/``projector.``).
  ``SENSELAB_STRICT_HEAD_LOAD=1`` promotes the warning to a hard error.

- Phase 6 (resampling in standard pipeline): verified already implemented
  at ``classify_audios_with_transformers`` lines 141-142; no code change
  required. Plan updated to reflect the discovery.

- Phase 7 (docs + diagnostics CLI):
  - ``doc.md`` enumerates the three dispatch paths (custom in-process
    head / subprocess venv / standard pipeline) and points at the CLI.
  - New ``python -m
    senselab.audio.tasks.classification.speech_emotion_recognition
    --probe <repo_id>`` reports model_type, architectures, dispatch
    decision, head source, and emits advisories. Pure config / manifest
    inspection — no weights loaded.

Backwards-compat shims kept in place:
- ``_make_wav2vec2_emotion_model_class(final_layer)`` (delegates to
  ``_make_emotion_model_class("wav2vec2", _HeadEntry(final_layer=...))``)
- ``_wav2vec2_emotion_head_kind`` (returns just the final-layer name from
  the new ``_emotion_head_kind`` tuple).
- ``final_layer=`` kwarg on ``_classify_wav2vec2_speech_cls_ser``
  short-circuits to ``_HeadEntry(final_layer=...)``.

Verified locally: all 8 classification tests pass (1 GPU-skip), audeering
happy path passes, ehcalabres CLI probe correctly identifies its head
from ``_KNOWN_HEAD_LAYOUTS`` registry, audeering CLI probe peeks the
manifest, superb correctly routes to the standard pipeline with an
advisory.

Test coverage gaps (deferred per plan):
- Phase 2 unit test for ``_check_head_loaded_cleanly`` (covered
  end-to-end by audeering/ehcalabres tests for now).
- Phase 5 unit test for ``_resolve_apply_softmax`` (no fixture model
  with problem_type=regression + confounding labels).
- Phase 4 numerical-regression test on existing checkpoints.

https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66

* Reviewer-pass cleanups: explicit encoder cls names, CLI null-config note

Two small fixes from the second-pass architect review:

1. Make ``_BASE_REGISTRY`` a 3-tuple ``(base_pretrained_cls_name,
   encoder_model_cls_name, encoder_attr_name)`` instead of deriving the
   encoder class name via ``base_name.replace("PreTrainedModel", "Model")``.
   The string-mutation worked for current Wav2Vec2/HuBERT/WavLM/W2V-BERT
   but would silently break the day a family ships an encoder with a
   different naming convention. Explicit > clever.

   Updated both indexers (``_emotion_head_kind`` in api.py and the
   probe CLI's status print) to read attr name from index ``[2]``.

2. Probe CLI: when ``AutoConfig.from_pretrained`` fails (e.g. audeering's
   ``vocab_size: null`` tripping the strict validator), make the
   diagnostic note explicit that the dispatcher's raw-config-dict
   fallback handles this — the routing decision and advisories that
   follow still work correctly because ``getattr(None, ...)`` short-
   circuits the ``config.model_type`` checks.

Verified: existing guard tests pass; ``uv run python -m
senselab.audio.tasks.classification.speech_emotion_recognition --probe
audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim`` still produces a
correct routing report end-to-end.

https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66

* Annotate model type in probe CLI to satisfy mypy --extra-checks

CI pre-commit hook runs mypy with --extra-checks which flagged the
inferred type for ``model = HFModel(...)`` in __main__.py. Add the
explicit annotation. cpu-tests already pass; this is the only blocker.

https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66

* Strict-reviewer second pass: 5 silent-failure surfaces fixed + Phase 2 unit tests

The strict reviewer identified 5 critical defects against the user's
"zero tolerance for silent failures" criterion. All addressed in this
commit:

D1. ``wav2vec2-bert`` removed from ``_BASE_REGISTRY``.
    The family consumes log-mel ``input_features`` (via
    ``SeamlessM4TFeatureExtractor``) rather than raw ``input_values``.
    The loader hardcodes the latter and accesses ``inputs["input_values"]``
    so any wav2vec2-bert checkpoint would have produced ``KeyError`` at
    best, silent garbage at worst. Add a per-family preprocessing
    adapter before re-introducing.

D2. Phase-1 guard now covers encoder-key misses.
    Previous filter was ``classifier.*`` only, which meant a wrong
    ``_BASE_REGISTRY`` entry (wrong encoder attribute name) would leave
    every encoder weight missing yet pass the guard — silent garbage
    output. The guard now treats any missing key under either
    ``classifier.`` or ``{encoder_attr}.`` as suspect, with a small
    whitelist of HF post_init buffers (``masked_spec_embed``).

D3. ``_resolve_apply_softmax`` peeks ``architectures`` on
    ``problem_type``-absent checkpoints. Any model declaring
    ``Wav2Vec2ForSpeechClassification`` (audeering signature) is treated
    as regression. Reduces the silent-corruption window for regression
    heads with English-emotion-word labels.

D4. ``_peek_head_final_layer`` raises when ``classifier.dense.weight``
    is present but no recognized final-layer attribute matches.
    Previously returned None and let the standard pipeline silently
    random-initialize the head. The error message lists the actual
    classifier keys so a maintainer can extend ``_FINAL_LAYER_NAMES``
    or ``_KNOWN_HEAD_LAYOUTS``.

D5. ``SENSELAB_STRICT_HEAD_LOAD`` default inverted to ``=1`` (raise).
    Warn-only meant downstream consumers like
    ``classify_audios`` windowed segmentation kept aggregating
    corrupted scores. Set ``SENSELAB_STRICT_HEAD_LOAD=0`` to demote to
    a warning if you intentionally want a partial-head load (e.g.
    fine-tuning warm-start).

T203. Three unit tests for ``_check_head_loaded_cleanly``: parametrised
    raise on suspect missing/mismatched-shape keys (default strict),
    warn-only mode under ``SENSELAB_STRICT_HEAD_LOAD=0``, and silent
    pass on clean loads (encoder-buffer misses don't trip).

Plan checkboxes T203 / T304 / T502 promoted from deferred-shortcut to
actually-shipped. Audeering happy-path test still passes; mypy clean
with ``--extra-checks``.

https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66

* Reviewer round 3: align user-facing docs with strict-by-default behavior

Reviewer round 3 verified D1-D5 + T203 fixes are functionally correct,
but flagged three pieces of user-facing text that still contradict the
new behavior — themselves a silent-assumption violation per the user's
"no silent failures and assumptions beneath those" criterion:

* ``doc.md`` listed ``wav2vec2-bert`` as a supported encoder for the
  custom-head path (it isn't; D1 dropped it from ``_BASE_REGISTRY``)
  and described ``SENSELAB_STRICT_HEAD_LOAD=1`` as a way to "promote
  the warning to a RuntimeError" — but ``=1`` is now the default
  (D5). A user who reads the doc, doesn't set the env var, and
  expects warn-mode would be surprised by a hard fail.

* ``__main__.py`` probe CLI advisory had the same inverted message
  about the env var.

* ``api.py:503`` dispatcher comment + ``api.py:144`` encoder-attribute
  comment still mentioned ``wav2vec2-bert``.

This commit:

1. Removes ``wav2vec2-bert`` from the doc's supported-encoder list,
   adds an explicit "intentionally not supported" sentence with the
   ``SeamlessM4TFeatureExtractor`` reason, so users know why and what
   it would take to add.
2. Inverts the env-var description in both the doc and the CLI to
   match the strict-by-default semantics.
3. Drops stale ``wav2vec2-bert`` mentions from the api.py comments.

No code-path or behavior change. Verified clean: ruff, ruff format,
mypy --extra-checks all pass.

https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66

* Plan cleanup: drop T203/T502 'legacy text for traceability' duplicates

When the strict reviewer's second pass forced T203 and T502 from
deferred-shortcut to actually-shipped, I appended the new ``[x]`` text
and kept the original ``[ ]`` lines tagged "legacy text, kept for
traceability". On re-read those duplicates just made the plan look
like there were unfinished items when there weren't. Drop the
duplicates; the ``[x]`` entries are sufficient.

Remaining ``[ ]`` items after cleanup:
  T305 — attention-mask shim for batched HuBERT/WavLM (don't optimize for unknowns)
  T403 — numerical regression test against fixed scores (Phase-1 guard + bounded-score covers)
  T602 — sampling-rate test (Phase 6 was a no-change phase, end-to-end coverage exists)

All three are honest deferrals, not shortcuts.

https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66

* PR #511 follow-up: AutoConfig memo (per-key locked) + SpeechBrain CWD lock + registry smoke tests (#515)

Stacks on top of PR #511 to address @wilke0818's review.

Changes:
1. **CWD race fix**: `speechbrain_loading_cwd` now serializes via a
   module-level `threading.Lock` so concurrent threads can't race on the
   process-global `os.chdir`. Docstring documents the trade-off (use
   processes, not threads, for parallel SpeechBrain init).

2. **AutoConfig deduplication**: introduces `_load_config_cached(model)`
   returning `(typed_config_or_None, raw_dict_or_None)`, memoized
   per-(path, revision). The four call sites (`_get_ser_type`,
   `_emotion_head_kind`, `_resolve_apply_softmax`,
   `_classify_wav2vec2_speech_cls_ser`) now share one network round-trip
   instead of four. Type annotated with `PretrainedConfig`.

3. **Per-key locking** (not a single global lock): mirrors the strategy
   used by `dependencies.ensure_hf_model` (`_HeartbeatLock` per
   `(repo_id, revision)`). Two threads loading the *same* model dedupe
   to one round-trip; two threads loading *different* models proceed in
   parallel. In-memory `threading.Lock` per key here, since the memo is
   per-process (cross-process is already handled by HF_HOME cache and
   `ensure_hf_model`).

4. **`_BASE_REGISTRY` smoke tests**: three parametrised unit tests verify
   `_resolve_base("wav2vec2"|"hubert"|"wavlm")` resolve to real
   transformers classes — catches typos at test-collection time without
   downloading model weights.

5. **Test isolation fix**: autouse fixture clears the module-level caches
   (`_config_memo`, `_config_memo_locks`, `_wav2vec2_emotion_models`)
   between tests. Previously the parametrised
   `test_wav2vec2_speech_cls_ser_raises_on_random_head` and
   `_raises_on_shape_mismatch` would silently serve a cached MagicMock
   from a prior test's monkeypatch instead of re-invoking their own;
   tests only passed because the mocks happened to agree.

Verified locally: ruff, format, mypy --extra-checks clean. 18 fast tests
pass including 4 new ones. CI: ✅ cpu-tests, pre-commit, build-and-deploy
all green on `be77c5d`.

https://claude.ai/code/session_01DyXMQNsF7F5Ze8j3GGVw66

---------

Co-authored-by: Satrajit Ghosh <satrajit.ghosh@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
New module ``senselab.utils.cuda_probe`` provides:

- ``detect_host_cuda()`` — parses ``CUDA Version: X.Y`` from ``nvidia-smi``'s
  default header, falls back to ``release X.Y`` from ``nvcc --version``,
  then ``HostCuda(version=None)`` when neither is available. Each probe
  has a 5s subprocess timeout. No driver→CUDA lookup table to maintain —
  both probes report the CUDA version directly.
- ``pick_torch_index(host_cuda, env_override=None)`` — picks the highest
  static-map entry whose CUDA is ``<=`` host's, from
  ``[cu128, cu126, cu124, cu121]``; falls back to ``cpu`` index. When
  ``env_override`` is set + non-empty, returns that URL verbatim with
  ``tag="override"``.
- ``SenselabCudaCompatibilityError`` — wraps wheel-not-found install
  failures with host CUDA + attempted index + failing packages and a
  one-line recommended action (downgrade CUDA / set
  ``SENSELAB_TORCH_INDEX_URL=https://download.pytorch.org/whl/cpu`` / wait
  for upstream wheels).

Two frozen dataclasses (``HostCuda``, ``TorchIndex``) carry the probe and
picker results so they're individually testable and the resolved index URL
is auditable in the subprocess-venv marker file.

Tests: ``src/tests/utils/cuda_probe_test.py`` — 15 unit tests, all
``subprocess.run`` calls mocked, no shell-out on CI. Coverage: both probe
paths, both probe failures, all 5 index map entries, env override
short-circuit + empty-string-treated-as-unset, error message contents.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
``ensure_venv`` now resolves a PyTorch wheel index for the host before
running ``uv pip install`` and routes the install through that index. On
hosts with system CUDA newer than the PyTorch default-wheels CUDA (e.g.
CUDA 12.9 against PyTorch's ``cu128`` default), this prevents ``torch``
and ``torchaudio`` from resolving to mismatched CUDA toolchains and
breaking their ABI contract at import time.

Resolution order inside the lock:

1. ``env_override = os.getenv("SENSELAB_TORCH_INDEX_URL") or None`` (empty
   string treated as unset).
2. If ``env_override`` is set: skip the host CUDA probe — the override
   wins regardless of host CUDA.
3. Otherwise: ``host_cuda = detect_host_cuda()`` followed by
   ``pick_torch_index(host_cuda, env_override=env_override)``.

The marker file gains a ``torch_index`` field; absence of that field in
an existing marker counts as a mismatch and triggers a rebuild — so
users upgrading through this fix get one automatic rebuild on first run.
No explicit schema-version field; the marker is internal cache state,
not a published contract.

Install argv now prefixes ``--index-url <torch_index.url>
--extra-index-url https://pypi.org/simple ...`` — the PyTorch index for
torch/torchaudio resolution, PyPI as the extra index so non-torch
packages (nemo_toolkit, qwen-asr, etc.) still resolve.

Failure handling: ``uv pip install`` failures pass through
``_classify_uv_failure`` to distinguish "no matching distribution" /
"could not find a version" errors (wrap into
``SenselabCudaCompatibilityError`` with named host/index/packages/action)
from unrelated failures (re-raise the original ``CalledProcessError``).
Either way the half-built venv directory is wiped before the raise so
the next run starts clean — addresses the recovery-from-broken-state
case.

Added in-source warnings to ``canary_qwen.py``, ``nemo.py``, and
``qwen.py`` near each ``*_REQUIREMENTS`` list: developers editing the
torch pins should not bypass ``ensure_venv`` with a per-backend install
shellout, since the CUDA routing is what guarantees the toolchain match.

Tests: ``src/tests/utils/subprocess_venv_test.py`` — 14 integration
tests, ``subprocess.run`` mocked. Coverage: marker-mismatch rebuild
paths (pre-fix-shape marker, matching index, different index), install
argv routing, env override short-circuit + empty-string-unset, wheel
not-found wrapping, unrelated-error pass-through, ``_classify_uv_failure``
unit cases, parameterized regression test walking all three real backend
requirement lists.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
README "System Requirements" section now mentions:

- Subprocess-venv backends auto-route their ``torch``/``torchaudio``
  install through the matching PyTorch wheel index based on host CUDA;
  no manual configuration needed.
- ``SENSELAB_TORCH_INDEX_URL`` operator override for internal mirrors,
  unsupported CUDA versions, or forced CPU fallback.
- Unsupported-CUDA failures surface as ``SenselabCudaCompatibilityError``
  with named host CUDA, attempted index, and recommended action.

Spec docs land under ``specs/20260512-204619-fix-canary-cuda-conflict/``:
spec.md (3 prioritized user stories), plan.md (Constitution check + 6
research decisions), research.md, data-model.md, contracts/, quickstart.md
(validation recipes for happy path, CPU fallback, negative path,
regression, recovery), tasks.md (28 tasks across 6 phases), and
checklists/requirements.md (passed twice).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Local-reviewer findings on the cuda_probe + subprocess_venv commits:

- MEDIUM (subprocess_venv): on the env-override path, host_cuda was
  hardcoded to ``HostCuda(version=None)`` so the wrapped
  ``SenselabCudaCompatibilityError`` reported "Host CUDA: none" even on
  real CUDA hosts using internal mirrors. Run ``detect_host_cuda()``
  unconditionally now; the override still bypasses the static index map
  but the probe result is preserved for diagnostic surface.

- MEDIUM (subprocess_venv): the failing-package extraction regex was
  pulling EVERY backticked token from uv's stderr, so unrelated hints
  like ``\`uv cache clean\`\`\` could leak into the
  ``SenselabCudaCompatibilityError`` message as fake "failing packages".
  Anchored the capture to the "no matching distribution" / "could not
  find a version" phrases so only the actual failing requirement is
  surfaced.

- LOW (subprocess_venv): ``uv venv`` failure (e.g. requested Python
  version unavailable) re-raised without wiping a partially-populated
  venv directory. Mirror the install-failure cleanup so both failure
  paths leave a clean cache state.

- LOW (cuda_probe): ``_PYTORCH_INDEX_MAP`` ordering ("highest CUDA
  first") was asserted by comment only. Wrapped the literal in
  ``sorted(..., reverse=True)`` so a future contributor appending an
  entry in the wrong position can't silently break routing.

- LOW (subprocess_venv): on the wrap-into-compat-error path, the
  pre-classification ``logger.error`` was duplicating the wrapped
  exception's already-formatted message and dumping unbounded uv
  stderr. Demoted to ``logger.debug``; the pass-through path still logs
  at error level since that's the only signal the user gets for
  non-wheel failures.

Test updates: ``test_env_override_short_circuits_probe_*`` renamed and
reworked to assert the probe DOES run (not skips) on the override path
— the contract is the URL routing, not the probe-skip side effect.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pre-fix-time local pre-commit pass was reading a stale mypy cache.
Once invalidated (CI / fresh clone), five errors surfaced:

- src/tests/utils/cuda_probe_test.py:44,46,67,69 — `_runner(*args: object)`
  was too generic for the indexed access `args[0]` (mypy "Value of type
  'object' is not indexable"). Tightened to the actual ``subprocess.run``
  positional signature `_runner(args: list[str], **kwargs: object)` —
  matches reality and unblocks indexing.
- src/tests/utils/subprocess_venv_test.py:80 — `_SubprocessRecorder.hook`
  was typed `Optional[object]`, which mypy refuses to call. Tightened to
  `Optional[Callable[[list[list[str]]], subprocess.CompletedProcess]]` —
  the type the hook actually implements.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two occurrences in research.md flagged by codespell on CI. Same fix
shape — the hyphenated forms are non-standard per the project's
codespell allowlist.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Distinguish ``text=""`` (model was called, returned no transcript — a
real signal) from ``text=None`` (field absent / model never called).
The earlier validator used falsiness (``not values.get("text")``),
treating both the same and rejecting any ``ScriptLine`` whose text was
empty even when its speaker was also unset.

In practice this only bit ASR backends honest enough to emit empty
output on unintelligible / silent audio. The trigger we observed: IBM
Granite-Speech (``ibm-granite/granite-speech-3.3-8b``) on heavily-
enhanced short clips (1.4–3.4s after speech enhancement) — it
correctly decoded "no speech" and emitted ``{"text": ""}``, which the
``ScriptLine`` validator then refused with ``"At least text or speaker
must be provided."``. Less-honest ASR backends (Whisper hallucinates
filler on silence, the LLM-augmented Canary-Qwen / Qwen3-ASR pass the
prompt's implicit "say something" pressure through and almost always
emit text) silently slipped past the same validator by producing a
non-empty string. The vulnerability lives in every backend that
constructs ``ScriptLine(text=...)`` directly — Granite was just the
only one that surfaced it in this dataset.

There was also a subtle inconsistency: a whitespace-only input like
``text=" "`` survived the validator (``" "`` is truthy) and was then
stripped to ``""`` by the field validator, while an explicit ``""``
was rejected up front. The new presence-based rule (``text is None
and speaker is None``) treats both consistently.

Negative-timestamp validation and the whitespace-stripping field
validator are unchanged.

Tests in ``src/tests/utils/data_structures/script_line_test.py``
cover the Granite-style empty-transcript path, the whitespace-normalize
case, both-fields-absent rejection, speaker-only construction, and
preservation of existing happy-path behavior (timestamps, stripping,
``from_dict``).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…equirements

PR #516 routes the install through ``--index-url <cuda> --extra-index-url
https://pypi.org/simple`` so non-torch packages can still resolve from
PyPI. But uv treats ``--extra-index-url`` as having higher priority than
``--index-url`` (opposite of pip), so PyPI wins for every package — torch
and torchaudio included. On hosts where PyPI ships those with mismatched
``+cu`` local-version tags (currently ``torch==X+cu129`` vs
``torchaudio==X`` with no tag, internally cu128), the resulting venv
reproduces the same ABI mismatch this PR is meant to fix:

    RuntimeError: Detected that PyTorch and TorchAudio were compiled with
    different CUDA versions. PyTorch has CUDA version 12.9 whereas
    TorchAudio has CUDA version 12.8.

Reproduced on MIT ORCD with the env override set
(``SENSELAB_TORCH_INDEX_URL=https://download.pytorch.org/whl/cu128``) —
override URL was honored in the marker but uv still resolved both wheels
from PyPI. So the override path was non-functional too.

Fix: split the install into two stages and decide whether Stage 1 fires
from the caller's ``requirements`` itself.

  Stage 1 — ``uv pip install --index-url <chosen>`` listing every
    ``torch`` / ``torchaudio`` spec found in ``requirements``, with NO
    ``--extra-index-url``. The chosen CUDA index is unambiguously
    primary; both wheels and their ``nvidia-cuda-runtime-cu12``
    transitives come from the same toolchain. Multiple constraints
    for the same package (``["torch>=2.8", "torch<2.9"]``) are all
    forwarded so uv combines them at resolve time.

  Stage 2 — ``uv pip install <rest of requirements> safetensors numpy``
    with no index flags. Backend-pinned torch / torchaudio specs are
    filtered out so uv can't be tempted to re-resolve them from PyPI
    against the matched ``+cu128`` wheels installed in Stage 1.

Drop the unconditional ``torchaudio`` IPC append, in two ways:

1. ``_torch_install_specs`` no longer pads its return with bare
   ``torch`` / ``torchaudio`` names. If neither is in ``requirements``,
   it returns ``[]``.
2. ``ensure_venv`` treats ``_torch_install_specs(requirements) == []``
   as the trigger to skip the probe entirely and run a single
   torch-free install pass against default PyPI. No ``nvidia-smi``
   shellout, no ``torchaudio`` force-appended.

This addresses @satra's review concern that the previous default forced
``torchaudio`` into every subprocess venv, even backends that don't use
it. With the change, ``yamnet`` and ``continuous-ser`` — both of which
read audio via ``soundfile`` in their worker scripts and never import
torchaudio — get ~200 MB leaner venvs. Backends that genuinely need
``torch`` / ``torchaudio`` (including via a transitive dep) MUST pin
them explicitly so Stage 1 routes them through the matched CUDA index;
this is also why ``qwen.py``'s ``_REQUIREMENTS`` now names them
directly even though ``qwen-asr==0.0.6`` would otherwise pull them
transitively — relying on the transitive would let Stage 2's PyPI
resolution split the local-version tags. The 7 backends that already
pin ``torchaudio`` explicitly (canary, nemo, ppgs, sparc, coqui,
s3prl) are unaffected.

The earlier ``requires_torch=False`` parameter from a previous round
of this review (an explicit opt-out flag) is removed in favor of the
auto-detection — it's structurally equivalent and the explicit flag
becomes redundant noise.

Behavioral guarantees preserved from PR #516: marker schema (extended,
not changed), cache-hit fast path, env override
(``SENSELAB_TORCH_INDEX_URL``), ``SenselabCudaCompatibilityError``
wrapping on wheel-not-found errors (now scoped to Stage 1 where it
semantically belongs), pass-through of unrelated ``CalledProcessError``,
half-built-venv cleanup on failure, host-CUDA probe diagnostic surface,
cross-backend routing.

Tests: 34 in total covering (1) Stage 1 names only ``--index-url`` with
torch + torchaudio pinned per requirements; (2) Stage 2 carries no
index flags + IPC deps; (3) Stage 2 filters torch / torchaudio specs
from caller's requirements; (4) multiple constraints for the same
package all forward through Stage 1 verbatim; (5) requirements without
torch / torchaudio skip the probe and Stage 1 entirely; (6) yamnet-
style requirements get NO torchaudio in their install argv; (7)
switching a venv name from torch-free to torch-using correctly
invalidates the cache and rebuilds. The cross-backend regression test
asserts both that Stage 1 routes through the chosen index AND that
Stage 2 contains no torch / torchaudio specs at all.

Review feedback addressed:
- @satra (PR #518 review): drop the unconditional ``torchaudio`` IPC
  append; auto-detect torch routing from the caller's ``requirements``
  instead of forcing every venv to pay for it. Adds explicit ``torch``
  + ``torchaudio`` to qwen's ``_REQUIREMENTS`` so its transitive-dep
  resolution still flows through the CUDA index.
- gemini-code-assist #1: filter torch / torchaudio specs from Stage 2.
- gemini-code-assist #2: list-based ``_torch_install_specs`` keeps
  every matching constraint instead of clobbering through a dict.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
subprocess_venv: two-stage install to fix torch/torchaudio CUDA-tag s…
…conflict

subprocess_venv: route torch/torchaudio via CUDA-aware PyTorch index
…opt-in

Hosts on Python versions newer than spaCy ships wheels for (3.14 today)
silently fell back to the no-detector path in ``pii.py`` and produced
``contains_pii: false`` reports across every clip — a false-negative
on PII-bearing audio for any host that can't install Presidio in-process.
Move detection into an isolated Python-3.13 subprocess venv so the host
can stay on whatever recent Python it likes without losing PII checks.

Detection content
-----------------
Two detectors run inside the new ``pii-detection`` venv and their spans
are merged before cross-ASR-model corroboration runs in the host:

- **Microsoft Presidio Analyzer** — regex + spaCy-NER recognizers for
  emails, phone numbers, SSNs, credit cards, IPs, dates, locations.
- **GLiNER PII** (``nvidia/gliner-pii`` by default, configurable) — a
  570M-param transformer fine-tuned on ~100k synthetic PII/PHI records.
  Catches medical / health entities Presidio doesn't natively cover,
  plus the long tail of unstructured personal-context references that
  zero-shot well from the labels we pass.

GLiNER's lowercase labels are normalized to Presidio's uppercase scheme
inside the worker so the existing ``(category, text.lower())`` dedupe
key in ``pii.py`` treats both detectors' hits on the same entity as the
same finding.

Per-detector opt-in
-------------------
``detect_pii_in_pass`` and ``detect_pii_via_subprocess`` gain a
``detectors`` argument symmetrical to the project's ASR-model selection
pattern:

- ``detectors=None`` (default) → both Presidio and GLiNER run.
- ``detectors=["presidio"]`` → skip GLiNER's model load (~5 s saved).
- ``detectors=["gliner"]`` → skip Presidio.
- ``detectors=[]`` → explicit-disable; no subprocess spawned, no venv
  build attempted, report carries ``failures["pii_disabled"]`` so an
  auditor can distinguish "didn't run on purpose" from "ran and found
  nothing" and "subprocess crashed".

``gliner_model``, ``gliner_labels``, and ``gliner_threshold`` are also
exposed on ``detect_pii_in_pass`` so callers can swap in a different
GLiNER variant (e.g. ``urchade/gliner_multi-v2.1`` for multilingual
coverage) or restrict the label set without touching the worker.

Architectural changes
---------------------
- New module ``pii_subprocess.py`` — venv constants
  (``_PII_VENV = "pii-detection"``, Python 3.13, presidio + spacy +
  en_core_web_lg wheel pinned via direct URL + gliner + torch),
  GLiNER -> Presidio category map, worker script, and the
  ``detect_pii_via_subprocess`` dispatch function. ``torch`` is named
  explicitly in ``_PII_REQUIREMENTS`` so ``ensure_venv``'s auto-detect
  (post-#516) routes it through the matched CUDA index — GLiNER's
  transformer wants matched ``torch`` / nvidia-cuda-runtime wheels.
  ``torchaudio`` is intentionally omitted (no audio decoding).
- ``pii.py`` simplified — removed ``_load_presidio_analyzer``,
  ``_load_spacy_nlp``, ``_scan_with_presidio``, ``_scan_with_spacy``,
  and the in-process tier-fallback chain. Dispatch now goes to the
  subprocess; everything else (transcript concatenation,
  ``PiiSpan`` materialization, cross-ASR-model corroboration,
  ``PiiPassReport`` construction) stays in-process.
- Subprocess-only — no in-process fallback. Matches the
  canary/nemo/qwen subprocess-venv pattern for hermetic deployment.
  A host that can't build the venv at all gets a single
  ``failures["pii_subprocess"]`` entry, not a confusing partial-fail
  state.

Worker model loads are paid per subprocess invocation. For a typical
``analyze_audio.py`` run (138 ``detect_pii_in_pass`` calls per
subject) that's ~15 min of model-load overhead per subject; batching
all of a subject's transcripts into a single subprocess invocation is
a worthwhile follow-up but isn't required for correctness.

Tests
-----
``src/tests/audio/workflows/audio_analysis/pii_subprocess_test.py``,
10 tests covering: default-both-detectors plumbing, single-detector
selection, empty-list short-circuit (no venv build, no subprocess
spawn), unknown-detector ``ValueError``, dedupe with order preservation,
default-arg pass-through (threshold, GLiNER model id, labels, full
GLiNER -> Presidio category map), caller overrides for the GLiNER
settings, response field pass-through.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The PII axis previously surfaced only ``contains_pii ? 1.0 : 0.0`` to
``global_summary``, throwing away every per-span detector score that the
worker produced. Worse, when the subprocess crashed or no detector
loaded, the report fell to ``0.0`` — indistinguishable from "we
checked, found nothing." That's the wrong default; it tells a reviewer
"this clip is clean" when the truth is "we don't know."

Add a continuous ``detection_confidence`` field on ``PiiPassReport``,
combined from three signals per unique ``(category, normalized_text)``
finding:

- **per-span detector score** — Presidio analyzer score or GLiNER
  prediction probability, ``max()`` within the finding's group.
- **cross-detector agreement** — Presidio AND GLiNER both flagged
  the same finding (factor 1.0) vs only one detector (0.5). The
  strongest "real entity vs hallucination?" signal at this layer,
  mirroring the model-voting pattern presence/identity already use.
- **cross-ASR-model agreement** — fraction of available ASR
  transcripts that contained the finding. Spans only one ASR
  transcribed and no sibling ASR confirms are the prototypical
  hallucination case.

Then ``max()`` across findings, matching how the transcript and
single-speaker axes combine their internal sub-signals.

Deliberately NO category-severity weighting. In pediatric / clinical
voice data the categories nominally most "severe" (US_SSN,
CREDIT_CARD, MEDICAL_RECORD_NUMBER) have near-zero true-positive rate
— children don't speak SSNs aloud — and the synthetic-PII-trained
detectors gladly flag any 9-digit ASR hallucination as SSN. Weighting
them up would inflate exactly the findings a reviewer should
de-prioritize. Heuristic enhancements (category weights, find-count
severity) are deferred as later goals; this commit just plumbs through
what the detectors and ASRs actually report.

``None`` propagation
--------------------
``detection_confidence`` is ``None`` when no detector ran:

- subprocess raised at dispatch (CalledProcessError / RuntimeError)
- both detectors failed to load inside the subprocess venv
- caller passed ``detectors=[]`` (explicit-disable short-circuit)

``0.0`` is reserved for "detectors ran, found nothing" — the
confident-negative case. ``global_summary.compute_pass_global_summary``
propagates the same distinction into ``no_pii_uncertainty`` so the
``combined_uncertainty`` consumer can tell "didn't actually check"
apart from "checked clean." No silent demotion to 0.

Tests
-----
``src/tests/audio/workflows/audio_analysis/pii_test.py``, 12 tests:

- ``_compute_detection_confidence`` unit cases: empty spans -> 0.0,
  single detector / single ASR -> 0.5 x score, both detectors -> 1.0
  factor, cross-ASR scaling by fraction, max across independent
  findings (not sum), whitespace-only text filtered, missing per-
  span score collapses to 0.
- ``detect_pii_in_pass`` integration with mocked
  ``detect_pii_via_subprocess``: happy path populates the field,
  ``detectors_used=[]`` -> None, subprocess raise -> None,
  detectors-ran-no-spans -> 0.0, ``detectors=[]`` short-circuit ->
  None plus ``failures["pii_disabled"]``.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Empirical finding from a diagnostic on ``john.doe@example.com``:
nvidia/gliner-pii exhibits competing-claim interference. When the
prompt includes overlapping labels that can both plausibly cover the
same span — e.g. ``person`` + ``first_name`` + ``last_name`` plus
``email`` + ``email_address`` — the model commits the span to one
label and silently drops it from consideration under the other,
*even when the other would have been the right answer*. The
diagnostic at ``temp/diag_gliner_email.py`` walks this through
explicitly at threshold=0.0:

  Labels: [person, first_name, last_name, email, email_address, ...]
    → John = first_name (1.0), Doe = last_name (1.0),
      john.doe@example.com = no email span at any score above 0.0

  Labels: [name, address, date, phone_number, email, ssn, ...]  (flat)
    → John, Doe = name (1.0), john.doe@example.com = email (1.0)

Same model. Same threshold. Same input. The only difference was
label-set granularity. Email goes missing under the over-decomposed
set; both name and email surface cleanly under the flat HIPAA set.

This is the same set b2aiprep PR #256 ships in its
``hipaa_labels.json``. Aligning on it has the secondary benefit
that the two projects' PII outputs now share a vocabulary, which
matters for downstream consumers that aggregate across both.

Changes
-------
- ``_DEFAULT_GLINER_LABELS`` replaced with the HIPAA Safe Harbor 18
  identifiers (45 CFR §164.514(b)(2)) plus two clinical-voice
  extensions (``health_condition``, ``medication``) that don't
  overlap with the HIPAA labels.
- Long block comment above the constant captures the empirical
  finding so future contributors don't walk back into the trap
  (i.e. "this would catch more if I added first_name and
  last_name..."). Names the diagnostic, names the symptom, names
  the fix.
- ``gliner_labels`` docstring on ``detect_pii_via_subprocess``
  points at the warning so anyone customizing the label set hits
  it before re-introducing overlap.

The category map (``_GLINER_TO_PRESIDIO_CATEGORY``) already handles
every new HIPAA label that has a Presidio analogue (``name``,
``email``, ``date``, ``phone_number``, ``address``,
``social_security_number``, ``ip_address``,
``medical_record_number``). New labels without Presidio analogues
(``fax_number``, ``health_plan_number``, ``account_number``,
``license_number``, ``vehicle_identifier``, ``device_identifier``,
``url``, ``biometric_identifier``, ``photographic_image``,
``unique_identifier``) fall through to ``str.upper()`` and stand on
their own — Presidio doesn't emit competing categories for them so
there's nothing to align with. Entries for the now-removed labels
(``first_name``, ``last_name``, ``street_address``, ``city``,
``date_of_birth``, ``patient_id``, etc.) stay in the map for
backward-compat — a caller can still pass them explicitly via the
``gliner_labels`` override.

Tests
-----
22 tests in pii_subprocess_test.py + pii_test.py still pass — they
check the request shape and detection_confidence math, not the
content of the default label list, so the swap is transparent at
the dispatch layer. The behavioral verification is in the real
end-to-end run via temp/test_pii_subprocess.py + temp/diag_gliner_email.py.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous commit on this branch (a2f384c) shipped the HIPAA Safe
Harbor 18 identifiers PLUS two clinical-voice-specific extensions
(``health_condition``, ``medication``) tacked on the end. Running the
smoke test against ``temp/test_pii_subprocess.py`` showed Step 3
(detectors=["gliner"]) still didn't catch ``john.doe@example.com``
even with the new flat HIPAA-aligned base.

Cross-referenced against the threshold=0.0 diagnostic
(``temp/diag_gliner_email.py``): the bare HIPAA-18 set catches the
email at GLiNER score 1.0. Adding ``health_condition`` and
``medication`` on top is enough to push the email's score below the
0.5 threshold and make it disappear from the report, even though
those two labels are not semantically competing with ``email`` at
all. The mechanism is unclear — possibly a label-list length effect
on the model's projection space, possibly an artefact of nvidia/
gliner-pii's training data — but the symptom is reproducible.

Two-detector mode (the default) is robust to this: Presidio catches
the email at score 1.0 and feeds it through ``pii.py``'s cross-
detector dedup, so the report still surfaces the EMAIL_ADDRESS span.
The user-visible regression is confined to ``detectors=["gliner"]``
mode and to the contribution of the email to ``detection_confidence``
when both detectors run (cross-detector agreement collapses from 1.0
to 0.5 for that one span). For the b2ai voice-data workflow the cost
of losing the clinical-extension coverage was lower than the cost of
weakening the email signal — so this commit removes the extensions
and matches b2aiprep #256's HIPAA-18 list verbatim.

The empirical finding is captured as a second ``DO NOT extend this
list`` warning above the constant, so a future contributor doesn't
re-add ``health_condition`` / ``medication`` / similar adjacent
labels without first re-running the diagnostic against the email-
detection case.

Callers who actually want the clinical-extension coverage (or any
other labels beyond the HIPAA-18) can still pass them via
``gliner_labels=[...]`` on the dispatch function — they just take on
the responsibility of verifying their specific additions don't trip
the interference on their corpus.

Module docstrings on both pii.py and pii_subprocess.py updated to
reflect that GLiNER's coverage is now strictly HIPAA-aligned — no
longer the broader "medical / health entities and unstructured
personal-context references" framing the previous commit used.

22 tests in pii_subprocess_test.py + pii_test.py still pass; they
key off ``_DEFAULT_GLINER_LABELS`` as a literal, not its content.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
#2 (pii.py:159) — replace literal `2` in detector-agreement divisor
with `len(_KNOWN_DETECTORS)`. Today they're equal, but the symbolic
form removes the silent-miscalculation trap if a third detector is
ever added to `_KNOWN_DETECTORS`.

#3 (pii_subprocess.py:327) — drop the `or None` fallback when passing
`entities` to `AnalyzerEngine.analyze`. Presidio interprets
`entities=None` as "scan for ALL entity types"; `[] or None` collapses
to `None`, so a caller passing `entities=[]` (explicit "scan for
nothing") got the opposite of intent. The dispatcher already
substitutes the default full list when the caller-side
`presidio_entities` is None, so the worker never needs to pass None
itself — empty list now passes straight through to Presidio.

#4 (pii_subprocess.py:334) — change the post-scan category-filter
guard from `if presidio_entities` to `if presidio_entities is not
None`. Same falsy-empty-list trap as #3: with the old guard, an
explicit empty list silently skipped the filter so any Presidio
result would pass through. With `is not None`, the empty-list path
filters everything out, matching the "scan for nothing" semantics.
The comment is updated to clarify this is a belt-and-suspenders check
on Presidio's output rather than a category fallback.

Gemini's review comment #1 (the torch version pin) is being rejected
in a separate PR reply — torch 2.8 has been on PyPI for months
(latest is 2.12.0), the pin is intentional for NeMo trunk SALM
compatibility, and Gemini's training data predates the release.

Tests: src/tests/audio/workflows/audio_analysis/{pii,pii_subprocess}_test.py
22/22 still pass; the empty-list path was already covered by the
detector-agreement tests with `len(_KNOWN_DETECTORS)` in scope.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
#1 (high) — `_is_meaningful` filter in `from_dict` still used truthy
checks (`bool(c.get("text"))`), which silently dropped `{"text": ""}`
child chunks even though the root validator now accepts them. That
defeats the PR's whole point for payloads that pass through
`from_dict` (subprocess IPC, saved JSON, MMS-aligner outputs that go
through the dict shape). Switched to `is not None` so the filter only
drops children whose text AND speaker are wholly absent / None — the
same invariant the root validator enforces. Comment block updated to
explain why empty-text chunks must survive and what the filter is now
guarding against.

#2 (medium) — added two tests that would have caught the inconsistency
above:
  - test_from_dict_preserves_empty_text_in_nested_chunks: payload with
    `chunks=[{"text": ""}]` survives `from_dict` and the empty chunk
    is preserved.
  - test_from_dict_still_drops_chunks_missing_both_text_and_speaker:
    the original `_is_meaningful` purpose (filtering placeholder
    children that would trip the root validator) is preserved — only
    its over-aggressive truthiness behavior is loosened.

Tests: 13/13 pass (was 11; +2 new). Pre-commit clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ScriptLine: accept empty text as "model produced no transcript" signal
Under many parallel jobs (e.g. analyze_audio across SLURM nodes), every
from_pretrained/pipeline call makes a network revision check against the
HF Hub even when the model is already cached. Concurrent jobs collectively
trip HTTP 429 rate limits, which surface as hard ASR/alignment task
failures (a recent batch showed ~3k Qwen3-ASR failures, all 429 on
huggingface.co/api/models/...).

Add in-library resilience — no external env vars required:
- hf_offline_loading(): ensure a snapshot is cached once across
  processes/nodes (existing cross-node heartbeat lock + retry), then load
  from the local cache only (HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE) so no
  network revision check fires. Restores env afterward (even on error).
- load_hf_resilient(): loader-agnostic offline-load + retry-on-429 wrapper.
- hf_subprocess_env(): subprocess analogue; sets the offline flags in the
  child env only when all referenced models are cached (supports workers
  that load >1 model, e.g. Qwen3-ASR + its companion forced aligner).

Apply at the ASR + forced-alignment load sites that were 429-ing:
Whisper/HF ASR pipeline, Granite, MMS aligner + per-language wav2vec2
(in-process); Qwen, Canary, NeMo (subprocess workers).

Add unit tests for the new helpers (no network).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive audio analysis workflow, a three-axis uncertainty framework, multilingual MMS forced-alignment, and support for new ASR backends (Canary-Qwen, Qwen3-ASR, and Granite Speech) with robust CUDA-aware virtual environment provisioning and PII detection. The review feedback highlights several critical bugs and improvement opportunities, including a timestamp ratio calculation bug for stereo audio in forced_alignment.py, potential process crashes in cuda_probe.py due to narrow exception handling, a potential AttributeError in script_line.py, and opportunities to reduce code duplication and remove unused helper functions in aggregate.py.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

# multiplies the ratio by N and inflates timestamps into the millions of
# seconds, which is what produced the spurious chunk start/end values
# observed in earlier Granite/Canary alignment outputs.
ratio = duration * waveform_segment.size(0) / (trellis.size(0) - 1)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The calculation of ratio multiplies duration by waveform_segment.size(0). Since waveform_segment is shaped (channels, samples), size(0) represents the number of channels (which is 1 for mono, but 2 for stereo). If the input audio is stereo, this will double the ratio and result in incorrect timestamps. It is safer and cleaner to calculate the ratio directly as duration / (trellis.size(0) - 1) without multiplying by the channel count.

Suggested change
ratio = duration * waveform_segment.size(0) / (trellis.size(0) - 1)
ratio = duration / (trellis.size(0) - 1)

f"Attempted index: {self.attempted_index.url}\n"
f"Failing packages: {self.failing_packages}\n"
"Action: downgrade CUDA, set SENSELAB_TORCH_INDEX_URL=https://download.pytorch.org/whl/cpu "
"for CPU fallback, or wait for upstream wheels."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Catching only FileNotFoundError when running nvidia-smi can lead to unhandled exceptions (such as PermissionError or other system-level OSError variants) which would crash the process. It is safer to catch OSError to handle all process execution failures gracefully.

    except (OSError, subprocess.TimeoutExpired):

text=True,
timeout=_PROBE_TIMEOUT_S,
)
except (FileNotFoundError, subprocess.TimeoutExpired):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Catching only FileNotFoundError when running nvcc can lead to unhandled exceptions (such as PermissionError or other system-level OSError variants) which would crash the process. It is safer to catch OSError to handle all process execution failures gracefully.

Suggested change
except (FileNotFoundError, subprocess.TimeoutExpired):
except (OSError, subprocess.TimeoutExpired):

Comment on lines +238 to +239
def _is_meaningful(c: Dict[str, Any]) -> bool:
return c.get("text") is not None or c.get("speaker") is not None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To prevent potential AttributeError if a chunk element c is None, add a defensive null check before calling .get() on it.

Suggested change
def _is_meaningful(c: Dict[str, Any]) -> bool:
return c.get("text") is not None or c.get("speaker") is not None
def _is_meaningful(c: Dict[str, Any]) -> bool:
return c is not None and (c.get("text") is not None or c.get("speaker") is not None)

Comment on lines +32 to +96
def aggregate_presence(votes: dict[str, dict[str, Any]]) -> float | None:
"""Calibrated "is voice present?" uncertainty in ``[0, 1]``.

The presence question is binary, but the goal is *not* to measure
disagreement among voters (Shannon entropy did that) — it's to tell the
user whether voice is present and how strongly the evidence supports
that conclusion. A 6-of-8 split should read as "moderate evidence for
voice", not "high uncertainty".

Algorithm:

1. For each voter, derive a per-voter probability of voice:

- With ``native_confidence`` ``c`` and ``speaks=True``: ``p = c``
(the model says "voice with confidence c").
- With ``native_confidence`` ``c`` and ``speaks=False``: ``p = 1 - c``
(the model says "silence with confidence c", so voice prob is ``1-c``).
- Without ``native_confidence``: ``p = 1.0`` if ``speaks=True``, else
``0.0`` (binary vote — the model is fully committed either way).

2. ``p_voice = mean(per-voter probabilities)`` — equal weight per voter.
Models with a ``native_confidence`` that aren't sure pull ``p_voice``
toward 0.5 even when their boolean ``speaks`` flips one way; binary-only
voters get equal weight to confidence-bearing ones.

3. Uncertainty = ``1 − |2 · p_voice − 1|``: 0 when all evidence agrees
(either way), 1 at a perfect 50/50 split. Whether voice is more likely
present or absent is recoverable from ``p_voice`` itself; this metric
only grades how decisive the evidence is.
"""
p_voice_per_voter: list[float] = []

for v in votes.values():
if not isinstance(v, dict):
continue
if "speaks" not in v:
continue
speak_val = v.get("speaks")
if speak_val is None:
continue
raw_nc = v.get("native_confidence")
nc: float | None
if raw_nc is None:
nc = None
else:
try:
nc = max(0.0, min(1.0, float(raw_nc)))
except (TypeError, ValueError):
nc = None
# ASR hallucination flag: the voter said "speaks" but the model's own
# silence head says otherwise — treat as a vote AGAINST voice instead
# of for it (reflecting that the transcript came from a hallucination,
# not actual audio content).
if v.get("hallucinated"):
p_voter = 0.1 # near-zero with a hint of caution rather than full 0
elif nc is None:
p_voter = 1.0 if speak_val else 0.0
else:
p_voter = nc if speak_val else (1.0 - nc)
p_voice_per_voter.append(p_voter)

if not p_voice_per_voter:
return None
p_voice = sum(p_voice_per_voter) / len(p_voice_per_voter)
return max(0.0, min(1.0, 1.0 - abs(2.0 * p_voice - 1.0)))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The loop in aggregate_presence is identical to the one in presence_p_voice. To reduce code duplication and improve maintainability, aggregate_presence can directly call presence_p_voice and apply the final transformation on the result.

def aggregate_presence(votes: dict[str, dict[str, Any]]) -> float | None:
    p_voice = presence_p_voice(votes)
    if p_voice is None:
        return None
    return max(0.0, min(1.0, 1.0 - abs(2.0 * p_voice - 1.0)))

Comment on lines +21 to +26
def _normalize_transcript_for_wer(text: str) -> str:
"""Lowercase, strip non-word punctuation, collapse whitespace."""
if not text:
return ""
cleaned = _PUNCTUATION_PATTERN.sub(" ", text.lower())
return _WHITESPACE_PATTERN.sub(" ", cleaned).strip()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The private helper function _normalize_transcript_for_wer is defined but never used anywhere in this module. Consider removing it to keep the codebase clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants