Skip to content

Commit bcf612c

Browse files
satraclaude
andauthored
comparison & uncertainty stage for analyze_audio.py (#512)
* analyze_audio: foundational comparator framework + CLI flags Lays the foundation for the comparison & uncertainty stage. Adds the new CLI flags, cache key composition, ComparisonRow column lists, parquet writer with comparator_provenance metadata, ComparisonGrid dataclass + iter_buckets, _token_overlaps_window helper, uncertainty aggregator (min/mean/harmonic_mean/disagreement_weighted), LS XML extension hooks for comparator tracks, and disagreements.json index builder. This commit alone produces no reviewer-visible output — the user-story phases (US1–US4) plug differencer logic into this plumbing in subsequent commits. Constitution checks: - §VIII (No Hardcoded Parameters): every comparator threshold and grid resolution is a CLI flag; defaults documented. - §IV (CI Must Stay Green): existing 12 script tests still pass; 11 new comparator-foundational tests added (23 total, all green). - §VII (Simplicity First): comparator code lives inline in analyze_audio.py; no new senselab task module needed. Adds g2p-en to the [nlp] extra (small wheel; needed by the future US3 ASR↔PPG comparator). NLTK averaged_perceptron_tagger_eng auto-downloads on first use. Bumps cache schema 2 → 3 to reflect the new comparator key shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * analyze_audio: US1 raw-vs-enhanced comparator + tests Adds the per-task differencer registry (_diff_diarization, _diff_classification, _diff_asr) and the top-level compare_raw_vs_enhanced driver. Wires it into main() so a default two-pass run now emits parquet sidecars under <run_dir>/comparisons/raw_vs_enhanced/<task>/<model>.parquet plus a disagreements.json index ranking the top-N flagged regions. LS bundle additions: one Labels track per (task, model) actually compared, named pass_pair__compare__<task>__<model> with the fixed enum {agree, disagree, incomparable, one_sided}; ASR-vs-ASR pairs also get a sibling TextArea track carrying WER + both transcripts. Track names are appended via build_labelstudio_config(... , comparator_tracks=tracks); a run with --skip comparisons produces bit-identical output to current main (FR-005, SC-005 anchored by test_comparator_skip_no_op_preserves_ls_bundle_shape). Cache key composition follows the comparison_cache_key helper from Phase 2, using the upstream-task cache_keys so re-running an upstream task cleanly invalidates downstream comparisons. Tests added (29 total now in src/tests/scripts/, all green): - test_raw_vs_enhanced_diarization_diff covers boundary_shift - test_raw_vs_enhanced_asr_text_diff covers per-window WER + a/b text - test_raw_vs_enhanced_handles_failed_pass covers the incomparable edge case from FR-010 / US1 acceptance scenario 3 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * analyze_audio: US2 within-stream comparator + tests Adds compare_within_stream that, for each task with two or more successful models in a pass, iterates itertools.combinations and emits one parquet per ordered model pair under <run_dir>/<pass>/comparisons/<task>/<a>_vs_<b>.parquet (US2 layout in contracts/comparison-row.parquet.md). For ASR pairs the WER row schema is reused; for diarization the speaks_a/speaks_b shape; for classification (AST vs YAMNet) the top1/score/entropy shape, with the existing scene_agreement.json semantics preserved as a strict subset. LS bundle: one Labels track per (pass, task, model_pair) with the fixed enum {agree, disagree, incomparable, one_sided}; ASR pairs also emit a sibling TextArea track carrying WER + both transcripts. ASR-vs-ASR honors --asr-reference-model (default Whisper turbo) so the soft-reference side is consistent across runs. Tests added (29 total now, all green): - test_within_stream_asr_pair_emits_wer covers the WER per-bucket path - test_within_stream_single_model_no_op covers the graceful no-op when a task has only one successful model - test_within_stream_classification_superset_of_scene_agreement covers the AST-vs-YAMNet matching-grid path Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * analyze_audio: US3 cross-stream comparator + tests Adds compare_cross_stream and three sub-comparators: - _diff_asr_vs_diarization: per cross-stream-grid bucket, sets asr_says_speech via _token_overlaps_window (Q4 clarification: any transcript token whose timestamp range overlaps the bucket counts) and diar_says_speech via diarization-segment overlap. - _diff_classification_vs_diarization: per bucket, picks the AST/YAMNet window whose center is closest to the bucket center and asks whether its top-1 label is in the configurable --speech-presence-labels allowlist (default: AudioSet 'Speech' subtree); compares to diarization the same way. - _diff_asr_vs_ppg: per bucket, derives ARPAbet phonemes from the ASR transcript via g2p_en, argmaxes the PPG output frames, and reports a continuous phoneme_per (jiwer.wer) plus a boolean phoneme_disagreement flag at >= --phoneme-disagreement-threshold (default 0.50; Q5 two-tier clarification). PPG path gracefully degrades when no PPG result is available — records "PPG backend not provisioned" in disagreements.json's incomparable_reasons rather than crashing (FR-010, US3 acceptance scenario 4). LS bundle: one Labels track per (pass, stream_pair, model_pair) with the fixed enum {agree, disagree, incomparable, one_sided}. Tests added (37 total now in src/tests/scripts/, all green): - test_cross_stream_asr_speech_in_silent_window - test_cross_stream_classification_speech_allowlist - test_cross_stream_ppg_unavailable_degrades_gracefully - test_cross_stream_phoneme_per_two_tier (continuous PER + boolean flag) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * analyze_audio: US4 per-region uncertainty + ranked disagreements.json Adds per-region confidence harvesters and threads combined_uncertainty through every comparator emitted by the previous phases: - _whisper_chunk_confidence converts Whisper's per-chunk avg_logprob -> exp(avg_logprob) confidence in [0, 1] and surfaces no_speech_prob alongside. - _whisper_bucket_confidence averages overlapping chunks within a comparison-grid bucket (FR-007: arithmetic mean of native events inside one bucket). - _enrich_diff_asr_with_confidence + _enrich_diff_classification_with_confidence populate confidence_a/b and combined_uncertainty in-place on the rows produced by the existing differencers. - _models_without_native_confidence walks the summary and lists the pyannote/Sortformer/Granite/Canary-Qwen/Qwen3-ASR backends — these contribute to comparisons but lack a native scalar in v1, per research.md §2. The disagreements.json index now ranks regions by combined_uncertainty descending (NaN-last). The aggregator is the configurable --uncertainty-aggregator (default min — most-doubtful model wins; mean / harmonic_mean / disagreement_weighted available). Per FR-007 the missing-confidence model list rides on the index for transparency. Tests added (38 total now in src/tests/scripts/, all green): - test_uncertainty_whisper_avg_logprob_extracted - test_uncertainty_aggregator_min_default_via_compare (end-to-end: build two ASR results with different avg_logprob, verify the parquet's combined_uncertainty matches 1 - min(...)) - test_uncertainty_missing_signal_dropped_not_zeroed - test_disagreements_index_top_n_and_ordering - test_models_without_native_confidence_lists_known_null Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * analyze_audio: aligned timeline plot Adds build_aligned_timeline_plot which reads a finished run dir and emits <run_dir>/timeline.png — a stacked, time-aligned figure showing diarization speaker bars per (pass, model), ASR token spans per (pass, model), AST/YAMNet scene-window strips colored by top-1 score, and one comparator row per parquet with disagreement bars colored by combined_uncertainty (Reds; deeper red = higher uncertainty). Hooked into main() at the very end of a run (after disagreements.json is written) so every default invocation now produces the figure alongside the existing parquets / JSON. Best-effort: a plot failure is logged to stderr and the rest of the outputs are unaffected. Plot row layout, top-down: - One row per (pass, diarization_model) — speaker-colored bars. - One row per (pass, asr_model) — token spans (filled bars from chunk timestamps, or a single span for text-only ASR after auto-align). - One row per (pass, scene-classifier) — score-shaded windows. - One row per comparator parquet — disagreement bars; comparator agreement is rendered as the empty grey background. Constitution checks: - §VII (Simplicity First): one new function, no new module; uses matplotlib (already pulled in by pyannote-audio). - §VIII (No Hardcoded Parameters): figure size scales with duration; no hardcoded paths beyond the conventional run_dir. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * polish: mark Phase 7 tasks done T052-T058 marked complete in tasks.md after full E2E validation. E2E (run twin-1_20260509-032631): - All 13 tasks succeed across both raw_16k + enhanced_16k passes - 38 comparator parquets emitted (within_stream / cross_stream / raw_vs_enhanced) - disagreements.json: 16,513 rows, 10,133 disagreements, top-100 - timeline.png: aligned multi-row plot rendered Senselab full suite: 399 passed, 1 failed (the same voice_cloning order-flake from PR #510; passes in isolation; unrelated to comparator). SC-005 (no regression in existing senselab outputs) holds. Smoke tests: 38 passed, 7 skipped. Known v1 gap documented in artifacts/compare_uncertainty_e2e_validation.md: the senselab HuggingFaceASR wrapper does not currently surface Whisper's avg_logprob, so the disagreements.json ranking has only NaN combined_uncertainty values across the whole run. The harvester is correct — it has nothing to read. Follow-up PR will opt the HF pipeline into per-chunk avg_logprob. T059 (PR open) intentionally left unchecked — pending a fresh push. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * polish: mark T059 (PR opened) done PR #512 opened against alpha: #512 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * PR review: pre-commit fix (mypy 1.20 stricter type inference) CI's mypy 1.20.1 flagged two var-annotated diagnostics that the local mypy 1.18.x missed: - test_comparator_skip_no_op_preserves_ls_bundle_shape: annotate ``summary: dict[str, Any]`` so the empty-dict literal doesn't collapse to ``dict[str, dict[str, dict[Never, Never]]]``. - test_comparison_cache_key_changes_with_inputs: convert the ``base = dict(...)`` factory call to a literal ``{...}`` with an explicit ``dict[str, Any]`` annotation so the inferred value type accepts the heterogeneous values (str / list / dict). Pull the unpacked-params expression out into a local so mypy doesn't try to type-check ``{**base["params"], ...}`` against the wrong base inference. No code changes; all pre-commit hooks pass locally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * PR #512 review: address Gemini findings Applies the 5 actionable Gemini comments on the comparator stage: - HIGH (line 1526): compare_raw_vs_enhanced now also runs for AST and YAMNet — they're top-level summary keys (single-model tasks) rather than under by_model. Their parquet rows carry real time spans driven by the per-task win/hop and combined_uncertainty derived from the top-1 score via _enrich_diff_classification_with_confidence. - MEDIUM (line 725): _enrich_diff_asr_with_confidence now passes the per-row WER into _aggregate_uncertainty as ``mismatch_severity`` so the disagreement_weighted aggregator actually scales with the textual disagreement (matches FR-003 intent; otherwise it collapsed to ``mean``). - MEDIUM (line 991/995): build_aligned_timeline_plot now reads AST and YAMNet win/hop from ``summary["scene_window"]`` rather than hardcoded 10.24/0.96 constants. CLI overrides surface in the plot correctly. - MEDIUM (line 1316): _diff_classification now uses ``win_length_a`` + new ``hop_length_a`` to compute real per-row start/end timestamps rather than integer placeholders. Both call sites (raw_vs_enhanced AST/YAMNet and within_stream AST-vs-YAMNet) pass the explicit hop. - MEDIUM (line 2234): _whisper_bucket_confidence now falls back to the segment-level avg_logprob when the parent ScriptLine has no chunks — covers post-aligned text-only ASR backends like Granite / Canary-Qwen if they ever surface a per-segment scalar in a future senselab release. Also tightens the matplotlib colormap usage to mypy 1.20-compliant ``matplotlib.colormaps.get_cmap`` form (was ``plt.cm.<name>`` which mypy now flags as attr-defined). Did NOT change: NLTK runtime download (line 1914). The download is a one-time cmudict + averaged_perceptron_tagger_eng fetch (small files); air-gapped envs can pre-populate ~/nltk_data via the existing ``T001`` setup task before running. Will revisit if a follow-up requires fully offline operation. All 38 script smoke tests still pass; mypy + ruff clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * audio_analysis workflow: speech-aware speaker clustering + 3-axis uncertainty The audio_analysis workflow lands as a standalone module under src/senselab/audio/workflows/audio_analysis/ with the analyze_audio script as a thin wrapper. Highlights from this iteration: - Speech-aware embedding clustering: a YAMNet → AST → openSMILE-loudness fallback mask is built per embedding window; non-speech windows are tagged NOISE and skip clustering, so silent / music / vehicle stretches no longer inflate the speaker count. - Spectral clustering on a precomputed cosine-similarity affinity is now the default, with k-means as the per-k fallback. New CLI flag --clustering-algorithm exposes the choice. - Min-cluster-size guard (max(5, 10% of valid windows)) rejects partitions where a 2-3 window outlier is propping up the silhouette score. - Cross-model speaker unification (clustering.py + plot.py) processes the synthetic embedding source first to seed the centroid pool, then freezes the pool — so a noisy 3-segment pyannote / sortformer label can't spawn a unified third "speaker" beyond what the embedding source validated. Validation: twin-1.wav unified-speaker count drops from 3 → 2; English two-speaker conversation stays at 2 (no regression). 57/57 workflow tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * PR review: pre-commit fixes (mypy annotations + test naming + toml format) - global_summary.py: explicit `list[float]` annotations on stoi/pesq/sisdr accumulators - test_aggregate.py: `dict[str, dict[str, Any]]` annotations on votes literals (mypy 1.20 narrows mixed-value literals to dict[str, object]) - Rename test_*.py → *_test.py to match repo convention - pyproject.toml: pretty-format-toml auto-fix Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * PR review: address 6 findings + same-speaker post-merge step Review findings addressed: - HIGH: ``min_cluster_size`` floor was 5 windows but ``min_windows_for_clustering`` is 4 — for passes with 4-9 valid windows every k≥2 partition got rejected silently and we fell to single-cluster regime with no diagnostic. Lowered floor to ``max(2, round(0.10 * n))`` so the gate is reachable; record a ``failures`` entry when all partitions are rejected by min-size. - HIGH: ``compute_uncertainty_axes`` mutates the caller's ``passes`` dict (injects the synthetic embedding diar source). Removed the "pure function" claim from the module docstring and documented the mutation explicitly. - MEDIUM: ``_speech_window_mask`` is a YAMNet veto, not a fallback ladder. Documented the veto explicitly (YAMNet decides; AST only when YAMNet missing; loudness only when both missing) and the known YAMNet-confusion failure modes (child voices flagged as "Music") plus the mitigation. - MEDIUM: extracted ``assign_unified_clusters_with_seed_phase`` shared helper in clustering.py; both ``cluster_speaker_labels_by_embedding`` and the plot's ``_cluster_speakers_by_embedding`` now share the seed-then- freeze logic. - LOW: replaced the schema-progression block comment + hardcoded ``"schema_version": 1`` parquet literal with a reference to the ``_CACHE_SCHEMA_VERSION`` constant so they can never drift. - LOW: surfaced the spectral→k-means fallback through the ``failures`` dict (per-k entries) and recorded which algorithm actually produced the partition in the cluster output. New: same-speaker post-merge step. Spectral / k-means at k≥2 sometimes splits one speaker's recording into sub-clusters when prosody / distance varies. After silhouette-driven k* selection, the closest centroid pair is iteratively merged while cosine similarity ≥ ``merge_threshold`` (default 0.55). The threshold keeps similar-voiced but distinct speakers (same gender + age + family resemblance ~cos_sim 0.3) separate while folding in clear-cut same-speaker prosodic outliers (typical adult VoxCeleb same- speaker cos_sim 0.6+). No target speaker count is assumed anywhere. New: single-utterance / quiet-environment path. When 1+ valid speech windows exist but the count is below ``min_windows_for_clustering``, the clusterer returns ``n_speakers=1`` with all valid windows tagged ``S0`` instead of skipping. This is the "single speaker, just one word in an otherwise quiet recording" case the prior fall-through silently dropped. Validation: both ``twin-1.wav`` (2 distinct child speakers, one speaking briefly) and the English two-speaker conversation still report ``n_speakers=2``; merge step does not fire on either (centroid cos_sim 0.31 and 0.15 respectively, both below 0.55). 57/57 workflow tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * unified speaker clustering: group-aware seed phase + cross-pass threshold Two-pass speaker-axis fix surfaced by ``audio_48khz_mono_16bits.wav`` (a 5-second clip where 4 distinct speakers each say ``This is <Name>``): The synthetic embedding source's per-pass clusterer correctly found 4 distinct speakers in both raw_16k and enhanced_16k. The plot's cross-pass alignment ``_cluster_speakers_by_embedding`` then collapsed them to 2 because it ran ``assign_unified_clusters_with_seed_phase`` with a single 0.5 cosine threshold that didn't distinguish "within-pass distinct speakers (validated by silhouette + min-size + post-merge)" from "cross-pass same-speaker match". For short similar-phonetic utterances, different speakers can sit at cos_sim > 0.5 and got merged. Fix: ``assign_unified_clusters_with_seed_phase`` is now group-aware: - ``seed_groups: list[list[(key, mean_emb)]]`` — each inner list comes from a single per-source clustering. Within a group, every label gets its own distinct centroid (no within-group merging — those clusters were already validated upstream). - Across groups, items merge when centroid cosine ≥ ``cross_group_threshold`` (default 0.75 — same-speaker cos_sim across raw/enh is typically 0.85+, different speakers within a pass sit around 0.30-0.50). - ``other_items`` (pyannote / sortformer) still snap to the seed pool at the legacy ``cosine_threshold`` (0.5) and never spawn new clusters. Both callers updated: - ``cluster_speaker_labels_by_embedding`` (per-pass): one seed group = embedding source's labels for this pass. - Plot's ``_cluster_speakers_by_embedding`` (cross-pass): one seed group per (pass, embedding_source) so raw and enhanced cluster sets stay intact and align speaker-to-speaker across passes at 0.75. Validation (cache-hit reruns after the fix): - audio_48khz_mono_16bits (4 distinct speakers, brief utterances): legend S0–S3, 4 colors visible per embedding-source row. - twin-1 (2 distinct child speakers): legend S0/S1, no collapse. - english_conversation_higgs (2 adult speakers): legend S0/S1, no regression. The per-pass speaker clustering is unchanged; only the cross-pass alignment in the timeline plot + the per-pass cross-model unification in clustering.py were affected. Embedding clustering remains independent for raw and enhanced passes — same-speaker matching across them happens only when cos_sim crosses the 0.75 cross-group bar. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 71a2a85 commit bcf612c

47 files changed

Lines changed: 10168 additions & 76 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

pyproject.toml

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,17 @@ dependencies = [
6161
nlp = [
6262
"jiwer>=3.0",
6363
"nltk>=3.9",
64-
"uroman>=1.3"
64+
"uroman>=1.3",
65+
"g2p-en>=2.1"
66+
]
67+
pii = [
68+
# Microsoft Presidio for PII detection over ASR transcripts.
69+
# Presidio bundles purpose-built recognizers (EMAIL, PHONE, US_SSN, CREDIT_CARD,
70+
# IP_ADDRESS, PERSON, LOCATION, ...) on top of a spaCy NLP backbone. The
71+
# spaCy model itself is not pulled by pip — install it separately:
72+
# uv run python -m spacy download en_core_web_lg
73+
"presidio-analyzer>=2.2",
74+
"spacy>=3.7"
6575
]
6676
text = [
6777
"sentence-transformers>=5.1",
@@ -198,4 +208,4 @@ skip = [
198208
"*.cha",
199209
"*.ipynb"
200210
]
201-
ignore-words-list = ["senselab", "nd", "astroid", "wil", "SER", "te", "EXPRESSO", "VAI", "vie"]
211+
ignore-words-list = ["senselab", "nd", "astroid", "wil", "SER", "te", "EXPRESSO", "VAI", "vie", "g2p"]

scripts/analyze_audio.py

Lines changed: 655 additions & 54 deletions
Large diffs are not rendered by default.
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# Specification Quality Checklist: Comparison & Uncertainty Stage
2+
3+
**Purpose**: Validate specification completeness and quality before proceeding to planning
4+
**Created**: 2026-05-08
5+
**Last verified**: 2026-05-09 (post three-axis pivot)
6+
**Feature**: [spec.md](../spec.md)
7+
8+
## Content Quality
9+
10+
- [X] No implementation details (languages, frameworks, APIs)
11+
- [X] Focused on user value and business needs
12+
- [X] Written for non-technical stakeholders
13+
- [X] All mandatory sections completed
14+
15+
## Requirement Completeness
16+
17+
- [X] No [NEEDS CLARIFICATION] markers remain
18+
- [X] Requirements are testable and unambiguous
19+
- [X] Success criteria are measurable
20+
- [X] Success criteria are technology-agnostic (no implementation details)
21+
- [X] All acceptance scenarios are defined
22+
- [X] Edge cases are identified
23+
- [X] Scope is clearly bounded
24+
- [X] Dependencies and assumptions identified
25+
26+
## Feature Readiness
27+
28+
- [X] All functional requirements have clear acceptance criteria
29+
- [X] User scenarios cover primary flows
30+
- [X] Feature meets measurable outcomes defined in Success Criteria
31+
- [X] No implementation details leak into specification
32+
33+
## Notes
34+
35+
- 2026-05-09 re-verification confirms all 16 items still pass against the rebuilt
36+
spec/plan/research after the three-axis pivot. No `[NEEDS CLARIFICATION]` markers
37+
remain (verified via grep).
38+
- The four user stories were rewritten in-place against the new design without renumbering:
39+
US1 (P1) "did enhancement help?" via the 3 raw_vs_enhanced parquets is the MVP;
40+
US2 (P2) per-pass parquets and US3 (P2) cross-stream contributions layer on; US4 (P3)
41+
adds the disagreements index + 5-row timeline plot.
42+
- Backwards compatibility (existing per-task outputs unchanged) is the hard constraint
43+
in SC-005 — the workflow is purely additive on top of the existing per-task pipeline.
44+
- PPG availability, AudioSet label allowlist, aggregator choice, and bucket grid are all
45+
CLI-configurable with documented defaults; the spec records the defaults in FRs rather
46+
than as open clarifications.
47+
- LS bin thresholds (`< 0.33` / `[0.33, 0.66)` / `≥ 0.66`) are hardcoded in FR-005 and
48+
flagged in plan.md "Out of Scope" as a Constitution VIII follow-up. Acceptable as a
49+
documented tradeoff, not a quality-checklist failure.
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# CLI Contract — analyze_audio.py comparator additions
2+
3+
All flags are additive on top of the script's existing analyze_audio flags.
4+
5+
## New flags
6+
7+
| Flag | Type | Default | Description |
8+
|---|---|---|---|
9+
| `--cross-stream-win-length` | float (seconds) | `0.5` | Bucket length for the comparator. |
10+
| `--cross-stream-hop-length` | float (seconds) | `0.5` | Hop length for the comparator (≤ win-length; default = non-overlapping). |
11+
| `--uncertainty-aggregator` `{min,mean,harmonic_mean,disagreement_weighted}` | choice | `min` | Function used to collapse per-axis sub-signals into `aggregated_uncertainty`. |
12+
| `--phoneme-disagreement-threshold` | float in `[0, 1]` | `0.50` | Threshold on `phoneme_per` above which `phoneme_disagreement = true` (utterance axis sub-signal). |
13+
| `--speech-presence-labels` `LABEL [LABEL ...]` | space-separated list of strings (`nargs="+"`) | seven AudioSet "Speech" subtree labels | AudioSet labels that count as "speech-present" for AST/YAMNet contributions to the presence axis. AudioSet labels themselves contain commas (e.g. `"Narration, monologue"`) — that's why this is `nargs="+"` not a comma-string. |
14+
| `--asr-reference-model` | string (HF model id) | `openai/whisper-large-v3-turbo` | Reserved for utterance-axis transcript-consensus tiebreaks (currently used to pick the soft reference for the LS TextArea sibling track). |
15+
| `--diarization-boundary-shift-ms` | float ≥ 0 | `50.0` | Boundary-shift tolerance for treating two diar segments as describing the same speech region; per Constitution §VIII (No Hardcoded Parameters). |
16+
| `--disagreements-top-n` | int ≥ 0 | `100` | Number of top-ranked rows in `disagreements.json`. `0` disables the index. |
17+
18+
## `--skip comparisons`
19+
20+
The script's existing `--skip` flag accepts `comparisons` as a new value to disable the
21+
entire comparator stage. With `--skip comparisons`, the script's output is identical to a
22+
run from before the comparator was added — no `<run_dir>/<pass>/uncertainty/` subtree, no
23+
`<run_dir>/uncertainty/raw_vs_enhanced/` subtree, no `disagreements.json`, no comparator
24+
tracks in the LS bundle. This satisfies SC-005.
25+
26+
## Defaults rationale
27+
28+
- `0.5 s` non-overlapping grid: matches the temporal resolution that's actually meaningful
29+
given the underlying signal granularities (Whisper word ≈ 20 ms, pyannote ≈ 62.5 ms,
30+
AST 10.24 s) without double-counting via overlap.
31+
- `min` aggregator: surfaces the most-doubtful contributing signal — the right default for
32+
reviewers reading the disagreements index ("show me where *any* model is unsure").
33+
- `0.50` phoneme PER threshold: half the phonemes in a transcript span had to be wrong
34+
before flagging — a strong signal, not a noisy one.
35+
36+
## Example invocations
37+
38+
```bash
39+
# Default — runs the full pipeline including comparator
40+
uv run python scripts/analyze_audio.py audio.wav
41+
42+
# Switch the aggregator from "max-doubtful-signal" to "average-doubt"
43+
uv run python scripts/analyze_audio.py audio.wav --uncertainty-aggregator mean
44+
45+
# Tighter grid for sub-second analysis (slower, double-counts via overlap)
46+
uv run python scripts/analyze_audio.py audio.wav \
47+
--cross-stream-win-length 0.2 --cross-stream-hop-length 0.1
48+
49+
# Treat singing/laughter as "speech" for AST/YAMNet contributions to presence
50+
uv run python scripts/analyze_audio.py audio.wav \
51+
--speech-presence-labels Speech Conversation Singing Laughter \
52+
"Narration, monologue"
53+
54+
# Skip the entire comparator stage
55+
uv run python scripts/analyze_audio.py audio.wav --skip comparisons
56+
57+
# Disable the disagreements index but keep the parquets + plot + LS bundle
58+
uv run python scripts/analyze_audio.py audio.wav --disagreements-top-n 0
59+
```
60+
61+
## Validation rules
62+
63+
The script rejects (via `argparse.error`):
64+
65+
- `--cross-stream-win-length <= 0`
66+
- `--cross-stream-hop-length > --cross-stream-win-length`
67+
- `--phoneme-disagreement-threshold` outside `[0, 1]`
68+
- `--diarization-boundary-shift-ms < 0`
69+
- `--disagreements-top-n < 0`
Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# JSON Contract — `<run_dir>/disagreements.json`
2+
3+
Top-level ranked index. Built once per run by reducing across the 9 uncertainty parquets.
4+
5+
## Shape
6+
7+
```json
8+
{
9+
"schema_version": 1,
10+
"generated_at": "2026-05-09T17:30:00Z",
11+
"wrapper_hash": "<sha256 of analyze_audio.py>",
12+
"senselab_version": "1.3.1a27.dev17",
13+
14+
"config": {
15+
"top_n": 100,
16+
"aggregator": "min",
17+
"phoneme_disagreement_threshold": 0.50,
18+
"bucket_grid": {"win_length": 0.5, "hop_length": 0.5},
19+
"speech_presence_labels": [
20+
"Speech",
21+
"Conversation",
22+
"Narration, monologue",
23+
"Female speech, woman speaking",
24+
"Male speech, man speaking",
25+
"Child speech, kid speaking",
26+
"Speech synthesizer"
27+
]
28+
},
29+
30+
"models_without_native_signal": [
31+
"pyannote/speaker-diarization-community-1",
32+
"nvidia/diar_sortformer_4spk-v1",
33+
"ibm-granite/granite-speech-3.3-8b",
34+
"nvidia/canary-qwen-2.5b",
35+
"Qwen/Qwen3-ASR-1.7B"
36+
],
37+
38+
"incomparable_reasons": {
39+
"raw_16k/utterance/ppg": "PPG backend not provisioned",
40+
"enhanced_16k/utterance/granite": "no ScriptLine produced (upstream task failed)"
41+
},
42+
43+
"totals": {
44+
"total_rows": 1080,
45+
"rows_by_axis": {"presence": 360, "identity": 360, "utterance": 360},
46+
"rows_by_pass": {"raw_16k": 360, "enhanced_16k": 360, "raw_vs_enhanced": 360},
47+
"high_uncertainty_rows": 47,
48+
"high_uncertainty_rate": 0.0435
49+
},
50+
51+
"entries": [
52+
{
53+
"rank": 1,
54+
"axis": "utterance",
55+
"pass": "raw_16k",
56+
"start": 12.5,
57+
"end": 13.0,
58+
"aggregated_uncertainty": 0.91,
59+
"contributing_models": ["openai/whisper-large-v3-turbo", "ibm-granite/granite-speech-3.3-8b", "nvidia/canary-qwen-2.5b", "Qwen/Qwen3-ASR-1.7B"],
60+
"parquet": "raw_16k/uncertainty/utterance.parquet",
61+
"row_idx": 25,
62+
"ls_region_id": "raw_16k__uncertainty__utterance__25",
63+
"summary": "pairwise WER 0.83 across 4 ASR; whisper avg_logprob low"
64+
}
65+
]
66+
}
67+
```
68+
69+
## Ranking rules
70+
71+
1. Primary key: `aggregated_uncertainty` desc. NaN values sort last.
72+
2. Tiebreak 1: axis priority `utterance > identity > presence`.
73+
3. Tiebreak 2: `start` ascending.
74+
4. Truncate to `--disagreements-top-n` (default 100). `--disagreements-top-n 0` skips the
75+
index (file not written).
76+
77+
## Per-entry fields
78+
79+
| Field | Type | Description |
80+
|---|---|---|
81+
| `rank` | int | 1-indexed rank in the truncated index. |
82+
| `axis` | string | One of `{"presence", "identity", "utterance"}`. |
83+
| `pass` | string | One of `{"raw_16k", "enhanced_16k", "raw_vs_enhanced"}`. |
84+
| `start`, `end` | float | Bucket boundaries in seconds. |
85+
| `aggregated_uncertainty` | float | The headline scalar in `[0, 1]`. |
86+
| `contributing_models` | list[string] | Models that voted on this bucket. |
87+
| `parquet` | string | Path of the source parquet relative to `<run_dir>`. |
88+
| `row_idx` | int | Row index inside the source parquet. |
89+
| `ls_region_id` | string | Region id assigned in `labelstudio_tasks.json`. Joins to the LS bundle. |
90+
| `summary` | string | One-line human-readable explanation of why this bucket scored high. |
91+
92+
## `models_without_native_signal`
93+
94+
Documents which models lacked a native confidence scalar so reviewers know why those
95+
models contribute only via cross-model aggregation rather than per-bucket native signals.
96+
This is a top-level field rather than per-entry to keep the entries terse.
97+
98+
## `incomparable_reasons`
99+
100+
Keyed by `<pass>/<axis>/<sub-signal>` (e.g. `raw_16k/utterance/ppg`). Each value is a
101+
one-line reason the sub-signal was unavailable so the reviewer can audit
102+
`comparison_status="incomparable"` / `"unavailable"` rows without re-running the script.
103+
104+
## `totals`
105+
106+
- `total_rows`: rows across all 9 parquets.
107+
- `rows_by_axis` / `rows_by_pass`: per-axis / per-pass counts.
108+
- `high_uncertainty_rows`: count of rows with `aggregated_uncertainty >= 0.66` (the LS
109+
`high` bin per FR-005).
110+
- `high_uncertainty_rate`: ratio over `total_rows` (NaN when `total_rows == 0`).
Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
# Label Studio Bundle Contract — comparator additions
2+
3+
The comparator stage extends `<run_dir>/labelstudio_tasks.json` and
4+
`<run_dir>/labelstudio_config.xml` with three Labels tracks per pass plus three raw-vs-enhanced
5+
tracks plus one TextArea track for utterance.
6+
7+
## XML config additions (`labelstudio_config.xml`)
8+
9+
Six `<Labels>` blocks (3 per pass × 2 passes) plus three for `raw_vs_enhanced`:
10+
11+
```xml
12+
<Labels name="raw_16k__uncertainty__presence" toName="audio">
13+
<Label value="low"/>
14+
<Label value="medium"/>
15+
<Label value="high"/>
16+
<Label value="incomparable"/>
17+
<Label value="unavailable"/>
18+
</Labels>
19+
20+
<Labels name="raw_16k__uncertainty__identity" toName="audio">
21+
<Label value="low"/>
22+
<Label value="medium"/>
23+
<Label value="high"/>
24+
<Label value="incomparable"/>
25+
<Label value="unavailable"/>
26+
</Labels>
27+
28+
<Labels name="raw_16k__uncertainty__utterance" toName="audio">
29+
<Label value="low"/>
30+
<Label value="medium"/>
31+
<Label value="high"/>
32+
<Label value="incomparable"/>
33+
<Label value="unavailable"/>
34+
</Labels>
35+
36+
<!-- … same three blocks for enhanced_16k … -->
37+
38+
<!-- raw_vs_enhanced delta tracks -->
39+
<Labels name="pass_pair__uncertainty__presence" toName="audio">
40+
<Label value="low"/><Label value="medium"/><Label value="high"/>
41+
<Label value="incomparable"/><Label value="unavailable"/>
42+
</Labels>
43+
<!-- … identity and utterance same shape … -->
44+
```
45+
46+
Plus one TextArea sibling for the utterance tracks (one per pass + one for `pass_pair`):
47+
48+
```xml
49+
<TextArea name="raw_16k__uncertainty__utterance__text"
50+
toName="audio" perRegion="true" editable="false"
51+
placeholder="Per-bucket transcript consensus + dissenting models"/>
52+
```
53+
54+
## Bin mapping
55+
56+
`aggregated_uncertainty` from each parquet row → label value:
57+
58+
| `aggregated_uncertainty` | LS label |
59+
|---|---|
60+
| `< 0.33` | `low` |
61+
| `[0.33, 0.66)` | `medium` |
62+
| `≥ 0.66` | `high` |
63+
| (any) with `comparison_status == "incomparable"` | `incomparable` |
64+
| (any) with `comparison_status == "unavailable"` | `unavailable` |
65+
66+
## Tasks JSON additions (`labelstudio_tasks.json`)
67+
68+
For each row in each of the 9 parquets, append one Labels region to the existing pass-level
69+
task (the `pass_pair` rows are appended to the `raw_16k` task by convention, since LS doesn't
70+
have a notion of "pass pair"):
71+
72+
```json
73+
{
74+
"id": "raw_16k__uncertainty__utterance__25",
75+
"from_name": "raw_16k__uncertainty__utterance",
76+
"to_name": "audio",
77+
"type": "labels",
78+
"value": {
79+
"start": 12.5,
80+
"end": 13.0,
81+
"labels": ["high"]
82+
}
83+
}
84+
```
85+
86+
Plus, for every row in the **utterance** parquets, also append one TextArea region carrying
87+
the per-bucket transcript consensus + dissenting-model transcripts:
88+
89+
```json
90+
{
91+
"id": "raw_16k__uncertainty__utterance__25__text",
92+
"from_name": "raw_16k__uncertainty__utterance__text",
93+
"to_name": "audio",
94+
"type": "textarea",
95+
"value": {
96+
"start": 12.5,
97+
"end": 13.0,
98+
"text": [
99+
"consensus: \"hello world\"\nwhisper-turbo: \"hello world\"\ngranite: \"hello word\"\ncanary-qwen: \"hello\"\nqwen3: \"hello world\""
100+
]
101+
}
102+
}
103+
```
104+
105+
## Region-id naming
106+
107+
`<pass>__uncertainty__<axis>__<row_idx>` — joins to `disagreements.json` entries via the
108+
`ls_region_id` field. The TextArea sibling appends `__text`.
109+
110+
## Region-emission policy
111+
112+
- Every parquet row is emitted as an LS region (no top-N filtering at the LS layer — the
113+
`disagreements.json` top-N drives the *ranking* but every bucket is still scrubbable).
114+
- For high-volume runs (long clips, many models), reviewers can hide low-uncertainty rows
115+
in LS via the standard label filter (e.g. show only `high` + `medium`).

0 commit comments

Comments
 (0)