comparison & uncertainty stage for analyze_audio.py#512
Merged
Conversation
Lays the foundation for the comparison & uncertainty stage. Adds the new CLI flags, cache key composition, ComparisonRow column lists, parquet writer with comparator_provenance metadata, ComparisonGrid dataclass + iter_buckets, _token_overlaps_window helper, uncertainty aggregator (min/mean/harmonic_mean/disagreement_weighted), LS XML extension hooks for comparator tracks, and disagreements.json index builder. This commit alone produces no reviewer-visible output — the user-story phases (US1–US4) plug differencer logic into this plumbing in subsequent commits. Constitution checks: - §VIII (No Hardcoded Parameters): every comparator threshold and grid resolution is a CLI flag; defaults documented. - §IV (CI Must Stay Green): existing 12 script tests still pass; 11 new comparator-foundational tests added (23 total, all green). - §VII (Simplicity First): comparator code lives inline in analyze_audio.py; no new senselab task module needed. Adds g2p-en to the [nlp] extra (small wheel; needed by the future US3 ASR↔PPG comparator). NLTK averaged_perceptron_tagger_eng auto-downloads on first use. Bumps cache schema 2 → 3 to reflect the new comparator key shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the per-task differencer registry (_diff_diarization,
_diff_classification, _diff_asr) and the top-level
compare_raw_vs_enhanced driver. Wires it into main() so a default
two-pass run now emits parquet sidecars under
<run_dir>/comparisons/raw_vs_enhanced/<task>/<model>.parquet plus a
disagreements.json index ranking the top-N flagged regions.
LS bundle additions: one Labels track per (task, model) actually
compared, named pass_pair__compare__<task>__<model> with the fixed
enum {agree, disagree, incomparable, one_sided}; ASR-vs-ASR pairs
also get a sibling TextArea track carrying WER + both transcripts.
Track names are appended via build_labelstudio_config(... ,
comparator_tracks=tracks); a run with --skip comparisons produces
bit-identical output to current main (FR-005, SC-005 anchored by
test_comparator_skip_no_op_preserves_ls_bundle_shape).
Cache key composition follows the comparison_cache_key helper from
Phase 2, using the upstream-task cache_keys so re-running an
upstream task cleanly invalidates downstream comparisons.
Tests added (29 total now in src/tests/scripts/, all green):
- test_raw_vs_enhanced_diarization_diff covers boundary_shift
- test_raw_vs_enhanced_asr_text_diff covers per-window WER + a/b text
- test_raw_vs_enhanced_handles_failed_pass covers the incomparable
edge case from FR-010 / US1 acceptance scenario 3
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds compare_within_stream that, for each task with two or more
successful models in a pass, iterates itertools.combinations and
emits one parquet per ordered model pair under
<run_dir>/<pass>/comparisons/<task>/<a>_vs_<b>.parquet (US2 layout
in contracts/comparison-row.parquet.md). For ASR pairs the WER row
schema is reused; for diarization the speaks_a/speaks_b shape; for
classification (AST vs YAMNet) the top1/score/entropy shape, with
the existing scene_agreement.json semantics preserved as a strict
subset.
LS bundle: one Labels track per (pass, task, model_pair) with the
fixed enum {agree, disagree, incomparable, one_sided}; ASR pairs
also emit a sibling TextArea track carrying WER + both transcripts.
ASR-vs-ASR honors --asr-reference-model (default Whisper turbo) so
the soft-reference side is consistent across runs.
Tests added (29 total now, all green):
- test_within_stream_asr_pair_emits_wer covers the WER per-bucket path
- test_within_stream_single_model_no_op covers the graceful no-op
when a task has only one successful model
- test_within_stream_classification_superset_of_scene_agreement
covers the AST-vs-YAMNet matching-grid path
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds compare_cross_stream and three sub-comparators:
- _diff_asr_vs_diarization: per cross-stream-grid bucket, sets
asr_says_speech via _token_overlaps_window (Q4 clarification:
any transcript token whose timestamp range overlaps the bucket
counts) and diar_says_speech via diarization-segment overlap.
- _diff_classification_vs_diarization: per bucket, picks the
AST/YAMNet window whose center is closest to the bucket center
and asks whether its top-1 label is in the configurable
--speech-presence-labels allowlist (default: AudioSet 'Speech'
subtree); compares to diarization the same way.
- _diff_asr_vs_ppg: per bucket, derives ARPAbet phonemes from the
ASR transcript via g2p_en, argmaxes the PPG output frames, and
reports a continuous phoneme_per (jiwer.wer) plus a boolean
phoneme_disagreement flag at >= --phoneme-disagreement-threshold
(default 0.50; Q5 two-tier clarification).
PPG path gracefully degrades when no PPG result is available —
records "PPG backend not provisioned" in disagreements.json's
incomparable_reasons rather than crashing (FR-010, US3 acceptance
scenario 4).
LS bundle: one Labels track per (pass, stream_pair, model_pair)
with the fixed enum {agree, disagree, incomparable, one_sided}.
Tests added (37 total now in src/tests/scripts/, all green):
- test_cross_stream_asr_speech_in_silent_window
- test_cross_stream_classification_speech_allowlist
- test_cross_stream_ppg_unavailable_degrades_gracefully
- test_cross_stream_phoneme_per_two_tier (continuous PER + boolean flag)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds per-region confidence harvesters and threads combined_uncertainty through every comparator emitted by the previous phases: - _whisper_chunk_confidence converts Whisper's per-chunk avg_logprob -> exp(avg_logprob) confidence in [0, 1] and surfaces no_speech_prob alongside. - _whisper_bucket_confidence averages overlapping chunks within a comparison-grid bucket (FR-007: arithmetic mean of native events inside one bucket). - _enrich_diff_asr_with_confidence + _enrich_diff_classification_with_confidence populate confidence_a/b and combined_uncertainty in-place on the rows produced by the existing differencers. - _models_without_native_confidence walks the summary and lists the pyannote/Sortformer/Granite/Canary-Qwen/Qwen3-ASR backends — these contribute to comparisons but lack a native scalar in v1, per research.md §2. The disagreements.json index now ranks regions by combined_uncertainty descending (NaN-last). The aggregator is the configurable --uncertainty-aggregator (default min — most-doubtful model wins; mean / harmonic_mean / disagreement_weighted available). Per FR-007 the missing-confidence model list rides on the index for transparency. Tests added (38 total now in src/tests/scripts/, all green): - test_uncertainty_whisper_avg_logprob_extracted - test_uncertainty_aggregator_min_default_via_compare (end-to-end: build two ASR results with different avg_logprob, verify the parquet's combined_uncertainty matches 1 - min(...)) - test_uncertainty_missing_signal_dropped_not_zeroed - test_disagreements_index_top_n_and_ordering - test_models_without_native_confidence_lists_known_null Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds build_aligned_timeline_plot which reads a finished run dir and emits <run_dir>/timeline.png — a stacked, time-aligned figure showing diarization speaker bars per (pass, model), ASR token spans per (pass, model), AST/YAMNet scene-window strips colored by top-1 score, and one comparator row per parquet with disagreement bars colored by combined_uncertainty (Reds; deeper red = higher uncertainty). Hooked into main() at the very end of a run (after disagreements.json is written) so every default invocation now produces the figure alongside the existing parquets / JSON. Best-effort: a plot failure is logged to stderr and the rest of the outputs are unaffected. Plot row layout, top-down: - One row per (pass, diarization_model) — speaker-colored bars. - One row per (pass, asr_model) — token spans (filled bars from chunk timestamps, or a single span for text-only ASR after auto-align). - One row per (pass, scene-classifier) — score-shaded windows. - One row per comparator parquet — disagreement bars; comparator agreement is rendered as the empty grey background. Constitution checks: - §VII (Simplicity First): one new function, no new module; uses matplotlib (already pulled in by pyannote-audio). - §VIII (No Hardcoded Parameters): figure size scales with duration; no hardcoded paths beyond the conventional run_dir. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
T052-T058 marked complete in tasks.md after full E2E validation. E2E (run twin-1_20260509-032631): - All 13 tasks succeed across both raw_16k + enhanced_16k passes - 38 comparator parquets emitted (within_stream / cross_stream / raw_vs_enhanced) - disagreements.json: 16,513 rows, 10,133 disagreements, top-100 - timeline.png: aligned multi-row plot rendered Senselab full suite: 399 passed, 1 failed (the same voice_cloning order-flake from PR #510; passes in isolation; unrelated to comparator). SC-005 (no regression in existing senselab outputs) holds. Smoke tests: 38 passed, 7 skipped. Known v1 gap documented in artifacts/compare_uncertainty_e2e_validation.md: the senselab HuggingFaceASR wrapper does not currently surface Whisper's avg_logprob, so the disagreements.json ranking has only NaN combined_uncertainty values across the whole run. The harvester is correct — it has nothing to read. Follow-up PR will opt the HF pipeline into per-chunk avg_logprob. T059 (PR open) intentionally left unchecked — pending a fresh push. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive 'Comparison & Uncertainty' stage to the audio analysis pipeline, enabling automated detection of discrepancies between raw and enhanced audio, disagreements between different models of the same task, and cross-stream inconsistencies. Feedback identifies several critical improvements: including 'ast' and 'yamnet' in raw-vs-enhanced comparisons, refining the 'disagreement_weighted' aggregator with meaningful severity metrics, and removing hardcoded temporal parameters from the timeline plot. Additionally, the reviewer suggests improving the robustness of classification timestamps, managing NLTK dependencies during environment setup to avoid runtime failures, and implementing fallback logic for confidence harvesting when word-level timestamps are unavailable.
CI's mypy 1.20.1 flagged two var-annotated diagnostics that the
local mypy 1.18.x missed:
- test_comparator_skip_no_op_preserves_ls_bundle_shape: annotate
``summary: dict[str, Any]`` so the empty-dict literal doesn't
collapse to ``dict[str, dict[str, dict[Never, Never]]]``.
- test_comparison_cache_key_changes_with_inputs: convert the
``base = dict(...)`` factory call to a literal ``{...}`` with
an explicit ``dict[str, Any]`` annotation so the inferred value
type accepts the heterogeneous values (str / list / dict).
Pull the unpacked-params expression out into a local so mypy
doesn't try to type-check ``{**base["params"], ...}`` against
the wrong base inference.
No code changes; all pre-commit hooks pass locally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Applies the 5 actionable Gemini comments on the comparator stage: - HIGH (line 1526): compare_raw_vs_enhanced now also runs for AST and YAMNet — they're top-level summary keys (single-model tasks) rather than under by_model. Their parquet rows carry real time spans driven by the per-task win/hop and combined_uncertainty derived from the top-1 score via _enrich_diff_classification_with_confidence. - MEDIUM (line 725): _enrich_diff_asr_with_confidence now passes the per-row WER into _aggregate_uncertainty as ``mismatch_severity`` so the disagreement_weighted aggregator actually scales with the textual disagreement (matches FR-003 intent; otherwise it collapsed to ``mean``). - MEDIUM (line 991/995): build_aligned_timeline_plot now reads AST and YAMNet win/hop from ``summary["scene_window"]`` rather than hardcoded 10.24/0.96 constants. CLI overrides surface in the plot correctly. - MEDIUM (line 1316): _diff_classification now uses ``win_length_a`` + new ``hop_length_a`` to compute real per-row start/end timestamps rather than integer placeholders. Both call sites (raw_vs_enhanced AST/YAMNet and within_stream AST-vs-YAMNet) pass the explicit hop. - MEDIUM (line 2234): _whisper_bucket_confidence now falls back to the segment-level avg_logprob when the parent ScriptLine has no chunks — covers post-aligned text-only ASR backends like Granite / Canary-Qwen if they ever surface a per-segment scalar in a future senselab release. Also tightens the matplotlib colormap usage to mypy 1.20-compliant ``matplotlib.colormaps.get_cmap`` form (was ``plt.cm.<name>`` which mypy now flags as attr-defined). Did NOT change: NLTK runtime download (line 1914). The download is a one-time cmudict + averaged_perceptron_tagger_eng fetch (small files); air-gapped envs can pre-populate ~/nltk_data via the existing ``T001`` setup task before running. Will revisit if a follow-up requires fully offline operation. All 38 script smoke tests still pass; mypy + ruff clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
@gemini-code-assist thanks for the thorough review. Pushed 12833a2c addressing the 5 actionable findings: Applied:
Not applied:
Mypy + ruff clean; all 38 script smoke tests still pass. |
Contributor
|
Thanks for the update, @satra. The implementation looks solid and addresses the actionable feedback from the review. The decision to keep the NLTK download as-is for v1 is reasonable given the existing T001 setup for air-gapped environments. The integration of the comparison stage into the existing cache and the inclusion of the |
…ertainty The audio_analysis workflow lands as a standalone module under src/senselab/audio/workflows/audio_analysis/ with the analyze_audio script as a thin wrapper. Highlights from this iteration: - Speech-aware embedding clustering: a YAMNet → AST → openSMILE-loudness fallback mask is built per embedding window; non-speech windows are tagged NOISE and skip clustering, so silent / music / vehicle stretches no longer inflate the speaker count. - Spectral clustering on a precomputed cosine-similarity affinity is now the default, with k-means as the per-k fallback. New CLI flag --clustering-algorithm exposes the choice. - Min-cluster-size guard (max(5, 10% of valid windows)) rejects partitions where a 2-3 window outlier is propping up the silhouette score. - Cross-model speaker unification (clustering.py + plot.py) processes the synthetic embedding source first to seed the centroid pool, then freezes the pool — so a noisy 3-segment pyannote / sortformer label can't spawn a unified third "speaker" beyond what the embedding source validated. Validation: twin-1.wav unified-speaker count drops from 3 → 2; English two-speaker conversation stays at 2 (no regression). 57/57 workflow tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rmat) - global_summary.py: explicit `list[float]` annotations on stoi/pesq/sisdr accumulators - test_aggregate.py: `dict[str, dict[str, Any]]` annotations on votes literals (mypy 1.20 narrows mixed-value literals to dict[str, object]) - Rename test_*.py → *_test.py to match repo convention - pyproject.toml: pretty-format-toml auto-fix Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Review findings addressed: - HIGH: ``min_cluster_size`` floor was 5 windows but ``min_windows_for_clustering`` is 4 — for passes with 4-9 valid windows every k≥2 partition got rejected silently and we fell to single-cluster regime with no diagnostic. Lowered floor to ``max(2, round(0.10 * n))`` so the gate is reachable; record a ``failures`` entry when all partitions are rejected by min-size. - HIGH: ``compute_uncertainty_axes`` mutates the caller's ``passes`` dict (injects the synthetic embedding diar source). Removed the "pure function" claim from the module docstring and documented the mutation explicitly. - MEDIUM: ``_speech_window_mask`` is a YAMNet veto, not a fallback ladder. Documented the veto explicitly (YAMNet decides; AST only when YAMNet missing; loudness only when both missing) and the known YAMNet-confusion failure modes (child voices flagged as "Music") plus the mitigation. - MEDIUM: extracted ``assign_unified_clusters_with_seed_phase`` shared helper in clustering.py; both ``cluster_speaker_labels_by_embedding`` and the plot's ``_cluster_speakers_by_embedding`` now share the seed-then- freeze logic. - LOW: replaced the schema-progression block comment + hardcoded ``"schema_version": 1`` parquet literal with a reference to the ``_CACHE_SCHEMA_VERSION`` constant so they can never drift. - LOW: surfaced the spectral→k-means fallback through the ``failures`` dict (per-k entries) and recorded which algorithm actually produced the partition in the cluster output. New: same-speaker post-merge step. Spectral / k-means at k≥2 sometimes splits one speaker's recording into sub-clusters when prosody / distance varies. After silhouette-driven k* selection, the closest centroid pair is iteratively merged while cosine similarity ≥ ``merge_threshold`` (default 0.55). The threshold keeps similar-voiced but distinct speakers (same gender + age + family resemblance ~cos_sim 0.3) separate while folding in clear-cut same-speaker prosodic outliers (typical adult VoxCeleb same- speaker cos_sim 0.6+). No target speaker count is assumed anywhere. New: single-utterance / quiet-environment path. When 1+ valid speech windows exist but the count is below ``min_windows_for_clustering``, the clusterer returns ``n_speakers=1`` with all valid windows tagged ``S0`` instead of skipping. This is the "single speaker, just one word in an otherwise quiet recording" case the prior fall-through silently dropped. Validation: both ``twin-1.wav`` (2 distinct child speakers, one speaking briefly) and the English two-speaker conversation still report ``n_speakers=2``; merge step does not fire on either (centroid cos_sim 0.31 and 0.15 respectively, both below 0.55). 57/57 workflow tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…hold Two-pass speaker-axis fix surfaced by ``audio_48khz_mono_16bits.wav`` (a 5-second clip where 4 distinct speakers each say ``This is <Name>``): The synthetic embedding source's per-pass clusterer correctly found 4 distinct speakers in both raw_16k and enhanced_16k. The plot's cross-pass alignment ``_cluster_speakers_by_embedding`` then collapsed them to 2 because it ran ``assign_unified_clusters_with_seed_phase`` with a single 0.5 cosine threshold that didn't distinguish "within-pass distinct speakers (validated by silhouette + min-size + post-merge)" from "cross-pass same-speaker match". For short similar-phonetic utterances, different speakers can sit at cos_sim > 0.5 and got merged. Fix: ``assign_unified_clusters_with_seed_phase`` is now group-aware: - ``seed_groups: list[list[(key, mean_emb)]]`` — each inner list comes from a single per-source clustering. Within a group, every label gets its own distinct centroid (no within-group merging — those clusters were already validated upstream). - Across groups, items merge when centroid cosine ≥ ``cross_group_threshold`` (default 0.75 — same-speaker cos_sim across raw/enh is typically 0.85+, different speakers within a pass sit around 0.30-0.50). - ``other_items`` (pyannote / sortformer) still snap to the seed pool at the legacy ``cosine_threshold`` (0.5) and never spawn new clusters. Both callers updated: - ``cluster_speaker_labels_by_embedding`` (per-pass): one seed group = embedding source's labels for this pass. - Plot's ``_cluster_speakers_by_embedding`` (cross-pass): one seed group per (pass, embedding_source) so raw and enhanced cluster sets stay intact and align speaker-to-speaker across passes at 0.75. Validation (cache-hit reruns after the fix): - audio_48khz_mono_16bits (4 distinct speakers, brief utterances): legend S0–S3, 4 colors visible per embedding-source row. - twin-1 (2 distinct child speakers): legend S0/S1, no collapse. - english_conversation_higgs (2 adult speakers): legend S0/S1, no regression. The per-pass speaker clustering is unchanged; only the cross-pass alignment in the timeline plot + the per-pass cross-model unification in clustering.py were affected. Embedding clustering remains independent for raw and enhanced passes — same-speaker matching across them happens only when cos_sim crosses the 0.75 cross-group bar. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 13, 2026
This was referenced May 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a fourth stage to
scripts/analyze_audio.pythat consumes the per-task outputs from the existing pipeline and emits comparison artefacts in four classes:--uncertainty-aggregator(defaultmin).Outputs (all additive —
--skip comparisonsproduces bit-identical legacy output):<run_dir>/<pass>/comparisons/<task>/<a>_vs_<b>.parquet(within-stream + raw_vs_enhanced)<run_dir>/<pass>/comparisons/cross_stream/...parquet<run_dir>/disagreements.json— top-N ranked by combined uncertainty<run_dir>/timeline.png— multi-row aligned plot showing diarization speakers, ASR token spans, scene-window strips, and per-comparator disagreement bars colored by combined uncertaintyNew CLI flags:
--skip-comparisons {raw_vs_enhanced,within_stream,cross_stream},--cross-stream-win-length(0.2 s),--cross-stream-hop-length(0.1 s),--uncertainty-aggregator {min,mean,harmonic_mean,disagreement_weighted},--phoneme-disagreement-threshold(0.50),--speech-presence-labels,--asr-reference-model,--diarization-boundary-shift-ms(50.0),--disagreements-top-n(100).Spec workflow
Followed the full spec-kit cycle:
/speckit.specify→/speckit.clarify(5 questions) →/speckit.plan→/speckit.tasks(57 tasks across 7 phases) →/speckit.analyze(11 findings, all remediated) →/speckit.implement. Every clarification, plan decision, and contract is captured underspecs/20260508-173136-compare-uncertainty/.E2E validation
Run on a 38.4 s real-speech WAV (twin-1.wav) with the full default model list:
38 comparison parquets produced;
disagreements.jsoncarries 16,513 total rows with 10,133 disagreements;timeline.pngrendered as a stacked aligned figure. Full validation report atartifacts/compare_uncertainty_e2e_validation.md(not committed).Test plan
uv run pytest src/tests/scripts/— 38 passed, 7 skipped (~24 s); covers all four comparison kinds plus the four documented degradation modes.uv run pytest --ignore=src/tests/scripts -x -q— 399 passed, 1 failed; the single failure is the samevoice_cloning_testorder-flake from PR audio analysis script + multilingual MMS aligner + 4 new ASR backends #510 (passes in isolation; touches nothing the comparator changes). SC-005 (no regression) holds.uv run mypy scripts/analyze_audio.py— clean.uv run ruff check scripts/analyze_audio.py src/tests/scripts/— clean.Known gap (follow-up)
disagreements.jsonranking currently sorts NaN entries to the top because senselab's HuggingFaceASR wrapper does not opt into Whisper's per-chunkavg_logprob. The harvester (_whisper_chunk_confidence) is correct — it just has nothing to read. Documented in the validation note; small follow-up PR will surfaceavg_logprobfrom the HF pipeline.🤖 Generated with Claude Code
Version
Published prerelease version:
1.3.1-alpha.29Changelog
🐛 Bug Fix
Authors: 1