|
| 1 | +# Instant audio quality experiment |
| 2 | + |
| 3 | +This is an offline, shadow-only experiment. No route, recording finalizer, player, |
| 4 | +export, upload, desktop capture path, or production flag imports the worker. |
| 5 | +`mode: "off"` returns before filesystem access. There is no publishing mode. |
| 6 | + |
| 7 | +The proposed first rollout is bounded, constant level correction, after production |
| 8 | +validation. EQ and denoising remain experimental because consistent perceptual |
| 9 | +improvement has not been established. This PR does not enable either profile. |
| 10 | + |
| 11 | +## Evidence and limits |
| 12 | + |
| 13 | +The September 7–8, 2026 study measured 60 additional public, unprotected Instant |
| 14 | +recordings with completed transcripts: 20 from each of September 5, 6, and 7, |
| 15 | +all from different owners and separate from the initial 12-recording study. |
| 16 | +The expanded cohort contains 4.464 hours of audio. Median playback-compensated |
| 17 | +loudness is -29.205 LUFS; 40/60 recordings are below -24 LUFS, 9 already exceed |
| 18 | +-18 LUFS, and 9 have true peaks above 0 dBTP. |
| 19 | + |
| 20 | +A fixed 39-recording tuning / 21-recording holdout split preceded processing. |
| 21 | +The final voice policy passed technical gates on 42/60, left 17 unchanged, and |
| 22 | +rejected one short holdout clip for excessive gain. Among the 42 passing clips, |
| 23 | +median loudness moved from -31.745 to -16.84 LUFS, median gain was 14.495 dB, |
| 24 | +and the highest encoded true peak was -1.34 dBTP. All decoded sample counts were |
| 25 | +preserved; container duration changes were at most 21 ms. The reserved holdout |
| 26 | +alone had 15 passes, five unchanged, and that one rejected candidate. |
| 27 | +Constant-gain processing passed 40/60 and left 20 unchanged, with no rejected |
| 28 | +outputs. These counts measure technical eligibility, not listening preference. |
| 29 | + |
| 30 | +Five initial policies produced 195 comparisons. Naive dynamic normalization, |
| 31 | +EQ/compression, and denoising each changed container duration by more than 25 ms |
| 32 | +on 29/39 recordings. Aggressive processing also damaged synthetic intelligibility |
| 33 | +scores. These failures remain in the local evidence; they are not shipping presets. |
| 34 | + |
| 35 | +The adjusted policy uses a 60 Hz high-pass, -0.75 dB at 250 Hz, +0.75 dB at 2.5 kHz, |
| 36 | +6 dB adaptive FFT denoising, and loudness normalization. It has no extra compressor. |
| 37 | +Input and output gates bound gain and peak level. Separate constant-gain processing |
| 38 | +is available for content whose suitability for voice processing is unknown. |
| 39 | + |
| 40 | +The exact final worker was tested on 39 controlled cases at 48 kHz: three reference |
| 41 | +voice excerpts, four additive noise types, and three SNRs, plus the unmodified |
| 42 | +references. Median STOI change was -0.000260, worst -0.005510; none exceeded the |
| 43 | +chosen -0.01 regression tolerance. Six inputs were conservatively left unchanged. |
| 44 | +These are relative tests against existing recordings, not clean studio ground |
| 45 | +truth, subjective quality ratings, or a matched Loom comparison. The calibration |
| 46 | +run with 12 dB denoising exceeded that tolerance in four cases, motivating 6 dB. |
| 47 | + |
| 48 | +Volume-matched RMS in uncaptioned intervals changed by a median +0.096 dB across |
| 49 | +41 passing clips, with a maximum increase of 6.841 dB. Twelve voice candidates |
| 50 | +changed LRA by more than two LU. Those observations require listening review for |
| 51 | +background noise swelling and altered dynamics before enabling voice processing. |
| 52 | + |
| 53 | +The reviewed intelligibility scorer aligns reference, noisy input, and processed |
| 54 | +audio to the same overlapping sample interval before computing STOI and SI-SDR. |
| 55 | +It retains unaligned scores and the measured lag separately; aligning a score does |
| 56 | +not waive timing gates. The 39 exact-worker cases were rescored from their original |
| 57 | +artifacts after this correction, with zero cases below the -0.01 tolerance. |
| 58 | + |
| 59 | +Full-recording LUFS and caption-aligned RMS answer different questions. Caption |
| 60 | +intervals approximate speech activity; uncaptioned audio is not necessarily noise |
| 61 | +or silence. A completed transcript does not establish that a recording contains |
| 62 | +only microphone speech. Mono loudness uses FFmpeg's `dual_mono=true` playback |
| 63 | +compensation consistently; it must not be mixed with uncompensated mono metrics. |
| 64 | + |
| 65 | +## Source and timing guarantees |
| 66 | + |
| 67 | +The worker only reads an absolute regular local source, hashes it before and after, |
| 68 | +and writes into a unique temporary directory. It copies video packets and verifies |
| 69 | +them with the existing packet-proof helper. Existing finalization checks are |
| 70 | +unchanged. Results carry source/output hashes, metrics, version, and validation |
| 71 | +failures; every nonempty validation failure list disqualifies that candidate. |
| 72 | + |
| 73 | +The worker restricts demuxers and protocols to local media files, rejecting playlists |
| 74 | +instead of following their references. Only AAC inputs are eligible for processing; |
| 75 | +other codecs are left unchanged. A MOV/PCM fixture exposed a video preservation |
| 76 | +mismatch, so the first rollout deliberately bypasses that format. |
| 77 | + |
| 78 | +The worker skips silence, extreme levels, existing clipping, unsupported formats, |
| 79 | +already loud content, nonzero audio start times, discontinuous source timestamps, |
| 80 | +and mismatched source audio/video durations. Voice processing additionally requires |
| 81 | +`speechOnlyConfirmed`; the benchmark explicitly overrides this only for local |
| 82 | +research. There is no production content classifier in this change. |
| 83 | + |
| 84 | +FFmpeg's denoiser delays content by two sample-advance blocks without adjusting |
| 85 | +PTS. Padding the tail and trimming that delay preserves boundary speech. Integer |
| 86 | +sample timebases avoid timestamp rounding at 44.1 kHz. The encoded AAC result is |
| 87 | +remeasured, with one bounded peak correction rendered from the original if needed. |
| 88 | +Failed validation never authorizes publication. Cancellation, timeouts, and exceptions |
| 89 | +clean up only the worker's own temporary files. |
| 90 | + |
| 91 | +46 tests cover policy gates, mono/stereo, 44.1/48 kHz, both profiles, speech-like |
| 92 | +markers at clip boundaries, exact video packets, source preservation, silence, |
| 93 | +nonzero/discontinuous timestamps, cancellation, and timeout. Scoped TypeScript and |
| 94 | +Biome checks also pass. Measurements used macOS FFmpeg 8.0.1 and Bun 1.4.0; |
| 95 | +the reviewed format suite also runs in both production-image architectures and |
| 96 | +in the Railway Docker build. The full production cohort was rerun locally. |
| 97 | + |
| 98 | +## Reproducing |
| 99 | + |
| 100 | +Keep source media, transcripts, per-recording measurements, and customer identifiers |
| 101 | +outside the repository. Aggregate results and the frozen worker source hash are in |
| 102 | +[audio-quality-benchmark-summary.json](audio-quality-benchmark-summary.json). |
| 103 | +The local study retains `final-summary.json`, `final-voice-results.json`, |
| 104 | +`final-levels-results.json`, and `report.md`, plus per-recording run receipts. |
| 105 | +Interrupted runs and retries are retained separately. |
| 106 | + |
| 107 | +The manifest is a JSON array with `id`, `split` (`tuning` or `holdout`), `stratum`, |
| 108 | +`createdAt`, and `duration`. Sources are `sources/<id>.m4a`; transcripts are |
| 109 | +`sources/<id>.vtt`. Initial download uses the authenticated Cap CLI for existing |
| 110 | +transcripts and the public playlist for audio. No new transcription is requested. |
| 111 | + |
| 112 | +```sh |
| 113 | +python3 scripts/benchmark-instant-audio.py /absolute/study --phase baseline |
| 114 | +python3 scripts/benchmark-instant-audio.py /absolute/study --phase tuning --policies gain6 gain12 dynamic equalized clean |
| 115 | +bun apps/media-server/scripts/benchmark-audio-quality.ts /absolute/study tuning unique-label voice |
| 116 | +bun apps/media-server/scripts/benchmark-audio-quality.ts /absolute/study holdout another-label levels |
| 117 | +``` |
| 118 | + |
| 119 | +Use a new label per run; the worker benchmark will not overwrite existing results. |
| 120 | +Rejected candidates are recorded with their validation failures but are not copied |
| 121 | +into the output set. Earlier historical runs retained rejected files for diagnosis. |
| 122 | +The intelligibility calibration requires NumPy, SciPy, and pystoi. Supply three |
| 123 | +reference IDs with `--reference-ids`; their M4A files must be two directories above |
| 124 | +the output directory. Output-directory suffix `-v2` selects the corrected mild |
| 125 | +policy; a name containing `strength` selects the 6/12 dB comparison. This calibration |
| 126 | +script records historical filter alternatives; the TypeScript worker benchmark is |
| 127 | +the authoritative final implementation. |
| 128 | + |
| 129 | +## Production-data revalidation |
| 130 | + |
| 131 | +The tighter local-input restrictions were applied to the original 60-recording |
| 132 | +cohort again. All 40 accepted outputs had identical hashes to the benchmark outputs; |
| 133 | +20 sources were left unchanged. Sixteen additional public production files, including |
| 134 | +browser captures and recordings without completed transcripts, were left unchanged |
| 135 | +by stream, level, headroom, or timestamp gates. All original hashes were preserved. |
| 136 | +These are bounded compatibility checks, not proof of safety for every possible file. |
| 137 | + |
| 138 | +## Before serving any enhanced audio |
| 139 | + |
| 140 | +Human review of the 12 volume-matched A/B excerpts is still required. Speech-only |
| 141 | +eligibility, noisy and mixed-system-audio cases, and recordings excluded by the |
| 142 | +public/transcribed selection need broader coverage. The short holdout clip rejected |
| 143 | +for excessive gain must remain on its original audio; do not relax its gate to make |
| 144 | +the benchmark pass. |
| 145 | + |
| 146 | +Run the exact policy in the production Linux image, then verify actual share-page, |
| 147 | +embed, seeking, downloads, edits, transcript alignment, and fallback behavior. |
| 148 | +Measure worker memory, throughput, storage, and tail latency before rollout. |
| 149 | + |
| 150 | +Future integration should create a separately versioned derivative after the |
| 151 | +original is available, using a durable idempotent job bound to the source hash. |
| 152 | +Publish atomically only after validation and only if the source still matches. |
| 153 | +Keep the original available throughout processing, on failure, and for rollback. |
| 154 | +Existing desktop installs could then benefit server-side without a capture update; |
| 155 | +this experiment does not yet implement that serving integration. |
0 commit comments