release: merge v1.8.11 — sanitizer round-2 + #287 + #271 + #291 fixes

Author: meizhong986

.gitignore

@@ -85,8 +85,10 @@ output/
 *.mkv
 *.srt
 !examples/*.srt
+!tests/fixtures/**/*.srt
 .tmp/
 raw_subs/
+!tests/fixtures/**/raw_subs/
 
 # Installer binaries
 installer/*.exe

docs/ISSUE_TRACKER_v1.8.x.md

@@ -0,0 +1,141 @@
+**Sanitizer hardening + small fixes**
+The Japanese SRT sanitizer is more robust. Two smaller issues with Colab
+install and Ollama translation are also fixed.
+
+---
+
+
+Subtitle Sanitization
+
+- Hardening of the Japanese SRT sanitization. (#287)
+
+
+Ollama
+
+- **Curated model list + output length control** — one model in the
+  curated list emitted chain-of-thought text that broke the translation
+  format; replaced with an instruct-tuned equivalent. Also added
+  `--ollama-max-tokens` so output length can be tuned for setups that
+  need a different default. (#271)
+
+
+Colab
+
+- **`--provider local` translation on Colab** — the `[llm]` extra was
+  missing from `install_colab.sh`, causing
+  `ModuleNotFoundError: starlette_context` when starting local LLM
+  translation. The extra is now installed. (#291)
+
+---
+
+### How to Upgrade or Install
+
+**Upgrade from 1.8.10:**
+
+```
+pip install -U --no-deps "whisperjav @ git+https://github.com/meizhong986/whisperjav.git@v1.8.11"
+```
+
+**Fresh Install:**
+
+### Windows — Standalone Installer (.exe)
+
+
+1. Download **WhisperJAV-1.8.11-Windows-x86_64.exe** from the Assets below
+2. Run the installer (no admin rights required)
+3. Wait 10-20 minutes for setup to complete
+4. Launch from the Desktop shortcut
+
+Installs to `%LOCALAPPDATA%\WhisperJAV`. A desktop shortcut is created automatically. Your GPU is detected automatically.
+
+### macOS
+
+Requires [Git](https://git-scm.com/downloads). The install script checks for everything else (Xcode CLI Tools, Python, FFmpeg, PortAudio) and tells you exactly what to install if anything is missing. Open Terminal and run:
+
+```bash
+cd ~
+git clone https://github.com/meizhong986/whisperjav.git
+cd whisperjav
+git checkout v1.8.11
+installer/install_mac.sh
+```
+
+After installation, open the `whisperjav` folder in Finder and double-click **WhisperJAV.command** to launch the GUI.
+
+### Linux
+
+Requires Git and Python 3.10-3.12. The install script handles PEP 668 (externally-managed) environments on Debian 12+ / Ubuntu 24.04+. Open a terminal and run:
+
+```bash
+cd ~
+git clone https://github.com/meizhong986/whisperjav.git
+cd whisperjav
+git checkout v1.8.11
+installer/install_linux.sh
+```
+
+After installation, launch the GUI with `./WhisperJAV.sh`.
+
+### Windows — Source Install
+
+Requires [Git](https://git-scm.com/downloads) and [Python 3.10-3.12](https://www.python.org/downloads/). Open a terminal and run:
+
+```
+cd %USERPROFILE%
+git clone https://github.com/meizhong986/whisperjav.git
+cd whisperjav
+git checkout v1.8.11
+installer\install_windows.bat
+```
+
+After installation, double-click **WhisperJAV.bat** to launch the GUI.
+
+
+
+
+## Compatibility
+
+Same as v1.8.10 — no dependency changes.
+
+| Component | Supported Versions |
+|-----------|-------------------|
+| Python | 3.10, 3.11, 3.12 |
+| PyTorch | 2.4.0 - 2.10.x |
+| CUDA | 11.8+ (12.4+ recommended) |
+| Ollama | 0.3.0+ recommended |
+
+## Known Issues
+
+- **Windows standalone installer does not add its bundled tools to user PATH.**
+  WhisperJAV ships its own ffmpeg 7.1 (pinned `>=6,<8`) inside the install
+  directory's `Library\bin`, but the installer does not persistently add
+  that folder to the user's PATH. If you have a separate ffmpeg on your
+  PATH — especially ffmpeg 8.x — WhisperJAV subprocesses that run outside
+  the GUI launcher may resolve to that other ffmpeg instead of the bundled
+  one. Behavior can be unexpected if the third-party ffmpeg has different
+  default flags or filter names. Workaround: launch via the Desktop
+  shortcut (which activates the environment) or manually add
+  `<install-dir>\Library\bin` to your user PATH. A proper fix is planned
+  for v1.8.12.
+
+- **Apple Silicon MPS + whisper-large-v3-turbo** — produces garbage output
+  on MPS for this specific model. Use `--hf-device cpu` or the default
+  kotoba model. (#198, #227)
+
+- **Ollama download progress** — the download progress bar in the popup is
+  indeterminate (pulsing). Real progress is shown in the terminal. A hint
+  in the popup directs you to check there.
+
+## What's Next
+
+**v1.8.12** — installer PATH fix (above), a ZipEnhancer Colab init bug
+(#290), Qwen3-ASR `transformers` version pin (#280), and verification of
+the Colab fix in this release.
+
+**v1.9.0** — full Ollama migration (remove llama-cpp-python), standalone
+subtitle merge CLI (#230), Chinese GUI partial i18n (#175, #180), speaker
+diarization (#248, #252).
+
+---
+
+Thanks to everyone who reported issues and tested on their own material.

installer/RELEASE_NOTES_v1.8.11.md

@@ -6,11 +6,11 @@
 [version]
 major = 1
 minor = 8
-patch = 10
+patch = 11
 # PEP 440 prerelease suffix for wheel/pip (a0, b0, rc0, .post1, or empty for stable)
-prerelease = .post3
+prerelease =
 # Human-readable label for display (alpha, beta, rc, hotfix1, or empty for clean version)
-display_label = hotfix3
+display_label =
 architecture = v4.4
 
 [metadata]

installer/VERSION

@@ -173,7 +173,7 @@ info "This may take 1-3 minutes..."
 # --system: required since we're not in a venv
 # --no-build-isolation: not needed, uv handles this
 # Colab's torch/numpy/etc. are reused — only WhisperJAV's own deps are installed.
-if ! uv pip install --system "git+${WHISPERJAV_REPO}@${WHISPERJAV_BRANCH}#egg=whisperjav[cli,enhance,translate,huggingface,qwen,analysis,compatibility]"; then
+if ! uv pip install --system "git+${WHISPERJAV_REPO}@${WHISPERJAV_BRANCH}#egg=whisperjav[cli,enhance,translate,huggingface,qwen,analysis,compatibility,llm]"; then
     error "WhisperJAV installation failed"
     exit 1
 fi

installer/install_colab.sh

@@ -28,8 +28,8 @@
     "pass1_model = \"automatic\" #@param [\"automatic\", \"large-v2\", \"large-v3\", \"turbo\", \"kotoba-bilingual\", \"kotoba-v2.0\", \"kotoba-v2.1\", \"kotoba-v2.2\"]\n",
     "\n",
     "#@markdown **Expert Audio Setup**\n",
-    "pass1_scene_detector = \"automatic\" #@param [\"automatic\", \"auditok\", \"silero\", \"semantic\"]\n",
-    "pass1_speech_segmenter = \"automatic\" #@param [\"automatic\", \"silero\", \"ten\", \"none\"]\n",
+    "pass1_scene_detector = \"semantic\" #@param [\"automatic\", \"auditok\", \"silero\", \"semantic\"]\n",
+    "pass1_speech_segmenter = \"silero-v3.1\" #@param [\"automatic\", \"silero-v3.1\", \"ten\", \"none\"]\n",
     "pass1_speech_enhancer = \"none\" #@param [\"none\", \"ffmpeg-dsp\", \"clearvoice\", \"zipenhancer\", \"bs-roformer\"]\n",
     "#@markdown <font size=\"1\">auditok=energy (fast), silero=VAD, semantic=texture (complex audio) | enhancer: ffmpeg-dsp(no GPU), clearvoice(48k), bs-roformer(vocal)</font>\n",
     "\n",
@@ -51,8 +51,8 @@
     "pass2_model = \"automatic\" #@param [\"automatic\", \"large-v2\", \"large-v3\", \"turbo\", \"kotoba-bilingual\", \"kotoba-v2.0\", \"kotoba-v2.1\", \"kotoba-v2.2\"]\n",
     "\n",
     "#@markdown **Expert Audio Setup**\n",
-    "pass2_scene_detector = \"automatic\" #@param [\"automatic\", \"auditok\", \"silero\", \"semantic\"]\n",
-    "pass2_speech_segmenter = \"automatic\" #@param [\"automatic\", \"silero\", \"ten\", \"none\"]\n",
+    "pass2_scene_detector = \"semantic\" #@param [\"automatic\", \"auditok\", \"silero\", \"semantic\"]\n",
+    "pass2_speech_segmenter = \"silero-v3.1\" #@param [\"automatic\", \"silero-v3.1\", \"ten\", \"none\"]\n",
     "pass2_speech_enhancer = \"none\" #@param [\"none\", \"ffmpeg-dsp\", \"clearvoice\", \"zipenhancer\", \"bs-roformer\"]\n",
     "\n",
     "#@markdown **FFmpeg Filters** *(only if enhancer is ffmpeg-dsp)*\n",
@@ -706,4 +706,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 0
-}
+}

notebook/WhisperJAV_colab_edition_expert.ipynb

@@ -0,0 +1,5 @@
+# Diff script working output — not committed (regenerated each run)
+current_*/
+
+# Round-2 verifier working output — regenerated each run
+round2_escapees/_run_output/

tests/fixtures/sanitizer_regression/.gitignore

@@ -0,0 +1,118 @@
+# Sanitizer Regression Corpus — Acceptance Matrix
+
+> Contract document for v1.8.11 sanitizer fixes.
+> Baseline captured at HEAD: **e224333** (2026-04-18)
+> Plan reference: `docs/plans/V1811_SANITIZER_FIX_PLAN.md`
+
+## Acceptance Categories
+
+Every fixture must produce an output that falls into exactly one of these categories:
+
+- **A. EXPECTED CHANGE** — output differs from baseline; the difference is an expected consequence of Fix 1, 2, 3, or 5a
+- **B. BYTE-IDENTICAL** — output is byte-identical to baseline
+- **C. EXPECTED DROP** — subtitle removed; artifact record shows a valid reason code (`hallucination`, `repetition`, `abnormally_fast_cps_*`, `abnormally_slow_cps_*`, `symbol_only_residue`)
+
+Any output not fitting A/B/C blocks the v1.8.11 release.
+
+---
+
+## Fix 1 — `${N:0:M}` slice syntax
+
+Durations are 0.5s so post-strip residues have CPS > 1.0 and survive the CPS filter.
+
+| Fixture | Input | Baseline (pre-fix) | Expected (post-fix) | Category |
+|---|---|---|---|---|
+| 01_vowel_run_alone.srt | `いいいいいいいいいいいい` | (dropped — empty) | `いい` (kept) | **A** |
+| 02_vowel_run_with_punct.srt | `いいいいいいいいいいいい?` | `?` (kept — #287 symptom) | `いい?` (kept) | **A** |
+| 02b_vowel_run_with_period.srt | `うううううううううううう。` | `。` | `うう。` | **A** |
+| 03_vocalization_hagh.srt | `はぁはぁはぁはぁはぁはぁはぁはぁはぁはぁ` | (dropped — empty) | `はぁ` (kept) | **A** |
+| 04_small_tsu_run.srt | `いっいっいっいっいっいっいっいっいっいっ` | (dropped — empty) | `いっ` (kept) | **A** |
+| 05_long_vowel_mark.srt | `〜〜〜〜〜〜〜〜〜〜〜〜` | (dropped — empty) | (dropped by Fix 2 — `〜〜〜` is symbol-only) | **C** |
+| 06_non_slice_replacement.srt | `こんにちは` | `こんにちは` | `こんにちは` | **B** |
+| 07_kana_in_context.srt | `わああうううううううううううう` | `わああ` | `わああうう` | **A** |
+
+---
+
+## Fix 2 — Symbol-only purge
+
+| Fixture | Input | Baseline (pre-fix) | Expected (post-fix) | Category |
+|---|---|---|---|---|
+| 01_bang_only.srt | `!!` | (dropped — L135 `nonsensical`) | (dropped — `symbol_only_residue` if reaches Fix 2, or same reason as baseline) | **B** or **C** |
+| 02_question_only.srt | `???` | (dropped — L135) | (dropped) | **B** or **C** |
+| 03_ellipsis_only.srt | `……` | `……` (kept — L135 doesn't match ellipsis) | (dropped — `symbol_only_residue`) | **A** |
+| 04_music_notes_only.srt | `♪♫♩` | `♫♩` (kept — ♪ stripped by L268) | (dropped — `symbol_only_residue` after Fix 5a also strips ♫♩) | **A** |
+| 05_animal_emoji_only.srt | `🐈🐈🐈` | `🐈🐈🐈` (kept) | (dropped — `symbol_only_residue`) | **A** |
+| 06_fullwidth_punct_only.srt | `！！！` | (dropped) | (dropped) | **B** or **C** |
+| 07_whitespace_plus_punct.srt | `  !!  ` | (dropped — L135) | (dropped) | **B** or **C** |
+| 08_mixed_kana_and_punct.srt | `こんにちは!!` | (dropped — `こんにちは` in filter list, `!!` stripped) | (dropped — same) | **B** |
+| 09_kanji_only.srt | `家` | `家` (kept — has kanji) | `家` (kept) | **B** |
+| 10_fullwidth_alnum.srt | `ｈｅｌｌｏ` | `ｈｅｌｌｏ` (kept) | `ｈｅｌｌｏ` (kept) | **B** |
+| 11_fix1_residue_kana_plus_q.srt | `いいいいいいいいいいいい?` | `?` (bug residue) | `いい?` (Fix 1 keeps kana; Fix 2 keeps — has kana) | **A** |
+
+Notes:
+- For cases where baseline already drops via a different path, post-fix may either (a) drop via the same path (B) or (b) drop via `symbol_only_residue` if control flow changes slightly (C). Either is acceptable provided the final outcome matches.
+
+---
+
+## Fix 3 — CPS newline exclusion
+
+| Fixture | Input | Baseline text_len | Post-fix text_len | Expected post-fix | Category |
+|---|---|---|---|---|---|
+| 01_multiline_normal.srt | `こんにちは\n今日は` @ 2s | 9 (incl. `\n`) | 8 | kept (both CPS in safe range) | **B** |
+| 02_multiline_fast.srt | `はい\nはい\nはい` @ 0.1s | 8 (incl. 2x`\n`) | 6 | dropped (both: CPS > 30) | **B** |
+| 03_multiline_slow_short.srt | `あ\nい\nう\nえ` @ 10s | 7 | 4 | **dropped (new behavior)** — slow CPS + len ≤ 4 | **A** |
+| 04_single_line_baseline.srt | `あいう` @ 1s | 3 | 3 | kept (no newlines) | **B** |
+
+---
+
+## Fix 5a — SDH patterns
+
+| Fixture | Input | Baseline | Expected post-fix | Category |
+|---|---|---|---|---|
+| 01_tortoise_shell.srt | `〔効果音〕こんにちは` | `〔効果音〕こんにちは` (unchanged — no pattern for 〔〕) | `こんにちは` | **A** |
+| 02_new_music_symbols.srt | `♫♩♬🎵🎶こんにちは` | `♫♩♬🎵🎶こんにちは` (unchanged — ♫♩♬🎵🎶 not in current L268) | `こんにちは` | **A** |
+| 03_sdh_bgm.srt | `[BGM]おはよう` | `[BGM]おはよう` (unchanged — specific `[音楽]` only) | `おはよう` | **A** |
+| 04_sdh_breathing.srt | `(喘息)` | (dropped — L65 strips halfwidth parens) | (dropped — same via L65 or new pattern) | **B** |
+| 05_sdh_noise.srt | `【ノイズ】` | (dropped — L86 strips fullwidth square) | (dropped — same) | **B** |
+| 06_regression_existing_music_symbol.srt | `♪★こんにちは` | `こんにちは` (♪ and ★ stripped by L268) | `こんにちは` (unchanged) | **B** |
+
+---
+
+## Combined fixtures
+
+### `01_all_four_fixes_together.srt`
+
+Exercises all four fixes in one SRT. Expected post-fix output:
+
+| Block | Input | Expected |
+|---|---|---|
+| 1 | `いいいいいいいいいいいい?` @ 0.5s | `いい?` (Fix 1 produces residue, Fix 2 keeps) |
+| 2 | `!!` @ 2s | dropped |
+| 3 | `あ\nい\nう\nえ` @ 10s | dropped (Fix 3: text_len=4, CPS=0.4) |
+| 4 | `〔効果音〕おはよう` @ 3s | `おはよう` (Fix 5a strips tortoise shell) |
+| 5 | `こんにちは` @ 3s | dropped (filter list hit — pre-existing) |
+
+Final SRT should contain subs 1 and 4 only (renumbered 1 and 2).
+
+### `02_negative_control_all_legitimate.srt`
+
+**Criterion**: post-fix output must be **BYTE-IDENTICAL to baseline**.
+
+Note: baseline already drops 6 of 22 subs due to filter-list inclusions (pre-existing behavior, not in scope for v1.8.11). The remaining 16 kept subs must stay kept.
+
+Pre-existing drops (informational only — not our fixes' concern):
+- `こんにちは`, `すみません`, `そうですね`, `気持ちいい`, `お母さん`, and one more (indexes 1, 4, 7, 8, 13 per baseline audit).
+
+Any NEW drop introduced by Fix 1/2/3/5a is a false positive and blocks release.
+
+---
+
+## Verification Protocol (Phase 6)
+
+1. Run `python tests/fixtures/sanitizer_regression/generate_synthetic.py` — regenerates fixtures (idempotent).
+2. Run `python tests/fixtures/sanitizer_regression/capture_baseline.py` — captures current-state output. Named after current HEAD SHA.
+3. After each code fix, re-run step 2 — creates new baseline directory.
+4. Diff the post-fix baseline against `baseline_e224333/` directory:
+   - For every fixture, classify diff as A / B / C per this document.
+   - Any fixture output not classifiable blocks the release.
+5. Archive diffs as evidence in plan document appendix.

tests/fixtures/sanitizer_regression/ACCEPTANCE.md

@@ -0,0 +1,37 @@
+Baseline captured at HEAD: e224333
+Total fixtures: 31
+Successful: 31
+Errors: 0
+
+--- Successful captures ---
+  combined/01_all_four_fixes_together.srt: 0 -> 0
+  combined/02_negative_control_all_legitimate.srt: 0 -> 0
+  fix1_slice_syntax/01_vowel_run_alone.srt: 0 -> 0
+  fix1_slice_syntax/02_vowel_run_with_punct.srt: 0 -> 0
+  fix1_slice_syntax/02b_vowel_run_with_period.srt: 0 -> 0
+  fix1_slice_syntax/03_vocalization_hagh.srt: 0 -> 0
+  fix1_slice_syntax/04_small_tsu_run.srt: 0 -> 0
+  fix1_slice_syntax/05_long_vowel_mark.srt: 0 -> 0
+  fix1_slice_syntax/06_non_slice_replacement.srt: 0 -> 0
+  fix1_slice_syntax/07_kana_in_context.srt: 0 -> 0
+  fix2_symbol_purge/01_bang_only.srt: 0 -> 0
+  fix2_symbol_purge/02_question_only.srt: 0 -> 0
+  fix2_symbol_purge/03_ellipsis_only.srt: 0 -> 0
+  fix2_symbol_purge/04_music_notes_only.srt: 0 -> 0
+  fix2_symbol_purge/05_animal_emoji_only.srt: 0 -> 0
+  fix2_symbol_purge/06_fullwidth_punct_only.srt: 0 -> 0
+  fix2_symbol_purge/07_whitespace_plus_punct.srt: 0 -> 0
+  fix2_symbol_purge/08_mixed_kana_and_punct.srt: 0 -> 0
+  fix2_symbol_purge/09_kanji_only.srt: 0 -> 0
+  fix2_symbol_purge/10_fullwidth_alnum.srt: 0 -> 0
+  fix2_symbol_purge/11_fix1_residue_kana_plus_q.srt: 0 -> 0
+  fix3_newline_cps/01_multiline_normal.srt: 0 -> 0
+  fix3_newline_cps/02_multiline_fast.srt: 0 -> 0
+  fix3_newline_cps/03_multiline_slow_short.srt: 0 -> 0
+  fix3_newline_cps/04_single_line_baseline.srt: 0 -> 0
+  fix5a_sdh/01_tortoise_shell.srt: 0 -> 0
+  fix5a_sdh/02_new_music_symbols.srt: 0 -> 0
+  fix5a_sdh/03_sdh_bgm.srt: 0 -> 0
+  fix5a_sdh/04_sdh_breathing.srt: 0 -> 0
+  fix5a_sdh/05_sdh_noise.srt: 0 -> 0
+  fix5a_sdh/06_regression_existing_music_symbol.srt: 0 -> 0

tests/fixtures/sanitizer_regression/baseline_e224333/BASELINE_REPORT.txt

@@ -0,0 +1,15 @@
+1
+00:00:00,200 --> 00:00:00,500
+?
+
+2
+00:00:18,000 --> 00:00:19,000
+あ
+い
+う
+え
+
+3
+00:00:20,000 --> 00:00:23,000
+〔効果音〕おはよう
+