Commit e459468
fix(sanitizer): round-2 hardening — emoji drop, normalized match, escapees
Adds two new stages to the Japanese hallucination remover pipeline
plus a defensive empty-drop gate, and extends the bundled filter list
with 89 user-reported escapees. Targets classes identified in v1.8.11
post-release testing: outro phrases with varied punctuation, channel
watermarks, SDH residue, paralinguistic noise, and micro-utterances.
Pipeline additions (whisperjav/modules/hallucination_remover.py):
- Stage 1.1.a.5 — Emoji-contains drop gate. Drops the whole sub if
it contains any char in the broad emoji ranges (U+2600-26FF,
U+2702-27B0, U+1F300-1FAFF). Runs right after bracket-context
detection so emoji lines skip exact-match, regex, repetition, and
CPS stages. Covers CAT2 channel watermarks ("🐯 Sound Hodori..."),
CAT1 emoji-tailed outros, and any sub sprinkled with music notes
or pictograms.
- Stage 1.1.d — Full-normalized exact match. Strips all whitespace,
quotes, and punctuation (ASCII + fullwidth + JP) anywhere in the
text and case-folds Latin, then matches against a pre-built
per-language normalized set. Catches escapees that differ from
list entries only by punctuation, ellipsis, internal whitespace,
or case ("by H." vs "BY H.", "はぁ……" vs list "はぁ…",
"はぁっ はぁっ" vs "はぁっはぁっ"). Preserves the prolonged-sound
mark (ー) and combining voicing mark (゛) — these are semantic,
not punctuation. Normalized set is built once at load time in
_build_normalized_exact_lists, O(1) lookup per sub.
Pipeline addition (whisperjav/modules/subtitle_sanitizer.py):
- Stage 2.5 — Defensive empty-text drop gate before renumbering.
Phase 1 already drops empties and the current timing_adjuster
cannot produce empty text; this guards against any future phase-2
stage (e.g. CAT4 soft gate planned for v1.8.12) that might drop
content.
Data additions (filter_list_v08.json):
- Appends 89 strings to japanese[] from user CAT1/CAT2-non-emoji/
CAT3/CAT5/CAT6/CAT7/CAT11 observations. Skips 9 already present
and 4 CAT2 emoji entries (caught by the new emoji gate).
Regression:
- 91 existing sanitizer tests still pass unchanged
(test_hallucination_remover_slice_syntax, _symbol_purge,
_cps_newline, _regexp_v09_additions).
- New tests/fixtures/sanitizer_regression/round2_escapees/ contains a
110-sub fixture (102 escapees + 8 control dialogue lines),
expected_map.json, and verify_round2.py which forces bundled data
via config URL overrides for offline-deterministic runs. Result:
102/102 escapees dropped, 8/8 controls preserved.
Gist sync: filter_list gist (4882bdb3f4f5aa4034a112cebd2e0845) is
updated manually by the user post-commit;
round2_escapees/gist_paste_snippet.json contains the paste-ready
content. Bundled file in repo is authoritative and works offline.
Out of scope for v1.8.11 (deferred): CAT4 contextual soft-rule drop
(needs timing plumbing, v1.8.12), CAT12 ambiguous dialogue (v1.8.12),
CAT13 mis-transcription (different problem class), CAT8/9
comma-separated syllable repetition (existing repetition_cleaner
already covers patterns that escape here).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent e4f9b52 commit e459468
8 files changed
Lines changed: 1507 additions & 6 deletions
File tree
- tests/fixtures/sanitizer_regression
- round2_escapees
- whisperjav
- data/hallucination_filters
- modules
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
0 commit comments