Skip to content

Commit e459468

Browse files
meizhong986claude
andcommitted
fix(sanitizer): round-2 hardening — emoji drop, normalized match, escapees
Adds two new stages to the Japanese hallucination remover pipeline plus a defensive empty-drop gate, and extends the bundled filter list with 89 user-reported escapees. Targets classes identified in v1.8.11 post-release testing: outro phrases with varied punctuation, channel watermarks, SDH residue, paralinguistic noise, and micro-utterances. Pipeline additions (whisperjav/modules/hallucination_remover.py): - Stage 1.1.a.5 — Emoji-contains drop gate. Drops the whole sub if it contains any char in the broad emoji ranges (U+2600-26FF, U+2702-27B0, U+1F300-1FAFF). Runs right after bracket-context detection so emoji lines skip exact-match, regex, repetition, and CPS stages. Covers CAT2 channel watermarks ("🐯 Sound Hodori..."), CAT1 emoji-tailed outros, and any sub sprinkled with music notes or pictograms. - Stage 1.1.d — Full-normalized exact match. Strips all whitespace, quotes, and punctuation (ASCII + fullwidth + JP) anywhere in the text and case-folds Latin, then matches against a pre-built per-language normalized set. Catches escapees that differ from list entries only by punctuation, ellipsis, internal whitespace, or case ("by H." vs "BY H.", "はぁ……" vs list "はぁ…", "はぁっ はぁっ" vs "はぁっはぁっ"). Preserves the prolonged-sound mark (ー) and combining voicing mark (゛) — these are semantic, not punctuation. Normalized set is built once at load time in _build_normalized_exact_lists, O(1) lookup per sub. Pipeline addition (whisperjav/modules/subtitle_sanitizer.py): - Stage 2.5 — Defensive empty-text drop gate before renumbering. Phase 1 already drops empties and the current timing_adjuster cannot produce empty text; this guards against any future phase-2 stage (e.g. CAT4 soft gate planned for v1.8.12) that might drop content. Data additions (filter_list_v08.json): - Appends 89 strings to japanese[] from user CAT1/CAT2-non-emoji/ CAT3/CAT5/CAT6/CAT7/CAT11 observations. Skips 9 already present and 4 CAT2 emoji entries (caught by the new emoji gate). Regression: - 91 existing sanitizer tests still pass unchanged (test_hallucination_remover_slice_syntax, _symbol_purge, _cps_newline, _regexp_v09_additions). - New tests/fixtures/sanitizer_regression/round2_escapees/ contains a 110-sub fixture (102 escapees + 8 control dialogue lines), expected_map.json, and verify_round2.py which forces bundled data via config URL overrides for offline-deterministic runs. Result: 102/102 escapees dropped, 8/8 controls preserved. Gist sync: filter_list gist (4882bdb3f4f5aa4034a112cebd2e0845) is updated manually by the user post-commit; round2_escapees/gist_paste_snippet.json contains the paste-ready content. Bundled file in repo is authoritative and works offline. Out of scope for v1.8.11 (deferred): CAT4 contextual soft-rule drop (needs timing plumbing, v1.8.12), CAT12 ambiguous dialogue (v1.8.12), CAT13 mis-transcription (different problem class), CAT8/9 comma-separated syllable repetition (existing repetition_cleaner already covers patterns that escape here). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent e4f9b52 commit e459468

8 files changed

Lines changed: 1507 additions & 6 deletions

File tree

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,5 @@
11
# Diff script working output — not committed (regenerated each run)
22
current_*/
3+
4+
# Round-2 verifier working output — regenerated each run
5+
round2_escapees/_run_output/

0 commit comments

Comments
 (0)