Strip Japanese voiced-kana marks in BasicTokenizer (#352, Bug 2) (#354)

* Strip Japanese voiced-kana marks in BasicTokenizer (#352, Bug 2)

`BasicTokenizer.maybeStripAccents` used `.folding(options: .diacriticInsensitive,
locale: nil)`, which strips Latin diacritics but not the U+3099/U+309A combining
sound marks that Japanese voiced kana carry. Precomposed `ザ` (U+30B6) and `で`
(U+3067) therefore reached WordPiece intact, but BERT-family vocabularies only
contain the dakuten-stripped forms (`##サ`, `##て`) — `##ザ`/`##で` are missing.
A single missing continuation forced WordPiece's greedy match to fall back to
`[UNK]` for the entire word, so long Japanese inputs containing any voiced kana
returned essentially nothing useful.

Switch to NFD-decompose-then-Mn-filter, matching HF Python's `_run_strip_accents`.
This strips every nonspacing-mark scalar regardless of block, so Latin diacritics
and Japanese dakuten/handakuten are handled identically.

Adds a regression test using BAAI/bge-small-en-v1.5 over a multi-script Japanese
input that exercises both hiragana and katakana including voiced kana.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Address review: widen BertNormalizer.stripAccents to all Mn; add direct-tokenizer test

Per @pcuenca's review on #354:

- Widen `BertNormalizer.stripAccents` in Normalizer.swift from filtering only
  U+0300..U+036F to filtering all nonspacing marks (Unicode general category Mn).
  This is the architecturally correct location, matching the Rust HF tokenizers
  `normalizers/bert.rs#strip_accents` path used by the fast Python tokenizer.
  Keeps `BasicTokenizer.maybeStripAccents` as defensive coverage for callers
  using `BertTokenizer` directly without `AutoTokenizer.from(pretrained:)`.

- Add `bertTokenizerStripsDakuten` to BertTokenizerTests exercising the direct
  path (no `from(pretrained:)`), per @pcuenca's suggested test.

- Add a single-character `#expect(tokenizer.encode(text: "ザ") == [101, 1705, 102])`
  to the existing `bertJapaneseDakuten` test for clarity, per @pcuenca's request.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Apply suggestions from code review

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* lint

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

Author: 3 people

Sources/Tokenizers/BertTokenizer.swift

@@ -240,7 +240,19 @@ final class BasicTokenizer: Sendable {
 
     func maybeStripAccents(_ text: String) -> String {
         guard doLowerCase else { return text }
-        return text.folding(options: .diacriticInsensitive, locale: nil)
+        // Equivalent of HF Python's `_run_strip_accents`: NFD-decompose then drop every
+        // nonspacing-mark scalar (Unicode general category Mn). The standard Foundation
+        // API `.folding(options: .diacriticInsensitive, locale: nil)` strips Latin
+        // diacritics but does NOT strip Japanese voiced-kana combining marks.
+        // Conceptual reference:
+        // https://github.com/huggingface/transformers/blob/542e65fae2fe9cc7ddeb816e540162bc5a8bff77/src/transformers/models/bert/tokenization_bert.py#L382
+        return String(
+            String.UnicodeScalarView(
+                text.decomposedStringWithCanonicalMapping.unicodeScalars.filter {
+                    $0.properties.generalCategory != .nonspacingMark
+                }
+            )
+        )
     }
 
     func maybeLowercase(_ text: String) -> String {

Sources/Tokenizers/Normalizer.swift

@@ -236,11 +236,18 @@ class BertNormalizer: Normalizer {
     }
 
     private func stripAccents(text: String) -> String {
-        // This might be the same as `text.folding(options: .diacriticInsensitive, locale: nil)`
+        // Equivalent of HF Python's `_run_strip_accents`: NFD-decompose then drop every
+        // nonspacing-mark scalar (Unicode general category Mn). Filtering only
+        // U+0300..U+036F handled Latin diacritics but missed marks in other blocks —
+        // notably the Japanese voiced-kana combining marks U+3099/U+309A.
+        // This matches Rust `tokenizers` and HF Python.
         String(
-            text.decomposedStringWithCanonicalMapping.unicodeScalars.filter { scalar in
-                !(scalar.value >= 0x0300 && scalar.value <= 0x036F)
-            })
+            String.UnicodeScalarView(
+                text.decomposedStringWithCanonicalMapping.unicodeScalars.filter {
+                    $0.properties.generalCategory != .nonspacingMark
+                }
+            )
+        )
     }
 }
 

Tests/TokenizersTests/BertTokenizerTests.swift

@@ -219,6 +219,26 @@ struct BertTokenizerTests {
         }
     }
 
+    /// Regression for https://github.com/huggingface/swift-transformers/issues/352 (Bug 2).
+    /// Exercises `BertTokenizer` directly (bypassing `AutoTokenizer.from(pretrained:)`)
+    /// to verify the `BasicTokenizer.maybeStripAccents` defensive fallback strips Japanese
+    /// voiced-kana dakuten even when a `BertNormalizer` pass hasn't already run. The
+    /// upstream-fix is in `BertNormalizer.stripAccents`; this test guards the direct path.
+    @Test("Direct BertTokenizer strips Japanese voiced-kana dakuten")
+    func bertTokenizerStripsDakuten() {
+        let vocab: [String: Int] = [
+            "[UNK]": 0,
+            "[CLS]": 1,
+            "[SEP]": 2,
+            "サ": 3,
+            "##サ": 4,
+        ]
+        let tokenizer = BertTokenizer(vocab: vocab, merges: nil)
+
+        #expect(tokenizer.tokenize(text: "ザ") == ["サ"])
+        #expect(tokenizer.tokenize(text: "ザザ") == ["サ", "##サ"])
+    }
+
     @Test("BERT encoder/decoder round-trip")
     func encoderDecoder() {
         let text = """

Tests/TokenizersTests/TokenizerTests.swift

@@ -376,6 +376,26 @@ struct TokenizerTests {
         #expect(tokenizer.encode(text: "안") == [101, 1463, 30006, 30021, 102])
     }
 
+    @Test
+    func bertJapaneseDakuten() async throws {
+        // Regression for https://github.com/huggingface/swift-transformers/issues/352 (Bug 2).
+        let tokenizerOpt = try await AutoTokenizer.from(pretrained: "BAAI/bge-small-en-v1.5") as? PreTrainedTokenizer
+        #expect(tokenizerOpt != nil)
+        let tokenizer = tokenizerOpt!
+
+        // Minimal repro: a single voiced-kana character.
+        #expect(tokenizer.encode(text: "ザ") == [101, 1705, 102])
+
+        // Long form covering hiragana, ideographs, punctuation, and katakana with
+        // multiple voiced kana (`ザ` and `で`).
+        #expect(
+            tokenizer.encode(text: "こんにちは、世界。トークナイザーのテストです。") == [
+                101, 1655, 30217, 30194, 30188, 30198, 1635, 1745, 100, 1636,
+                1714, 30265, 30228, 30241, 30221, 30231, 30265, 30197, 30239,
+                30233, 30240, 30191, 30184, 1636, 102,
+            ])
+    }
+
     @Test
     func robertaEncodeDecode() async throws {
         let tokenizerOpt = try await AutoTokenizer.from(pretrained: "FacebookAI/roberta-base") as? PreTrainedTokenizer