wasm-lib: refresh assets script and switch tests to consolidated testcases.json by frankslin · Pull Request #10 · frankslin/OpenCC

frankslin · 2026-01-03T03:39:34Z

add refresh_assets.sh to rebuild/copy only config-referenced .ocd2 files and testcases.json
convert wasm-lib tests to consume the new {cases:[...]} JSON format
update bundled .ocd2 dictionaries and testcases.json fixtures

…cases.json - add refresh_assets.sh to rebuild/copy only config-referenced .ocd2 files and testcases.json - convert wasm-lib tests to consume the new `{cases:[...]}` JSON format - update bundled .ocd2 dictionaries and testcases.json fixtures

…cases.json (#10) - add refresh_assets.sh to rebuild/copy only config-referenced .ocd2 files and testcases.json - convert wasm-lib tests to consume the new `{cases:[...]}` JSON format - update bundled .ocd2 dictionaries and testcases.json fixtures ---- * wasm-lib: refresh assets script and switch tests to consolidated testcases.json - add refresh_assets.sh to rebuild/copy only config-referenced .ocd2 files and testcases.json - convert wasm-lib tests to consume the new `{cases:[...]}` JSON format - update bundled .ocd2 dictionaries and testcases.json fixtures * Rebuild the wasm-lib and update the documentations

Correctness hardening, decoder unification, and API cleanup from review; also extends character-level filtering to the single-dictionary fast path, which the review identified as the largest remaining win. Detailed Changes: - **Shared UTF-8 decoding (review #1-#3)**: - The scanner's 3-byte branch is now explicitly guarded (charLength == 2 || charLength == 3) instead of relying on the implicit invariant that length 1 cannot reach it. - internal::DecodeCodePoint23() is the single decoder used by both the skip-table builder and the scanner, so a key's first character always maps to the bit the scanner tests; the incorrect comment about invalid continuation bytes is replaced with the byte-consumption equivalence argument. - New invariant test: for every dictionary key, SkipUnmatchable() must return 0 on both the fast path and the table path. - **Utf8SkipTable invariants (review #4, #5)**: - Character-level mode is now derived from bmpCandidates being non-empty; the separate charLevel flag is gone. - IDS operator marking iterates UTF8Util::kFirst/kLast IdeographicDescriptionOperator and consults IdeographicDescriptionOperatorArity(), replacing the hard-coded 0xE2 and 0x2FF0..0x2FFF literals; the constants live next to the arity switch with a sync note. - **API and ABI (review #6-#8)**: - PrefixMatch.hpp no longer includes Utf8SkipScan.hpp (and thus no SIMD intrinsic headers); the skip table lives in the opaque Tables pimpl for both paths and sizeof(PrefixMatch) is back to its previous value. Note PrefixMatch.hpp and Utf8SkipScan.hpp are in LIBOPENCC_PRIVATE_HEADERS and are not installed. - dynamic_cast<MarisaDict> is replaced by a virtual Dict::EnumerateKeys(cb) with a GetLexicon()-walking default; MarisaDict overrides it with a trie walk (using the materialized lexicon when one already exists) and DictGroup recurses into children, restoring group handling on the fast path. prefix_match_lib no longer depends on marisa_dict_lib. OPENCC_ABI_VERSION bumped 1.4 -> 1.5 for the new virtual. - **Fast-path character-level filtering and caching (review #9, #10)**: - Both paths now build the same character-level skip table via EnumerateKeys, and fast-path tables go through the existing Tables cache (with a distinct key prefix). Measured convert-phase speedups vs the byte-level baseline on the 1.9MB zuozhuan corpus: t2tw 5.3x, tw2sp 2.9x, t2s 1.7x, s2twp 1.3x, s2tw 1.2x, s2t unchanged; load times unchanged within a few ms. - **Scan details (review #11, #12)**: - SWAR fallback resolves the mismatch byte with ctz on little-endian instead of rescanning; 32-bit ARM NEON now takes the vector path. - **Tests (review section 4)**: - Differential fuzz test: 300 random inputs mixing dictionary keys, CJK, IDS operators, and raw invalid/truncated bytes, comparing Conversion::Convert against a per-character reference loop (same output or same exception). - 4-byte character coverage (lead-byte filtering with and without 4-byte keys), enumeration-failure fallback that clears a partially built bitmap mid-walk, and unsigned literals in skip-scan comparisons. Benchmarks are reproducible with src/tools/SpeedBenchmark.cpp or the CLI's --measured_result flag; corpora were test/benchmark/zuozhuan.txt and its s2t-converted Traditional variant, plus test/golden/input/ us_constitution_zhs.txt repeated 100x.

…cases.json (#10) - add refresh_assets.sh to rebuild/copy only config-referenced .ocd2 files and testcases.json - convert wasm-lib tests to consume the new `{cases:[...]}` JSON format - update bundled .ocd2 dictionaries and testcases.json fixtures ---- * wasm-lib: refresh assets script and switch tests to consolidated testcases.json - add refresh_assets.sh to rebuild/copy only config-referenced .ocd2 files and testcases.json - convert wasm-lib tests to consume the new `{cases:[...]}` JSON format - update bundled .ocd2 dictionaries and testcases.json fixtures * Rebuild the wasm-lib and update the documentations

frankslin added 2 commits January 2, 2026 19:15

Rebuild the wasm-lib and update the documentations

e0dc437

frankslin force-pushed the wasm-lib branch from b10b093 to e0dc437 Compare January 3, 2026 03:54

frankslin merged commit 3fd32e7 into master Jan 3, 2026
23 checks passed

Navigation Menu

wasm-lib: refresh assets script and switch tests to consolidated testcases.json - #10

wasm-lib: refresh assets script and switch tests to consolidated testcases.json#10
frankslin merged 2 commits into
masterfrom
wasm-lib

frankslin commented Jan 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

frankslin commented Jan 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant