新增 CONTRIBUTING.md 貢獻指南 - #11
Merged
Merged
Conversation
新增完整的貢獻指南文檔，包含： - 如何新增詞典條目（強調使用 Tab 字元分隔） - 如何使用排序工具確保詞典正確排序 - 如何安裝 Bazel 並執行測試 - 如何撰寫測試案例（測試驅動開發流程） - 簡轉繁轉換的特殊注意事項（需測試多個配置） 使用台灣繁體中文撰寫。
frankslin
pushed a commit
that referenced
this pull request
Jul 24, 2026
Correctness hardening, decoder unification, and API cleanup from review; also extends character-level filtering to the single-dictionary fast path, which the review identified as the largest remaining win. Detailed Changes: - **Shared UTF-8 decoding (review #1-#3)**: - The scanner's 3-byte branch is now explicitly guarded (charLength == 2 || charLength == 3) instead of relying on the implicit invariant that length 1 cannot reach it. - internal::DecodeCodePoint23() is the single decoder used by both the skip-table builder and the scanner, so a key's first character always maps to the bit the scanner tests; the incorrect comment about invalid continuation bytes is replaced with the byte-consumption equivalence argument. - New invariant test: for every dictionary key, SkipUnmatchable() must return 0 on both the fast path and the table path. - **Utf8SkipTable invariants (review #4, #5)**: - Character-level mode is now derived from bmpCandidates being non-empty; the separate charLevel flag is gone. - IDS operator marking iterates UTF8Util::kFirst/kLast IdeographicDescriptionOperator and consults IdeographicDescriptionOperatorArity(), replacing the hard-coded 0xE2 and 0x2FF0..0x2FFF literals; the constants live next to the arity switch with a sync note. - **API and ABI (review #6-#8)**: - PrefixMatch.hpp no longer includes Utf8SkipScan.hpp (and thus no SIMD intrinsic headers); the skip table lives in the opaque Tables pimpl for both paths and sizeof(PrefixMatch) is back to its previous value. Note PrefixMatch.hpp and Utf8SkipScan.hpp are in LIBOPENCC_PRIVATE_HEADERS and are not installed. - dynamic_cast<MarisaDict> is replaced by a virtual Dict::EnumerateKeys(cb) with a GetLexicon()-walking default; MarisaDict overrides it with a trie walk (using the materialized lexicon when one already exists) and DictGroup recurses into children, restoring group handling on the fast path. prefix_match_lib no longer depends on marisa_dict_lib. OPENCC_ABI_VERSION bumped 1.4 -> 1.5 for the new virtual. - **Fast-path character-level filtering and caching (review #9, #10)**: - Both paths now build the same character-level skip table via EnumerateKeys, and fast-path tables go through the existing Tables cache (with a distinct key prefix). Measured convert-phase speedups vs the byte-level baseline on the 1.9MB zuozhuan corpus: t2tw 5.3x, tw2sp 2.9x, t2s 1.7x, s2twp 1.3x, s2tw 1.2x, s2t unchanged; load times unchanged within a few ms. - **Scan details (review #11, #12)**: - SWAR fallback resolves the mismatch byte with ctz on little-endian instead of rescanning; 32-bit ARM NEON now takes the vector path. - **Tests (review section 4)**: - Differential fuzz test: 300 random inputs mixing dictionary keys, CJK, IDS operators, and raw invalid/truncated bytes, comparing Conversion::Convert against a per-character reference loop (same output or same exception). - 4-byte character coverage (lead-byte filtering with and without 4-byte keys), enumeration-failure fallback that clears a partially built bitmap mid-walk, and unsigned literals in skip-scan comparisons. Benchmarks are reproducible with src/tools/SpeedBenchmark.cpp or the CLI's --measured_result flag; corpora were test/benchmark/zuozhuan.txt and its s2t-converted Traditional variant, plus test/golden/input/ us_constitution_zhs.txt repeated 100x.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
新增完整的貢獻指南文檔，包含：
使用台灣繁體中文撰寫。