Skip to content

Add Chinese descriptions and examples to config files - #4

Closed
frankslin wants to merge 31 commits into
opencc-wasm-0.6.3from
claude/add-opencc-config-docs-3GMxx
Closed

Add Chinese descriptions and examples to config files#4
frankslin wants to merge 31 commits into
opencc-wasm-0.6.3from
claude/add-opencc-config-docs-3GMxx

Conversation

@frankslin

Copy link
Copy Markdown
Owner
  • Add description_zh field with Traditional Chinese descriptions to all 14 config files
  • Add examples field with typical conversion examples for each config
  • Add config-examples.test.js to validate conversion examples
  • Examples demonstrate differences between variants (e.g., s2tw vs s2twp)

Example conversions:

  • s2t: "鼠标" → "鼠標" (standard Traditional)
  • s2tw: "鼠标" → "滑鼠" (Taiwan standard)
  • s2twp: "数据库" → "資料庫" (Taiwan with phrases)

Also: clarify Traditional Chinese as OpenCC Standard in 2t and t2 configs

Update all 2t and t2 config files, except jp2t and t2jp, to specify 'OpenCC標準繁體' (OpenCC Standard Traditional Chinese) instead of just '繁體' (Traditional Chinese). This clarifies that it represents an intermediate standard format, not a final user-facing variant.

Modified configs:

  • s2t.json: 簡體到繁體 → 簡體到OpenCC標準繁體
  • t2s.json: 繁體到簡體 → OpenCC標準繁體到簡體
  • hk2t.json: 香港繁體到繁體 → 香港繁體到OpenCC標準繁體
  • t2hk.json: 繁體到香港繁體 → OpenCC標準繁體到香港繁體
  • t2tw.json: 繁體到台灣正體 → OpenCC標準繁體到台灣正體
  • tw2t.json: 台灣正體到繁體 → 台灣正體到OpenCC標準繁體

Note: The 't' in jp2t/t2jp refers to Japanese Kyūjitai (舊字體), not Chinese OpenCC Standard Traditional, so these should not be labeled as 'OpenCC標準繁體'.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enhances the OpenCC configuration files by adding Traditional Chinese descriptions and conversion examples to all 14 config files. It also clarifies terminology by specifying "OpenCC Standard Traditional Chinese" for intermediate 2t and t2 conversions (except Japanese variants).

Key Changes

  • Adds description_zh field with Traditional Chinese descriptions to all config files
  • Adds examples field with input/output conversion examples demonstrating variant differences
  • Adds test file config-examples.test.js to validate conversion examples
  • Updates config names to clarify "OpenCC Standard Traditional Chinese" terminology

Reviewed changes

Copilot reviewed 18 out of 35 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
wasm-lib/test/config-examples.test.js New test file to validate conversion examples from config files
wasm-lib/test/check-conversions.js Helper script for manually checking conversions (development tool)
wasm-lib/scripts/find-variant-examples.js Helper script for finding character variants (development tool)
wasm-lib/package.json Updates test script to include new config-examples test
wasm-lib/data/config/*.json (14 files) Adds description_zh and examples fields to all config files
wasm-lib/dist/data/config/*.json (14 files) Mirrored changes to dist config files
wasm-lib/dist/esm/index.js Updates module path resolution logic
wasm-lib/dist/esm/opencc-wasm.js Minified WASM glue code with updated file reference
wasm-lib/dist/cjs/index.cjs Updates CommonJS module path resolution

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +16 to +20
function getConverter(config) {
if (!converterCache.has(config)) {
converterCache.set(config, OpenCC.Converter({ config }));
}
return converterCache.get(config);

Copilot AI Jan 1, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The converter is being cached and reused across multiple async tests. Since OpenCC.Converter() returns a promise (as indicated by await convert(example.input) on line 37), there's a potential race condition where the converter might not be properly initialized before being used in a test.

The cache should store the result of awaiting the converter promise, not the promise itself. Consider changing line 18 to:
converterCache.set(config, await OpenCC.Converter({ config }));

And make the function async:
async function getConverter(config) { ... }

Copilot uses AI. Check for mistakes.
// Test each example
config.examples.forEach((example, idx) => {
test(`[${configFile}] ${config.description_zh || config.name} - example #${idx + 1}: "${example.input}"`, async () => {
const convert = getConverter(configFile);

Copilot AI Jan 1, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test is created at the top level (outside of a describe block) by iterating over config files synchronously, but the test callback is async. The getConverter() call should be awaited since it needs to be async to properly await the converter initialization.

Change line 36 to:
const convert = await getConverter(configFile);

Copilot uses AI. Check for mistakes.
Comment thread wasm-lib/data/config/tw2s.json
Comment thread wasm-lib/data/config/tw2s.json
frankslin and others added 9 commits January 3, 2026 07:27
* Add WASM demo scaffold and project notes
* Add OpenCC WASM demo with converter UI and test runner
  - 补充 WASM 编译结果在前端 JS 中的用法
* Polish WASM demo UI and paths, run tests, and streamline converter export
* Add wasm-based OpenCC package and update demo to consume it
* Add wasm-based OpenCC package, static demo bundle, and benchmarking page
* Add copyright notice and LICENSE
…eparation

This commit enhances the opencc-wasm library with TypeScript support and
implements a cleaner build architecture with semantic separation between
intermediate build artifacts and publishable distribution.

TypeScript Support:
- Add comprehensive type definitions (index.d.ts) with full JSDoc documentation
- Define interfaces: ConverterOptions, ConverterFunction, OpenCCNamespace, etc.
- Provide complete type safety for better IDE support and developer experience

Build Architecture Redesign (semantic separation):
- build/ - Intermediate WASM artifacts (gitignored, for tests/development)
  * build/opencc-wasm.esm.js - ESM WASM glue
  * build/opencc-wasm.cjs - CJS WASM glue
  * build/opencc-wasm.wasm - WASM binary
- dist/ - Publishable distribution (committed, for npm)
  * dist/esm/ - ESM package entry
  * dist/cjs/ - CJS package entry
  * dist/data/ - OpenCC config and dictionary files

Invariants and Semantics:
- Tests import source (index.js) → loads from build/
- Published package exports dist/ only
- build/ = internal intermediate artifacts
- dist/ = publishable artifacts
- Clear separation ensures tests validate actual build output

Enhanced .gitignore:
- Add build/ to gitignore (intermediate artifacts)
- Add node_modules/, logs, OS-specific files (.DS_Store, Thumbs.db)
- Exclude editor configurations (.vscode/, .idea/)
- Add cache and temporary file exclusions

Two-Stage Build Process:
Stage 1 (build.sh):
  - Compiles C++ to WASM using Emscripten
  - Outputs to build/ directory

Stage 2 (build-api.js):
  - Copies WASM artifacts from build/ to dist/
  - Transforms source paths for production
  - Generates API wrappers for ESM and CJS
  - Copies data files

Package Configuration (package.json):
- Add "types" field pointing to index.d.ts
- Update "main" and "module" to point to API wrappers in dist/
- Add comprehensive "exports" map:
  * "." - Main API (ESM/CJS wrappers)
  * "./wasm" - Direct access to WASM glue for advanced users
  * "./dist/*" - Wildcard for flexible file access
- Include LICENSE and NOTICE in published files

Documentation:
- Add comprehensive README section explaining build architecture
- Document project structure with invariants
- Explain semantic separation between build/ and dist/

Benefits:
- Better TypeScript integration and IDE autocomplete
- Cleaner, more maintainable directory structure
- Tests validate actual build output, not stale dist files
- Clear semantic separation between internal and publishable artifacts
- Professional project setup following modern npm best practices
- Long-term maintainability through clear invariants
## Summary
- add a `//data/config:config_dict_validation_test` to test dictionaries and configs against a `testcases.json` file
- switch all CLI/Python/Node tests to consume `testcases.json` as the single source of truth; drop `.in/.ans` dependencies and adjust Bazel/CMake wiring
- streamline dictionary build outputs (no standalone `TWPhrases{IT,Name,Other}.ocd2`) and align DictionaryTest with the actual generated dict set
- add maintenance helpers (refresh_assets.sh cleanup and fix, rapidjson dep/path for CLI test) and keep wasm assets in sync via `testcases.json`

## Testing
- bazel test //data/dictionary:dictionary_test
- bazel test //test:command_line_converter_test
- bazel test //python/tests:test_opencc
- node/test.js (sync/async/promise) using updated testcases.json
----

* feature: add a new ConfigDictValidationTest.cpp to be executed in bazel
* Changeover to JSON-based testcases and clean dictionary outputs
  - Switch all tests (C++ CLI, Python, Node) to consume `testcases.json` and drop `.in`/`.ans` dependencies; keep filegroup for the JSON.
  - Prune TWPhrases sub-dictionary artifacts and align DictionaryTest to current generated dict set.
  - Add rapidjson dep/path for CLI test, refresh_assets script fixes, and keep Bazel Python toolchain note.
* Normalize CommandLineConvertTest for CRLF comparisons on Windows
* Address review feedback for tests and Bazel-only validation
  - Rename and guard streams in CommandLineConvertTest; ensure input file opens and normalize CRLF.
  - Fix node test promise handling to propagate errors correctly.
  - Mark ConfigDictValidationTest as Bazel-only to skip CMake builds.
…cases.json (#10)

- add refresh_assets.sh to rebuild/copy only config-referenced .ocd2 files and testcases.json
- convert wasm-lib tests to consume the new `{cases:[...]}` JSON format
- update bundled .ocd2 dictionaries and testcases.json fixtures

----

* wasm-lib: refresh assets script and switch tests to consolidated testcases.json
  - add refresh_assets.sh to rebuild/copy only config-referenced .ocd2 files and testcases.json
  - convert wasm-lib tests to consume the new `{cases:[...]}` JSON format
  - update bundled .ocd2 dictionaries and testcases.json fixtures
* Rebuild the wasm-lib and update the documentations
新增完整的貢獻指南文檔,包含:
- 如何新增詞典條目(強調使用 Tab 字元分隔)
- 如何使用排序工具確保詞典正確排序
- 如何安裝 Bazel 並執行測試
- 如何撰寫測試案例(測試驅動開發流程)
- 簡轉繁轉換的特殊注意事項(需測試多個配置)

使用台灣繁體中文撰寫。
@frankslin
frankslin force-pushed the claude/add-opencc-config-docs-3GMxx branch from 8291610 to 1ff1888 Compare January 3, 2026 13:37
claude and others added 2 commits January 3, 2026 05:56
1. 新增演算法與理論局限性分析文件
   - 詳細說明最大正向匹配分詞演算法
   - 分析轉換鏈機制與詞典系統
   - 探討理論局限性(一對多歧義、缺乏上下文理解、維護負擔)
   - 與現代方法(統計模型、神經網路)的比較

2. 更新 AGENTS.md
   - 新增「延伸閱讀」章節
   - 連結到技術文件和貢獻指南

3. 新增 Claude Code 配置
   - .claude/hooks/session_start.sh - 會話啟動時顯示專案資訊
   - .claude/skills/opencc-dict-edit.md - 詞典編輯技能
   - .claude/skills/opencc-algorithm-explain.md - 演算法解釋技能

這些配置幫助 AI 代理更好地理解 OpenCC 專案架構與開發流程。
@frankslin
frankslin force-pushed the claude/add-opencc-config-docs-3GMxx branch 2 times, most recently from a6d37f8 to 7d1f7ed Compare January 3, 2026 22:57
claude and others added 5 commits January 3, 2026 20:43
🚨 BREAKING CHANGE: New distribution layout

The .wasm files have been moved to be co-located with their corresponding
glue code files, fixing loading issues and enabling proper CDN usage.

New layout:
  dist/
    esm/
      opencc-wasm.js
      opencc-wasm.wasm      ← Now here (same directory)
    cjs/
      opencc-wasm.cjs
      opencc-wasm.wasm      ← Now here (same directory)
    opencc-wasm.wasm        ← Kept for legacy compatibility

Features:
- ✅ CDN support: Can now import directly from jsDelivr/unpkg
- ✅ Fixed WASM loading in various bundlers and environments
- ✅ Comprehensive test suite with CDN usage tests
- ✅ Complete documentation (CDN_USAGE.md, TESTING.md, CHANGELOG.md)

Test suite:
- npm test         → Run all tests (core + CDN)
- npm run test:core → Run 56 core functionality tests
- npm run test:cdn  → Run CDN usage tests

All 56 core tests + CDN tests pass successfully.

Usage example:
```js
import OpenCC from "https://cdn.jsdelivr.net/npm/opencc-wasm@0.3.0/dist/esm/index.js";
const converter = OpenCC.Converter({ from: "cn", to: "t" });
const result = await converter("简体中文");
```

Co-authored-by: Claude <claude@anthropic.com>
- 在頭部新增「專案說明」章節,說明本項目為 BYVoid/OpenCC 的 fork
- 闡述兩個主要目的:WASM 實現與詞表擴充
- 新增「背景」小節,說明現有第三方實作的維護狀況與本專案定位
- 原有 README 內容完整保留在分隔線下方作為參考
This commit adds significant improvements to opencc-wasm:

**API Enhancements:**
- Add `config` parameter to Converter() as intuitive alternative to `from`/`to`
- Support direct OpenCC config file names (e.g., `{ config: "s2twp" }`)
- Expand CONFIG_MAP to support all conversion types and aliases
- Maintain backward compatibility with `from`/`to` parameters

**Documentation Improvements:**
- Consolidate all API documentation into comprehensive README.md
- Add Traditional Chinese README (README.zh-TW.md) with Taiwan localization
- Emphasize "zero configuration" and "3-line start" features
- Include practical examples for React, Vue, Node.js, and Web Workers
- Add best practices and FAQ sections
- Create interactive demo (test/demo-out-of-box.html)

**User Experience:**
- Clarify auto-loading of configs and dictionaries from CDN
- Show both API methods side-by-side for user choice
- Provide TypeScript usage examples

All 56 core tests + new config parameter tests passing.
- Add '方程式' to TWPhrasesOther.txt to prevent '程式' -> '程序' misconversion
- Add regression test case in testcases.json

Ref: BYVoid#714
@frankslin
frankslin force-pushed the master branch 12 times, most recently from b6d5b84 to 5333e5c Compare January 16, 2026 16:13
@frankslin
frankslin force-pushed the master branch 4 times, most recently from 58b2124 to 683fb4e Compare January 28, 2026 15:16
@frankslin
frankslin force-pushed the master branch 6 times, most recently from 6384603 to a6c6bb5 Compare March 15, 2026 18:31
@frankslin
frankslin force-pushed the master branch 3 times, most recently from ddc006b to c75b5e6 Compare March 18, 2026 19:15
@frankslin
frankslin force-pushed the master branch 2 times, most recently from e192243 to 0cfb89d Compare March 31, 2026 05:41
@frankslin frankslin closed this Jun 27, 2026
@frankslin
frankslin deleted the claude/add-opencc-config-docs-3GMxx branch June 28, 2026 14:05
frankslin pushed a commit that referenced this pull request Jul 24, 2026
Correctness hardening, decoder unification, and API cleanup from review;
also extends character-level filtering to the single-dictionary fast path,
which the review identified as the largest remaining win.

Detailed Changes:
- **Shared UTF-8 decoding (review #1-#3)**:
  - The scanner's 3-byte branch is now explicitly guarded (charLength == 2
    || charLength == 3) instead of relying on the implicit invariant that
    length 1 cannot reach it.
  - internal::DecodeCodePoint23() is the single decoder used by both the
    skip-table builder and the scanner, so a key's first character always
    maps to the bit the scanner tests; the incorrect comment about invalid
    continuation bytes is replaced with the byte-consumption equivalence
    argument.
  - New invariant test: for every dictionary key, SkipUnmatchable() must
    return 0 on both the fast path and the table path.
- **Utf8SkipTable invariants (review #4, #5)**:
  - Character-level mode is now derived from bmpCandidates being non-empty;
    the separate charLevel flag is gone.
  - IDS operator marking iterates UTF8Util::kFirst/kLast
    IdeographicDescriptionOperator and consults
    IdeographicDescriptionOperatorArity(), replacing the hard-coded 0xE2
    and 0x2FF0..0x2FFF literals; the constants live next to the arity
    switch with a sync note.
- **API and ABI (review #6-#8)**:
  - PrefixMatch.hpp no longer includes Utf8SkipScan.hpp (and thus no SIMD
    intrinsic headers); the skip table lives in the opaque Tables pimpl for
    both paths and sizeof(PrefixMatch) is back to its previous value. Note
    PrefixMatch.hpp and Utf8SkipScan.hpp are in LIBOPENCC_PRIVATE_HEADERS
    and are not installed.
  - dynamic_cast<MarisaDict> is replaced by a virtual
    Dict::EnumerateKeys(cb) with a GetLexicon()-walking default; MarisaDict
    overrides it with a trie walk (using the materialized lexicon when one
    already exists) and DictGroup recurses into children, restoring group
    handling on the fast path. prefix_match_lib no longer depends on
    marisa_dict_lib. OPENCC_ABI_VERSION bumped 1.4 -> 1.5 for the new
    virtual.
- **Fast-path character-level filtering and caching (review #9, #10)**:
  - Both paths now build the same character-level skip table via
    EnumerateKeys, and fast-path tables go through the existing Tables
    cache (with a distinct key prefix). Measured convert-phase speedups vs
    the byte-level baseline on the 1.9MB zuozhuan corpus: t2tw 5.3x,
    tw2sp 2.9x, t2s 1.7x, s2twp 1.3x, s2tw 1.2x, s2t unchanged; load
    times unchanged within a few ms.
- **Scan details (review #11, #12)**:
  - SWAR fallback resolves the mismatch byte with ctz on little-endian
    instead of rescanning; 32-bit ARM NEON now takes the vector path.
- **Tests (review section 4)**:
  - Differential fuzz test: 300 random inputs mixing dictionary keys, CJK,
    IDS operators, and raw invalid/truncated bytes, comparing
    Conversion::Convert against a per-character reference loop (same
    output or same exception).
  - 4-byte character coverage (lead-byte filtering with and without 4-byte
    keys), enumeration-failure fallback that clears a partially built
    bitmap mid-walk, and unsigned literals in skip-scan comparisons.

Benchmarks are reproducible with src/tools/SpeedBenchmark.cpp or the CLI's
--measured_result flag; corpora were test/benchmark/zuozhuan.txt and its
s2t-converted Traditional variant, plus test/golden/input/
us_constitution_zhs.txt repeated 100x.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants