Add OpenCC Pure Python implementation - #33
Merged
frankslin merged 13 commits intoJun 28, 2026
Conversation
…p third-party library (#29) Replace the Python native extension wrapper with a self-contained pure Python implementation that mirrors the OpenCC conversion pipeline. Bundle mmseg JSON configs and text dictionaries into the Python package, expose only supported non-jieba configs, and simplify Python wheel/release workflows to build a py3-none-any package. The plugin jieba is not supported at this time. Loading config from custom config path is not supported at this time.
Remove files added by the pure Python PR (#29) commits during rebase (benchmark_opencc.py, _opencc_pure.py, test_opencc_golden.py) and restore the BUILD.bazel and test files to their opencc-wasm-develop state. The pure Python implementation now lives exclusively in python-pure/, not in the C-extension python/ package.
Introduce python-pure/ as a parallel installable package (opencc-py)
containing the pure Python OpenCC implementation, with resource zip
support for generated dictionaries and a clean Bazel build target.
Detailed Changes:
- **opencc_resources_zip**:
- Add dict_zip_name() to preserve subdirectory paths (fixes cngov/
basename collision with main STCharacters.txt).
- Add --exclude-dict-subdirs flag to skip dicts in subdirectories,
and suppress configs that reference those dicts.
- Add exclude_dict_subdirs attribute to the Starlark rule.
- Add opencc_py_resources_zip Bazel target using this flag so the
opencc-py bundle omits cngov entirely.
- **python-pure/opencc/_opencc_pure.py**:
- _get_trie: filesystem-first lookup with zip fallback for generated
files (STPhrases_GeneratedFromRegionalPhrases, TSCharactersExt).
- OpenCC.__init__: filesystem configs take priority over zip; always
pass zip_loader for dict fallback; support no-segmentation configs
(t2jp, jp2t).
- list_configs: scan jieba config dirs even in zip mode; filter out
configs referencing subdirectory dicts (_config_uses_subdir_dicts).
- _GroupMatcher: preserve group hierarchy; implement union/
short_circuit match_policy.
- _ZipLoader: new class for reading configs and dicts from zip.
- OpenCC.convert: apply normalization pre-pass before segmentation.
- **python-pure/**: package scaffold (setup.py, tests, benchmarks,
.gitignore) for the opencc-py distribution.
`_parse_dict_node` now accepts `include_tofu_risk_dicts` and returns `None` for any dict node whose `may_output_tofu` is true when the flag is false, mirroring `ConfigInternal::ParseDict` in Config.cpp. `OpenCC.__init__` gains `include_tofu_risk_dictionaries` (default `True`, matching the C++ Python binding); jieba comparison tests pass `False` explicitly since those golden values are CLI outputs, where tofu-risk dictionaries are excluded by default.
Replace source-tree and bundled-resource data discovery with the opencc-data runtime dependency so the pure Python package no longer depends on generated OpenCC artifacts. Keep the opencc-py package version and its pinned opencc-data dependency version identical for data compatibility. Detailed Changes: - **Resource Loading**: - Load built-in configs and dictionaries through opencc_data resources. - Keep custom config support for local text dictionaries next to user configs. - **Packaging and License**: - Set the opencc-py version to match the opencc-data version. - Pin opencc-data to the identical runtime dependency version. - Set package author metadata to OpenCC contributors. - Include an Apache-2.0 LICENSE file in python-pure package builds. - **Documentation and Tests**: - Add English and Traditional Chinese README files with cross-links. - Link to the official OpenCC project and the opencc-data PyPI package. - Document that this package is an OpenCC derivative work. - Update pure Python tests to use opencc-data test fixtures.
Build PKG-INFO and wheel metadata from both English and Traditional Chinese README files while stripping language-switch links and Markdown URLs. Include both README files in the source distribution manifest. Detailed Changes: - **Package Metadata**: - Compose long_description from README.md and README.zh-TW.md. - Remove README language-switch links and Markdown link targets from generated metadata. - **Source Distribution**: - Add MANIFEST.in so README.md, README.zh-TW.md, and LICENSE are explicitly included.
Update the English and Traditional Chinese pure Python READMEs to name the published package explicitly and show the user-facing pip install command. Detailed Changes: - **Installation Docs**: - State that the PyPI package name is opencc-py. - Add python -m pip install opencc-py to both README files. - Keep local and editable install instructions for development.
Link the pure Python README comparison section to the official opencc PyPI package and state that it is the official Python implementation. Detailed Changes: - **Documentation**: - Add the official opencc PyPI package link to the English README. - Add the official opencc PyPI package link to the Traditional Chinese README. - Clarify that opencc, not opencc-py, is the official Python implementation.
Add a manual workflow for validating and publishing the opencc-py pure Python package. The workflow defaults to dry-run validation and publishes through PyPI Trusted Publishing only when dry_run is disabled. Detailed Changes: - **Release Workflow**: - Install pure Python build and test dependencies. - Run the python-pure pytest suite before packaging. - Build sdist and wheel artifacts from python-pure. - Run twine check and upload built distributions as GitHub artifacts. - Publish to PyPI via pypa/gh-action-pypi-publish when dry_run is false.
frankslin
added a commit
that referenced
this pull request
Jun 28, 2026
* Make `opencc` Python package pure Python: rewrite core algorithm, drop third-party library (#29) Replace the Python native extension wrapper with a self-contained pure Python implementation that mirrors the OpenCC conversion pipeline. Bundle mmseg JSON configs and text dictionaries into the Python package, expose only supported non-jieba configs, and simplify Python wheel/release workflows to build a py3-none-any package. The plugin jieba is not supported at this time. Loading config from custom config path is not supported at this time. * Add Python golden tests and benchmarks, and update README.md * Fix Python dictionary path parsing * Add optional Python jieba support * Preserve suffixless Python config paths * Restore python/ to match upstream/master baseline Remove files added by the pure Python PR (#29) commits during rebase (benchmark_opencc.py, _opencc_pure.py, test_opencc_golden.py) and restore the BUILD.bazel and test files to their opencc-wasm-develop state. The pure Python implementation now lives exclusively in python-pure/, not in the C-extension python/ package. * Add opencc-py pure Python package and fix resource zip build Introduce python-pure/ as a parallel installable package (opencc-py) containing the pure Python OpenCC implementation, with resource zip support for generated dictionaries and a clean Bazel build target. Detailed Changes: - **opencc_resources_zip**: - Add dict_zip_name() to preserve subdirectory paths (fixes cngov/ basename collision with main STCharacters.txt). - Add --exclude-dict-subdirs flag to skip dicts in subdirectories, and suppress configs that reference those dicts. - Add exclude_dict_subdirs attribute to the Starlark rule. - Add opencc_py_resources_zip Bazel target using this flag so the opencc-py bundle omits cngov entirely. - **python-pure/opencc/_opencc_pure.py**: - _get_trie: filesystem-first lookup with zip fallback for generated files (STPhrases_GeneratedFromRegionalPhrases, TSCharactersExt). - OpenCC.__init__: filesystem configs take priority over zip; always pass zip_loader for dict fallback; support no-segmentation configs (t2jp, jp2t). - list_configs: scan jieba config dirs even in zip mode; filter out configs referencing subdirectory dicts (_config_uses_subdir_dicts). - _GroupMatcher: preserve group hierarchy; implement union/ short_circuit match_policy. - _ZipLoader: new class for reading configs and dicts from zip. - OpenCC.convert: apply normalization pre-pass before segmentation. - **python-pure/**: package scaffold (setup.py, tests, benchmarks, .gitignore) for the opencc-py distribution. * Honour may_output_tofu flag in pure Python converter `_parse_dict_node` now accepts `include_tofu_risk_dicts` and returns `None` for any dict node whose `may_output_tofu` is true when the flag is false, mirroring `ConfigInternal::ParseDict` in Config.cpp. `OpenCC.__init__` gains `include_tofu_risk_dictionaries` (default `True`, matching the C++ Python binding); jieba comparison tests pass `False` explicitly since those golden values are CLI outputs, where tofu-risk dictionaries are excluded by default. * Load pure Python resources from opencc-data Replace source-tree and bundled-resource data discovery with the opencc-data runtime dependency so the pure Python package no longer depends on generated OpenCC artifacts. Keep the opencc-py package version and its pinned opencc-data dependency version identical for data compatibility. Detailed Changes: - **Resource Loading**: - Load built-in configs and dictionaries through opencc_data resources. - Keep custom config support for local text dictionaries next to user configs. - **Packaging and License**: - Set the opencc-py version to match the opencc-data version. - Pin opencc-data to the identical runtime dependency version. - Set package author metadata to OpenCC contributors. - Include an Apache-2.0 LICENSE file in python-pure package builds. - **Documentation and Tests**: - Add English and Traditional Chinese README files with cross-links. - Link to the official OpenCC project and the opencc-data PyPI package. - Document that this package is an OpenCC derivative work. - Update pure Python tests to use opencc-data test fixtures. * Combine pure Python package descriptions Build PKG-INFO and wheel metadata from both English and Traditional Chinese README files while stripping language-switch links and Markdown URLs. Include both README files in the source distribution manifest. Detailed Changes: - **Package Metadata**: - Compose long_description from README.md and README.zh-TW.md. - Remove README language-switch links and Markdown link targets from generated metadata. - **Source Distribution**: - Add MANIFEST.in so README.md, README.zh-TW.md, and LICENSE are explicitly included. * Document opencc-py installation Update the English and Traditional Chinese pure Python READMEs to name the published package explicitly and show the user-facing pip install command. Detailed Changes: - **Installation Docs**: - State that the PyPI package name is opencc-py. - Add python -m pip install opencc-py to both README files. - Keep local and editable install instructions for development. * Document official Python opencc package Link the pure Python README comparison section to the official opencc PyPI package and state that it is the official Python implementation. Detailed Changes: - **Documentation**: - Add the official opencc PyPI package link to the English README. - Add the official opencc PyPI package link to the Traditional Chinese README. - Clarify that opencc, not opencc-py, is the official Python implementation. * Add pure Python PyPI release workflow Add a manual workflow for validating and publishing the opencc-py pure Python package. The workflow defaults to dry-run validation and publishes through PyPI Trusted Publishing only when dry_run is disabled. Detailed Changes: - **Release Workflow**: - Install pure Python build and test dependencies. - Run the python-pure pytest suite before packaging. - Build sdist and wheel artifacts from python-pure. - Run twine check and upload built distributions as GitHub artifacts. - Publish to PyPI via pypa/gh-action-pypi-publish when dry_run is false.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f0970f7849
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Build sdist only once, to avoid duplicate uploads from matrix jobs. | ||
| python -m build --sdist --outdir wheelhouse | ||
| fi | ||
| python -m build --sdist --wheel --outdir wheelhouse |
There was a problem hiding this comment.
在 .github/workflows/release-pypi.yml 中這個 script 會同時由 ubuntu-latest 和 ubuntu-22.04-arm 兩個 Linux matrix job 執行；舊版有 uname -m = x86_64 guard，所以 sdist 只上傳一次。這行現在讓兩個 Linux job 都產生同名 sdist，而下方 twine upload wheelhouse/* 沒有 --skip-existing，同一版本第二個 job 上傳時會被 PyPI 以檔案已存在拒絕，導致 release workflow 失敗。
Useful? React with 👍 / 👎.
Comment on lines
+567
to
+569
| def __init__(self, config: str = 't2s', | ||
| resource_zip: 'str | None' = None, | ||
| include_tofu_risk_dictionaries: bool = True) -> None: |
There was a problem hiding this comment.
原本的 Python API 是 OpenCC(config, include_tofu_risk_dictionaries=True, resource_zip=None)，但這裡把 resource_zip 放到第二個 positional 參數；因此既有呼叫如 OpenCC('t2s', False) 在純 Python package 中會把 False 當成 resource_zip，隨即因 resource_zip is not None 丟出錯誤，而不是關閉 tofu-risk 字典。這會破壞使用 positional 參數的既有使用者。
Useful? React with 👍 / 👎.
Comment on lines
+459
to
+461
| else: | ||
| parts.append(text[i]) | ||
| i += 1 |
There was a problem hiding this comment.
遇到完整 Ideographic Description Sequence 時，C++ Conversion 會在沒有字典命中時一次跳過整個 IDS，避免把描述字元的組字元件改掉；例如 ⿰钅只只 應只轉換最後一個 只。這裡 miss 時固定只前進一個 code point，所以會繼續匹配並轉換 IDS 內的 钅/只，輸出會變成類似 ⿰釒隻隻，破壞 IDS 文字（_segment 的 miss path 也有同樣問題）。
Useful? React with 👍 / 👎.
Comment on lines
+434
to
+437
| if dtype in ('ocd2', 'ocd', 'txt', 'text'): | ||
| return _get_trie(d['file'], config_dir) | ||
|
|
||
| raise ValueError(f'Unknown dict type: {dtype!r}') |
There was a problem hiding this comment.
OpenCC 的 schema、測試與 examples/config/*.json 都允許 "type": "inline" 的自訂字典，但這個 parser 只接受 file-backed ocd2/ocd/txt/text，任何 inline segmentation 或 conversion dict 都會落到 ValueError('Unknown dict type')。因此使用現有 inline 範例或依 README 建立小型自訂 config 的使用者無法在新的純 Python 實作中載入這些 config。
Useful? React with 👍 / 👎.
Merging this PR will not alter performance
Comparing 
Footnotes
|
frankslin
added a commit
that referenced
this pull request
Jun 30, 2026
* Make `opencc` Python package pure Python: rewrite core algorithm, drop third-party library (#29) Replace the Python native extension wrapper with a self-contained pure Python implementation that mirrors the OpenCC conversion pipeline. Bundle mmseg JSON configs and text dictionaries into the Python package, expose only supported non-jieba configs, and simplify Python wheel/release workflows to build a py3-none-any package. The plugin jieba is not supported at this time. Loading config from custom config path is not supported at this time. * Add Python golden tests and benchmarks, and update README.md * Fix Python dictionary path parsing * Add optional Python jieba support * Preserve suffixless Python config paths * Restore python/ to match upstream/master baseline Remove files added by the pure Python PR (#29) commits during rebase (benchmark_opencc.py, _opencc_pure.py, test_opencc_golden.py) and restore the BUILD.bazel and test files to their opencc-wasm-develop state. The pure Python implementation now lives exclusively in python-pure/, not in the C-extension python/ package. * Add opencc-py pure Python package and fix resource zip build Introduce python-pure/ as a parallel installable package (opencc-py) containing the pure Python OpenCC implementation, with resource zip support for generated dictionaries and a clean Bazel build target. Detailed Changes: - **opencc_resources_zip**: - Add dict_zip_name() to preserve subdirectory paths (fixes cngov/ basename collision with main STCharacters.txt). - Add --exclude-dict-subdirs flag to skip dicts in subdirectories, and suppress configs that reference those dicts. - Add exclude_dict_subdirs attribute to the Starlark rule. - Add opencc_py_resources_zip Bazel target using this flag so the opencc-py bundle omits cngov entirely. - **python-pure/opencc/_opencc_pure.py**: - _get_trie: filesystem-first lookup with zip fallback for generated files (STPhrases_GeneratedFromRegionalPhrases, TSCharactersExt). - OpenCC.__init__: filesystem configs take priority over zip; always pass zip_loader for dict fallback; support no-segmentation configs (t2jp, jp2t). - list_configs: scan jieba config dirs even in zip mode; filter out configs referencing subdirectory dicts (_config_uses_subdir_dicts). - _GroupMatcher: preserve group hierarchy; implement union/ short_circuit match_policy. - _ZipLoader: new class for reading configs and dicts from zip. - OpenCC.convert: apply normalization pre-pass before segmentation. - **python-pure/**: package scaffold (setup.py, tests, benchmarks, .gitignore) for the opencc-py distribution. * Honour may_output_tofu flag in pure Python converter `_parse_dict_node` now accepts `include_tofu_risk_dicts` and returns `None` for any dict node whose `may_output_tofu` is true when the flag is false, mirroring `ConfigInternal::ParseDict` in Config.cpp. `OpenCC.__init__` gains `include_tofu_risk_dictionaries` (default `True`, matching the C++ Python binding); jieba comparison tests pass `False` explicitly since those golden values are CLI outputs, where tofu-risk dictionaries are excluded by default. * Load pure Python resources from opencc-data Replace source-tree and bundled-resource data discovery with the opencc-data runtime dependency so the pure Python package no longer depends on generated OpenCC artifacts. Keep the opencc-py package version and its pinned opencc-data dependency version identical for data compatibility. Detailed Changes: - **Resource Loading**: - Load built-in configs and dictionaries through opencc_data resources. - Keep custom config support for local text dictionaries next to user configs. - **Packaging and License**: - Set the opencc-py version to match the opencc-data version. - Pin opencc-data to the identical runtime dependency version. - Set package author metadata to OpenCC contributors. - Include an Apache-2.0 LICENSE file in python-pure package builds. - **Documentation and Tests**: - Add English and Traditional Chinese README files with cross-links. - Link to the official OpenCC project and the opencc-data PyPI package. - Document that this package is an OpenCC derivative work. - Update pure Python tests to use opencc-data test fixtures. * Combine pure Python package descriptions Build PKG-INFO and wheel metadata from both English and Traditional Chinese README files while stripping language-switch links and Markdown URLs. Include both README files in the source distribution manifest. Detailed Changes: - **Package Metadata**: - Compose long_description from README.md and README.zh-TW.md. - Remove README language-switch links and Markdown link targets from generated metadata. - **Source Distribution**: - Add MANIFEST.in so README.md, README.zh-TW.md, and LICENSE are explicitly included. * Document opencc-py installation Update the English and Traditional Chinese pure Python READMEs to name the published package explicitly and show the user-facing pip install command. Detailed Changes: - **Installation Docs**: - State that the PyPI package name is opencc-py. - Add python -m pip install opencc-py to both README files. - Keep local and editable install instructions for development. * Document official Python opencc package Link the pure Python README comparison section to the official opencc PyPI package and state that it is the official Python implementation. Detailed Changes: - **Documentation**: - Add the official opencc PyPI package link to the English README. - Add the official opencc PyPI package link to the Traditional Chinese README. - Clarify that opencc, not opencc-py, is the official Python implementation. * Add pure Python PyPI release workflow Add a manual workflow for validating and publishing the opencc-py pure Python package. The workflow defaults to dry-run validation and publishes through PyPI Trusted Publishing only when dry_run is disabled. Detailed Changes: - **Release Workflow**: - Install pure Python build and test dependencies. - Run the python-pure pytest suite before packaging. - Build sdist and wheel artifacts from python-pure. - Run twine check and upload built distributions as GitHub artifacts. - Publish to PyPI via pypa/gh-action-pypi-publish when dry_run is false.
frankslin
added a commit
that referenced
this pull request
Jul 2, 2026
* Make `opencc` Python package pure Python: rewrite core algorithm, drop third-party library (#29) Replace the Python native extension wrapper with a self-contained pure Python implementation that mirrors the OpenCC conversion pipeline. Bundle mmseg JSON configs and text dictionaries into the Python package, expose only supported non-jieba configs, and simplify Python wheel/release workflows to build a py3-none-any package. The plugin jieba is not supported at this time. Loading config from custom config path is not supported at this time. * Add Python golden tests and benchmarks, and update README.md * Fix Python dictionary path parsing * Add optional Python jieba support * Preserve suffixless Python config paths * Restore python/ to match upstream/master baseline Remove files added by the pure Python PR (#29) commits during rebase (benchmark_opencc.py, _opencc_pure.py, test_opencc_golden.py) and restore the BUILD.bazel and test files to their opencc-wasm-develop state. The pure Python implementation now lives exclusively in python-pure/, not in the C-extension python/ package. * Add opencc-py pure Python package and fix resource zip build Introduce python-pure/ as a parallel installable package (opencc-py) containing the pure Python OpenCC implementation, with resource zip support for generated dictionaries and a clean Bazel build target. Detailed Changes: - **opencc_resources_zip**: - Add dict_zip_name() to preserve subdirectory paths (fixes cngov/ basename collision with main STCharacters.txt). - Add --exclude-dict-subdirs flag to skip dicts in subdirectories, and suppress configs that reference those dicts. - Add exclude_dict_subdirs attribute to the Starlark rule. - Add opencc_py_resources_zip Bazel target using this flag so the opencc-py bundle omits cngov entirely. - **python-pure/opencc/_opencc_pure.py**: - _get_trie: filesystem-first lookup with zip fallback for generated files (STPhrases_GeneratedFromRegionalPhrases, TSCharactersExt). - OpenCC.__init__: filesystem configs take priority over zip; always pass zip_loader for dict fallback; support no-segmentation configs (t2jp, jp2t). - list_configs: scan jieba config dirs even in zip mode; filter out configs referencing subdirectory dicts (_config_uses_subdir_dicts). - _GroupMatcher: preserve group hierarchy; implement union/ short_circuit match_policy. - _ZipLoader: new class for reading configs and dicts from zip. - OpenCC.convert: apply normalization pre-pass before segmentation. - **python-pure/**: package scaffold (setup.py, tests, benchmarks, .gitignore) for the opencc-py distribution. * Honour may_output_tofu flag in pure Python converter `_parse_dict_node` now accepts `include_tofu_risk_dicts` and returns `None` for any dict node whose `may_output_tofu` is true when the flag is false, mirroring `ConfigInternal::ParseDict` in Config.cpp. `OpenCC.__init__` gains `include_tofu_risk_dictionaries` (default `True`, matching the C++ Python binding); jieba comparison tests pass `False` explicitly since those golden values are CLI outputs, where tofu-risk dictionaries are excluded by default. * Load pure Python resources from opencc-data Replace source-tree and bundled-resource data discovery with the opencc-data runtime dependency so the pure Python package no longer depends on generated OpenCC artifacts. Keep the opencc-py package version and its pinned opencc-data dependency version identical for data compatibility. Detailed Changes: - **Resource Loading**: - Load built-in configs and dictionaries through opencc_data resources. - Keep custom config support for local text dictionaries next to user configs. - **Packaging and License**: - Set the opencc-py version to match the opencc-data version. - Pin opencc-data to the identical runtime dependency version. - Set package author metadata to OpenCC contributors. - Include an Apache-2.0 LICENSE file in python-pure package builds. - **Documentation and Tests**: - Add English and Traditional Chinese README files with cross-links. - Link to the official OpenCC project and the opencc-data PyPI package. - Document that this package is an OpenCC derivative work. - Update pure Python tests to use opencc-data test fixtures. * Combine pure Python package descriptions Build PKG-INFO and wheel metadata from both English and Traditional Chinese README files while stripping language-switch links and Markdown URLs. Include both README files in the source distribution manifest. Detailed Changes: - **Package Metadata**: - Compose long_description from README.md and README.zh-TW.md. - Remove README language-switch links and Markdown link targets from generated metadata. - **Source Distribution**: - Add MANIFEST.in so README.md, README.zh-TW.md, and LICENSE are explicitly included. * Document opencc-py installation Update the English and Traditional Chinese pure Python READMEs to name the published package explicitly and show the user-facing pip install command. Detailed Changes: - **Installation Docs**: - State that the PyPI package name is opencc-py. - Add python -m pip install opencc-py to both README files. - Keep local and editable install instructions for development. * Document official Python opencc package Link the pure Python README comparison section to the official opencc PyPI package and state that it is the official Python implementation. Detailed Changes: - **Documentation**: - Add the official opencc PyPI package link to the English README. - Add the official opencc PyPI package link to the Traditional Chinese README. - Clarify that opencc, not opencc-py, is the official Python implementation. * Add pure Python PyPI release workflow Add a manual workflow for validating and publishing the opencc-py pure Python package. The workflow defaults to dry-run validation and publishes through PyPI Trusted Publishing only when dry_run is disabled. Detailed Changes: - **Release Workflow**: - Install pure Python build and test dependencies. - Run the python-pure pytest suite before packaging. - Build sdist and wheel artifacts from python-pure. - Run twine check and upload built distributions as GitHub artifacts. - Publish to PyPI via pypa/gh-action-pypi-publish when dry_run is false.
frankslin
added a commit
that referenced
this pull request
Jul 4, 2026
* Make `opencc` Python package pure Python: rewrite core algorithm, drop third-party library (#29) Replace the Python native extension wrapper with a self-contained pure Python implementation that mirrors the OpenCC conversion pipeline. Bundle mmseg JSON configs and text dictionaries into the Python package, expose only supported non-jieba configs, and simplify Python wheel/release workflows to build a py3-none-any package. The plugin jieba is not supported at this time. Loading config from custom config path is not supported at this time. * Add Python golden tests and benchmarks, and update README.md * Fix Python dictionary path parsing * Add optional Python jieba support * Preserve suffixless Python config paths * Restore python/ to match upstream/master baseline Remove files added by the pure Python PR (#29) commits during rebase (benchmark_opencc.py, _opencc_pure.py, test_opencc_golden.py) and restore the BUILD.bazel and test files to their opencc-wasm-develop state. The pure Python implementation now lives exclusively in python-pure/, not in the C-extension python/ package. * Add opencc-py pure Python package and fix resource zip build Introduce python-pure/ as a parallel installable package (opencc-py) containing the pure Python OpenCC implementation, with resource zip support for generated dictionaries and a clean Bazel build target. Detailed Changes: - **opencc_resources_zip**: - Add dict_zip_name() to preserve subdirectory paths (fixes cngov/ basename collision with main STCharacters.txt). - Add --exclude-dict-subdirs flag to skip dicts in subdirectories, and suppress configs that reference those dicts. - Add exclude_dict_subdirs attribute to the Starlark rule. - Add opencc_py_resources_zip Bazel target using this flag so the opencc-py bundle omits cngov entirely. - **python-pure/opencc/_opencc_pure.py**: - _get_trie: filesystem-first lookup with zip fallback for generated files (STPhrases_GeneratedFromRegionalPhrases, TSCharactersExt). - OpenCC.__init__: filesystem configs take priority over zip; always pass zip_loader for dict fallback; support no-segmentation configs (t2jp, jp2t). - list_configs: scan jieba config dirs even in zip mode; filter out configs referencing subdirectory dicts (_config_uses_subdir_dicts). - _GroupMatcher: preserve group hierarchy; implement union/ short_circuit match_policy. - _ZipLoader: new class for reading configs and dicts from zip. - OpenCC.convert: apply normalization pre-pass before segmentation. - **python-pure/**: package scaffold (setup.py, tests, benchmarks, .gitignore) for the opencc-py distribution. * Honour may_output_tofu flag in pure Python converter `_parse_dict_node` now accepts `include_tofu_risk_dicts` and returns `None` for any dict node whose `may_output_tofu` is true when the flag is false, mirroring `ConfigInternal::ParseDict` in Config.cpp. `OpenCC.__init__` gains `include_tofu_risk_dictionaries` (default `True`, matching the C++ Python binding); jieba comparison tests pass `False` explicitly since those golden values are CLI outputs, where tofu-risk dictionaries are excluded by default. * Load pure Python resources from opencc-data Replace source-tree and bundled-resource data discovery with the opencc-data runtime dependency so the pure Python package no longer depends on generated OpenCC artifacts. Keep the opencc-py package version and its pinned opencc-data dependency version identical for data compatibility. Detailed Changes: - **Resource Loading**: - Load built-in configs and dictionaries through opencc_data resources. - Keep custom config support for local text dictionaries next to user configs. - **Packaging and License**: - Set the opencc-py version to match the opencc-data version. - Pin opencc-data to the identical runtime dependency version. - Set package author metadata to OpenCC contributors. - Include an Apache-2.0 LICENSE file in python-pure package builds. - **Documentation and Tests**: - Add English and Traditional Chinese README files with cross-links. - Link to the official OpenCC project and the opencc-data PyPI package. - Document that this package is an OpenCC derivative work. - Update pure Python tests to use opencc-data test fixtures. * Combine pure Python package descriptions Build PKG-INFO and wheel metadata from both English and Traditional Chinese README files while stripping language-switch links and Markdown URLs. Include both README files in the source distribution manifest. Detailed Changes: - **Package Metadata**: - Compose long_description from README.md and README.zh-TW.md. - Remove README language-switch links and Markdown link targets from generated metadata. - **Source Distribution**: - Add MANIFEST.in so README.md, README.zh-TW.md, and LICENSE are explicitly included. * Document opencc-py installation Update the English and Traditional Chinese pure Python READMEs to name the published package explicitly and show the user-facing pip install command. Detailed Changes: - **Installation Docs**: - State that the PyPI package name is opencc-py. - Add python -m pip install opencc-py to both README files. - Keep local and editable install instructions for development. * Document official Python opencc package Link the pure Python README comparison section to the official opencc PyPI package and state that it is the official Python implementation. Detailed Changes: - **Documentation**: - Add the official opencc PyPI package link to the English README. - Add the official opencc PyPI package link to the Traditional Chinese README. - Clarify that opencc, not opencc-py, is the official Python implementation. * Add pure Python PyPI release workflow Add a manual workflow for validating and publishing the opencc-py pure Python package. The workflow defaults to dry-run validation and publishes through PyPI Trusted Publishing only when dry_run is disabled. Detailed Changes: - **Release Workflow**: - Install pure Python build and test dependencies. - Run the python-pure pytest suite before packaging. - Build sdist and wheel artifacts from python-pure. - Run twine check and upload built distributions as GitHub artifacts. - Publish to PyPI via pypa/gh-action-pypi-publish when dry_run is false.
frankslin
added a commit
that referenced
this pull request
Jul 13, 2026
* Make `opencc` Python package pure Python: rewrite core algorithm, drop third-party library (#29) Replace the Python native extension wrapper with a self-contained pure Python implementation that mirrors the OpenCC conversion pipeline. Bundle mmseg JSON configs and text dictionaries into the Python package, expose only supported non-jieba configs, and simplify Python wheel/release workflows to build a py3-none-any package. The plugin jieba is not supported at this time. Loading config from custom config path is not supported at this time. * Add Python golden tests and benchmarks, and update README.md * Fix Python dictionary path parsing * Add optional Python jieba support * Preserve suffixless Python config paths * Restore python/ to match upstream/master baseline Remove files added by the pure Python PR (#29) commits during rebase (benchmark_opencc.py, _opencc_pure.py, test_opencc_golden.py) and restore the BUILD.bazel and test files to their opencc-wasm-develop state. The pure Python implementation now lives exclusively in python-pure/, not in the C-extension python/ package. * Add opencc-py pure Python package and fix resource zip build Introduce python-pure/ as a parallel installable package (opencc-py) containing the pure Python OpenCC implementation, with resource zip support for generated dictionaries and a clean Bazel build target. Detailed Changes: - **opencc_resources_zip**: - Add dict_zip_name() to preserve subdirectory paths (fixes cngov/ basename collision with main STCharacters.txt). - Add --exclude-dict-subdirs flag to skip dicts in subdirectories, and suppress configs that reference those dicts. - Add exclude_dict_subdirs attribute to the Starlark rule. - Add opencc_py_resources_zip Bazel target using this flag so the opencc-py bundle omits cngov entirely. - **python-pure/opencc/_opencc_pure.py**: - _get_trie: filesystem-first lookup with zip fallback for generated files (STPhrases_GeneratedFromRegionalPhrases, TSCharactersExt). - OpenCC.__init__: filesystem configs take priority over zip; always pass zip_loader for dict fallback; support no-segmentation configs (t2jp, jp2t). - list_configs: scan jieba config dirs even in zip mode; filter out configs referencing subdirectory dicts (_config_uses_subdir_dicts). - _GroupMatcher: preserve group hierarchy; implement union/ short_circuit match_policy. - _ZipLoader: new class for reading configs and dicts from zip. - OpenCC.convert: apply normalization pre-pass before segmentation. - **python-pure/**: package scaffold (setup.py, tests, benchmarks, .gitignore) for the opencc-py distribution. * Honour may_output_tofu flag in pure Python converter `_parse_dict_node` now accepts `include_tofu_risk_dicts` and returns `None` for any dict node whose `may_output_tofu` is true when the flag is false, mirroring `ConfigInternal::ParseDict` in Config.cpp. `OpenCC.__init__` gains `include_tofu_risk_dictionaries` (default `True`, matching the C++ Python binding); jieba comparison tests pass `False` explicitly since those golden values are CLI outputs, where tofu-risk dictionaries are excluded by default. * Load pure Python resources from opencc-data Replace source-tree and bundled-resource data discovery with the opencc-data runtime dependency so the pure Python package no longer depends on generated OpenCC artifacts. Keep the opencc-py package version and its pinned opencc-data dependency version identical for data compatibility. Detailed Changes: - **Resource Loading**: - Load built-in configs and dictionaries through opencc_data resources. - Keep custom config support for local text dictionaries next to user configs. - **Packaging and License**: - Set the opencc-py version to match the opencc-data version. - Pin opencc-data to the identical runtime dependency version. - Set package author metadata to OpenCC contributors. - Include an Apache-2.0 LICENSE file in python-pure package builds. - **Documentation and Tests**: - Add English and Traditional Chinese README files with cross-links. - Link to the official OpenCC project and the opencc-data PyPI package. - Document that this package is an OpenCC derivative work. - Update pure Python tests to use opencc-data test fixtures. * Combine pure Python package descriptions Build PKG-INFO and wheel metadata from both English and Traditional Chinese README files while stripping language-switch links and Markdown URLs. Include both README files in the source distribution manifest. Detailed Changes: - **Package Metadata**: - Compose long_description from README.md and README.zh-TW.md. - Remove README language-switch links and Markdown link targets from generated metadata. - **Source Distribution**: - Add MANIFEST.in so README.md, README.zh-TW.md, and LICENSE are explicitly included. * Document opencc-py installation Update the English and Traditional Chinese pure Python READMEs to name the published package explicitly and show the user-facing pip install command. Detailed Changes: - **Installation Docs**: - State that the PyPI package name is opencc-py. - Add python -m pip install opencc-py to both README files. - Keep local and editable install instructions for development. * Document official Python opencc package Link the pure Python README comparison section to the official opencc PyPI package and state that it is the official Python implementation. Detailed Changes: - **Documentation**: - Add the official opencc PyPI package link to the English README. - Add the official opencc PyPI package link to the Traditional Chinese README. - Clarify that opencc, not opencc-py, is the official Python implementation. * Add pure Python PyPI release workflow Add a manual workflow for validating and publishing the opencc-py pure Python package. The workflow defaults to dry-run validation and publishes through PyPI Trusted Publishing only when dry_run is disabled. Detailed Changes: - **Release Workflow**: - Install pure Python build and test dependencies. - Run the python-pure pytest suite before packaging. - Build sdist and wheel artifacts from python-pure. - Run twine check and upload built distributions as GitHub artifacts. - Publish to PyPI via pypa/gh-action-pypi-publish when dry_run is false.
frankslin
added a commit
that referenced
this pull request
Jul 21, 2026
* Make `opencc` Python package pure Python: rewrite core algorithm, drop third-party library (#29) Replace the Python native extension wrapper with a self-contained pure Python implementation that mirrors the OpenCC conversion pipeline. Bundle mmseg JSON configs and text dictionaries into the Python package, expose only supported non-jieba configs, and simplify Python wheel/release workflows to build a py3-none-any package. The plugin jieba is not supported at this time. Loading config from custom config path is not supported at this time. * Add Python golden tests and benchmarks, and update README.md * Fix Python dictionary path parsing * Add optional Python jieba support * Preserve suffixless Python config paths * Restore python/ to match upstream/master baseline Remove files added by the pure Python PR (#29) commits during rebase (benchmark_opencc.py, _opencc_pure.py, test_opencc_golden.py) and restore the BUILD.bazel and test files to their opencc-wasm-develop state. The pure Python implementation now lives exclusively in python-pure/, not in the C-extension python/ package. * Add opencc-py pure Python package and fix resource zip build Introduce python-pure/ as a parallel installable package (opencc-py) containing the pure Python OpenCC implementation, with resource zip support for generated dictionaries and a clean Bazel build target. Detailed Changes: - **opencc_resources_zip**: - Add dict_zip_name() to preserve subdirectory paths (fixes cngov/ basename collision with main STCharacters.txt). - Add --exclude-dict-subdirs flag to skip dicts in subdirectories, and suppress configs that reference those dicts. - Add exclude_dict_subdirs attribute to the Starlark rule. - Add opencc_py_resources_zip Bazel target using this flag so the opencc-py bundle omits cngov entirely. - **python-pure/opencc/_opencc_pure.py**: - _get_trie: filesystem-first lookup with zip fallback for generated files (STPhrases_GeneratedFromRegionalPhrases, TSCharactersExt). - OpenCC.__init__: filesystem configs take priority over zip; always pass zip_loader for dict fallback; support no-segmentation configs (t2jp, jp2t). - list_configs: scan jieba config dirs even in zip mode; filter out configs referencing subdirectory dicts (_config_uses_subdir_dicts). - _GroupMatcher: preserve group hierarchy; implement union/ short_circuit match_policy. - _ZipLoader: new class for reading configs and dicts from zip. - OpenCC.convert: apply normalization pre-pass before segmentation. - **python-pure/**: package scaffold (setup.py, tests, benchmarks, .gitignore) for the opencc-py distribution. * Honour may_output_tofu flag in pure Python converter `_parse_dict_node` now accepts `include_tofu_risk_dicts` and returns `None` for any dict node whose `may_output_tofu` is true when the flag is false, mirroring `ConfigInternal::ParseDict` in Config.cpp. `OpenCC.__init__` gains `include_tofu_risk_dictionaries` (default `True`, matching the C++ Python binding); jieba comparison tests pass `False` explicitly since those golden values are CLI outputs, where tofu-risk dictionaries are excluded by default. * Load pure Python resources from opencc-data Replace source-tree and bundled-resource data discovery with the opencc-data runtime dependency so the pure Python package no longer depends on generated OpenCC artifacts. Keep the opencc-py package version and its pinned opencc-data dependency version identical for data compatibility. Detailed Changes: - **Resource Loading**: - Load built-in configs and dictionaries through opencc_data resources. - Keep custom config support for local text dictionaries next to user configs. - **Packaging and License**: - Set the opencc-py version to match the opencc-data version. - Pin opencc-data to the identical runtime dependency version. - Set package author metadata to OpenCC contributors. - Include an Apache-2.0 LICENSE file in python-pure package builds. - **Documentation and Tests**: - Add English and Traditional Chinese README files with cross-links. - Link to the official OpenCC project and the opencc-data PyPI package. - Document that this package is an OpenCC derivative work. - Update pure Python tests to use opencc-data test fixtures. * Combine pure Python package descriptions Build PKG-INFO and wheel metadata from both English and Traditional Chinese README files while stripping language-switch links and Markdown URLs. Include both README files in the source distribution manifest. Detailed Changes: - **Package Metadata**: - Compose long_description from README.md and README.zh-TW.md. - Remove README language-switch links and Markdown link targets from generated metadata. - **Source Distribution**: - Add MANIFEST.in so README.md, README.zh-TW.md, and LICENSE are explicitly included. * Document opencc-py installation Update the English and Traditional Chinese pure Python READMEs to name the published package explicitly and show the user-facing pip install command. Detailed Changes: - **Installation Docs**: - State that the PyPI package name is opencc-py. - Add python -m pip install opencc-py to both README files. - Keep local and editable install instructions for development. * Document official Python opencc package Link the pure Python README comparison section to the official opencc PyPI package and state that it is the official Python implementation. Detailed Changes: - **Documentation**: - Add the official opencc PyPI package link to the English README. - Add the official opencc PyPI package link to the Traditional Chinese README. - Clarify that opencc, not opencc-py, is the official Python implementation. * Add pure Python PyPI release workflow Add a manual workflow for validating and publishing the opencc-py pure Python package. The workflow defaults to dry-run validation and publishes through PyPI Trusted Publishing only when dry_run is disabled. Detailed Changes: - **Release Workflow**: - Install pure Python build and test dependencies. - Run the python-pure pytest suite before packaging. - Build sdist and wheel artifacts from python-pure. - Run twine check and upload built distributions as GitHub artifacts. - Publish to PyPI via pypa/gh-action-pypi-publish when dry_run is false.
frankslin
added a commit
that referenced
this pull request
Aug 23, 2026
* Make `opencc` Python package pure Python: rewrite core algorithm, drop third-party library (#29) Replace the Python native extension wrapper with a self-contained pure Python implementation that mirrors the OpenCC conversion pipeline. Bundle mmseg JSON configs and text dictionaries into the Python package, expose only supported non-jieba configs, and simplify Python wheel/release workflows to build a py3-none-any package. The plugin jieba is not supported at this time. Loading config from custom config path is not supported at this time. * Add Python golden tests and benchmarks, and update README.md * Fix Python dictionary path parsing * Add optional Python jieba support * Preserve suffixless Python config paths * Restore python/ to match upstream/master baseline Remove files added by the pure Python PR (#29) commits during rebase (benchmark_opencc.py, _opencc_pure.py, test_opencc_golden.py) and restore the BUILD.bazel and test files to their opencc-wasm-develop state. The pure Python implementation now lives exclusively in python-pure/, not in the C-extension python/ package. * Add opencc-py pure Python package and fix resource zip build Introduce python-pure/ as a parallel installable package (opencc-py) containing the pure Python OpenCC implementation, with resource zip support for generated dictionaries and a clean Bazel build target. Detailed Changes: - **opencc_resources_zip**: - Add dict_zip_name() to preserve subdirectory paths (fixes cngov/ basename collision with main STCharacters.txt). - Add --exclude-dict-subdirs flag to skip dicts in subdirectories, and suppress configs that reference those dicts. - Add exclude_dict_subdirs attribute to the Starlark rule. - Add opencc_py_resources_zip Bazel target using this flag so the opencc-py bundle omits cngov entirely. - **python-pure/opencc/_opencc_pure.py**: - _get_trie: filesystem-first lookup with zip fallback for generated files (STPhrases_GeneratedFromRegionalPhrases, TSCharactersExt). - OpenCC.__init__: filesystem configs take priority over zip; always pass zip_loader for dict fallback; support no-segmentation configs (t2jp, jp2t). - list_configs: scan jieba config dirs even in zip mode; filter out configs referencing subdirectory dicts (_config_uses_subdir_dicts). - _GroupMatcher: preserve group hierarchy; implement union/ short_circuit match_policy. - _ZipLoader: new class for reading configs and dicts from zip. - OpenCC.convert: apply normalization pre-pass before segmentation. - **python-pure/**: package scaffold (setup.py, tests, benchmarks, .gitignore) for the opencc-py distribution. * Honour may_output_tofu flag in pure Python converter `_parse_dict_node` now accepts `include_tofu_risk_dicts` and returns `None` for any dict node whose `may_output_tofu` is true when the flag is false, mirroring `ConfigInternal::ParseDict` in Config.cpp. `OpenCC.__init__` gains `include_tofu_risk_dictionaries` (default `True`, matching the C++ Python binding); jieba comparison tests pass `False` explicitly since those golden values are CLI outputs, where tofu-risk dictionaries are excluded by default. * Load pure Python resources from opencc-data Replace source-tree and bundled-resource data discovery with the opencc-data runtime dependency so the pure Python package no longer depends on generated OpenCC artifacts. Keep the opencc-py package version and its pinned opencc-data dependency version identical for data compatibility. Detailed Changes: - **Resource Loading**: - Load built-in configs and dictionaries through opencc_data resources. - Keep custom config support for local text dictionaries next to user configs. - **Packaging and License**: - Set the opencc-py version to match the opencc-data version. - Pin opencc-data to the identical runtime dependency version. - Set package author metadata to OpenCC contributors. - Include an Apache-2.0 LICENSE file in python-pure package builds. - **Documentation and Tests**: - Add English and Traditional Chinese README files with cross-links. - Link to the official OpenCC project and the opencc-data PyPI package. - Document that this package is an OpenCC derivative work. - Update pure Python tests to use opencc-data test fixtures. * Combine pure Python package descriptions Build PKG-INFO and wheel metadata from both English and Traditional Chinese README files while stripping language-switch links and Markdown URLs. Include both README files in the source distribution manifest. Detailed Changes: - **Package Metadata**: - Compose long_description from README.md and README.zh-TW.md. - Remove README language-switch links and Markdown link targets from generated metadata. - **Source Distribution**: - Add MANIFEST.in so README.md, README.zh-TW.md, and LICENSE are explicitly included. * Document opencc-py installation Update the English and Traditional Chinese pure Python READMEs to name the published package explicitly and show the user-facing pip install command. Detailed Changes: - **Installation Docs**: - State that the PyPI package name is opencc-py. - Add python -m pip install opencc-py to both README files. - Keep local and editable install instructions for development. * Document official Python opencc package Link the pure Python README comparison section to the official opencc PyPI package and state that it is the official Python implementation. Detailed Changes: - **Documentation**: - Add the official opencc PyPI package link to the English README. - Add the official opencc PyPI package link to the Traditional Chinese README. - Clarify that opencc, not opencc-py, is the official Python implementation. * Add pure Python PyPI release workflow Add a manual workflow for validating and publishing the opencc-py pure Python package. The workflow defaults to dry-run validation and publishes through PyPI Trusted Publishing only when dry_run is disabled. Detailed Changes: - **Release Workflow**: - Install pure Python build and test dependencies. - Run the python-pure pytest suite before packaging. - Build sdist and wheel artifacts from python-pure. - Run twine check and upload built distributions as GitHub artifacts. - Publish to PyPI via pypa/gh-action-pypi-publish when dry_run is false.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.