Skip to content

Prepare v0.22.0 #207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bab2min merged 21 commits into from
Nov 16, 2025
Merged

Prepare v0.22.0 #207

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
4729a9b
Make the module thread-safe for no-gil Python
bab2min Nov 15, 2025
1d4a6df
Add supports for dialect
bab2min Nov 15, 2025
79395d1
Add supports for dialect & sense_id
bab2min Nov 15, 2025
d37b5fc
Update constants
bab2min Nov 15, 2025
62b9055
Add dataclass `KiwiConfig` and improve configuration handling in `Kiwi`
bab2min Nov 15, 2025
3a02eba
Fix wrong template parsing of `Kiwi.template()`
bab2min Nov 15, 2025
4f96af1
Improve robustness of `KiwiObject::join` by checking for expired `kiw…
bab2min Nov 15, 2025
f415603
Update HSDatasets for new Kiwi API
bab2min Nov 15, 2025
bdc5255
Update `Kiwi.make_hsdataset` for new Kiwi API
bab2min Nov 15, 2025
74ccedb
Add unit tests for no-gil safety
bab2min Nov 15, 2025
dcc90ba
Add more tests for `MorphemeSet`'s safety
bab2min Nov 15, 2025
c45ff30
Add more test cases for `Kiwi.join()`'s safety
bab2min Nov 15, 2025
d52a980
Add test cases for complex pattern of `Kiwi.template()`
bab2min Nov 15, 2025
68c2cbc
Add test cases for dialects
bab2min Nov 15, 2025
e041171
Update old API usages in test cases to new ones
bab2min Nov 15, 2025
1b76ca3
bump to 0.22.0
bab2min Nov 15, 2025
62a3175
Update workflows
bab2min Nov 15, 2025
329851f
Update submodule `Kiwi`
bab2min Nov 15, 2025
883bee5
Fix test cases
bab2min Nov 15, 2025
a70e3ea
Update numpy versioning
bab2min Nov 15, 2025
8f7f2da
Update workflows
bab2min Nov 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 40 additions & 64 deletions .github/workflows/deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,101 +9,77 @@ on:


jobs:
build_manylinux:
name: Build for manylinux2010
build_manylinux2014:
name: Build for manylinux2014
runs-on: ubuntu-latest
container:
image: docker://quay.io/pypa/manylinux2010_x86_64
image: docker://quay.io/pypa/manylinux2014_x86_64

steps:
- uses: actions/checkout@v1
with:
submodules: recursive
- name: Install Git LFS
run: |
mkdir gitlfs && pushd gitlfs
curl -L https://github.com/git-lfs/git-lfs/releases/download/v2.13.2/git-lfs-linux-amd64-v2.13.2.tar.gz | tar -zxv
./install.sh
popd
- name: Pull LFS files
run: cd Kiwi && git config --global --add safe.directory /__w/kiwipiepy/kiwipiepy/Kiwi && git lfs pull
- name: Deploy
continue-on-error: True
env:
TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
run: |
mv Kiwi/models/base/sj.* model/kiwipiepy_model/
mv Kiwi/models/base/extract.mdl model/kiwipiepy_model/
mv Kiwi/models/base/*.dict model/kiwipiepy_model/
mv Kiwi/models/base/combiningRule.txt model/kiwipiepy_model/
mv Kiwi/models/base/skipbigram.mdl model/kiwipiepy_model/
mv Kiwi/models/cong/base/sj.* model/kiwipiepy_model/
mv Kiwi/models/cong/base/*.mdl model/kiwipiepy_model/
mv Kiwi/models/cong/base/*.dict model/kiwipiepy_model/
mv Kiwi/models/cong/base/combiningRule.txt model/kiwipiepy_model/

/opt/python/cp38-cp38/bin/pip install "cmake<4"
/opt/python/cp311-cp311/bin/pip install "cmake<4"
rm /usr/local/bin/cmake || true
ln -s /opt/python/cp38-cp38/bin/cmake /usr/local/bin/cmake
ln -s /opt/python/cp311-cp311/bin/cmake /usr/local/bin/cmake

yum install libffi-devel -y
/opt/python/cp38-cp38/bin/python -m pip install --upgrade pip "setuptools<71"
/opt/python/cp38-cp38/bin/python -m pip install "readme-renderer==41.0" "cryptography<38" "twine<4" wheel numpy==`/opt/python/cp38-cp38/bin/python .github/workflows/numpy_version.py`
/opt/python/cp38-cp38/bin/python setup.py sdist
/opt/python/cp38-cp38/bin/python -m twine upload dist/*.tar.gz
for cp in cp38-cp38
/opt/python/cp311-cp311/bin/python -m pip install --upgrade pip setuptools
/opt/python/cp311-cp311/bin/python -m pip install twine wheel numpy==`/opt/python/cp311-cp311/bin/python .github/workflows/numpy_version.py`
for cp in cp39-cp39 cp310-cp310 cp311-cp311 cp312-cp312 cp313-cp313 cp314-cp314 cp314-cp314t
do
/opt/python/${cp}/bin/python -m pip install wheel numpy==`/opt/python/${cp}/bin/python .github/workflows/numpy_version.py`
/opt/python/${cp}/bin/python -m pip install wheel setuptools numpy==`/opt/python/${cp}/bin/python .github/workflows/numpy_version.py`
USE_MIMALLOC=1 /opt/python/${cp}/bin/python setup.py build bdist_wheel
auditwheel repair dist/*-${cp}-linux_x86_64.whl
done
/opt/python/cp38-cp38/bin/python -m twine upload wheelhouse/*.whl

cd model
/opt/python/cp38-cp38/bin/python setup.py sdist
/opt/python/cp38-cp38/bin/python -m twine upload dist/*.tar.gz
/opt/python/cp311-cp311/bin/python -m twine upload wheelhouse/*.whl

build_manylinux2014:
name: Build for manylinux2014
runs-on: ubuntu-latest
container:
image: docker://quay.io/pypa/manylinux2014_x86_64
build_macos:
name: Build for macOS
runs-on: macos-15
strategy:
max-parallel: 4
matrix:
python-version: [3.9, "3.10", 3.11, 3.12, 3.13, 3.14, 3.14t]

steps:
- uses: actions/checkout@v1
steps:
- uses: actions/checkout@v2
with:
submodules: recursive
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python-version }}
architecture: arm64
- name: Deploy
continue-on-error: True
env:
TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
run: |
mv Kiwi/models/base/sj.* model/kiwipiepy_model/
mv Kiwi/models/base/extract.mdl model/kiwipiepy_model/
mv Kiwi/models/base/*.dict model/kiwipiepy_model/
mv Kiwi/models/base/combiningRule.txt model/kiwipiepy_model/
mv Kiwi/models/base/skipbigram.mdl model/kiwipiepy_model/

/opt/python/cp311-cp311/bin/pip install "cmake<4"
rm /usr/local/bin/cmake || true
ln -s /opt/python/cp311-cp311/bin/cmake /usr/local/bin/cmake
python -m pip install twine wheel numpy==`python .github/workflows/numpy_version.py` setuptools
MACOSX_DEPLOYMENT_TARGET=11.0 KIWI_CPU_ARCH=arm64 USE_MIMALLOC=1 python setup.py bdist_wheel
twine upload dist/*

yum install libffi-devel -y
/opt/python/cp311-cp311/bin/python -m pip install --upgrade pip setuptools
/opt/python/cp311-cp311/bin/python -m pip install twine wheel numpy==`/opt/python/cp311-cp311/bin/python .github/workflows/numpy_version.py`
for cp in cp39-cp39 cp310-cp310 cp311-cp311 cp312-cp312 cp313-cp313
do
/opt/python/${cp}/bin/python -m pip install wheel setuptools numpy==`/opt/python/${cp}/bin/python .github/workflows/numpy_version.py`
USE_MIMALLOC=1 /opt/python/${cp}/bin/python setup.py build bdist_wheel
auditwheel repair dist/*-${cp}-linux_x86_64.whl
done
/opt/python/cp311-cp311/bin/python -m twine upload wheelhouse/*.whl

build_macos_13:
name: Build for macOS 13
runs-on: macOS-13
build_macos_intel:
name: Build for macOS Intel
runs-on: macos-15-intel
strategy:
max-parallel: 4
matrix:
python-version: [3.8, 3.9, "3.10", 3.11, 3.12, 3.13]
python-version: [3.9, "3.10", 3.11, 3.12, 3.13, 3.14, 3.14t]

steps:
- uses: actions/checkout@v2
Expand All @@ -113,6 +89,7 @@ jobs:
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python-version }}
architecture: x64
- name: Deploy
continue-on-error: True
env:
Expand All @@ -121,17 +98,16 @@ jobs:
run: |
python -m pip install twine wheel numpy==`python .github/workflows/numpy_version.py` setuptools
MACOSX_DEPLOYMENT_TARGET=10.14 KIWI_CPU_ARCH=x86_64 USE_MIMALLOC=1 python setup.py bdist_wheel
MACOSX_DEPLOYMENT_TARGET=10.14 KIWI_CPU_ARCH=arm64 USE_MIMALLOC=1 python setup.py bdist_wheel
twine upload dist/*

build_windows:
name: Build for Windows
runs-on: windows-2019
runs-on: windows-2022
strategy:
max-parallel: 4
matrix:
python-version: [3.8, 3.9, "3.10", 3.11, 3.12, 3.13]
architecture: [x86, x64]
python-version: [3.9, "3.10", 3.11, 3.12, 3.13, 3.14, 3.14t]
architecture: [x64]

steps:
- uses: actions/checkout@v2
Expand Down Expand Up @@ -183,7 +159,7 @@ jobs:
/opt/python/cp311-cp311/bin/pip install "cmake<4"
rm /usr/local/bin/cmake || true
ln -s /opt/python/cp311-cp311/bin/cmake /usr/local/bin/cmake
for cp in cp39-cp39 cp310-cp310 cp311-cp311 cp312-cp312 cp313-cp313
for cp in cp39-cp39 cp310-cp310 cp311-cp311 cp312-cp312 cp313-cp313 cp314-cp314 cp314-cp314t
do
/opt/python/${cp}/bin/python -m pip install wheel setuptools numpy==`/opt/python/${cp}/bin/python .github/workflows/numpy_version.py`
USE_MIMALLOC=1 /opt/python/${cp}/bin/python setup.py build bdist_wheel
Expand Down
104 changes: 40 additions & 64 deletions .github/workflows/deploy_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,101 +6,77 @@ on:
- 'v*.*.*d'

jobs:
build_manylinux:
name: Build for manylinux2010
build_manylinux2014:
name: Build for manylinux2014
runs-on: ubuntu-latest
container:
image: docker://quay.io/pypa/manylinux2010_x86_64
image: docker://quay.io/pypa/manylinux2014_x86_64

steps:
- uses: actions/checkout@v1
with:
submodules: recursive
- name: Install Git LFS
run: |
mkdir gitlfs && pushd gitlfs
curl -L https://github.com/git-lfs/git-lfs/releases/download/v2.13.2/git-lfs-linux-amd64-v2.13.2.tar.gz | tar -zxv
./install.sh
popd
- name: Pull LFS files
run: cd Kiwi && git config --global --add safe.directory /__w/kiwipiepy/kiwipiepy/Kiwi && git lfs pull
- name: Deploy
continue-on-error: True
env:
TWINE_USERNAME: ${{ secrets.TEST_PYPI_USERNAME }}
TWINE_PASSWORD: ${{ secrets.TEST_PYPI_PASSWORD }}
run: |
mv Kiwi/models/base/sj.* model/kiwipiepy_model/
mv Kiwi/models/base/extract.mdl model/kiwipiepy_model/
mv Kiwi/models/base/*.dict model/kiwipiepy_model/
mv Kiwi/models/base/combiningRule.txt model/kiwipiepy_model/
mv Kiwi/models/base/skipbigram.mdl model/kiwipiepy_model/
mv Kiwi/models/cong/base/sj.* model/kiwipiepy_model/
mv Kiwi/models/cong/base/*.mdl model/kiwipiepy_model/
mv Kiwi/models/cong/base/*.dict model/kiwipiepy_model/
mv Kiwi/models/cong/base/combiningRule.txt model/kiwipiepy_model/

/opt/python/cp38-cp38/bin/pip install "cmake<4"
/opt/python/cp311-cp311/bin/pip install "cmake<4"
rm /usr/local/bin/cmake || true
ln -s /opt/python/cp38-cp38/bin/cmake /usr/local/bin/cmake
ln -s /opt/python/cp311-cp311/bin/cmake /usr/local/bin/cmake

yum install libffi-devel -y
/opt/python/cp38-cp38/bin/python -m pip install --upgrade pip "setuptools<71"
/opt/python/cp38-cp38/bin/python -m pip install "readme-renderer==41.0" "cryptography<38" "twine<4" wheel numpy==`/opt/python/cp38-cp38/bin/python .github/workflows/numpy_version.py`
/opt/python/cp38-cp38/bin/python setup.py sdist
/opt/python/cp38-cp38/bin/python -m twine upload --repository testpypi dist/*.tar.gz
for cp in cp38-cp38
/opt/python/cp311-cp311/bin/python -m pip install --upgrade pip setuptools
/opt/python/cp311-cp311/bin/python -m pip install twine wheel numpy==`/opt/python/cp311-cp311/bin/python .github/workflows/numpy_version.py`
for cp in cp39-cp39 cp310-cp310 cp311-cp311 cp312-cp312 cp313-cp313 cp314-cp314 cp314-cp314t
do
/opt/python/${cp}/bin/python -m pip install wheel numpy==`/opt/python/${cp}/bin/python .github/workflows/numpy_version.py`
/opt/python/${cp}/bin/python -m pip install wheel setuptools numpy==`/opt/python/${cp}/bin/python .github/workflows/numpy_version.py`
USE_MIMALLOC=1 /opt/python/${cp}/bin/python setup.py build bdist_wheel
auditwheel repair dist/*-${cp}-linux_x86_64.whl
done
/opt/python/cp38-cp38/bin/python -m twine upload --repository testpypi wheelhouse/*.whl

cd model
/opt/python/cp38-cp38/bin/python setup.py sdist
/opt/python/cp38-cp38/bin/python -m twine upload --repository testpypi dist/*.tar.gz
/opt/python/cp311-cp311/bin/python -m twine upload --repository testpypi wheelhouse/*.whl

build_manylinux2014:
name: Build for manylinux2014
runs-on: ubuntu-latest
container:
image: docker://quay.io/pypa/manylinux2014_x86_64
build_macos:
name: Build for macOS
runs-on: macOS-15
strategy:
max-parallel: 4
matrix:
python-version: [3.9, "3.10", 3.11, 3.12, 3.13, 3.14, 3.14t]

steps:
- uses: actions/checkout@v1
steps:
- uses: actions/checkout@v2
with:
submodules: recursive
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python-version }}
architecture: arm64
- name: Deploy
continue-on-error: True
env:
TWINE_USERNAME: ${{ secrets.TEST_PYPI_USERNAME }}
TWINE_PASSWORD: ${{ secrets.TEST_PYPI_PASSWORD }}
run: |
mv Kiwi/models/base/sj.* model/kiwipiepy_model/
mv Kiwi/models/base/extract.mdl model/kiwipiepy_model/
mv Kiwi/models/base/*.dict model/kiwipiepy_model/
mv Kiwi/models/base/combiningRule.txt model/kiwipiepy_model/
mv Kiwi/models/base/skipbigram.mdl model/kiwipiepy_model/

/opt/python/cp311-cp311/bin/pip install "cmake<4"
rm /usr/local/bin/cmake || true
ln -s /opt/python/cp311-cp311/bin/cmake /usr/local/bin/cmake
python -m pip install twine wheel numpy==`python .github/workflows/numpy_version.py` setuptools
MACOSX_DEPLOYMENT_TARGET=11.0 KIWI_CPU_ARCH=arm64 USE_MIMALLOC=1 python setup.py bdist_wheel
twine upload --repository testpypi dist/*

yum install libffi-devel -y
/opt/python/cp311-cp311/bin/python -m pip install --upgrade pip setuptools
/opt/python/cp311-cp311/bin/python -m pip install twine wheel numpy==`/opt/python/cp311-cp311/bin/python .github/workflows/numpy_version.py`
for cp in cp39-cp39 cp310-cp310 cp311-cp311 cp312-cp312 cp313-cp313
do
/opt/python/${cp}/bin/python -m pip install wheel setuptools numpy==`/opt/python/${cp}/bin/python .github/workflows/numpy_version.py`
USE_MIMALLOC=1 /opt/python/${cp}/bin/python setup.py build bdist_wheel
auditwheel repair dist/*-${cp}-linux_x86_64.whl
done
/opt/python/cp311-cp311/bin/python -m twine upload --repository testpypi wheelhouse/*.whl

build_macos_13:
name: Build for macOS 13
runs-on: macOS-13
build_macos_intel:
name: Build for macOS Intel
runs-on: macOS-15-intel
strategy:
max-parallel: 4
matrix:
python-version: [3.8, 3.9, "3.10", 3.11, 3.12, 3.13]
python-version: [3.9, "3.10", 3.11, 3.12, 3.13, 3.14, 3.14t]

steps:
- uses: actions/checkout@v2
Expand All @@ -110,6 +86,7 @@ jobs:
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python-version }}
architecture: x64
- name: Deploy
continue-on-error: True
env:
Expand All @@ -118,17 +95,16 @@ jobs:
run: |
python -m pip install twine wheel numpy==`python .github/workflows/numpy_version.py` setuptools
MACOSX_DEPLOYMENT_TARGET=10.14 KIWI_CPU_ARCH=x86_64 USE_MIMALLOC=1 python setup.py bdist_wheel
MACOSX_DEPLOYMENT_TARGET=10.14 KIWI_CPU_ARCH=arm64 USE_MIMALLOC=1 python setup.py bdist_wheel
twine upload --repository testpypi dist/*

build_windows:
name: Build for Windows
runs-on: windows-2019
runs-on: windows-2022
strategy:
max-parallel: 4
matrix:
python-version: [3.8, 3.9, "3.10", 3.11, 3.12, 3.13]
architecture: [x86, x64]
python-version: [3.9, "3.10", 3.11, 3.12, 3.13, 3.14, 3.14t]
architecture: [x64]

steps:
- uses: actions/checkout@v2
Expand Down Expand Up @@ -180,7 +156,7 @@ jobs:
/opt/python/cp311-cp311/bin/pip install "cmake<4"
rm /usr/local/bin/cmake || true
ln -s /opt/python/cp311-cp311/bin/cmake /usr/local/bin/cmake
for cp in cp39-cp39 cp310-cp310 cp311-cp311 cp312-cp312 cp313-cp313
for cp in cp39-cp39 cp310-cp310 cp311-cp311 cp312-cp312 cp313-cp313 cp314-cp314 cp314-cp314t
do
/opt/python/${cp}/bin/python -m pip install --upgrade pip setuptools
/opt/python/${cp}/bin/python -m pip install wheel numpy==`/opt/python/${cp}/bin/python .github/workflows/numpy_version.py`
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/numpy_version.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,10 @@
def get_old_numpy_version(use_v1=False):
py_version = sys.version_info
if not use_v1:
if py_version >= (3, 11): return '2.3.*'
if py_version >= (3, 10): return '2.1.*'
if py_version >= (3, 9): return '2.0.*'
if py_version >= (3, 13): return '2.1.*'
if py_version >= (3, 13): return '2.3.*'
if py_version >= (3, 12): return '1.26.0'
if py_version >= (3, 11): return '1.24.0'
if py_version >= (3, 10): return '1.22.0'
Expand Down
Loading