Rewrite Gurmukhi Utils to Rust by harjot1singh · Pull Request #223 · shabados/gurmukhi-utils

harjot1singh · 2025-09-11T01:33:20Z

Summary

Wouldn't it be great if we had a singular codebase, and could generate the same gurmukhi-utils for each language?

Well, now we can! This PR rewrites gurmukhi-utils to Rust. Bindings are generated by Uniffi for multiple language targets: Python, Ruby, Kotlin, Swift, and JavaScript.

The JavaScript story is a single gurmukhi npm package serving every runtime through conditional exports:

Web / Node / Bun — WASM (size-optimized: ~950 KB raw, ~335 KB gzip), loaded via top-level await so the API is fully synchronous after import
React Native — a JSI Turbo Module generated by uniffi-bindgen-react-native, backed by the compiled Rust library (no WASM; Metro resolves the react-native export condition). Android jniLibs (arm64-v8a, armeabi-v7a, x86_64) and an iOS xcframework are cross-compiled in CI

Both entry points present an identical API surface (string-literal enums), enforced by a typechecked adapter.

Why?

Pros

Single language codebase to maintain
Function and interface parity across all language libraries — including web vs React Native, from one npm package
Signficantly reduced test, sharing, and build complexity
Rust is fast and has great support for bindings
CI smoke-tests every generated binding against real native builds on Linux, macOS, and Windows (React Native bindings are exercised in plain Node via ubrn's N-API runtime — no emulators)

Drawbacks

Correct bindings need to be generated
WASM carries install-time weight for web JS (mitigated: ~335 KB gzip over the wire)
Requires usage of enums (idiomatic in some languages, not recommended in others; JS exposes them as string literals)

See MIGRATING.md for the old→new API mapping per language.

Hopefully, a win 🤞🏼

dsomel21 · 2025-09-11T02:09:52Z

Wow

Implement to_ascii() converting Unicode Gurmukhi to ASCII for Open Gurbani Akhar font, mirroring the JS/Python implementations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Mark as private with name "root" to avoid confusion with the published package, remove unused tsx dependency and broken test script. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Consolidate all language bindings under a single bindings/ directory. Update path references in CODEOWNERS, CONTRIBUTING.md, README.md, constants generators, test fixture paths, and guut adapters. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add mise tasks for building Rust and generating UniFFI bindings for Ruby, Kotlin, and Swift. Update .gitignore to exclude generated binding files. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Delete pure Ruby reimplementation (ascii.rb, unicode.rb, remove.rb, constants.rb) and replace with UniFFI-generated FFI bindings from the Rust core. Update gemspec to depend on ffi gem, rewrite tests to use the new API surface (to_ascii, to_unicode, normalize_unicode). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add Kotlin package with build.gradle.kts (JNA + Kotlin JVM), guut test adapter for the universal JSON test suite, and mise.toml for Java 21 toolchain. Bindings are generated from Rust via uniffi-bindgen at build time. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add Swift package with Package.swift (system library target for the C FFI header + modulemap), XCTest-based guut adapter for the universal JSON test suite. Bindings are generated from Rust via uniffi-bindgen at build time, split into Sources/GurmukhiUtils/ (.swift) and Sources/gurmukhi_utilsFFI/ (.h + .modulemap). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Delete empty csharp/ directory (no implementation was started). Update progress matrix to show Rust, Kotlin, and Swift columns. Remove C# maintainer reference and contributing link. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Generalizes vishram/line-ending operations into a composable Markup enum. Callers pass arrays of markup types to remove() or detect() instead of calling separate per-type functions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Generalize the markup module into a feature module that covers all removable/detectable categories of Gurmukhi text: vishrams, granular line endings (rahao/numbered/bare), vowel signs, vowel carriers, nukta, adhak, nasals, accents, and visarga. Add uniffi-exported grouping functions (vishraams, line_endings, vowels, modifiers, all_features) and feature_chars for character lookup — available in all binding languages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Needed by the transcribe module to normalize input before dispatching to scheme-specific conversion logic. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…schemes Single `transcribe(input, script)` API for converting Gurmukhi into other scripts. Each Script enum variant bundles opinionated conversion behavior for its target audience: - LatinScholar: mechanical ISO/IAST-like transliteration for academics - Latin: pronunciation-aware conversion for English readers in sangat (haha rules, grammatical vowel dropping, adhak gemination) - Devanagari: offset-based script mapping for Hindi readers 440 test cases ported from Python + new Devanagari cases. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Ports the 2 remaining guru_latn_pa assertion cases that were initially skipped. The Python has a bug where variation selectors leak into the output for these inputs — the Rust implementation handles them correctly. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Rust core (via UniFFI) is now the single source of truth. Removes all native language implementations that duplicated Rust functionality: - JavaScript: src/, test/, tsconfig, prettierignore - Python: gurmukhiutils/, tests/, poetry.lock, .flake8 - Dart: entire binding (no viable UniFFI generator exists) Moves language-specific gitignore entries from root to each binding folder (UniFFI generated paths for Kotlin/Ruby/Swift, standard patterns for JS/Python). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The constants/ directory generated language-specific constant files for the native JS/Python/Ruby implementations. Now that all bindings use Rust via UniFFI, constants are embedded in the Rust library and this pipeline is orphaned. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…gen-node-js Replace ubrn (uniffi-bindgen-react-native) with two purpose-built generators: - uniffi-bindgen-node-js: native FFI for Node/Bun (loads .so via koffi) - uniffi-bindgen-js: WASM for browser (self-contained, 2 files + .wasm) Both produce matching APIs: camelCase functions, string-valued enums, clean TypeScript types. Post-processing scripts handle camelCase-ification of native output and namespace flattening of WASM output. Adds mise tasks: generate:javascript, generate:javascript:native, generate:javascript:wasm, build:wasm. Removes ubrn dependency and patch-enums.ts workaround. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Wire up `mise run generate:python` using the standard uniffi-bindgen CLI with `--language python`. Generates ctypes-based bindings into bindings/python/gurmukhiutils/. Thin __init__.py re-exports everything for clean `from gurmukhiutils import to_ascii, Script` usage. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Rust tests are the source of truth. Per-language guut tests that validated native implementations against universal JSON test data are redundant now that all bindings are generated from UniFFI. Removes: test/ JSON suite, guut runners (Ruby/Kotlin/Swift), per-language test files, Rakefile/Gemfile, test tasks from mise.toml, test deps from build.gradle.kts and Package.swift. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Remove per-binding CONTRIBUTING.md and README.md files that described old native development workflows (poetry, guut, npm test, spec/). Fix Python pyproject.toml URLs to point to monorepo. Remove deprecated Python .vscode settings. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Run cargo fmt, fix $crate in macro, redundant closure, empty doc comment, and remove unused is_below_letter function. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add uniffi-bindgen-js and uniffi-bindgen-node-js as [[bin]] targets pinned via Cargo.lock for version-correct codegen, matching the existing uniffi-bindgen.rs pattern. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Each test imports the binding and calls toUnicode("gurU", SantLipi), asserting the result equals "ਗੁਰੂ". Validates the full chain: Rust → FFI → binding → runtime → correct output. - JS WASM, Python, Ruby, Kotlin, Swift smoke tests - Mise smoke:* tasks for local execution - Fix Swift Package.swift target names after crate rename - Strip use "Darwin" from generated modulemap at generation time Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Extends the generate matrix job to smoke test each language after generation. Adds conditional setup for Kotlin (JDK) and Swift (compiler) since those aren't in the base mise toolset. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Adds cargo-zigbuild CI matrix to compile libgurmukhi for 5 platforms (linux x64/arm64, macOS x64/arm64, Windows x64) from a single Linux runner. Extends publish workflow with Python wheels, Ruby gems, and Kotlin fat JAR packaging. - Reusable build-native.yml workflow called from CI and publish - Python: setuptools with platform-tagged wheels per target - Ruby: renamed gem to gurmukhi, patched ffi_lib to load from gem dir - Kotlin: maven-publish plugin, JNA resource dirs for fat JAR - Local: mise run build:native / build:native:all tasks Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- smoke:stage only creates symlinks if binding dir exists (CI runs per-language, so other bindings' dirs aren't present) - sudo gem install for CI Ruby ffi gem - rename job from "Generate + Smoke" to "Generate" Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Fix Ruby symlink: 4 levels up from bindings/ruby/lib/gurmukhi/ - Switch Kotlin smoke to Gradle runSmoke task (resolves JNA dep) - Add setup-gradle action for Kotlin CI Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ffi_lib with an absolute path doesn't append extensions. Use FFI.map_library_name('gurmukhi') to get the correct platform name (libgurmukhi.so, libgurmukhi.dylib, gurmukhi.dll). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Generate bindings once on Linux, then smoke test each language on all 3 platforms. Validates native library loading actually works on each OS, not just the CI build platform. - Generate job uploads bindings as artifacts - Smoke job: language × os matrix downloads bindings + native lib - Exclude swift on windows (no Swift runtime) - Remove legacy gurmukhi_utils wrapper files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Ruby FFI gem needs sudo on Linux CI runners - macOS runners have Swift built-in; setup-swift causes linker conflicts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…rror libswiftCompatibility56.a causes arm64 fixup errors on modern macOS linkers when SPM doesn't know the deployment target. Setting macOS 13 / iOS 16 avoids the compatibility library entirely. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@rpath

Cross-compiled dylibs have absolute Linux paths baked as install names. Use install_name_tool to fix to @rpath, and pass -rpath to swift build so the runtime linker finds the library. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

SPM resolves packages from the repo root, not subdirectories. Move Package.swift to root and update paths to reference bindings/swift/Sources. Update smoke:swift task and gitignore accordingly. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add tsc compilation to generate:javascript so the npm package ships compiled JS with type declarations instead of raw .ts — works in Node, Bun, Deno, and all bundlers without extra tooling. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Generate (javascript) failed in CI with TS2688 because @types/node was never installed before tsc ran. Add bun install to the task and migrate to bun.lock, matching the repo's bun toolchain. Ship a size-optimized WASM: dedicated wasm-release profile (opt-level=z, fat LTO, panic=abort, strip) plus wasm-opt -Oz after bindgen, taking the binary from 1.63 MB to 950 KB raw (335 KB gzip). Native library builds keep the speed-optimized release profile. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012giEdJSNKHM8sEuahhmonn

binaryen's version_XXX tag scheme makes ubi's "latest" resolve to a stale 2019 semver tag (1.39.1) on newer mise, which ships no wasm-opt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012giEdJSNKHM8sEuahhmonn

… notes JS bindings export enums as string-literal union types, not runtime objects — the examples showed imports that don't exist. Also document ESM-only/top-level-await requirements and bundler support for the WASM asset pattern. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012giEdJSNKHM8sEuahhmonn

…package One package, one import: Metro resolves the react-native export condition to a JSI Turbo Module backed by the compiled Rust library; every other bundler falls through to the wasm entry. - ubrn 0.31.0-3 (matches uniffi 0.31) generates the JSI bindings, C++ glue, Android/iOS scaffolding, and Codegen spec - Hand-maintained src/index.rn.ts adapter presents the same string-literal enum surface as the wasm bindings so both entry points are interchangeable, typechecked via tsconfig.rn.json - Smoke test runs the generated bindings under Node through ubrn's N-API runtime (@ubjs/node) — no emulator needed - CI: Generate (react-native) job + Android (cargo-ndk, 3 ABIs) and iOS (xcframework) cross-compile jobs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012giEdJSNKHM8sEuahhmonn

publish.yml gains a JavaScript pipeline: wasm + JSI generation, Android jniLibs and iOS xcframework cross-compiles, assembled into one npm package (dry-run always; real publish on release with NPM_TOKEN). release-please now bumps every binding's version via extra-files (package.json, pyproject.toml, gemspec, build.gradle.kts) so second and subsequent releases don't collide on any registry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012giEdJSNKHM8sEuahhmonn

harjot1singh added 2 commits September 10, 2025 14:03

build: add vscode rust settings

59b0c63

feat: add rust unicode functions

391969e

harjot1singh and others added 27 commits March 1, 2026 20:59

feat: add rust ascii functions

5a8f125

Implement to_ascii() converting Unicode Gurmukhi to ASCII for Open Gurbani Akhar font, mirroring the JS/Python implementations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

refactor: split unicode tests into separate functions per section

4ae710f

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

chore: remove resolved comment in unicode constants

f2756dc

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

feat: upgrade uniffi to 0.30 and export to_unicode

41b738f

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

build: add wasm build config and gitignore generated files

e8ada53

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

build: add js packaging and post-generation enum patch script

49a6427

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

chore: clean up root package.json

8733ff7

Mark as private with name "root" to avoid confusion with the published package, remove unused tsx dependency and broken test script. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

build: add uniffi binding generation tasks to mise

9178e20

Add mise tasks for building Rust and generating UniFFI bindings for Ruby, Kotlin, and Swift. Update .gitignore to exclude generated binding files. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

build: remove python/.vscode

1e0d543

refactor: make Sant Lipi replacement constant pub(crate)

2746410

Needed by the transcribe module to normalize input before dispatching to scheme-specific conversion logic. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

style: format codebase and resolve clippy warnings

5efb265

Run cargo fmt, fix $crate in macro, redundant closure, empty doc comment, and remove unused is_below_letter function. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

harjot1singh and others added 21 commits April 9, 2026 00:21

fix: install zig for cargo-zigbuild in CI and local tasks

38e8c79

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: sudo gem install on Linux, skip setup-swift on macOS

2938306

- Ruby FFI gem needs sudo on Linux CI runners - macOS runners have Swift built-in; setup-swift causes linker conflicts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

chore: bump setup-java and setup-gradle to v5 for Node 24

64915bb

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

chore: update package names, authors, and repo URLs

87d39e5

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: add node types to tsconfig for uniffi runtime compilation

e878a8c

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

harjot1singh marked this pull request as ready for review July 18, 2026 19:00

harjot1singh requested review from akshdeep-singh, bhajneet, dsomel21 and onkarjit as code owners July 18, 2026 19:00

harjot1singh enabled auto-merge (rebase) July 18, 2026 19:00

harjot1singh disabled auto-merge July 18, 2026 19:05

harjot1singh merged commit 1969a52 into main Jul 18, 2026
30 checks passed

harjot1singh deleted the to-rust branch July 18, 2026 19:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Rewrite Gurmukhi Utils to Rust - #223

Rewrite Gurmukhi Utils to Rust#223
harjot1singh merged 67 commits into
mainfrom
to-rust

harjot1singh commented Sep 11, 2025 •

edited

Loading

Uh oh!

dsomel21 commented Sep 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

harjot1singh commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why?

Pros

Drawbacks

Uh oh!

dsomel21 commented Sep 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

harjot1singh commented Sep 11, 2025 •

edited

Loading