Basically - would you like a wasm and/or native node binding?
I've been doing a little tinkering with a wasm-bindgen and napi-rs binding, with quite positive (albeit not as robust as the extensive test suite) results.
TLDR: just on wasm, 50MB/s batch encoding is very much achievable (70MB/s via NAPI) on commodity hardware. On an M5 Pro, frequently exceeds 130MB/s (in rare/flukey runs, 200MB/s).
- wasm, evaluated via Node 26, on a Ryzen 5900X (i.e. old). simd128 enabled.
- Tokenizer init ~45ms (for a typical Bert, WordPiece pretokenizer)
- Single ~170 char encode - ~1-2 microseconds + ~1.5-2.5 microseconds of FFI-induced overhead (some of it is just noise from the event loop). short randomly generated english sentence. Much more variable than batch, and of course even more
encode_batch(Vec<String>)->Encoding[] , batch=10, average length 170 chars - per string effective ~1-2 microseconds + ~1.5 microseconds of FFI overhead (creating class instances for each rs pointer is fairly expensive, with a slight discount thanks to the single (rather than 10) FFI boundary crosses).encode_batch(Vec<String)->BatchEncoding, batch=10, etc - per string effective ~1-2 microseconds + 0.2 microseconds + ~2-3 microseconds overall FFI overhead. Essentially this packs the individual member vectors of Encoding into a flattened Uint32Array (allocated on the JS side) paired with a 2 element dims array (very much like your average JS-facing tensor library does); fortunately downstream consumers typically want this structure anyway.encode_batch(&[u8], offsets: &[u32])->BatchEncoding - per string/slice 1-2 microseconds + 2-3 microseconds return. Ideal if you just so happen to be working with a file or readablestream (especially if its something newline delimited).- wasm binary ~1.6-2MB, depending on compilation settings. ~20kB of wrapping JS.
- typical NAPI (which I won't go into in exhaustive detail here) binary ~3-4MB.
Basically - would you like a wasm and/or native node binding?
I've been doing a little tinkering with a wasm-bindgen and napi-rs binding, with quite positive (albeit not as robust as the extensive test suite) results.
TLDR: just on wasm, 50MB/s batch encoding is very much achievable (70MB/s via NAPI) on commodity hardware. On an M5 Pro, frequently exceeds 130MB/s (in rare/flukey runs, 200MB/s).
encode_batch(Vec<String>)->Encoding[], batch=10, average length 170 chars - per string effective ~1-2 microseconds + ~1.5 microseconds of FFI overhead (creating class instances for each rs pointer is fairly expensive, with a slight discount thanks to the single (rather than 10) FFI boundary crosses).encode_batch(Vec<String)->BatchEncoding, batch=10, etc - per string effective ~1-2 microseconds + 0.2 microseconds + ~2-3 microseconds overall FFI overhead. Essentially this packs the individual member vectors of Encoding into a flattened Uint32Array (allocated on the JS side) paired with a 2 element dims array (very much like your average JS-facing tensor library does); fortunately downstream consumers typically want this structure anyway.encode_batch(&[u8], offsets: &[u32])->BatchEncoding- per string/slice 1-2 microseconds + 2-3 microseconds return. Ideal if you just so happen to be working with a file or readablestream (especially if its something newline delimited).