Tokenizer discards the Buffer from fs.readFile, halving SentencePiece parse speed (~2x)

[ScottMansfield]

Summary

The Node tokenizer platform converts the Buffer returned by fs.readFile() into a plain
Uint8Array before handing it to ModelProto.decode(). That conversion silently opts out of
protobufjs's fastest code path, roughly doubling SentencePiece model parse time.

Measured on a 256,000-piece / 5.8 MB model: 105 ms → 55 ms (1.93×) from removing the
wrap.

Cause

protobufjs picks its reader by type (src/reader.js):

Reader.create = util.Buffer
    ? function (buffer) {
        return util.Buffer.isBuffer(buffer)
            ? new BufferReader(buffer)   // native path
            : create_array(buffer);      // generic JS path
      }
    : create_array;

BufferReader exists almost entirely to override string decoding
(src/reader_buffer.js):

BufferReader.prototype.string = function read_string_buffer() {
    var len = this.uint32();
    return this.buf.utf8Slice
        ? this.buf.utf8Slice(this.pos, this.pos = Math.min(this.pos + len, this.len))
        : ...
};

Buffer.prototype.utf8Slice is backed by V8's native UTF-8 decoder. The generic Reader
falls back to a JS decode loop. For a SentencePiece model — which is ~256k short strings and
almost nothing else — that single method is the whole cost of parsing.

Where the Buffer is discarded

src/node/_node_tokenizer_platform.ts:38:

const data = await fs.readFile(filePath);   // Buffer
const hash = crypto.createHash('sha256').update(data).digest('hex');
if (hash === expectedHash) {
  return new Uint8Array(data);              // <-- copies into a plain Uint8Array
}

and fetchFromUrl() at src/node/_node_tokenizer_platform.ts:85:

const arrayBuffer = await response.arrayBuffer();
return new Uint8Array(arrayBuffer);

Both return values flow to SentencePieceProcessor →
parseModelProto(data: Uint8Array) → sentencepiece.ModelProto.decode(data)
(src/cross/sentencepiece/_processor.ts:415).

fs.readFile result   -> Buffer.isBuffer: true
after new Uint8Array -> Buffer.isBuffer: false   <-- what decode() receives

Benchmark

256,000 pieces, 5.8 MB wire, Node v26.7.0, median of 9 interleaved runs:

input to ModelProto.decode()	reader selected	median
Buffer (straight from fs.readFile)	BufferReader (native utf8Slice)	55 ms
new Uint8Array(...) (current behaviour)	Reader (JS loop)	105 ms

Cross-checked by isolating the variable in protobufjs itself — the same wire bytes decoded as
Buffer vs Uint8Array differ by 2.17×, and BufferReader overrides only string() and
_slice(), so string decoding accounts for essentially all of it.

Suggested fix

Buffer is a Uint8Array, so the declared Promise<Uint8Array> return types stay valid
with no signature changes — the wrap can simply be dropped:

-        return new Uint8Array(data);
+        // `Buffer` is a `Uint8Array`; keeping it lets protobufjs pick `BufferReader`,
+        // which decodes strings via native `utf8Slice` (~2x faster on large vocabs).
+        return data;

and on the fetch path:

-    const arrayBuffer = await response.arrayBuffer();
-    return new Uint8Array(arrayBuffer);
+    const arrayBuffer = await response.arrayBuffer();
+    return typeof Buffer !== 'undefined' ? Buffer.from(arrayBuffer) : new Uint8Array(arrayBuffer);

Both changes are Node-only (src/node/), so browser/web builds are unaffected — those keep
plain Uint8Array and the generic reader, as today.

If the explicit new Uint8Array(...) was deliberate — e.g. to normalise the type across
platforms, or to detach from a pooled Buffer — then a narrower fix would be to keep the
conversion at the platform boundary but pass the original Buffer through to
parseModelProto. Happy to send a PR either way.

[Varun-S10]

Hi @ScottMansfield, Thank you for your feedback. We appreciate you taking the time to share your thoughts with us, and we will be filing a feature request.