It's possible to tokenize from the input byte slice to remove the per-token String allocation. The tokenizer currently allocates a String for every token: read_until_whitespace builds String::with_capacity(12) per number and per word (src/tokenizer.rs). From my measures this dominates the parse cost.
Measured on the crate's own bench data (benches/big.wkt)
| tokenizer |
parse big.wkt |
current (Peekable<Chars> + per-token String) |
3.42 ms |
| byte-slice scan, zero allocation (prototype) |
1.35 ms |
The prototype makes three changes:
Tokens scans &[u8] with an index instead of Peekable<str::Chars>.Token::Word borrows the input (Word(&'a str)) instead of owning a String.- Numbers parse directly from their byte range:
fast_float accepts &[u8] input, so no UTF-8 conversion is needed on the hot path (see #153 for the fast_float part).
Tokens only ever end on ASCII bytes (whitespace, \0, (, ), ,), and ASCII bytes can never be part of a multi-byte sequence, so slices into the input are always valid at char boundaries.
This addresses the same bottleneck as the nom-rewrite direction in #24, but by removing the allocation rather than changing the parser structure.
Please note that this is just a quick prototype, not a finished contribution. It was written to validate the direction and the measurement, and has not been checked against this project's quality bar. For now I would be grateful just for a feedback on whether this is something you could consider.
It's possible to tokenize from the input byte slice to remove the per-token
Stringallocation. The tokenizer currently allocates aStringfor every token:read_until_whitespacebuildsString::with_capacity(12)per number and per word (src/tokenizer.rs). From my measures this dominates the parse cost.Measured on the crate's own bench data (
benches/big.wkt)Peekable<Chars>+ per-tokenString)The prototype makes three changes:
Tokensscans&[u8]with an index instead ofPeekable<str::Chars>.Token::Wordborrows the input (Word(&'a str)) instead of owning aString.fast_floataccepts&[u8]input, so no UTF-8 conversion is needed on the hot path (see #153 for the fast_float part).Tokens only ever end on ASCII bytes (whitespace,
\0,(,),,), and ASCII bytes can never be part of a multi-byte sequence, so slices into the input are always valid at char boundaries.This addresses the same bottleneck as the nom-rewrite direction in #24, but by removing the allocation rather than changing the parser structure.
Please note that this is just a quick prototype, not a finished contribution. It was written to validate the direction and the measurement, and has not been checked against this project's quality bar. For now I would be grateful just for a feedback on whether this is something you could consider.