Skip to content

Implement path lookahead and optimize normalization#188

Open
NotAShelf wants to merge 3 commits into
nix-community:masterfrom
NotAShelf:master
Open

Implement path lookahead and optimize normalization#188
NotAShelf wants to merge 3 commits into
nix-community:masterfrom
NotAShelf:master

Conversation

@NotAShelf

@NotAShelf NotAShelf commented Jul 16, 2026

Copy link
Copy Markdown
Member

I've gotten really into parser development recently (for the purposes of witing a nftables parser) and ultimately ended up needing/wanting/dreaming of one for Nix. While looking for inspiration in rnix's codebase I've identified some areas that I could be "optimized". If my math is correct, they return rather positive results:

Bench Before After Change
tokenizer/all-packages 25.45 ms 6.02 ms -76.14%
normalized-parts/all-packages 300.83 µs 86.65 µs -77.62%

My motivation for the change is rather simple. The way I see it, our path/URI disambiguation code scans an initial run of path characters before it knows what kind of token it is looking at. That scan used to be materialized as a String, despite the following decision only needing two things from it:

  1. where the scan ended, and
  2. whether the prefix contained an underscore

Recording those while scanning keeps the existing classification rules without allocating or walking the prefix again. Thus, this PR removes the temporary String allocation, and lookahead now records the byte offset and whether it saw an underscore while scanning the input. On a similar optimization now (which is minor in comparison Str::normalized_parts no longer collects all string parts before processing them but instead, it makes separate passes over the immutable syntax tree for indentation detection and output generation, and preallocates the result buffers for us. On rnix's own fixtures, tokenizer time dropped from about 25.5 ms to 6.0 ms, and string normalization dropped from about 301 µs to 87 µs. Which I see as an absolute win even though the numbers are rather small.

I've went back in the git history to see if this changes are intentional by any chance and looks like the temporary String dates to all the way back tothe original 2019 URI heuristic. Admittedly it was a straightforward way to retain both "how far did I scan?" and "did I see _?" but I think we can do better.

The public API remains the same; the optimization is internal, but I do want to state that unescape now reserves input.len() even when an escape-heavy string will shrink substantially. This can, in theory, reserve more transient memory for a huge, escape-dense string than the old growth strategy. Should probably not matter for most cases.


I've written a tiny benchmark suite to get the numbers. I'm keeping this PR as a draft while I beat it into shape, but in the meantime please do feel free to tear apart my changes. I've written this most of those changes while rather exhausted and mostly with the goal of getting them off my mind before I call it a day. If I misread anything or misunderstood any intent anywhere, please let me know. I'm happy to iterate.

P.S. I'm sorry for the wall of text. I do hope this doesn't defer from reviewing the actual change, as it's very simple in theory and in application. I like typing a bit too much.

Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Id54f99e66b308f5464138b55ae599fb96a6a6964
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: I71d00d96426a07117a63fc2a764dfe336a6a6964
Signed-off-by: NotAShelf <raf@notashelf.dev>
Change-Id: Ifa62fc405065b078a93f3fe8445479d16a6a6964
@NotAShelf
NotAShelf marked this pull request as ready for review July 19, 2026 11:11
@NotAShelf

Copy link
Copy Markdown
Member Author

Okay, should be ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant