Skip to content

Add allocation-free UTF-8 string iteration#10173

Open
lukewilliamboswell wants to merge 4 commits into
mainfrom
str-utf8-iteration-json-scanning
Open

Add allocation-free UTF-8 string iteration#10173
lukewilliamboswell wants to merge 4 commits into
mainfrom
str-utf8-iteration-json-scanning

Conversation

@lukewilliamboswell

@lukewilliamboswell lukewilliamboswell commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Important

This PR is stacked on #10145, which should merge first.

Closes #7010

Summary

  • add lazy, allocation-free UTF-8 byte iteration and O(1) byte-drop APIs to Str
  • route unchecked UTF-8 byte access and shared substring operations through LIR and all supported backends
  • rewrite JSON whitespace trimming and scalar splitting to scan once without intermediate byte lists
  • add builtin, runtime ownership, allocation-count, and JSON regression coverage

Testing

  • zig build minici (62/62 passed)

@lukewilliamboswell
lukewilliamboswell marked this pull request as ready for review July 16, 2026 02:30
Comment thread src/build/roc/Builtin.roc
## Drop a byte count from the start of a string. Counts at or beyond the
## byte length return `Ok("")`; an in-range cut inside a UTF-8 code point
## returns `Err(BadUtf8)`. Valid slices do not allocate.
drop_first_bytes : Str, U64 -> Try(Str, [BadUtf8, ..])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would you think of having the error be something like NotACharacterBoundary instead?

The argument being that BadUtf8 makes it sound (to me) like there was something wrong with the original string itself, whereas the error actually comes from the count param cutting into it in a bad place.

@lukewilliamboswell lukewilliamboswell Jul 20, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

~~Sounds good, I'll change it. ~~ actually I decided to leave it so it's consistent with all the others. I think this kind of idea would be better discussed in Zulip with everyone before we commit to an API deviation

@lukewilliamboswell
lukewilliamboswell force-pushed the str-utf8-iteration-json-scanning branch from 16dc53d to d0a0d3b Compare July 20, 2026 01:05
lukewilliamboswell and others added 3 commits July 20, 2026 12:07
…on-scanning

# Conflicts:
#	design.md
#	src/canonicalize/NodeStore.zig
#	src/compile/static_data_exports.zig
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement builtin number parsing

3 participants