refactor(formatter): introduce Token element #7968
Open
+975
−958
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🦋 Changeset detectedLatest commit: edd7a45 The changes in this PR will be included in the next version bump. This PR includes changesets to release 13 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
github-actions
bot
added
A-Project
Parser conformance results onjs/262
jsx/babel
symbols/microsoft
ts/babel
ts/microsoft
|
CodSpeed Performance ReportMerging #7968 will not alter performanceComparing Summary
Footnotes
|
WalkthroughThe PR renames and reshapes the formatter's textual primitives across the workspace: Possibly related PRs
Suggested labels
Suggested reviewers
Pre-merge checks and finishing touches✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro 📒 Files selected for processing (66)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (35)
🧰 Additional context used📓 Path-based instructions (5)crates/biome_js_formatter/**/*.rs📄 CodeRabbit inference engine (crates/biome_js_formatter/CONTRIBUTING.md)
Files:
**/*.{rs,toml}📄 CodeRabbit inference engine (CONTRIBUTING.md)
Files:
**/*.rs📄 CodeRabbit inference engine (CONTRIBUTING.md)
Files:
.changeset/*.md📄 CodeRabbit inference engine (CONTRIBUTING.md)
Files:
crates/biome_formatter/src/**/*.rs📄 CodeRabbit inference engine (crates/biome_formatter/CONTRIBUTING.md)
Files:
🧠 Learnings (42)📓 Common learnings📚 Learning: 2025-10-15T09:22:46.002ZApplied to files:
📚 Learning: 2025-10-15T09:22:46.002ZApplied to files:
📚 Learning: 2025-10-15T09:22:46.002ZApplied to files:
📚 Learning: 2025-10-15T09:22:46.002ZApplied to files:
📚 Learning: 2025-10-15T09:23:33.055ZApplied to files:
📚 Learning: 2025-10-15T09:22:46.002ZApplied to files:
📚 Learning: 2025-10-24T21:24:58.650ZApplied to files:
📚 Learning: 2025-10-15T09:22:15.851ZApplied to files:
📚 Learning: 2025-10-15T09:22:15.851ZApplied to files:
📚 Learning: 2025-10-15T09:23:33.055ZApplied to files:
📚 Learning: 2025-10-24T21:24:58.650ZApplied to files:
📚 Learning: 2025-10-15T09:22:15.851ZApplied to files:
📚 Learning: 2025-10-15T09:22:15.851ZApplied to files:
📚 Learning: 2025-10-15T09:22:15.851ZApplied to files:
📚 Learning: 2025-10-15T09:22:15.851ZApplied to files:
📚 Learning: 2025-10-15T09:22:15.851ZApplied to files:
📚 Learning: 2025-10-15T09:22:46.002ZApplied to files:
📚 Learning: 2025-10-15T09:22:15.851ZApplied to files:
📚 Learning: 2025-10-26T15:28:00.951ZApplied to files:
📚 Learning: 2025-10-24T21:24:58.650ZApplied to files:
📚 Learning: 2025-10-26T15:28:00.951ZApplied to files:
📚 Learning: 2025-10-24T21:24:58.650ZApplied to files:
📚 Learning: 2025-10-24T21:24:58.650ZApplied to files:
📚 Learning: 2025-10-24T21:24:58.650ZApplied to files:
📚 Learning: 2025-10-15T09:22:15.851ZApplied to files:
📚 Learning: 2025-10-25T07:22:18.540ZApplied to files:
📚 Learning: 2025-10-15T09:22:15.851ZApplied to files:
📚 Learning: 2025-10-15T09:24:31.042ZApplied to files:
📚 Learning: 2025-09-13T16:16:06.459ZApplied to files:
📚 Learning: 2025-09-13T16:16:06.459ZApplied to files:
📚 Learning: 2025-10-03T12:28:56.788ZApplied to files:
📚 Learning: 2025-10-15T09:22:15.851ZApplied to files:
📚 Learning: 2025-10-15T09:22:15.851ZApplied to files:
📚 Learning: 2025-10-15T09:22:15.851ZApplied to files:
📚 Learning: 2025-10-15T09:23:33.055ZApplied to files:
📚 Learning: 2025-08-20T16:24:59.781ZApplied to files:
📚 Learning: 2025-10-15T09:23:33.055ZApplied to files:
📚 Learning: 2025-09-13T16:16:06.459ZApplied to files:
📚 Learning: 2025-10-15T09:23:33.055ZApplied to files:
📚 Learning: 2025-10-15T09:23:33.055ZApplied to files:
📚 Learning: 2025-10-15T09:23:33.055ZApplied to files:
🧬 Code graph analysis (28)crates/biome_js_formatter/src/js/classes/extends_clause.rs (1)
crates/biome_html_formatter/src/html/auxiliary/self_closing_element.rs (1)
crates/biome_js_formatter/src/jsx/expressions/tag_expression.rs (1)
crates/biome_grit_formatter/src/comments.rs (1)
crates/biome_js_formatter/src/utils/member_chain/groups.rs (1)
crates/biome_js_formatter/src/utils/jsx.rs (1)
crates/biome_graphql_formatter/src/comments.rs (3)
crates/biome_graphql_formatter/src/graphql/lists/object_value_member_list.rs (1)
crates/biome_css_formatter/src/css/auxiliary/declaration_with_semicolon.rs (1)
crates/biome_js_formatter/src/context/trailing_commas.rs (1)
crates/biome_graphql_formatter/src/verbatim.rs (1)
crates/biome_graphql_formatter/src/graphql/value/string_value.rs (3)
crates/biome_js_formatter/src/js/expressions/bigint_literal_expression.rs (1)
crates/biome_html_formatter/src/verbatim.rs (1)
crates/biome_html_formatter/src/html/auxiliary/string.rs (1)
crates/biome_css_formatter/src/css/value/url_value_raw.rs (2)
crates/biome_js_formatter/src/ts/types/bigint_literal_type.rs (1)
crates/biome_formatter/src/macros.rs (3)
crates/biome_js_type_info/src/format_type_info.rs (2)
crates/biome_js_formatter/src/js/statements/expression_statement.rs (2)
crates/biome_js_formatter/src/ts/expressions/type_assertion_expression.rs (1)
crates/biome_formatter/src/format_element/document.rs (1)
crates/biome_html_formatter/src/utils/formatters.rs (1)
crates/biome_formatter/src/separated.rs (1)
crates/biome_formatter/src/format_element.rs (2)
crates/biome_js_formatter/src/js/expressions/unary_expression.rs (1)
crates/biome_formatter/src/printer/mod.rs (1)
crates/biome_module_graph/src/format_module_graph.rs (2)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (19)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
crates/biome_css_formatter/src/verbatim.rs (1)
117-121:textpanics on multi-line verbatim content
builders::textcallsdebug_assert_no_newlines, but verbatim chunks routinely include\neven afternormalize_newlines. The first multiline suppressed CSS node will now trip the assert. Please stick with a newline-tolerant code path (e.g. keep the old constructor or split the content before passing it totext).crates/biome_graphql_formatter/src/verbatim.rs (1)
118-122: Same newline assertion issue as the CSS verbatim printer
text(..)rejects strings containing line terminators, yet GraphQL verbatim nodes are often multi-line. This will assert in debug builds. Please keep using a constructor that tolerates newlines (or split the content) before landing this refactor.
🧹 Nitpick comments (4)
.changeset/lemon-planes-knock.md (1)
7-9: Add full stops to list items for consistency.The guideline requires ending every sentence in a changeset with a full stop. The constraint list items should end with periods:
- ASCII only (no Unicode characters) - No newlines (`\n`, `\r`) - No tab characters (`\t`) + ASCII only (no Unicode characters). + No newlines (`\n`, `\r`). + No tab characters (`\t`).crates/biome_module_graph/src/format_module_graph.rs (1)
126-126: Inconsistent path qualification.The fully-qualified path is unnecessary here since
biome_formatter::prelude::*is imported at line 3. Usetoken("=>")to match the rest of the file.Apply this diff:
- biome_formatter::prelude::token("=>"), + token("=>"),crates/biome_css_semantic/src/format_semantic_model.rs (1)
128-137: Verify theto_string().as_str()pattern.Whilst CI is green, the pattern
self.0.to_string().as_str()(lines 129, 132, 135) typically creates lifetime issues in Rust, as the temporaryStringis dropped whilst still borrowed. If thewrite!macro handles this specially, it might be worth documenting. Otherwise, consider binding to variables:let s0 = self.0.to_string(); let s1 = self.1.to_string(); let s2 = self.2.to_string(); write!( f, [ token("("), text(s0.as_str(), TextSize::default()), token(","), space(), text(s1.as_str(), TextSize::default()), token(","), space(), text(s2.as_str(), TextSize::default()), token(")") ] )crates/biome_formatter/src/macros.rs (1)
91-91: Tidy the dbg example to match the new variant nameWe no longer print
StaticToken. Let’s update the blurb so future readers aren’t chasing ghosts.-/// // ^-- prints: [src/main.rs:7][0] = StaticToken("Hello") +/// // ^-- prints: [src/main.rs:7][0] = Token("Hello")
denbezrukov
force-pushed
the
dbezrukov/token-formatter
branch
from
e528443 to
5525b72
Compare
denbezrukov
force-pushed
the
dbezrukov/token-formatter
branch
from
5525b72 to
a58e46a
Compare
denbezrukov
changed the title
ematipico
approved these changes
Nov 3, 2025
|
@ematipico |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Port performance improvement from Ruff. astral-sh/ruff#7048
Refactored formatter to use strict
Tokenelement for better performance. The newTokenvariant is optimized for static, ASCII-only text (keywords, operators, punctuation) with the following constraints:\n,\r)\t)This enables faster printing and fitting logic by using bulk string operations (
push_str,len()) instead of character-by-character iteration with Unicode width calculations.The documentation changes were partly written by Claude Code
Test Plan
CI green
Docs