refactor(syntax): in-place lexer, lexical scan, Module/Comment nodes

[16bit-ykiko]

Summary

This round refactors the lexing layer and gives the semantics table first-class coverage of everything the AST and preprocessor callbacks fail to record.

Lexer

• In-place lexing: Lexer::from_line(content, offset) starts at the line containing offset; token ranges stay in file coordinates, so they compose directly with feature offsets. The hand-written rfind('\n') → substr → rebase dance at call sites is gone.
• Positional bool parameters replaced by LexerOptions; the dead ignore_end_of_directive flag and the unused is_directive_keyword helper removed.
• Header-name automation now covers the __has_include/__has_embed family (mode switches after the opening paren) and the #import directive; the residual "pick the filename argument" grammar moved next to its only consumer (document links).
• Retained comments are transparent to the directive state machine: a line-leading comment no longer consumes the start-of-line state or ends the module-declaration context.
• clang's NUL-terminated-buffer contract is documented and asserted: lex full buffers or suffixes, never a prefix slice; bound the lexing logically instead.

lexical_scan (new)

One pass over the file collects what neither the AST nor PPCallbacks report, as structured data: comments (range + line/block kind) and the three module-declaration forms (module;, [export] module x.y:z;, module :private;) with per-token ranges.

Semantics table: Module and Comment nodes

New Module and Comment node kinds are fed by the lexical scan at build time, cross-checked against the compiler before becoming nodes: named-module gate, DefinitionLoc anchor for the declaration form (macro-spelled names survive, disabled-branch duplicates die), spelled-token liveness for the private fragment. Module nodes own their written tokens like other directive nodes.

Consumers then deleted their own scanners:

• semantic_tokens: the module-declaration matcher and the per-request whole-file comment scan are gone; comments come precomputed from the table (build-time once instead of per request), module tokens paint from the node like imports do.
• tu_index: the module-declaration lexer state machine is gone; the occurrence is emitted from the validated table data, partition span included.

Fixes along the way

• #pragma region/endregion is classified by the first argument token instead of substring matching — a pragma merely mentioning "endregion" no longer closes a fold early (pinned by a new folding fixture).
• is_preamble_complete: a trailing comment no longer hides a terminating semicolon (or fakes one), and angled #import <...> completes.
• detect_completion_context rewritten on the lexer: comment-tolerant, exact keyword boundaries, and the cursor-at-line-start case no longer leaks the previous line.
• A wire-only divergence where the global module fragment lost its keyword highlight under a preamble PCH (its spelled token counts as preprocessed-away there) — gates are now per-form and the case is pinned by a PCH unit test.

Import-channel audit

The preprocessor callback channel and the AST ImportDecl channel agree on all three import forms (named, export-import, partition); pinned by a unit test. The preamble-PCH boundary remains the shared, by-design gap handled per feature by PreambleState.

Tests

• New unit coverage: lexer in-place lexing / header-name automation / incomplete input, lexical_scan with negative controls (identifier named module, strings, comments, disabled branches), semantics-table nodes and gates, module painting under a real PCH split, partition and implementation-unit indexing, preamble/completion edge cases including CRLF and cursor-mid-keyword.
• New folding fixture for the pragma classification fix; feature docs regenerated.
• All four suites green on both Debug and RelWithDebInfo; zero snapshot drift outside the new fixture.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: dbbc8db5-a4af-499b-a392-af8d7c07b841

📥 Commits

Reviewing files that changed from the base of the PR and between 46a664a and ccb83c2.

📒 Files selected for processing (2)

• src/syntax/scan.cpp
• tests/unit/syntax/scan_tests.cpp

🚧 Files skipped from review as they are similar to previous changes (1)

• src/syntax/scan.cpp

---

📝 Walkthrough

Walkthrough

The lexer now supports offset-aware tokenization and header-name directives. Lexical scans provide C++ module and comment data to semantic analysis. Semantic tokens, indexing, completion, document links, preamble detection, and pragma folding now use token-based data.

Changes

Lexer and semantic integration

Layer / File(s)	Summary
Lexer and lexical scanner foundation src/syntax/lexer.*, src/syntax/lexical_scan.*, tests/unit/syntax/*	The lexer gains configurable, offset-aware tokenization. The lexical scanner records comments and C++ module declarations.
Semantic lexical entities src/semantic/semantics.*, tests/unit/semantic/semantics_tests.cpp	Semantics owns lexical information and exposes module and comment nodes with token ownership and compiler-state filtering.
Semantic token and index consumers src/feature/semantic_tokens.cpp, src/index/tu_index.cpp, tests/unit/feature/semantic_tokens_tests.cpp, tests/unit/index/tu_index_tests.cpp	Semantic tokens and indexing consume module and comment data. Module names, partitions, imports, definitions, and references receive updated handling.
Directive, completion, and preamble parsing src/compile/directive.cpp, src/feature/document_links.cpp, src/syntax/completion.cpp, src/syntax/scan.cpp, tests/unit/feature/document_link_tests.cpp, tests/unit/syntax/*	Directive classification, document-link argument lookup, completion detection, and preamble checks use lexer tokens.
Folding classification validation docs/en/features/folding-ranges.md, tests/snap/folding_range/*	The folding fixtures document first-argument pragma classification and update fold-order values.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

• clice-io/clice#417: Adds earlier lexer-based module semantic-token handling.
• clice-io/clice#421: Provides the directive-argument scanning approach refactored by this change.
• clice-io/clice#535: Shares folding-range documentation and snapshot fixture changes.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 13.21% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main syntax lexer refactor and addition of lexical scan and Module/Comment nodes.
Description check	✅ Passed	The description directly explains the lexer refactor, lexical scanning, semantics nodes, fixes, and related tests.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch refactor/lexer-module-semantics

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e7fdbc76ae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (4)

src/syntax/lexer.h (1)

20-46: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Document the lang_opts lifetime requirement.

clang::Lexer keeps a reference to the LangOptions object. The pointed-to object must therefore outlive the Lexer. The NUL-termination contract is documented; add the same explicit note for lang_opts so callers do not pass a temporary.

📝 Proposed doc addition

 struct LexerOptions {
     /// Emit comment tokens instead of dropping them.
     bool keep_comments = false;
 
+    /// Borrowed, not copied: clang's lexer keeps a reference to it, so the
+    /// pointee must outlive the Lexer. Null selects a default instance.
     const clang::LangOptions* lang_opts = nullptr;
 };

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/syntax/lexer.h` around lines 20 - 46, Update the LexerOptions
documentation for lang_opts to state that the referenced clang::LangOptions
object must outlive the Lexer and must not be provided as a temporary, matching
the explicit lifetime guidance style used for Lexer content.

src/syntax/lexical_scan.h (1)

31-60: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Give ModuleDeclaration::kind a default value.

lexical_scan declares ModuleDeclaration decl; and assigns kind only on the paths that push the record, so the current code is correct. A future path that pushes a record before assigning kind would store an indeterminate value, and src/index/tu_index.cpp switches on it. A default initializer removes that risk at no cost.

♻️ Proposed change

-        Kind kind;
+        Kind kind = Kind::Declaration;

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/syntax/lexical_scan.h` around lines 31 - 60, Initialize
ModuleDeclaration::kind with a safe default enum value at its declaration,
preserving the existing assignments in lexical_scan and ensuring any record
pushed before explicit assignment has a determinate kind for consumers such as
the switch in tu_index.cpp.

src/semantic/semantics.h (1)

377-380: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

State the mutation invariant next to the storage.

Move-only construction protects the payload pointers against copies, and moving the two std::vector members keeps element addresses stable. The remaining hazard is a later push_back or erase on lexical.comments or lexical.modules after append_directives recorded pointers into them. SemanticsBuilder is the only friend, so this is enforceable today; record the invariant so it stays that way.

📝 Proposed doc addition

     /// The lexically scanned entities; Module and Comment nodes point into
-    /// this storage.
+    /// this storage. Both vectors must not be resized after the nodes are
+    /// built (the builder scans and filters before it takes any pointer).
     LexicalInfo lexical;

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/semantic/semantics.h` around lines 377 - 380, Add documentation next to
the LexicalInfo lexical storage member stating that after append_directives
records payload pointers, lexical.comments and lexical.modules must not be
mutated with push_back, erase, or equivalent operations. Note that
SemanticsBuilder is responsible for preserving this invariant and keep the
existing storage declaration unchanged.

src/syntax/lexer.cpp (1)

83-95: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Correct the comment: export import also reaches this branch.

Line 71 sets parse_pp_keyword = true for export at the start of a line while module_declaration_context holds. The next token then enters this branch. For export import <vector>; or export import "hdr.h"; the header-name mode is in fact what you want, and a plain module name still lexes as an identifier. So the behavior is fine, but the claim "a module import never sets parse_pp_keyword" is inaccurate and will mislead the next reader.

📝 Proposed comment fix

-        // `import` here is the directive form (`#import`), which takes a
-        // filename; a module import never sets parse_pp_keyword.
+        // `import` here is either the directive form (`#import`) or the
+        // `export import` opener; both take a header name when one is
+        // written, and a plain module name still lexes as an identifier.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/syntax/lexer.cpp` around lines 83 - 95, Correct the comment above the
parse_header_name assignment to acknowledge that export import can also reach
this branch and that header-name mode is intentional for directive-style and
exported imports, while plain module names remain identifiers. Do not change the
existing logic.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/feature/document_links.cpp`:
- Around line 29-36: Add "import" to the recognized keyword comparisons in
find_directive_argument so `#import` directives set after_keyword before locating
their filename argument. Preserve the existing handling for all other directive
keywords.

In `@tests/unit/syntax/lexer_tests.cpp`:
- Around line 152-161: In the MacroArgument test, assert that tokens contains
the expected number of entries before indexing tokens[2], matching the guard
used by the IncompleteInput cases. Also protect the tokens.back() access near
the related test by asserting the vector is non-empty before using it.

---

Nitpick comments:
In `@src/semantic/semantics.h`:
- Around line 377-380: Add documentation next to the LexicalInfo lexical storage
member stating that after append_directives records payload pointers,
lexical.comments and lexical.modules must not be mutated with push_back, erase,
or equivalent operations. Note that SemanticsBuilder is responsible for
preserving this invariant and keep the existing storage declaration unchanged.

In `@src/syntax/lexer.cpp`:
- Around line 83-95: Correct the comment above the parse_header_name assignment
to acknowledge that export import can also reach this branch and that
header-name mode is intentional for directive-style and exported imports, while
plain module names remain identifiers. Do not change the existing logic.

In `@src/syntax/lexer.h`:
- Around line 20-46: Update the LexerOptions documentation for lang_opts to
state that the referenced clang::LangOptions object must outlive the Lexer and
must not be provided as a temporary, matching the explicit lifetime guidance
style used for Lexer content.

In `@src/syntax/lexical_scan.h`:
- Around line 31-60: Initialize ModuleDeclaration::kind with a safe default enum
value at its declaration, preserving the existing assignments in lexical_scan
and ensuring any record pushed before explicit assignment has a determinate kind
for consumers such as the switch in tu_index.cpp.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 05c94cc1-d07c-4620-95e0-e532f55270ce

📥 Commits

Reviewing files that changed from the base of the PR and between cf358a7 and e7fdbc7.

📒 Files selected for processing (32)

• docs/en/features/folding-ranges.md
• src/compile/directive.cpp
• src/feature/document_links.cpp
• src/feature/semantic_tokens.cpp
• src/index/tu_index.cpp
• src/semantic/semantics.cpp
• src/semantic/semantics.h
• src/syntax/completion.cpp
• src/syntax/lexer.cpp
• src/syntax/lexer.h
• src/syntax/lexical_scan.cpp
• src/syntax/lexical_scan.h
• src/syntax/scan.cpp
• tests/snap/folding_range/abbreviated_function_template.cpp
• tests/snap/folding_range/comment_folding.cpp
• tests/snap/folding_range/coroutine_body.cpp
• tests/snap/folding_range/include_region.cpp
• tests/snap/folding_range/initializer_list_construction.cpp
• tests/snap/folding_range/macro_folding.cpp
• tests/snap/folding_range/pragma_classification.cpp
• tests/snap/folding_range/pragma_classification.snap.yml
• tests/snap/folding_range/raw_string_literal.cpp
• tests/snap/folding_range/template_instantiations.cpp
• tests/snap/folding_range/template_parameter_list.cpp
• tests/snap/folding_range/using_declaration_block.cpp
• tests/unit/feature/semantic_tokens_tests.cpp
• tests/unit/index/tu_index_tests.cpp
• tests/unit/semantic/semantics_tests.cpp
• tests/unit/syntax/completion_tests.cpp
• tests/unit/syntax/lexer_tests.cpp
• tests/unit/syntax/lexical_scan_tests.cpp
• tests/unit/syntax/scan_tests.cpp

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 987286396f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7868d39db1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 46a664a55f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".