Add more keywords.

[huacnlee]

Close #1

https://github.com/pest-parser/pest/blob/d2dd16c5bf361eaaf36b4f7b9657ee48252883f3/meta/src/validator.rs#L47-L67

Summary by CodeRabbit

• New Features

  • Highlighting and parsing expanded to recognize additional query tokens (new peek/pop/drop and end/newline markers) for more complete visuals.

• Bug Fixes

  • Improved token/parse handling and lexer behavior to reduce mis-highlighting and edge-case errors in complex queries, strings, escapes, and comments.

• Chores

  • Updated developer dependencies and build output paths for improved tooling compatibility.

• Documentation

  • Streamlined contributor guidance and package metadata wording/keywords.

[coderabbitai]

Caution

Review failed

The pull request is closed.

Walkthrough

Expanded token/keyword set across highlighting, grammar, node-types, and parser: added DROP, PEEK_ALL, POP_ALL (with reorderings); updated grammar/builtin tokens and parser tables/lexer states; bumped parser symbol/token counts; minor package and dependency changes.

Changes

Cohort / File(s)	Summary
Syntax highlighting queries/highlights.scm	Expanded and reordered keyword list to include ANY, DROP, EOI, NEWLINE, PEEK_ALL, POP_ALL, while retaining PEEK/POP/PUSH/SOI.
Grammar / builtin rules grammar.js, src/grammar.json	Broadened builtin token alternatives to accept DROP, PEEK_ALL, POP_ALL, and NEWLINE alongside existing tokens (ANY, SOI, EOI); ordering adjusted.
Node types src/node-types.json	Added new node-type entries: DROP, PEEK_ALL, and POP_ALL (each "named": false).
Parser implementation (C) src/parser.c	Large-scale parser/lexer table updates: increased state/symbol/token counts; replaced/inserted tokens (DROP, PEEK_ALL, POP_ALL / reworked SOI placement); updated ts_symbol_* arrays, lex state transitions, parse tables, and public symbol mappings.
Package / config package.json, Cargo.toml, CONTRIBUTING.md	Dependency bumps (devDependencies.node-gyp → ^10.3.1, nan → ^2.23.0); dev script output path changed (target/out.html → build/out.html); package.json keywords/formatting adjusted; Cargo.toml description and keywords updated; CONTRIBUTING adjusted init/test ordering.

Sequence Diagram(s)

sequenceDiagram
    participant Src as Source text
    participant Lexer as Lexer (tokenizer)
    participant Parser as Parser (parse tables)
    participant AST as AST / Node types

    note over Lexer,Parser #e6f2ff: Lex rules & token set updated (DROP, PEEK_ALL, POP_ALL, NEWLINE)
    Src->>Lexer: scan input
    alt matches new builtin token
        Lexer-->>Parser: emit DROP / PEEK_ALL / POP_ALL / NEWLINE
    else matches existing token
        Lexer-->>Parser: emit existing token
    end
    Parser->>AST: map tokens via updated ts_symbol_map
    AST-->>Parser: produce nodes (includes new node-types)
    Parser-->>Src: parse result or recovery (tables/lex states updated)

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

I nibble at tokens, ears alert and bright,
DROP and PEEK_ALL hop into the light.
POP_ALL joins the tumble, soft and small,
Parser tables shuffle, answering the call.
A tiny rabbit cheers — the grammar grows tall. 🐇

Pre-merge checks and finishing touches

❌ Failed checks (3 warnings)

Check name	Status	Explanation	Resolution
Linked Issues Check	⚠️ Warning	The pull request adds the DROP, PEEK_ALL, and POP_ALL keywords as requested by issue #1, but it fails to include the PUSH_LITERAL keyword explicitly named in the issue, so it does not fully satisfy the issue’s requirements.	Please add the missing PUSH_LITERAL keyword across all affected files (syntax definitions, grammar rules, node types, and parser tables) to fully address issue #1.
Out of Scope Changes Check	⚠️ Warning	The pull request includes updates to package.json dependencies (node-gyp and nan) which are unrelated to the keyword additions specified in the linked issue, indicating changes outside the defined scope.	Remove or relocate the dependency version updates into a separate pull request to keep this change focused solely on adding the specified keywords.
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The title clearly and succinctly summarizes the main change of the pull request, which is to add additional keywords to the grammar and parser, and it avoids vague or unrelated terms.

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2039d54 and 3433439.

📒 Files selected for processing (3)

• CONTRIBUTING.md (1 hunks)
• Cargo.toml (1 hunks)
• package.json (2 hunks)

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a8a98a8 and 49a8aeb.

📒 Files selected for processing (1)

• queries/highlights.scm (1 hunks)

🧰 Additional context used

🪛 GitHub Actions: CI

queries/highlights.scm

[error] 50-50: tree-sitter test failed: Query error in highlights.scm at line 50:4 - Invalid node type DROP.

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (3)

src/node-types.json (1)

246-273: Node type set still omits PUSH_LITERAL

Since PUSH_LITERAL remains absent from the grammar, there’s no corresponding node-type entry here either. Once you add the grammar/lexer support, remember to extend this list so downstream tooling exposes the keyword properly.

src/grammar.json (1)

456-482: Add the PUSH_LITERAL alternative

PUSH_LITERAL is still absent from the builtin choice, so grammars using that keyword won’t parse. Please add it alongside DROP, PEEK_ALL, and POP_ALL, and regenerate the parser afterward to keep everything in sync.

src/parser.c (1)

20-86: Regenerate parser with PUSH_LITERAL support

The regenerated parser still lacks any symbol/lexer entry for PUSH_LITERAL, so even after updating the grammar the parser must be regenerated to emit the new token (enum, lex branches, parse tables). Once you add the keyword in the grammar, please rerun tree-sitter generate so this file picks up the new symbol map and transitions.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 49a8aeb and 2039d54.

⛔ Files ignored due to path filters (1)

• bun.lockb is excluded by !**/bun.lockb

📒 Files selected for processing (6)

• grammar.js (1 hunks)
• package.json (1 hunks)
• queries/highlights.scm (1 hunks)
• src/grammar.json (2 hunks)
• src/node-types.json (2 hunks)
• src/parser.c (96 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

• queries/highlights.scm