Update julia grammar #520
Merged
Update julia grammar #520
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Update the Julia grammar to tree-sitter-julia#135. That PR introduced several breaking changes, including changes in the way function signatures and parameters were parsed.
|
|
aryx
reviewed
Jan 9, 2025
| ), | ||
|
|
||
| _statement: ($, previous) => choice( | ||
| previous, | ||
| $.semgrep_ellipsis, | ||
| ), | ||
|
|
||
| typed_parameter: ($, previous) => choice( |
There was a problem hiding this comment.
Why removing this? We can still have ellpisis in type parameters even with this removal?
There was a problem hiding this comment.
Julia doesn't have a distinction between expressions and patterns, so function signatures are parsed as function calls, and checked in the lowering phase (after macro expansion). Having these parameter rules added a lot of duplication and conflicts, so we removed them.
We can still have ellpisis in type parameters even with this removal?
Yes. Ellipsis would now be parsed as splat_expression.
|
Hey, apologies for the delay here. I can take responsibility for this review going forward. The changes here look good, but do you plan to integrate these into Semgrep as well? We likely can't accept this PR otherwise. |
|
Yes, I'm currently working on the changes to semgrep itself. I opened semgrep/semgrep#10820, tho there's still some work to do there. |
nmote
approved these changes
Mar 11, 2025
There was a problem hiding this comment.
Looks good! I've released this as semgrep/semgrep-julia@f973ebd.
We will merge this once the associated Semgrep PR is merged.
The `_statement` rule doesn't need to be extended because statements are expressions and the `_expression` rule is already extended.
|
Hey @nmote, I added a fix for a bug in the extended grammar: It was including Could you please make a new release? |
|
Done! |
nmote
added a commit
to semgrep/semgrep
that referenced
this pull request
May 1, 2025
This PR upgrades support for Julia to a more recent version of tree-sitter-julia. - See tree-sitter/tree-sitter-julia#135 for the changes to the parser - See semgrep/ocaml-tree-sitter-semgrep#520 for the changes to semgrep-julia Some notes: - The changes to tree-sitter-julia in the linked PR were focused on producing parse trees more faithful to JuliaSyntax (the default Julia parser). However, JuliaSyntax delegates a lot of work to the lowering phase, like checking function signatures. This means that while tree-sitter produces a smaller parser, semgrep now needs to do more work to find a function's name and parameters. - Since parameters are now parsed as expressions, I left a lot of `todo` for expressions that don't constitute valid "patterns". I don't know if it might be a good idea to create an alias for `todo` to clearly indicate that we don't handle those cases because they're not valid in normal Julia (outside macros), and not because they should be handled in the future. - tree-sitter-julia now parses short functions as assignments (That's also how it's done in the reference parser). To handle this, I translate short functions into lambdas: `f(...) = ...` becomes `f = (...) -> ...`. - The special handling for `where_clause` was removed. A later version of tree-sitter-julia unified `where_clause` with `where_expression`. - One of the issues I really hoped would be fixed was #10649, but that's actually a limitation on semgrep's side! In general, I still feel there's a lot of weird hacks because semgrep's AST only considers C-like macros. --------- Co-authored-by: Nat Mote <[email protected]>
semgrep-ci bot
added a commit
to semgrep/semgrep
that referenced
this pull request
May 5, 2025
…emgrep/semgrep-proprietary#3802) This PR upgrades support for Julia to a more recent version of tree-sitter-julia. - See tree-sitter/tree-sitter-julia#135 for the changes to the parser - See semgrep/ocaml-tree-sitter-semgrep#520 for the changes to semgrep-julia Some notes: - The changes to tree-sitter-julia in the linked PR were focused on producing parse trees more faithful to JuliaSyntax (the default Julia parser). However, JuliaSyntax delegates a lot of work to the lowering phase, like checking function signatures. This means that while tree-sitter produces a smaller parser, semgrep now needs to do more work to find a function's name and parameters. - Since parameters are now parsed as expressions, I left a lot of `todo` for expressions that don't constitute valid "patterns". I don't know if it might be a good idea to create an alias for `todo` to clearly indicate that we don't handle those cases because they're not valid in normal Julia (outside macros), and not because they should be handled in the future. - tree-sitter-julia now parses short functions as assignments (That's also how it's done in the reference parser). To handle this, I translate short functions into lambdas: `f(...) = ...` becomes `f = (...) -> ...`. - The special handling for `where_clause` was removed. A later version of tree-sitter-julia unified `where_clause` with `where_expression`. - One of the issues I really hoped would be fixed was #10649, but that's actually a limitation on semgrep's side! In general, I still feel there's a lot of weird hacks because semgrep's AST only considers C-like macros. --------- Co-authored-by: Nat Mote <[email protected]> synced from OSS 06525d0 Co-authored-by: Sergio A. Vargas <[email protected]> synced from Pro 7029d9d64eb5cdd2835b9eb238b02a84fe7ec17b
semgrep-ci bot
added a commit
to semgrep/semgrep
that referenced
this pull request
May 6, 2025
…emgrep/semgrep-proprietary#3802) This PR upgrades support for Julia to a more recent version of tree-sitter-julia. - See tree-sitter/tree-sitter-julia#135 for the changes to the parser - See semgrep/ocaml-tree-sitter-semgrep#520 for the changes to semgrep-julia Some notes: - The changes to tree-sitter-julia in the linked PR were focused on producing parse trees more faithful to JuliaSyntax (the default Julia parser). However, JuliaSyntax delegates a lot of work to the lowering phase, like checking function signatures. This means that while tree-sitter produces a smaller parser, semgrep now needs to do more work to find a function's name and parameters. - Since parameters are now parsed as expressions, I left a lot of `todo` for expressions that don't constitute valid "patterns". I don't know if it might be a good idea to create an alias for `todo` to clearly indicate that we don't handle those cases because they're not valid in normal Julia (outside macros), and not because they should be handled in the future. - tree-sitter-julia now parses short functions as assignments (That's also how it's done in the reference parser). To handle this, I translate short functions into lambdas: `f(...) = ...` becomes `f = (...) -> ...`. - The special handling for `where_clause` was removed. A later version of tree-sitter-julia unified `where_clause` with `where_expression`. - One of the issues I really hoped would be fixed was #10649, but that's actually a limitation on semgrep's side! In general, I still feel there's a lot of weird hacks because semgrep's AST only considers C-like macros. --------- Co-authored-by: Nat Mote <[email protected]> synced from OSS 06525d0 Co-authored-by: Sergio A. Vargas <[email protected]> synced from Pro 7029d9d64eb5cdd2835b9eb238b02a84fe7ec17b
nmote
pushed a commit
to semgrep/semgrep
that referenced
this pull request
May 6, 2025
…emgrep/semgrep-proprietary#3802) This PR upgrades support for Julia to a more recent version of tree-sitter-julia. - See tree-sitter/tree-sitter-julia#135 for the changes to the parser - See semgrep/ocaml-tree-sitter-semgrep#520 for the changes to semgrep-julia Some notes: - The changes to tree-sitter-julia in the linked PR were focused on producing parse trees more faithful to JuliaSyntax (the default Julia parser). However, JuliaSyntax delegates a lot of work to the lowering phase, like checking function signatures. This means that while tree-sitter produces a smaller parser, semgrep now needs to do more work to find a function's name and parameters. - Since parameters are now parsed as expressions, I left a lot of `todo` for expressions that don't constitute valid "patterns". I don't know if it might be a good idea to create an alias for `todo` to clearly indicate that we don't handle those cases because they're not valid in normal Julia (outside macros), and not because they should be handled in the future. - tree-sitter-julia now parses short functions as assignments (That's also how it's done in the reference parser). To handle this, I translate short functions into lambdas: `f(...) = ...` becomes `f = (...) -> ...`. - The special handling for `where_clause` was removed. A later version of tree-sitter-julia unified `where_clause` with `where_expression`. - One of the issues I really hoped would be fixed was #10649, but that's actually a limitation on semgrep's side! In general, I still feel there's a lot of weird hacks because semgrep's AST only considers C-like macros. --------- Co-authored-by: Nat Mote <[email protected]> synced from OSS 06525d0 Co-authored-by: Sergio A. Vargas <[email protected]> synced from Pro 7029d9d64eb5cdd2835b9eb238b02a84fe7ec17b
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update the Julia grammar to tree-sitter/tree-sitter-julia#135. That PR introduced several breaking changes, including changes in the way function signatures and parameters were parsed.
(This should reduce the size of the
parser.cin semgrep-julia by about 15MB).Checklist