unified: Switch over to using swift-syntax for parsing#22233
Draft
tausbn wants to merge 27 commits into
Draft
Conversation
The AST dump previously emitted named fields in field-id order, which made it dependant on registration order and so it could differ between front-ends. We now emit them in the order declared in the node-types YAML instead, so that the order is kept stable.
In our swift-syntax wrapper, we now attempt to fold all operator sequences (i.e. `sequenceExpr` nodes) into appropriate `infixOperatorExpr` nodes, assuming the requisite operator definitions are present. Currently, we only consider operators that are defined in the standard library, and operators that are defined in the current file, leaving operators defined in separate modules as future work. The folding is done maximally -- if an argument of an unknown operator can be folded in isolation, then this is done. Each top-level sequence is folded independently, so a single unknown operator leaves only its own sequence flat rather than aborting folding elsewhere.
Prepare yeast for front-ends that do not parse with tree-sitter (e.g. the swift-syntax front-end, whose parser hands us a ready-built `yeast::Ast`): - `Runner`/`ConcreteDesugarer` now hold `Option<tree_sitter::Language>`. New constructors `Runner::with_schema_no_language`, `ConcreteDesugarer::without_language`, and `DesugaringConfig::build_schema_no_language` build the schema from the output node-types YAML alone. The parsing entry points (`run`/`run_from_tree`) error when no language is present; `run_from_ast` needs none. - `BuildCtx::source_text` is a small convenience for Rust-block rules that read a captured token's source text. - AST-dump type validation now resolves field constraints and required fields by field NAME rather than by field id. A field id is local to the schema that assigned it, so an AST built by one schema (e.g. an external parser's adapter) could not be validated against another (the output node-types schema) without re-keying. Looking up by name keeps the two schemas full independent: they share field names, not ids. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously the `simple` multi-language extractor carried an optional
desugarer, so every language (including plain tree-sitter ones such as
ql, dbscheme, json and blame) went through the same desugaring-aware
extraction path.
This commit splits it into two front-ends that share a private driver:
- `simple`: pure tree-sitter extraction with no desugaring. Comments
and other `extra` nodes are emitted inline as tokens. (The extractor
then extracts these as usual.)
- `desugaring`: parses source into a `ParsedTree` (a yeast AST plus
side-channel `extra` tokens) and rewrites the AST through a
`yeast::Desugarer` before extraction. The parser is a closure, so
both tree-sitter grammars (via `tree_sitter_parser`) and custom
parsers plug in the same way.
The shared multi-file plumbing (threading, glob matching, source-archive
copying, TRAP writing) lives in a new private `driver` module behind a
`LanguageExtractor` trait, so neither front-end duplicates it.
`extract` no longer takes an optional desugarer (it always walks the
parse tree directly); `extract_parsed` takes a required desugarer. ql
and ruby use the direct path; the unified Swift extractor uses the
desugaring path.
Also rename the new side-channel identifiers from "trivia" to "extra"
(ExtraToken, ParsedTree.extras, emit_extra, ...) to match tree-sitter's
own `is_extra()` terminology. The pre-existing `*_trivia_tokeninfo`
relation is left unchanged for a separate change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… (dormant) Adds the plumbing for the swift-syntax front-end, without yet switching the runtime over to it (the Swift front-end still parses with tree-sitter): - `languages/swift/parse.rs` shells out to the separate `swift-syntax-parse` binary and adapts its JSON into a `yeast::Ast` (plus side-channel `extra` tokens) via `swift_adapter`. Running the parser out-of-process keeps the Swift toolchain out of the extractor's own build. It is wired in as a module but left `allow(dead_code)` until the runtime uses it. - `swift_node_types.yml` is the authoritative swift-syntax input schema (generated from swift-syntax by a one-off tool). The adapter seeds every parse with it, pre-registering every input kind and field so that rule matching never references a name absent from a given file's tree. The adapter now emits `ExtraToken`s directly during its single traversal, so `parse.rs` hands the parsed tree straight through with no second pass. - `ast_types.yml` gains an `unresolved_operator_sequence` type (with an `expr_or_operator` union) for flat operator chains the parser can't resolve — e.g. a chain using an operator imported from another module, whose precedence is unknown. Nothing produces it yet; the mapping rules that do are added when the rules are ported. The dbscheme and QL library are regenerated to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget the top-level, literal, and name mapping rules from the tree-sitter grammar to the swift-syntax AST. The output each rule builds is unchanged; only the input pattern differs, reflecting the different AST shape: - `source_file` -> `sourceFile` (statements live in an elided `statements` collection of `codeBlockItem` wrappers); the tree-sitter `global_declaration` / `local_declaration` wrappers have no swift-syntax counterpart. - The lexical integer/string variants (`hex_literal`, `oct_literal`, `multi_line_string_literal`, ...) collapse into single `integerLiteralExpr` / `stringLiteralExpr` kinds. - `simple_identifier` and `referenceable_operator` both become `declReferenceExpr` (its `baseName` is the referenced name or operator). This is the first step of an in-place, rule-by-rule migration; intermediate commits do not pass the corpus test (the runtime front-end is still tree-sitter) — the corpus is regenerated once at the end. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget operator handling to the swift-syntax AST. Because Swift's grammar has no operator precedence, the parser front-end folds operator chains into nested `infixOperatorExpr`s (see swift-syntax-rs), so a single rule replaces the tree-sitter grammar's per-precedence binary rules (additive, multiplicative, comparison, equality, conjunction, disjunction, bitwise, range, nil-coalescing). The output is unchanged; only the input matching differs: - `binaryOperatorExpr` unwraps to the `infix_operator` leaf. - A `binaryOperator`-based `infixOperatorExpr` becomes `binary_expr`, or `compound_assign_expr` when the operator's spelling is a compound assignment — merging the tree-sitter grammar's separate binary and compound-assignment rules (the operator kinds are structurally identical, distinguishable only by spelling). - An `assignmentExpr`-based `infixOperatorExpr` becomes `assign_expr`. - An unresolved chain stays a flat `sequenceExpr` -> `unresolved_operator_sequence`. - `prefixOperatorExpr` -> prefix `unary_expr`; `tupleExpr` -> opaque `tuple_expr`; `codeBlock` -> `block`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget `let`/`var` bindings to the swift-syntax AST, preserving the output: - A `variableDecl` publishes its `bindingSpecifier` (`let`/`var`) as the binding modifier, followed by its attributes and modifiers (`@objc`, `public`, `static`, …), and flattens each `patternBinding` into its own `variable_declaration`, tagging non-first ones `chained_declaration`. - One `patternBinding` rule with optional `typeAnnotation`/`initializer` covers `let x`, `let x = e`, `let x: T`, and `let x: T = e`. - `identifierPattern` -> `name_pattern`; `tuplePattern` / `tuplePatternElement` -> `tuple_pattern` / `pattern_element` (tuple destructuring), carrying an optional element label through as the `pattern_element` key. - `codeBlockItem` now captures `_*` / annotates `stmt*` so a multi-binding declaration splices as several statements. - Add a `declModifier` -> `modifier` rule (swift-syntax unifies the visibility/function/member/mutation/ownership modifiers into one kind). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget type expressions to the swift-syntax AST, output unchanged: - `user_type` -> `identifierType` (its `name` is the type-name token). - The sugared types keep desugaring to `generic_type_expr`: `optionalType` -> Optional<T>, `arrayType` -> Array<T>, `dictionaryType` -> Dictionary<K, V>. - A generic type with explicit arguments (`Set<Int>`) stays opaque (its whole source text as the name), matched before the plain `identifierType` rule. This matches the tree-sitter `user_type` rule, which was also opaque. - Tuple types (`(Int, String)`) -> `tuple_type_expr` and function types (`(Int) -> Bool`) -> `function_type_expr`. swift-syntax holds both as `tupleTypeElement`s but a tuple element maps to `tuple_type_element` while a function parameter maps to `parameter`; the containers set `SwiftContext::in_function_type` for their direct children so the shared `tupleTypeElement` rule emits the right kind (and nested types stay correct). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget function declarations, calls, member access, and control transfer to the swift-syntax AST, output unchanged: - `functionDecl` -> `function_declaration` (parameters and return type nest under `signature`; the body is a `codeBlock`). A bodyless function (a protocol requirement) still emits an empty `block`. - `functionParameter` -> `parameter`: two names give the external label and internal name, one name just the internal name; the default value is handled inline, so the `ctx.default_value` threading (and its `SwiftContext` field) is removed. The declared type is dropped: in the tree-sitter path the untyped-parameter rule was ordered first and shadowed the typed one, so the baseline emits no parameter type. - A function reference spelled with argument labels (`f(x:y:z:)`) is a `declReferenceExpr` with `argumentNames`; it is mapped to `unsupported_node` (matched before the bare-name rule) so downstream QL isn't handed a malformed reference, as in the tree-sitter path. - `functionCallExpr` -> `call_expr` (a trailing closure becomes a final unlabelled argument); `labeledExpr` -> `argument`; `memberAccessExpr` -> `member_access_expr` (base-ful matched before leading-dot). - `returnStmt`/`breakStmt`/`continueStmt`/`throwStmt` -> `return_expr`/`break_expr`/`continue_expr`/`throw_expr`, collapsing the labelled/unlabelled variants via optional captures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget closures to the swift-syntax AST, output unchanged. swift-syntax nests the whole closure header under an optional `closureSignature`, so a single `closureExpr` rule (with optional attributes, capture list, parameter clause, and return clause) replaces the tree-sitter `lambda_literal` rule. The parameter clause is a union of the parenthesised form (`closureParameterClause`, unwrapped to its `closureParameter` children) and the shorthand form (`closureShorthandParameter`, a bare name); one rule each replaces the four `lambda_parameter` variants. `closureCapture` -> `variable_declaration` (an optional ownership specifier becomes a modifier; an explicit capture initializer becomes the bound value). The trailing-closure call form is already handled by the `functionCallExpr` `trailingClosure` variant, so the tree-sitter trailing-closure call rule is dropped. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget `if`/`guard`/`switch`/ternary and the case/binding patterns to the swift-syntax AST, output unchanged. swift-syntax distinguishes a binding pattern (`valueBindingPattern`, `let x`) from a match pattern (`expressionPattern`, `someConstant`) by node kind, so the tree-sitter path's context-based `in_binding_pattern` disambiguation is removed: - `ifExpr`/`guardStmt`/`ternaryExpr` -> `if_expr`/`guard_if_stmt`/`if_expr`; `switchExpr` + `switchCase` -> `switch_expr` + `switch_case` (comma cases become an `or_pattern`); `conditionElement`/`switchCaseItem` unwrap; a statement-position `if`/`switch`/`do` is unwrapped from its `expressionStmt`. - `optionalBindingCondition` (`if let`) and `matchingPatternCondition` (`if case`) -> `pattern_guard_expr`. - An `expressionPattern` wrapping a leading-dot or qualified call -> `constructor_pattern` (setting `ctx.in_pattern`); `valueBindingPattern` unwraps; a bare `expressionPattern` -> `expr_equality_pattern`; a wildcard (`discardAssignmentExpr`) -> `ignore_pattern`; a `tupleExpr` match pattern -> `tuple_pattern`; `isTypePattern` (`case is T`) -> `unsupported_node`. - The `labeledExpr` argument rules gain `in_pattern`-aware pattern variants so an enum-case pattern's arguments (`case .foo(let x, _)`) become `pattern_element`s. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget loops to the swift-syntax AST, output unchanged: `forStmt` -> `for_each_stmt` (the optional `where` clause becomes the `guard`), `whileStmt` -> `while_stmt`, `repeatStmt` -> `do_while_stmt`, and `labeledStmt` -> `labeled_stmt`. Unlike the tree-sitter grammar, swift-syntax stores a labeled statement's label and colon as separate tokens, so the label token is already the bare name (no trailing `:` to strip). A `repeat`-`while` loop has a single condition in swift-syntax (not a condition list), so it needs no `and_chain`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget collection literals and subscripts to the swift-syntax AST, output unchanged: `arrayExpr` -> `array_literal` (each `arrayElement` unwraps to its expression); `dictionaryExpr` -> an opaque `map_literal` leaf (its source span, matching the tree-sitter path); and `subscriptCallExpr` (`xs[0]`) -> `call_expr`, mirroring the tree-sitter grammar's treatment of a subscript as a call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget optional chaining, `try`, `do`/`catch`, casts, type tests, `await`, and force-unwrap to the swift-syntax AST, output unchanged: - `optionalChainingExpr` (`x?`) unwraps transparently (the enclosing member access / call carries the semantics). - `tryExpr` -> prefix `unary_expr`; swift-syntax splits the operator into a `try` keyword and an optional `?`/`!` mark, recombined into one `prefix_operator` spelling. - `doStmt` -> `try_expr` with `catchClause` -> `catch_clause`; a `catch` binds the first `catchItem`'s pattern and optional `where` guard. - `asExpr` (`x as`/`as?`/`as!` `T`) -> `type_cast_expr`, `isExpr` (`x is T`) -> `type_test_expr`, and `awaitExpr` -> prefix `unary_expr`. - `forceUnwrapExpr` (`x!`) -> postfix `unary_expr` (swift-syntax has a dedicated node; the tree-sitter path used the generic postfix operator). A couple of rewritten rules keep their opening inlined so this commit's diff stays readable; a later commit restores the canonical formatting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget imports to the swift-syntax AST, output unchanged. swift-syntax represents the dotted path as a list of `importPathComponent`s, folded into a `name_expr`/`member_access_expr` chain via `member_chain`. A single rule handles both forms via an optional `importKindSpecifier`: a scoped import (`import struct Foo.Bar`) has one and binds the last path component as a `name_pattern`; a plain import (`import Foundation`) has none and uses a `bulk_importing_pattern`. Leading attributes (`@_exported`) and access modifiers (`public`) become `modifier`s. The tree-sitter multi-part `identifier` rule is dropped: swift-syntax qualified names are already `memberAccessExpr` chains, and import paths are `importPathComponent`s. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget the nominal type declarations to the swift-syntax AST, output unchanged. `classDecl`/`structDecl`/`enumDecl`/`protocolDecl`/`extensionDecl` each become a `class_like_declaration` tagged with a modifier naming the declaration keyword, with members drawn from the `memberBlock`; each `memberBlockItem` unwraps to its contained declaration. `superExpr` maps to `super_expr`; `self` needs no rule (swift-syntax models it as an ordinary `declReferenceExpr`, already a `name_expr`). Following the tree-sitter path (PARITY), the inheritance clause is not emitted as a `base_type` (the tree-sitter rule captured it positionally, but the grammar nests it under a field, so it never actually matched); swift-syntax exposes it cleanly, so that is a correctness improvement to make once tree-sitter is retired. The tree-sitter grammar's standalone `self`, dead modifier (`visibility_modifier`, etc.), key-path, and inheritance-specifier rules are dropped; `#selector`/`#keyPath` (now a `macroExpansionExpr`) stays an `unsupported_node`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget property accessors to the swift-syntax AST, output unchanged.
An accessor-bearing `variableDecl` publishes the property name/type into
`ctx`; a computed property (`var v: T { get set }`) emits accessors
carrying the type, while a stored property with observers (`var x = e {
didSet {…} }`) emits the backing `variable_declaration` first.
swift-syntax models get/set/willSet/didSet uniformly as `accessorDecl`,
so a single `accessorDecl` rule — with an optional body distinguishing a
computed accessor from a bodyless protocol requirement — replaces the
tree-sitter grammar's separate computed-getter/setter/modify,
willset/didset, and getter-/setter-specifier rules.
The tree-sitter protocol property and function requirement rules
(`protocol_property_declaration`, `protocol_function_declaration`) are
dropped: swift-syntax models those requirements as ordinary
`variableDecl`s (with bodyless accessors) and `functionDecl`s, already
handled by the general rules.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget enum cases to the swift-syntax AST. An `enumCaseDecl` flattens its comma-separated `enumCaseElement`s (non-first tagged `chained_declaration`) and publishes any case modifiers (e.g. `indirect`) into `ctx`; an element with a payload becomes a nested `class_like_declaration` + constructor, an element with a raw value (`case a = 1`) or a plain element a `variable_declaration`; `enumCaseParameter` becomes a `parameter`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retarget the remaining member declarations to the swift-syntax AST. An `initializerDecl` becomes a `constructor_declaration` (its body optional, so a bodyless protocol requirement still maps); `deinitializerDecl`, `typeAliasDecl`, and `associatedTypeDecl` map to `destructor_declaration`, `type_alias_declaration`, and `associated_type_declaration` respectively. The tree-sitter subscript and preprocessor-diagnostic rules are dropped: their swift-syntax counterparts (`subscriptDecl`, `ifConfigDecl`) fall through to the `unsupported_node` fallback, producing the same output. This also removes the now-unused `type` unwrap rule (swift-syntax has no such wrapper node) and retires the last `ctx.literal` helper use in favour of a `tree!` leaf. PARITY(tree-sitter): the initializer's parameters are still not emitted, because the tree-sitter path dropped them too. Emitting them is a future improvement. This completes the rule migration: every declaration now maps through the swift-syntax front-end. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Flip the runtime Swift front-end from tree-sitter to swift-syntax. The mapping rules were already ported, so the mapped AST is unchanged; this makes the switch live. - `language_spec` now builds a language-free desugarer (`ConcreteDesugarer::without_language`) and wires the swift-syntax parser (`swift_parse::parse`) as the front-end, dropping the tree-sitter language and node types. The desugarer supplies the output schema, so `node_types` is left empty. - The `swift_adapter`/`swift_parse` modules are no longer `allow(dead_code)`: they are now reached from the live extraction path. - `corpus_tests` skips (rather than fails) when the external `swift-syntax-parse` binary is unavailable, since it cannot run without the Swift-backed parser. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Now that the Swift front-end is swift-syntax, regenerate the second (raw parse tree) section of the corpus `.output` files to hold the swift-syntax AST the adapter builds, instead of the old tree-sitter parse tree. These 98 cases map to a byte-for-byte identical mapped AST (the third section), so only their raw section changes — the mapping rules were ported to produce the same output. The one case whose mapped AST differs (`types/property-with-getter-and-setter`) is handled separately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Regenerate `types/property-with-getter-and-setter`, whose mapped AST now differs from the tree-sitter output: the backing `private var _v` retains its `private` modifier (`modifier "var"` + `modifier "private"`), whereas the tree-sitter path dropped it (its `visibility_modifier` node carried no text). The swift-syntax front-end preserves the modifier, so the mapped AST is strictly richer here. The raw (second) section is regenerated to the swift-syntax AST like the other cases; the mapped (third) section gains the retained `private` modifier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`swift-syntax-rs` is a workspace member, so its build script runs on a plain `cargo check`/`fmt`/`clippy` at the repo root. Previously it panicked when `swift build` could not be run, breaking those Swift-free workflows for anyone without a Swift toolchain. Instead, when `swift build` cannot be spawned, emit a `cargo:warning` and skip the link directives rather than panicking. `cargo check`/`fmt`/`clippy` don't link, so they keep working; only `cargo build`/`cargo test` then fail, at link time — which is fair, since those genuinely need Swift (and CI builds go through Bazel). A Swift toolchain that is present but whose build fails is still surfaced as a hard error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| final override string getAPrimaryQlClass() { result = "UnresolvedOperatorSequence" } | ||
|
|
||
| /** Gets the node corresponding to the field `element`. */ | ||
| final ExprOrOperator getElement(int i) { |
Contributor
There was a problem hiding this comment.
Pull request overview
Switches unified Swift parsing from tree-sitter to a swift-syntax subprocess and adapts/desugars its AST into commonAST.
Changes:
- Adds precedence-aware operator folding and unresolved-sequence modeling.
- Introduces a parser-agnostic desugaring extractor path.
- Regenerates Swift corpus snapshots and generated QL schema bindings.
Show a summary per file
| File | Description |
|---|---|
unified/swift-syntax-rs/swift/Sources/SwiftSyntaxFFI/SwiftSyntaxFFI.swift |
Folds operator sequences before JSON serialization. |
unified/swift-syntax-rs/swift/Package.swift |
Adds SwiftOperators dependency. |
unified/swift-syntax-rs/src/lib.rs |
Tests operator folding behavior. |
unified/swift-syntax-rs/README.md |
Documents operator folding. |
unified/swift-syntax-rs/build.rs |
Allows Swift-free Cargo checks. |
unified/swift-syntax-rs/BUILD.bazel |
Adds Bazel SwiftOperators dependency. |
unified/ql/lib/unified.dbscheme |
Adds unresolved operator sequences. |
unified/ql/lib/codeql/unified/Ast.qll |
Exposes the generated sequence API. |
unified/extractor/src/languages/swift/swift.rs |
Ports Swift desugaring rules. |
unified/extractor/src/languages/swift/parse.rs |
Adds the external Swift parser frontend. |
unified/extractor/src/languages/mod.rs |
Activates the new frontend. |
unified/extractor/src/extractor.rs |
Uses the desugaring extractor. |
unified/extractor/ast_types.yml |
Models unresolved operator sequences. |
unified/extractor/tests/swift_syntax_pipeline.rs |
Updates pipeline API usage. |
unified/extractor/tests/corpus_tests.rs |
Runs corpus cases through the new parser. |
unified/extractor/tests/corpus/swift/variables/*.output |
Refreshes variable snapshots. |
unified/extractor/tests/corpus/swift/types/*.output |
Refreshes type/declaration snapshots. |
unified/extractor/tests/corpus/swift/optionals-and-errors/*.output |
Refreshes optional/error snapshots. |
unified/extractor/tests/corpus/swift/operators/*.output |
Refreshes operator snapshots. |
unified/extractor/tests/corpus/swift/loops/*.output |
Refreshes loop snapshots. |
unified/extractor/tests/corpus/swift/literals/*.output |
Refreshes literal snapshots. |
unified/extractor/tests/corpus/swift/functions/*.output |
Refreshes function snapshots. |
unified/extractor/tests/corpus/swift/desugar/*.output |
Refreshes desugaring snapshots. |
unified/extractor/tests/corpus/swift/control-flow/*.output |
Refreshes control-flow snapshots. |
unified/extractor/tests/corpus/swift/collections/*.output |
Refreshes collection snapshots. |
unified/extractor/tests/corpus/swift/closures/*.output |
Refreshes closure snapshots. |
shared/yeast/src/dump.rs |
Orders and validates fields by name. |
shared/yeast/src/build.rs |
Adds source-text convenience access. |
shared/yeast-schema/src/schema.rs |
Stores authored field ordering. |
shared/yeast-schema/src/node_types_yaml.rs |
Preserves YAML field order. |
shared/tree-sitter-extractor/src/extractor/desugaring.rs |
Adds parser-agnostic desugaring extraction. |
shared/tree-sitter-extractor/tests/multiple_languages.rs |
Updates the simplified language API. |
shared/tree-sitter-extractor/tests/integration_test.rs |
Updates extractor test construction. |
ruby/extractor/src/extractor.rs |
Adapts to the extraction API. |
ql/extractor/src/extractor.rs |
Removes obsolete desugar fields. |
ql/Cargo.lock |
Records yeast-schema dependency. |
Review details
- Files reviewed: 129/130 changed files
- Comments generated: 3
- Review effort level: Medium
|
|
||
| /// The `swift-syntax-parse` executable to invoke. | ||
| fn parse_bin() -> String { | ||
| std::env::var(PARSE_BIN_ENV).unwrap_or_else(|_| "swift-syntax-parse".to_string()) |
Comment on lines
+109
to
+115
| if !parser_available() { | ||
| eprintln!( | ||
| "skipping test_corpus: the swift-syntax parser is unavailable \ | ||
| (set CODEQL_EXTRACTOR_UNIFIED_SWIFT_SYNTAX_PARSE or put \ | ||
| `swift-syntax-parse` on PATH)" | ||
| ); | ||
| return; |
| // another module) stays a flat `sequenceExpr`. Preserve it as an | ||
| // `unresolved_operator_sequence` whose elements alternate operands and | ||
| // infix operators, rather than guessing a structure. | ||
| rule!((sequenceExpr elements: _* @els) => (unresolved_operator_sequence element: {els})), |
Two fixes prompted by review of the swift-syntax switch-over.
Parser resolution (`parse.rs`): `parse_bin` now resolves the
`swift-syntax-parse` executable in priority order — the
`CODEQL_EXTRACTOR_UNIFIED_SWIFT_SYNTAX_PARSE` override, then a copy next
to the extractor executable (as a shipped extractor pack lays it out:
`tools/<platform>/{extractor,swift-syntax-parse}`), then a
bare `PATH` lookup. This lets a packaged extractor find its parser with
no environment setup. (Bundling the binary into the pack, together with
its Swift runtime, is a separate follow-up.)
Corpus test guard (`corpus_tests.rs`): `parser_available` previously
treated *any* parser error as "unavailable" and skipped the entire
corpus suite, so a parser that was present but crashed or emitted
invalid JSON would silently skip the exact regressions the suite exists
to catch. It now uses the new `binary_available`, which reports whether
the *executable* can be launched (false only when it cannot be found,
e.g. no Swift toolchain); a launchable-but-failing parser makes the
suite run and fail.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`languages::swift::adapter` embeds the swift-syntax node-types schema
with `include_str!("../../../swift_node_types.yml")`, but the file was
never listed in the extractor's Bazel `compile_data`. The `cargo` build
finds it on disk, so this went unnoticed, but the sandboxed Bazel build
cannot see it and fails to compile the extractor. List it alongside
`ast_types.yml`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The extractor shells out to a separate `swift-syntax-parse` binary, but
nothing placed it in the extractor pack, so a shipped Swift extraction
failed at the first spawn. Package it next to the extractor, the same
way `//swift/extractor` ships its Swift-linked binary: a small wrapper
points the dynamic loader at its own directory and execs the real
binary, whose Swift runtime libraries travel alongside it.
- `swift-syntax-parse.sh`: wrapper that sets `LD_LIBRARY_PATH` /
`DYLD_LIBRARY_PATH` to its directory and execs
`swift-syntax-parse.real`
(mirrors `swift/extractor/extractor.sh`).
- `runtime.bzl`: a `swift_runtime_libs` rule that selects just the Linux
Swift runtime shared objects (`usr/lib/swift/linux/*.so`) out of the
full toolchain, so only they — not the whole toolchain — travel with
the binary.
- `swift-syntax-rs/BUILD.bazel`: the `rust_binary` becomes
`swift-syntax-parse.real`
and carries the runtime libraries as runfiles on Linux; a `sh_binary`
(`swift-syntax-parse`) is the wrapper; `codeql_pkg_runfiles` flattens
the three (wrapper, real binary, runtime) into one directory.
- `BUILD.bazel`: ship that group under `tools/{CODEQL_PLATFORM}` next to
the extractor, on the platforms where swift-syntax builds
(Linux/macOS).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rewrites all of the
tree-sitter-swift-based yeast rules to instead use the AST produced byswift-syntax-rs. The output commonAST is identical across the entire test corpus, apart from a small enhancement (that has been separated out into its own commit). A later PR will fix up a bunch of cases where we -- due to parity -- ended up throwing away parts of the AST because thetree-sitterrules inadvertently did so as well.This should be reviewed commit-by-commit. I have endeavoured to keep the actual rule porting somewhat manageable by splitting it into a sequence of commits.
In addition to the 99 tests in the test corpus, I also ran a comparison of the outputs when run on 427 Swift files taken from the
swift-syntaxtest corpus and our own Swift tests. For all of these, the output is either identical, or improved by theswift-syntax-based rules.This should be reviewed commit-by-commit!