fix: support YAML-only snapshots (dbt v1.9+) - #83
Merged
Conversation
dbt v1.9+ allows defining snapshots in YAML without a corresponding
.sql file. These were previously silently dropped because SchemaFile
had no `snapshots` field and builder only read snapshot_sql_files.
- Add `SnapshotDefinition` struct to yaml_schema.rs with `name`,
`description`, and `relation` fields
- Add `snapshots: Vec<SnapshotDefinition>` to `SchemaFile`
- Collect YAML snapshot definitions in `process_yaml_files()`
- Add `process_yaml_snapshot_nodes()` to register nodes for snapshots
that have no matching SQL file, and add edges from the upstream model
via the `relation: ref('...')` field (reuses `parse_exposure_ref()`)
- SQL-based snapshots are unaffected; YAML defs for them are skipped
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Collapse nested if-let in process_yaml_snapshot_nodes() to satisfy clippy::collapsible_if (-D warnings) - Fix yaml snapshot tests: override stg_orders.sql with a source-free query so phantom source nodes don't inflate the expected node count Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
YAML-only snapshots using `relation: source('schema', 'table')` now
get a Source edge to the matching source node. If the source is not
defined in YAML, a phantom node is created (consistent with how SQL
snapshot source refs are handled by process_sql_edges).
- Add parse_relation_source() helper wrapping extract_sources()
- process_yaml_snapshot_nodes() checks source() before ref() and adds
EdgeType::Source edges; falls through to ref() parsing otherwise
- Add tests for defined-source and phantom-source relation variants
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates dlin-core to support dbt v1.9+ “YAML-only” snapshots (snapshots defined in schema.yml without a corresponding .sql snapshot file) by parsing snapshot definitions from schema YAML and registering them as NodeType::Snapshot nodes in the lineage graph, including upstream edges derived from a relation: expression.
Changes:
- Extend YAML schema parsing to include
snapshots:entries via a newSnapshotDefinitiontype. - Add a builder pass that registers YAML-only snapshot nodes and derives upstream edges from
relation: ref(...)/relation: source(...). - Add graph-builder tests covering YAML-only snapshots, SQL-precedence, and
source(...)relations (including phantom-source behavior).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
crates/dlin-core/src/parser/yaml_schema.rs |
Adds snapshots support to schema YAML parsing and tests parsing of YAML-only snapshot definitions. |
crates/dlin-core/src/graph/builder/mod.rs |
Collects YAML snapshot defs during YAML processing and registers YAML-only snapshot nodes + relation-derived upstream edges during graph build. |
crates/dlin-core/src/graph/builder/tests/build_graph.rs |
Adds end-to-end tests validating node creation, edge creation, and SQL-vs-YAML precedence for snapshots. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- For source() relations: use get_or_create_phantom_source() instead of inline phantom creation, so unresolved sources emit the standard warning and share the same canonical phantom node as SQL parsing - For ref() relations: use get_or_create_phantom_ref() so unresolved refs create a phantom node with a warning instead of silently dropping the edge Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.sqlfile. These were previously silently ignored becauseSchemaFilehad nosnapshotsfield and the builder only readsnapshot_sql_files.SnapshotDefinitionstruct toyaml_schema.rswithname,description, andrelationfieldsprocess_yaml_snapshot_nodes()in the builder to register nodes for YAML-only snapshots and add upstream edges via therelation: ref('...')fieldBehavior
NodeType::Snapshotnodesrelation: ref('model_name')is present.sqlfile and a YAML definition exist for the same snapshot name, the SQL file takes precedence (YAML def is skipped, no duplicate node)Test plan
test_build_graph_yaml_only_snapshot— verifies node creation and upstream edge fromrelationtest_build_graph_yaml_snapshot_sql_takes_precedence— verifies no duplicate when SQL file existstest_parse_yaml_only_snapshots— verifies YAML parsing ofSnapshotDefinitioncargo testpasses (83 tests)🤖 Generated with Claude Code