Skip to content

fix: preserve dependencies across runtime branches - #185

Merged
eitsupi merged 17 commits into
mainfrom
fix/jinja-semantic-certainty
Aug 30, 2026
Merged

fix: preserve dependencies across runtime branches#185
eitsupi merged 17 commits into
mainfrom
fix/jinja-semantic-certainty

Conversation

@eitsupi

@eitsupi eitsupi commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Summary

MiniJinja can render a dbt model successfully with dlin's placeholder runtime values while selecting only one side of a runtime-dependent branch. Render completion therefore does not imply that the selected branch is semantically representative of dbt parse, compile, and run contexts.

This change separates render completion from semantic certainty and conservatively recovers literal ref() and source() dependencies from runtime-dependent branches.

What changes

  • Runtime callbacks mark the current render scope uncertain when evaluated, including env_var(), unresolved var() calls without a default, run_query(), statement(), and placeholder runtime object attribute/item access such as target.*.
  • Runtime-specific scalar globals (execute, dbt_version, invocation_id, and run_started_at) retain their existing placeholder values, but their use no longer makes branch selection semantically certain.
  • Called model-local macros propagate uncertainty through execution markers without changing macro output or whitespace behavior.
  • MiniJinja's undeclared-variable analysis builds a conservative symbol graph for model-local and project macros. This covers aliases, higher-order callbacks, collections, conditional assignments, transitive calls, duplicate definitions, and lexical shadowing without implementing a regex call parser.
  • Complete uncertain renders recover macro-free model SQL plus only reachable local and project macro definition spans. Unused macro definitions remain excluded.
  • Uncertainty inside a local macro can reach project macros, and project macros can reach local macros. If reachability analysis fails, recovery falls back to whole-source scanning instead of risking a false negative.
  • Project macro definition metadata and per-definition analysis are prepared once per graph build and reused across models. Individual definitions are analyzed lazily only when reachable.
  • The Jinja/SQL extraction and macro-reachability implementation is split into directory modules by responsibility.

The implementation remains conservative rather than attempting symbolic Jinja execution. When runtime reachability cannot be established by the implemented analysis, recovery falls back to broader source scanning rather than risking a false negative.

Correctness coverage

Regression tests cover runtime branches in model SQL, called and transitive local macros, aliases and higher-order calls, project macro chains, local-to-project and project-to-local transitions, duplicate definitions, unused macro exclusion, comments/raw blocks/string literals, shadowing, and conservative analysis-failure behavior.

Three deterministic runtime-macro workloads exercise sparse local reachability, dense local reachability, and a large project macro prefix. CI validates exact graph counts, direct dependency distance, complete downstream sets, expected runtime branch refs/sources, and exclusion of 250 unused project-macro dependency literals.

Performance

Release binaries were compared with summary --no-cache over 120 randomized rounds per profile and binary, with 5 warmups (seed 20260830). Times below are medians; lower is better.

Profile Previous This PR Change Observable graph result
small 30.377 ms 31.029 ms +2.1% unchanged: 64 edges / 65 nodes; paired median was +0.7%
macro-heavy 82.372 ms 78.110 ms -5.2% unchanged: 64 edges / 65 nodes
runtime project-macro-heavy 138.711 ms 133.754 ms -3.6% 192 -> 320 edges; both runtime branches recovered
runtime local-macro sparse 69.814 ms 92.926 ms +33.1% 64 -> 128 edges; 64 missing alternate dependencies recovered
runtime local-macro dense 77.234 ms 107.059 ms +38.6% 64 -> 128 edges; 64 missing alternate dependencies recovered

The normal small path is noise-equivalent and the large project-macro paths are faster because prepared prefix analysis is shared and lazy. The adversarial local profiles deliberately place 128 macro definitions in every model; their added cost is approximately 0.36-0.47 ms per model while recovering one previously omitted dependency per model. Further reducing that cost would require sharing compiled model templates/free-symbol analysis across models or replacing MiniJinja parsing with a custom parser, which is not justified for this fix.

Validation

  • cargo test --workspace --locked
  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
  • release builds and deterministic workload generation/validation
  • runtime semantic validation with both positive and forbidden dependency assertions
  • the pre-change binary fails the new sparse runtime gate with 65 nodes / 64 edges versus the required 129 nodes / 128 edges

@eitsupi eitsupi changed the title fix(jinja): preserve dependencies across runtime branches fix: preserve dependencies across runtime branches Aug 30, 2026
@eitsupi
eitsupi marked this pull request as draft August 30, 2026 03:34
@eitsupi
eitsupi marked this pull request as ready for review August 30, 2026 08:33
@eitsupi
eitsupi merged commit 3248035 into main Aug 30, 2026
16 checks passed
@eitsupi
eitsupi deleted the fix/jinja-semantic-certainty branch August 30, 2026 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant