test(benchmarks): add macro-heavy Jinja baseline - #181
Merged
Conversation
eitsupi
marked this pull request as ready for review
August 29, 2026 15:18
eitsupi
added a commit
that referenced
this pull request
Aug 29, 2026
## Summary - compile each effective macro-prefix plus model template once - reuse the same MiniJinja environment, template, and callbacks for full-load and incremental renders - reset pass-local extraction after every render, including failures - preserve ref/source deduplication and full-load config precedence - remove the former Arc ownership/unwrapping requirement ## Correctness The new regressions verify: - full-load config remains authoritative when both branches set config - a failed full-load pass retains its partial extraction - the successful incremental pass has independent refs and config before merge Existing Jinja expectations are unchanged. ## Performance Release benchmarks use the exact fixtures introduced in #181. The macro-heavy no-cache median improved from 35.007 ms on the exact main binary to 31.036 ms over a nearby 30-run comparison (about 11%). Small and medium fixtures have tiny macro prefixes and remain dominated by short-command scheduling/filesystem variance; they showed no stable gain or regression. Persistent-warm behavior is effectively unchanged, as expected. The standard 10-run semantic benchmark also passed for small, medium, and macro-heavy, including SQL file, macro, and vars invalidation probes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Baseline
Release binary built from
3723b79, hyperfine 10 runs / 2 warmups:All SQL, manifest, and column semantic probes passed for every profile. SQL file, macro, and vars invalidation probes also passed.
Validation
--skip-timingvalidation for small and macro-heavygit diff --check