Skip to content

test(benchmarks): add macro-heavy Jinja baseline - #181

Merged
eitsupi merged 2 commits into
mainfrom
benchmark/jinja-extraction-baseline
Aug 29, 2026
Merged

test(benchmarks): add macro-heavy Jinja baseline#181
eitsupi merged 2 commits into
mainfrom
benchmark/jinja-extraction-baseline

Conversation

@eitsupi

@eitsupi eitsupi commented Aug 29, 2026

Copy link
Copy Markdown
Owner

Summary

  • add a deterministic macro-heavy cache benchmark profile with 64 models, 16 macro source files, and 128 valid macros
  • preserve the existing small and medium workload shapes
  • validate macro source/definition counts and expose them in benchmark summaries
  • run semantic cache validation in CI for both small and macro-heavy profiles without timing gates
  • document the profile for measuring cumulative macro validation and per-model prefix parsing

Baseline

Release binary built from 3723b79, hyperfine 10 runs / 2 warmups:

SQL profile No cache Persistent cold Persistent warm
small (64 models, 1 file, 1 macro) 9.972 ms 11.094 ms 9.669 ms
medium (512 models, 1 file, 1 macro) 18.813 ms 20.076 ms 16.819 ms
macro-heavy (64 models, 16 files, 128 macros) 32.670 ms 47.488 ms 12.167 ms

All SQL, manifest, and column semantic probes passed for every profile. SQL file, macro, and vars invalidation probes also passed.

Validation

  • deterministic generation self-check for small, medium, and macro-heavy
  • workload validation for all three profiles
  • timed cache benchmark for all three profiles
  • local CI-equivalent --skip-timing validation for small and macro-heavy
  • Python bytecode compilation
  • git diff --check

@eitsupi
eitsupi marked this pull request as ready for review August 29, 2026 15:18
@eitsupi
eitsupi merged commit 7c8dcd8 into main Aug 29, 2026
7 checks passed
@eitsupi
eitsupi deleted the benchmark/jinja-extraction-baseline branch August 29, 2026 15:26
eitsupi added a commit that referenced this pull request Aug 29, 2026
## Summary

- compile each effective macro-prefix plus model template once
- reuse the same MiniJinja environment, template, and callbacks for full-load and incremental renders
- reset pass-local extraction after every render, including failures
- preserve ref/source deduplication and full-load config precedence
- remove the former Arc ownership/unwrapping requirement

## Correctness

The new regressions verify:

- full-load config remains authoritative when both branches set config
- a failed full-load pass retains its partial extraction
- the successful incremental pass has independent refs and config before merge

Existing Jinja expectations are unchanged.

## Performance

Release benchmarks use the exact fixtures introduced in #181.

The macro-heavy no-cache median improved from 35.007 ms on the exact main binary to 31.036 ms over a nearby 30-run comparison (about 11%). Small and medium fixtures have tiny macro prefixes and remain dominated by short-command scheduling/filesystem variance; they showed no stable gain or regression. Persistent-warm behavior is effectively unchanged, as expected.

The standard 10-run semantic benchmark also passed for small, medium, and macro-heavy, including SQL file, macro, and vars invalidation probes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant