Skip to content

feat(benchmark): add cache behavior suite - #174

Merged
eitsupi merged 4 commits into
mainfrom
benchmark/cache-behavior-suite
Aug 29, 2026
Merged

feat(benchmark): add cache behavior suite#174
eitsupi merged 4 commits into
mainfrom
benchmark/cache-behavior-suite

Conversation

@eitsupi

@eitsupi eitsupi commented Aug 29, 2026

Copy link
Copy Markdown
Owner

Summary

  • add deterministic 64-model and 512-model workloads for SQL, manifest DAG, and column-lineage cache measurements
  • distinguish no-cache, persistent-cold, and persistent-warm runs without claiming to flush the OS page cache
  • keep timing on compact summary output while comparing SQL and manifest semantics with full graph JSON probes
  • exercise SQL file, macro, and vars.yml invalidation on isolated project copies
  • run a lightweight Ubuntu semantic-validation workflow and publish its results in the GitHub Actions Job Summary
  • record reproducibility metadata and raw hyperfine results without adding performance thresholds to CI

The manifest scenario is limited to model-level DAG commands. It does not replace or benchmark the typed manifest used by column lineage or MCP.

Baseline

Final-suite release measurements on the 512-model profile showed median times of:

Scenario No cache Persistent cold Persistent warm
SQL summary 20.11 ms 23.91 ms 15.92 ms
Manifest summary 13.88 ms 15.11 ms 12.30 ms
Column query 12.67 ms 12.81 ms 10.98 ms

These numbers are informational local measurements from 5 runs with 1 warmup using dlin 0.2.4 binary.
The raw hyperfine JSON and run metadata were retained locally for the upcoming cache redesign comparison.

CI does not measure or gate performance.

Validation

  • cargo test --all-targets --all-features --workspace --locked
  • deterministic small and medium workload generation and validation
  • final medium release semantic validation: SQL, manifest, and column no-cache/cold/warm equivalence all passed
  • final medium SQL invalidation validation: SQL file, macro, and vars.yml mutations all passed
  • final medium release benchmark: 5 runs, 1 warmup
  • runner invocation from outside the repository records the repository HEAD
  • cache-benchmark-validation GitHub Actions workflow passed and published its semantic results in the Job Summary
  • roborev follow-up review: no issues found

@eitsupi
eitsupi marked this pull request as ready for review August 29, 2026 09:32
@eitsupi
eitsupi merged commit de7436b into main Aug 29, 2026
6 checks passed
@eitsupi
eitsupi deleted the benchmark/cache-behavior-suite branch August 29, 2026 09:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant