Skip to content

fix!: validate SQL extraction by content - #175

Merged
eitsupi merged 3 commits into
mainfrom
fix/sql-cache-semantic-hash
Aug 29, 2026
Merged

fix!: validate SQL extraction by content#175
eitsupi merged 3 commits into
mainfrom
fix/sql-cache-semantic-hash

Conversation

@eitsupi

@eitsupi eitsupi commented Aug 29, 2026

Copy link
Copy Markdown
Owner

Summary

  • replace SQL extraction cache validation based on whole-second mtime and file size with a deterministic semantic input hash
  • hash SQL bytes together with the exact effective macro prefix and recursively canonicalized project vars
  • reuse one SQL read for cache validation, MiniJinja extraction, and column inference
  • retain only an opaque precomputed hash token after each parallel worker finishes, avoiding cold-path retention of the full missed SQL corpus
  • preserve physical relative-path keys, JSON persistence, and the dlin package version as the cache compatibility boundary

Correctness

The cache now misses for same-timestamp, same-size content rewrites and when unrelated projects share a custom cache directory with the same relative SQL path. Macro and vars changes still invalidate extraction conservatively. Corrupt, legacy, or version-mismatched cache files continue to fail open to cold computation.

The cache format intentionally has no independent schema version. A package version change remains the release-level compatibility boundary.

API scope

dlin_core::parser::cache::ExtractionCache is now explicitly crate-private. This is an intentional pre-1.0 cleanup rather than a compatibility regression to preserve: the cache originated as a binary-crate implementation detail and was moved unchanged when dlin-core was split out. It has no documented or known downstream use, and its lifecycle depends on internal macro-prefix, vars, extraction, persistence, and dirty-state invariants. Supported callers should use the graph builder cache options rather than operate the extraction cache backend directly.

No compatibility wrapper is added because doing so would preserve an accidental low-level API and constrain later cache redesign.

Scope

This PR does not persist inferred columns or add model-to-macro dependency tracking. A temporary-file plus rename save path was tested but omitted after it materially increased persistent-cold time; the existing best-effort direct JSON write remains.

Verification

  • cargo test --workspace --all-targets --all-features
    • 122 CLI unit tests
    • 88 CLI integration tests
    • 847 core tests
    • 9 column-lineage integration tests
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • medium cache suite semantic and invalidation probes

Benchmark

Deterministic medium workload, 512 models, release binary, 5 runs and 1 warmup:

SQL state Median
no-cache 21.435 ms
persistent-cold 26.913 ms
persistent-warm 18.558 ms

SQL, manifest, and column semantic probes passed for no-cache/cold/warm. Single SQL file, macro, and vars invalidation probes all passed. The SQL cache was 98,006 bytes, down from the 100,931-byte baseline.

Timing is informational: other scenarios were also slower than the earlier baseline in this run, so absolute cross-run differences are not used as a gate.

@eitsupi eitsupi changed the title fix(cache): validate SQL extraction by content fix: validate SQL extraction by content Aug 29, 2026
@eitsupi
eitsupi marked this pull request as ready for review August 29, 2026 10:13
@eitsupi eitsupi changed the title fix: validate SQL extraction by content fix!: validate SQL extraction by content Aug 29, 2026
@eitsupi
eitsupi merged commit c56abeb into main Aug 29, 2026
12 checks passed
@eitsupi
eitsupi deleted the fix/sql-cache-semantic-hash branch August 29, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant