Skip to content

fix: preserve partial column lineage diagnostics - #188

Merged
eitsupi merged 3 commits into
mainfrom
fix/partial-column-lineage-errors
Aug 30, 2026
Merged

fix: preserve partial column lineage diagnostics#188
eitsupi merged 3 commits into
mainfrom
fix/partial-column-lineage-errors

Conversation

@eitsupi

@eitsupi eitsupi commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Summary

  • classify unresolved but honest column lineage as structured column_indeterminate diagnostics instead of fatal column_not_found errors
  • carry column identity through cross-model renames so CLI, MCP, and impact filtering keep only diagnostics for the requested root output
  • preserve the nearest known model column as the terminal when deeper row-value lineage cannot be proven
  • skip redundant single-column retries after the all-column analysis already produced a scoped diagnostic

Behavior

For BigQuery row-value patterns such as ARRAY_AGG(t ...)[OFFSET(0)] AS event followed by event.field:

  • 0.2.4 returned a fabricated event.field source with no diagnostic
  • the 0.2.5 release candidate kept the honest nearest model terminal, but also emitted fatal column_not_found and exited 1
  • this change keeps the honest nearest model terminal, emits non-fatal column_indeterminate with the root output column, and exits 0 when no fatal diagnostics remain

Diagnostics from excluded outputs and unrelated same-named upstream branches are suppressed. Missing requested columns, ambiguous columns, parse failures, and internal failures remain fatal.

Performance

The release-candidate correctness path retried the full upstream SQL once per indeterminate field. Because the SQL also grows with the number of expanded fields, downstream analysis scaled quadratically.

Linux x86_64 dist builds, --no-cache, hyperfine warmup 1 / 5 runs:

Row-value fields Before After
35 55.3 ± 22.4 ms 4.7 ± 0.6 ms
100 179.9 ± 42.9 ms 11.8 ± 0.9 ms
300 1.405 ± 0.169 s 30.7 ± 1.3 ms

The before and after builds produce identical stdout, stderr, and exit status for all three fixtures. The existing on-demand recovery remains enabled when the all-column analysis did not produce a matching scoped diagnostic.

Validation

  • cargo test --workspace --locked
  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
  • cargo build --profile dist --locked
  • column-lineage artifact oracle and manifest/catalog validation
  • synthetic N=35/100/300 semantic and performance comparison
  • official 0.2.4 behavior comparison

@eitsupi
eitsupi merged commit 9048b5f into main Aug 30, 2026
@eitsupi
eitsupi deleted the fix/partial-column-lineage-errors branch August 30, 2026 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant