feat(query): extract query lineage from plans

[youngsofun]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Extract object-level and column-level lineage from optimized query plans and expose it through Plan::query_lineage().

This PR is the planner foundation for #20198. The larger lineage PR currently contains this same commit so its persistence and SQL tests remain executable end to end. This PR is intended to merge first; #20198 will then rebase onto main and drop the duplicate extraction commit.

Motivation

Persistent lineage needs a canonical representation of which input relations and columns contribute data to each output relation and column. Deriving that information later from formatted SQL would duplicate binder semantics and would be unreliable for aliases, casts, subqueries, CTEs, optimizer rewrites, and mutation expressions.

This change keeps extraction in the planner, where resolved table identities, column IDs, scalar expressions, and the optimized relational tree are available. The output is storage-independent and can be consumed by metadata persistence separately.

Output model

QueryLineage records the statement kind and a set of canonical data-flow edges:

upstream relation/column -> downstream relation/column

The public extraction DTO groups upstream relations under each downstream relation. Column edges are sorted and deduplicated. Relations carry catalog, database, name, optional stable ID, and whether the relation is a table or view.

Supported target statements are:

• CREATE TABLE ... AS SELECT (CTAS)
• CREATE VIEW
• INSERT ... SELECT
• REPLACE INTO ... SELECT
• multi-table insert
• mutation plans used by UPDATE ... FROM and MERGE

Statements without a data-producing query return no lineage.

Resolution algorithm

Extraction has two phases:

1. Walk the optimized SExpr and build symbol definitions for base scans and data-producing operators.
2. Starting from each target column expression, recursively resolve referenced symbols until base relation columns are reached.

Definitions are collected for scalar evaluation, aggregate functions, window functions, set-returning functions, UDFs, async functions, UNION ALL, expression scans, and automatic materialized CTE references. Scalar subqueries are followed through their output symbols.

Only target value expressions are roots of the traversal. Columns used exclusively by filters, join predicates, WHEN conditions, or ordering do not become lineage unless they also contribute to an output expression. Aggregate arguments such as count(col) therefore propagate col, while count(*) has no input column edge.

Cycle detection prevents recursive symbol definitions from looping, and ordered sets make multi-source expressions deterministic.

Relation boundaries

• Views are treated as lineage boundaries when read by another statement. Their output columns are attributed to the view rather than expanded through to base tables.
• Stream data columns are attributed to the stream's source table; stream metadata columns are ignored.
• Automatic materialized CTE consumer symbols are mapped back to producer symbols.
• Explicit AS MATERIALIZED CTE lineage is not included yet because the query-scoped temporary relation does not retain the original producer-column mapping.

Capture cost and execution semantics

The new query.lineage.capture_enabled configuration defaults to false. Additional view-body planning, relation annotations, and target-ID capture are gated by this flag.

For ordinary INSERT, the target table ID is captured in a lineage-only field. table_info remains unset, so execution continues to resolve the target by catalog/database/table name and does not become pinned to the table object seen during planning.

This PR only extracts lineage. Metadata schema, persistence, lifecycle handling, and GET_LINEAGE remain in #20198.

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Planner coverage includes CTAS, views, insert-select, view boundaries, CTEs, automatic materialized CTEs, joins and EXISTS filters, JOIN USING, replace, update, merge, and multi-table insert.

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

---

This change is 

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

[github-actions]

🤖 CI Job Analysis (Retry 1)

Workflow: 30134717828

📊 Summary

• Total Jobs: 89
• Failed Jobs: 1
• Retryable: 0
• Code Issues: 1

❌ NO RETRY NEEDED

All failures appear to be code/test issues requiring manual fixes.

🔍 Job Details

• ❌ linux / test_unit: Not retryable (Code/Test)

---

🤖 About

Automated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed).