Skip to content

tracking: flow incremental read #8125

@discord9

Description

@discord9

Summary

Support incremental read in flow engine — when a flow runs, only process the new data since the last run, rather than reprocessing the entire table. This significantly reduces recomputation overhead for continuous flow queries.

Related PRs

Component Breakdown

Component Description Status
Scan boundaries / context Defines IncrementalReadContext and scan range boundaries so the flow knows what data to scan
Scan bind sequence Binds scan sequence numbers to flow queries for incremental processing
Remote dyn filter Passes dynamic filter expressions to remote scans
Seq watermark metrics Records sequence watermark in record batch metrics for progress tracking
Terminal metrics transport Transports flow incremental query terminal metrics
Join rewrite helper Rewrites aggregate delta queries to LEFT JOIN with sink table for incremental merge 🔄
Flownode integration Integrate incremental read support into the flownode execution path 🔜

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions