Summary
Support incremental read in flow engine — when a flow runs, only process the new data since the last run, rather than reprocessing the entire table. This significantly reduces recomputation overhead for continuous flow queries.
Related PRs
Component Breakdown
| Component |
Description |
Status |
| Scan boundaries / context |
Defines IncrementalReadContext and scan range boundaries so the flow knows what data to scan |
✅ |
| Scan bind sequence |
Binds scan sequence numbers to flow queries for incremental processing |
✅ |
| Remote dyn filter |
Passes dynamic filter expressions to remote scans |
✅ |
| Seq watermark metrics |
Records sequence watermark in record batch metrics for progress tracking |
✅ |
| Terminal metrics transport |
Transports flow incremental query terminal metrics |
✅ |
| Join rewrite helper |
Rewrites aggregate delta queries to LEFT JOIN with sink table for incremental merge |
🔄 |
| Flownode integration |
Integrate incremental read support into the flownode execution path |
🔜 |
Summary
Support incremental read in flow engine — when a flow runs, only process the new data since the last run, rather than reprocessing the entire table. This significantly reduces recomputation overhead for continuous flow queries.
Related PRs
Component Breakdown
IncrementalReadContextand scan range boundaries so the flow knows what data to scan