[Feature]: Support dynamic refresh interval for refreshing table metadata #4037
Description
Description
No response
Use case/motivation
Amoro employs the "TableRuntimeRefreshExecutor" to perform table metadata scans on each table at fixed interval(default 1 minute). The next execution time interval is obtained via “getNextExecutingTime()”. Each execution involves loading table metadata, refreshing the status, and evaluating optimization requirements. Regardless of whether optimization is needed, the same process continues at the next interval.
The current fixed-interval scanning mechanism presents the following resource wastage issues:
- Ineffective scanning overhead: For tables with stable, long-term unchanged data, each execution still requires a full metadata load and status check. These operations are largely redundant in most scenarios.
- Uneven system load: All tables share the same scanning frequency, failing to differentiate between active and stable tables' distinct requirements, resulting in inefficient resource allocation.
- Lack of adaptability: The current mechanism cannot dynamically adjust scanning strategies based on historical change patterns or optimization demand frequency, thereby limiting overall system efficiency.
Describe the solution
Based on each table's evaluated result after "TableRuntimeRefreshExecutor#execute", tables are categorised and the next scan interval dynamically adjusted.
- Tables that need to be optimized: Scan interval rapidly shortened
- Tables that do not need to be optimized: Scan interval gradually increased
Subtasks
No response
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct