Skip to content

Add true time based flush behaviour to Coalesce transform#1997

Merged
tharinduamila-insta merged 6 commits intoshotover:mainfrom
tharinduamila-insta:1673-implement-true-flush_when_millis_since_last_flush
Mar 26, 2026
Merged

Add true time based flush behaviour to Coalesce transform#1997
tharinduamila-insta merged 6 commits intoshotover:mainfrom
tharinduamila-insta:1673-implement-true-flush_when_millis_since_last_flush

Conversation

@tharinduamila-insta
Copy link
Copy Markdown
Collaborator

@tharinduamila-insta tharinduamila-insta commented Mar 22, 2026

Closes : #1673

This PR introduces a background timer that will force the Transfrom chain to run if a time based flush definition is present when the time since last flush is greater than the time defined.

Last flush is considered to be the time

  1. When messages were truely flushed due meeting some condition.
  2. When the chain runs and the messages to flush is empty.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Mar 22, 2026

Merging this PR will not alter performance

✅ 36 untouched benchmarks
⏩ 2 skipped benchmarks1


Comparing tharinduamila-insta:1673-implement-true-flush_when_millis_since_last_flush (0443b3e) with main (e66d846)

Open in CodSpeed

Footnotes

  1. 2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds true time-based flushing to the Coalesce transform by introducing a per-connection background timer that can wake the chain to flush buffered messages, rather than only flushing opportunistically when new messages arrive.

Changes:

  • Implement a per-connection Tokio task in Coalesce that periodically signals force_run_chain and triggers a flush when the buffer is non-empty.
  • Update config validation/error messaging around flush settings (including requiring millis > 0 to enable the timer).
  • Refresh tests and documentation to reflect the new timer-driven behavior and semantics.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
shotover/src/transforms/coalesce.rs Adds timer-driven flush behavior, new validation logic, and expanded tests for timer/count interactions.
shotover/src/config/topology.rs Updates topology validation test expectations to match new Coalesce validation messaging.
docs/src/transforms.md Documents the new per-connection timer behavior and clarifies configuration semantics.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@justinweng-instaclustr
Copy link
Copy Markdown
Collaborator

The additional comments here look good and clear. Have approved the PR

Copy link
Copy Markdown
Collaborator

@yallen-ic yallen-ic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tharinduamila-insta tharinduamila-insta merged commit 836af57 into shotover:main Mar 26, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Coalesce transform - implement true flush_when_millis_since_last_flush

4 participants