Skip to content

[WIP] dq: idle watermarks: idle channel part #26841

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 26 commits into
base: main
Choose a base branch
from
Draft

[WIP] dq: idle watermarks: idle channel part #26841

wants to merge 26 commits into from

Conversation

yumkam
Copy link
Collaborator

@yumkam yumkam commented Oct 14, 2025 •
edited
Loading

Changelog entry

...

Changelog category

  • Not for changelog (changelog entry is not required)

Description for reviewers

Should be rebased on the top of #25540, only last 4 commits for review here

yumkam added 16 commits September 23, 2025 16:44
@yumkam
pq rd read actor: add idle watermark support
21653fd
@yumkam
Replace fake watermarks with idle watermarks
44af274 
Only sources handled by now (and that's inconsistent)

TODO remove/replace debug messages
TODO disentangle IdleDelay with WatermarkDelay
@yumkam
disentangle IdleDelay and LateArrivalDelay (incomplete)
14f18db 
this should really be moved from PRAGMA to WITH
(... but channel support require propagate this into dq graph; how this
should be performed?)
@yumkam
dq_source_tracker: new idle handling test
36a4087
@yumkam
cosmetics
d3f791d
@yumkam
cosmetics (drop unused method)
240753e
@yumkam
replace debug messages with log placeholders
33f5b66
@yumkam
fix review comment
68199b5
@yumkam
refactor common watermark code into base class
a2519bc
@yumkam
Merge remote-tracking branch 'origin/main' into pq-rd-idle-watermarks
7c4ce06 
Fixe Conflicts:
	ydb/library/yql/providers/pq/async_io/dq_pq_read_actor.cpp
@yumkam
ensure logging code compilable (even though ignored)
905923e
@yumkam
fixup! Merge remote-tracking branch 'origin/main' into pq-rd-idle-wat…
3ac2ba2 
…ermarks
@yumkam
fix ai review comment
12c6dfb
@yumkam
fix ai review comment
621a911
@yumkam
fix ai review comment
f472678
@yumkam
fixup! ensure logging code compilable (even though ignored)
626a707
@yumkam yumkam changed the title Watermark idle channels 3 dq: idle watermarks: idle channel part Oct 14, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 14, 2025 •
edited
Loading

⚪ 2025-10-14 11:56:49 UTC Pre-commit check linux-x86_64-release-asan for af7fe0e has started.
⚪ 2025-10-14 11:58:50 UTC Artifacts will be uploaded here
⚪ 2025-10-14 12:03:56 UTC ya make is running...
⚫ 2025-10-14 14:02:30 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 14, 2025 •
edited
Loading

⚪ 2025-10-14 11:57:37 UTC Pre-commit check linux-x86_64-relwithdebinfo for af7fe0e has started.
⚪ 2025-10-14 12:00:46 UTC Artifacts will be uploaded here
⚪ 2025-10-14 12:06:03 UTC ya make is running...
⚫ 2025-10-14 14:02:30 UTC Check cancelled

@yumkam yumkam changed the title dq: idle watermarks: idle channel part [WIP] dq: idle watermarks: idle channel part Oct 14, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 14, 2025 •
edited
Loading

🟢 2025-10-16 10:48:23 UTC The validation of the Pull Request description is successful.

@yumkam yumkam force-pushed the watermark-idle-channels-3 branch from 9717cd4 to f94a2d0 Compare October 14, 2025 14:02
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 14, 2025 •
edited
Loading

⚪ 2025-10-14 14:04:12 UTC Pre-commit check linux-x86_64-release-asan for 09815b2 has started.
⚪ 2025-10-14 14:04:21 UTC Artifacts will be uploaded here
⚪ 2025-10-14 14:08:40 UTC ya make is running...
🟡 2025-10-14 16:22:57 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
18210 17684 0 211 295 20

🟢 2025-10-14 16:23:03 UTC Build successful.
🟡 2025-10-14 16:23:27 UTC ydbd size 3.8 GiB changed* by +334.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 34f7f07 merge: 09815b2 diff diff %
ydbd size 4 031 469 520 Bytes 4 031 811 984 Bytes +334.4 KiB +0.008%
ydbd stripped size 1 497 304 384 Bytes 1 497 434 304 Bytes +126.9 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 14, 2025 •
edited
Loading

⚪ 2025-10-14 14:04:55 UTC Pre-commit check linux-x86_64-relwithdebinfo for 09815b2 has started.
⚪ 2025-10-14 14:05:25 UTC Artifacts will be uploaded here
⚪ 2025-10-14 14:10:16 UTC ya make is running...
🟡 2025-10-14 15:48:36 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
41397 38562 0 8 2795 32

⚪ 2025-10-14 15:48:46 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-10-14 16:07:59 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
1072 (only retried tests) 1052 0 0 0 20

🟢 2025-10-14 16:08:04 UTC Build successful.
🟡 2025-10-14 16:08:21 UTC ydbd size 2.3 GiB changed* by +197.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 34f7f07 merge: 09815b2 diff diff %
ydbd size 2 423 191 896 Bytes 2 423 394 560 Bytes +197.9 KiB +0.008%
ydbd stripped size 515 835 656 Bytes 515 878 760 Bytes +42.1 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@yumkam yumkam force-pushed the watermark-idle-channels-3 branch from f94a2d0 to 8364140 Compare October 16, 2025 10:44
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 16, 2025 •
edited
Loading

⚪ 2025-10-16 10:47:57 UTC Pre-commit check linux-x86_64-relwithdebinfo for d506e2f has started.
⚪ 2025-10-16 10:48:11 UTC Artifacts will be uploaded here
⚪ 2025-10-16 10:52:26 UTC ya make is running...
🟡 2025-10-16 13:06:48 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
41517 38745 0 4 2743 25

⚪ 2025-10-16 13:06:57 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-10-16 13:21:18 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
589 (only retried tests) 567 0 2 0 20

⚪ 2025-10-16 13:21:21 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-10-16 13:36:19 UTC Some tests failed, follow the links below.

Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
576 (only retried tests) 555 0 2 0 19

🟢 2025-10-16 13:36:22 UTC Build successful.
🟡 2025-10-16 13:36:42 UTC ydbd size 2.3 GiB changed* by +180.1 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: ed73f44 merge: d506e2f diff diff %
ydbd size 2 424 183 328 Bytes 2 424 367 800 Bytes +180.1 KiB +0.008%
ydbd stripped size 516 068 552 Bytes 516 101 224 Bytes +31.9 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 16, 2025 •
edited
Loading

⚪ 2025-10-16 10:48:37 UTC Pre-commit check linux-x86_64-release-asan for d506e2f has started.
⚪ 2025-10-16 10:48:51 UTC Artifacts will be uploaded here
⚪ 2025-10-16 10:53:18 UTC ya make is running...
🟡 2025-10-16 13:53:45 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
18292 17745 0 253 274 20

🟢 2025-10-16 13:53:51 UTC Build successful.
🟡 2025-10-16 13:54:20 UTC ydbd size 3.8 GiB changed* by +313.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: ed73f44 merge: d506e2f diff diff %
ydbd size 4 033 290 592 Bytes 4 033 611 392 Bytes +313.3 KiB +0.008%
ydbd stripped size 1 498 016 160 Bytes 1 498 136 672 Bytes +117.7 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 16, 2025 •
edited
Loading

⚪ 2025-10-16 16:52:21 UTC Pre-commit check linux-x86_64-relwithdebinfo for 72d4b56 has started.
⚪ 2025-10-16 16:52:35 UTC Artifacts will be uploaded here
⚪ 2025-10-16 16:56:55 UTC ya make is running...
⚫ 2025-10-16 17:49:19 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 16, 2025 •
edited
Loading

⚪ 2025-10-16 16:52:23 UTC Pre-commit check linux-x86_64-release-asan for 72d4b56 has started.
⚪ 2025-10-16 16:52:37 UTC Artifacts will be uploaded here
⚪ 2025-10-16 16:57:00 UTC ya make is running...
⚫ 2025-10-16 17:49:14 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 16, 2025 •
edited
Loading

⚪ 2025-10-16 17:50:36 UTC Pre-commit check linux-x86_64-release-asan for 193171f has started.
⚪ 2025-10-16 17:51:20 UTC Artifacts will be uploaded here
⚪ 2025-10-16 17:55:40 UTC ya make is running...
🟡 2025-10-16 20:18:38 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
18276 17781 0 229 248 18

🟢 2025-10-16 20:18:44 UTC Build successful.
🟡 2025-10-16 20:19:10 UTC ydbd size 3.8 GiB changed* by +384.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: eef470e merge: 193171f diff diff %
ydbd size 4 032 404 424 Bytes 4 032 798 584 Bytes +384.9 KiB +0.010%
ydbd stripped size 1 497 601 888 Bytes 1 497 733 728 Bytes +128.8 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 16, 2025 •
edited
Loading

⚪ 2025-10-16 17:50:51 UTC Pre-commit check linux-x86_64-relwithdebinfo for 193171f has started.
⚪ 2025-10-16 17:52:26 UTC Artifacts will be uploaded here
⚪ 2025-10-16 17:57:34 UTC ya make is running...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

not-for-changelog

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@yumkam