Skip to content

Conversation

@changliiu
Copy link
Contributor

@changliiu changliiu commented May 23, 2025

To improve the performance of the query for GetUnfinishedMinWatermark, we can add a filter to the query.
Note that the query run on a cache to refresh every 1s. We can remember the cached value every time and use the value as a filter in the next time query.

Also add class level timeout for all change stream integration tests.

Note that, the CacheLoader's refresh function cannot return null, because that will be deemed invalid, then the cache will not be updated. As the result, the connector will run forever. Hence we still keep the return type optional.

@changliiu changliiu force-pushed the add-filter-tester branch 2 times, most recently from f3470e8 to 344e4cf Compare May 23, 2025 22:22
@changliiu
Copy link
Contributor Author

Integration test can pass locally https://screenshot.googleplex.com/6aQBPAW8YT69VTd

@changliiu changliiu marked this pull request as ready for review May 23, 2025 22:23
@github-actions
Copy link
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@changliiu changliiu force-pushed the add-filter-tester branch 2 times, most recently from 922ce28 to e638136 Compare May 26, 2025 22:06
@changliiu
Copy link
Contributor Author

Run integration test locally based on earlier commit #34883 and it passed. Hence the integration timeout is not caused by this PR.

@changliiu
Copy link
Contributor Author

assign set of reviewers

@github-actions
Copy link
Contributor

Assigning reviewers:

R: @m-trieu for label java.
R: @nielm for label spanner.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@changliiu changliiu force-pushed the add-filter-tester branch 2 times, most recently from 1feabc1 to f7320e1 Compare May 29, 2025 18:11
@github-actions
Copy link
Contributor

github-actions bot commented Jun 9, 2025

Reminder, please take a look at this pr: @m-trieu @nielm

@github-actions
Copy link
Contributor

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @chamikaramj for label java.
R: @nielm for label spanner.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

@changliiu changliiu force-pushed the add-filter-tester branch 3 times, most recently from b1761a4 to c7d4c25 Compare June 12, 2025 17:38
@changliiu
Copy link
Contributor Author

Friendly ping :)

@changliiu changliiu force-pushed the add-filter-tester branch 2 times, most recently from 5a5f4a5 to 5f0a76a Compare June 13, 2025 17:02
@changliiu
Copy link
Contributor Author

Friendly ping :)

@changliiu changliiu force-pushed the add-filter-tester branch 2 times, most recently from 559ee3c to 3c388b5 Compare June 16, 2025 17:06
@changliiu
Copy link
Contributor Author

Friendly ping :)

@Abacn
Copy link
Contributor

Abacn commented Jun 18, 2025

It seems GCP IO integration test stucks indefinitely. Retriggered the test

update: second try still stuck, retriggering

If it timeouts for the third time, consider add class level test timeout for integraton tests that might be affected to debug

@changliiu changliiu force-pushed the add-filter-tester branch 5 times, most recently from c6deb4c to 26dcc4e Compare June 24, 2025 21:55
@changliiu changliiu force-pushed the add-filter-tester branch from 26dcc4e to d5bb6b4 Compare June 25, 2025 01:03
@Abacn Abacn merged commit ffd8945 into apache:master Jun 25, 2025
16 checks passed
@changliiu changliiu deleted the add-filter-tester branch June 25, 2025 20:04
shunping pushed a commit to shunping/beam that referenced this pull request Jun 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants