Skip to content

Conversation

@derrickaw
Copy link
Collaborator

  1. Currently, Pubsub transform doesn't support a maximum read time seconds parameter like Kafka transform has that limits the amount of time for the transform to read.
  2. This PR tries to add support for this feature.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@derrickaw derrickaw changed the title [yaml] - add max_read_time_seconds support [yaml] - add max_read_time_seconds support to PubSub Oct 31, 2025
@derrickaw
Copy link
Collaborator Author

Run Python PreCommit 3.13

@derrickaw
Copy link
Collaborator Author

Run Python PreCommit 3.12

@derrickaw
Copy link
Collaborator Author

Run Python PreCommit 3.11

@derrickaw
Copy link
Collaborator Author

Run Python PreCommit 3.10

@derrickaw
Copy link
Collaborator Author

Run Python_ML PreCommit 3.13

@derrickaw
Copy link
Collaborator Author

Run Python_Transforms PreCommit 3.9

@derrickaw
Copy link
Collaborator Author

Run PythonDocs PreCommit

@derrickaw derrickaw marked this pull request as ready for review November 3, 2025 16:46
@github-actions
Copy link
Contributor

github-actions bot commented Nov 3, 2025

Assigning reviewers:

R: @liferoad for label python.
R: @chamikaramj for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@liferoad
Copy link
Contributor

liferoad commented Nov 3, 2025

This wont work with Dataflow Runner since Dataflow overrides PubSub IO unless you use enable_custom_pubsub_source. So I do not think we need to support this for PubSub like Kafka.

@derrickaw
Copy link
Collaborator Author

derrickaw commented Nov 3, 2025

This wont work with Dataflow Runner since Dataflow overrides PubSub IO unless you use enable_custom_pubsub_source. So I do not think we need to support this for PubSub like Kafka.

To appropriately test in Yaml, I need this parameter. This was already discussed about.

@liferoad
Copy link
Contributor

liferoad commented Nov 3, 2025

This wont work with Dataflow Runner since Dataflow overrides PubSub IO unless you use enable_custom_pubsub_source. So I do not think we need to support this for PubSub like Kafka.

To appropriately test in Yaml, I need this parameter. This was already discussed about.

I probably miss something. If this is only for test, you can only use this with a local runner. You could use wait_until_finish with duration to test and then cancel the streaming pipeline.

@derrickaw
Copy link
Collaborator Author

This wont work with Dataflow Runner since Dataflow overrides PubSub IO unless you use enable_custom_pubsub_source. So I do not think we need to support this for PubSub like Kafka.

To appropriately test in Yaml, I need this parameter. This was already discussed about.

I probably miss something. If this is only for test, you can only use this with a local runner. You could use wait_until_finish with duration to test and then cancel the streaming pipeline.

This will be used in PostCommit tests.

@derrickaw derrickaw marked this pull request as draft November 3, 2025 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants