Add an experiment to make prism server a singleton #34623
Conversation
shunping
changed the title
shunping
force-pushed
the
prism-server-singleton
branch
from
4ef3c57 to
94c9cb0
Compare
shunping
force-pushed
the
prism-server-singleton
branch
from
94c9cb0 to
4122723
Compare
shunping
changed the title
shunping
force-pushed
the
prism-server-singleton
branch
from
4122723 to
3975e89
Compare
shunping
force-pushed
the
prism-server-singleton
branch
from
c670b1b to
892356d
Compare
shunping
force-pushed
the
prism-server-singleton
branch
from
7c48491 to
0956a9d
Compare
|
Assigning reviewers. If you would like to opt out of this review, comment R: @claudevdm for label python. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| debug_options = options.view_as(pipeline_options.DebugOptions) | ||
| get_job_server = lambda: job_server.StopOnExitJobServer( | ||
| PrismJobServer(options)) | ||
| if debug_options.lookup_experiment("enable_prism_server_singleton"): |
There was a problem hiding this comment.
when using this with Interactive Runner, does this need to be always a singleton?
There was a problem hiding this comment.
+1 can we detect or pass whether interactive runner is being used and if so use a singleton instead of requiring a flag?
There was a problem hiding this comment.
We decided to make it an experiment for the coming release 2.65.0 (#33623 (comment)), so we don't introduce breaking change 2 weeks before branch cut and have more time to bake in.
After we make prism runner as the default local runner, we can probably choose to make singleton the default as well, but this will be upon further discussion.
|
R: @tvalentyn for final approval |
Follow-up to #34616, addressing #33623.
Introduces an experimental feature for Prism Runner to maintain only a single background Prism server instance. This is beneficial in environments like Google Colab where long-lived Python processes may trigger Prism Runner multiple times across different pipeline executions.