[Bug][Prism]: Colab cannot run PrismRunner multiple times

[liferoad]

What happened?

Steps to reproduce the issue:

1. Go to https://colab.research.google.com/
2. Create a new notebook
3. Install beam

!pip3 install apache-beam[gcp]

4. ignore any warnings and no need to restart the session
5. Do this in a cell:

import apache_beam as beam
from apache_beam.runners.portability.prism_runner import PrismRunner
with beam.Pipeline(runner=PrismRunner()) as p:
  N =5 
  p | "Create Elements" >> beam.Create(range(N)) | "Squares" >> beam.Map(lambda x: x**2) | "Print" >> beam.Map(print)

6. the first run should work well but if you run the cell second time, the below error occurs:

OSError: [Errno 26] Text file busy: '/root/.apache_beam/cache/prism/bin/apache_beam-v2.61.0-prism-linux-amd64'

7. Check the running process:

/root/.apache_beam/cache/prism/bin/apache_beam-v2.61.0-prism-linux-amd64 is still running. If you kill this process, step 5 will run fine.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[lostluck]

Thank you for filing a clear issue. When you spoke to me offline, it wasn't clear this was a problem on second invocations of a pipeline run that caused the problem.

---

I don't see any harm in having the prism process persist. It's set to auto close when Python exits, which is probably a long lived interpreter in this context.

But it is a problem that we aren't taking advantage of the existing caching mechanism probably.

I think all that we need to do is set a class level handle that keeps the existing server instance on the first startup of prism, and then retain that for subsequent invocations, and it avoids the issue of trying to start a new one.

[lostluck]

Reading the error more closely: "Text file busy" https://stackoverflow.com/questions/16764946/what-generates-the-text-file-busy-message-in-unix

That happens when a binary is attempted to be modified while the binary is still running.

This better explains the symptoms:

1. The process is already executing because Python hasn't exited yet.
2. The re-run tries to download and start another instance.
3. Because of 1, the writing in 2 fails.

So, the solution there is fixing the caching logic to correctly use the cache. That's one bug.

Then we'd need to use the idle_timeout flag like we do for Java to avoid having a long set of prism processes sitting around the colab instance forever.

[shunping]

@lostluck, @damccorm: If I understand this correctly, either making prism server a singleton or keeping it as what it is but adding idle_timeout will solve this problem.

Regarding the singleton approach(draft PR at #34623), do we want to make it the default?

[damccorm]

Thanks - I'd say lets hold off on making it default until the following release (2.65 is cut on April 30, we can target 2.66); I'm planning on making Prism default everywhere at that point (I was going to exclude colab environments), so this fits well with that. It will also give us plenty of time to observe before the release and make sure we patch any obvious breakages.

We can make the patch itself now though

[shunping]

Thanks - I'd say lets hold off on making it default until the following release (2.65 is cut on April 30, we can target 2.66); I'm planning on making Prism default everywhere at that point (I was going to exclude colab environments), so this fits well with that. It will also give us plenty of time to observe before the release and make sure we patch any obvious breakages.

We can make the patch itself now though

In that case, I will keep the current prism behavior (i.e. creating new server for every pipeline runs) but add a singleton mode which will only be activated if a prism runner experiment option is set. Sounds good?