[RFC] Add `run_pipeline_in_subinterpreter` function

[mthrok]

Python 3.14 added a new type of executor called InterpreterPoolExecutor.
The following is excerpt from the release note and the API doc.

With multiple isolated interpreters, you can take advantage of a class of concurrency models, like Communicating Sequential Processes (CSP) or the actor model, that have found success in other programming languages, like Smalltalk, Erlang, Haskell, and Go. Think of multiple interpreters as threads but with opt-in sharing.

Regarding multi-core parallelism: as of Python 3.12, interpreters are now sufficiently isolated from one another to be used in parallel (see PEP 684). This unlocks a variety of CPU-intensive use cases for Python that were limited by the GIL.

Using multiple interpreters is similar in many ways to multiprocessing, in that they both provide isolated logical “processes” that can run in parallel, with no sharing by default. However, when using multiple interpreters, an application will use fewer system resources and will operate more efficiently (since it stays within the same process). Think of multiple interpreters as having the isolation of processes with the efficiency of threads.

the worker serializes the callable and arguments using pickle when sending them to its interpreter. The worker likewise serializes the return value when sending it back.

I quickly tried this with SPDL on my local env, and this new executor is compatible with SPDL Pipeline.

In the following code, one stage is delegated to the interpreter worker.

from spdl.pipeline import PipelineBuilder
from concurrent.futures import InterpreterPoolExecutor

executor = InterpreterPoolExecutor(max_workers=3)
pipeline = (
    PipelineBuilder()
    .add_source(range(10))
    .pipe(lambda x: x * 2, executor=executor)
    .add_sink()
    .build(num_threads=1)
)
with pipeline.auto_stop():
for i in pipeline:
    print(i)

What happens here is that for each invocation of the piped function, lambda and its argument x from the source, is pickled and unpickled.

Repeated pickling/unpickling has non-negligible impact on the performance, so likely this is not the way you want to use InterpreterPoolExecutor with SPDL.

In SPDL there is a run_pipeline_in_subprocess function, which executes multi-threaded pipeline in a subprocess. Only the final data are sent back to the main process. It still uses pickle but in some pipeline, but in some cases it is the faster than multi-threading or multi-processing.

I think the way SPDL would adopt InterpreterPoolExecutor is similar to this approach. When delegating the data loading to outside of the main process/interpreter, the number of Python objects that the main process has to deal with is decreased, so the overhead of garbage collection is reduced.

We can add run_pipeline_in_subinterpreter function in a similar manner as run_pipeline_in_subprocess.