Problem running multiple processes

[mingfang]

When num_processes > 1, running multiple pipelines results in this error:

Traceback (most recent call last):
  File "/home/coder/unstructured-api/run.py", line 171, in <module>
    ).run()
      ^^^^^
  File "/home/coder/.pyenv/versions/3.12.11/lib/python3.12/site-packages/unstructured_ingest/pipeline/pipeline.py", line 162, in run
    self._run()
  File "/home/coder/.pyenv/versions/3.12.11/lib/python3.12/site-packages/unstructured_ingest/pipeline/pipeline.py", line 223, in _run
    f"running local pipeline: {self} with configs: {self.context.model_dump_json()}"
                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/coder/.pyenv/versions/3.12.11/lib/python3.12/site-packages/pydantic/main.py", line 528, in model_dump_json
    return self.__pydantic_serializer__.to_json(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.PydanticSerializationError: Unable to serialize unknown type: <class 'multiprocessing.managers.DictProxy'>

My guess it's caused by _wrap_mp() here

unstructured-ingest/unstructured_ingest/pipeline/interfaces.py

Line 99 in 23ada7c

pool.imap_unordered(func=self._wrap_mp, iterable=iterable),

Workaround is to set num_processes = 1