Skip to content

Can't use num_workers > 0 on Mac #1562

Description

@lewfish

Raster Vision can be installed and used directly on a Mac (using macOS 12.1 and Python 3.9). Everything seems to work except I cannot use num_workers > 0. This has something to do with the fact that as of Python 3.8 on Mac (but not Linux), the default multiprocessing backend has changed from fork to spawn. See this issue.

This problem can be replicated using the following command and setting the num_workers argument in SemanticSegmentationGeoDataConfig in https://github.com/azavea/raster-vision/blob/master/integration_tests/semantic_segmentation/config.py.

python -m rastervision.pipeline.cli run inprocess integration_tests/semantic_segmentation/config.py \
  -a root_uri /Users/lewfish/data/int-test/ -a data_uri ./integration_tests/semantic_segmentation/

The error:

  File "/Users/lewfish/projects/raster-vision/rastervision_pytorch_backend/rastervision/pytorch_backend/pytorch_learner_backend.py", line 120, in train
    learner.main()
  File "/Users/lewfish/projects/raster-vision/rastervision_pytorch_learner/rastervision/pytorch_learner/learner.py", line 264, in main
    self.plot_dataloaders(self.cfg.data.preview_batch_limit)
  File "/Users/lewfish/projects/raster-vision/rastervision_pytorch_learner/rastervision/pytorch_learner/learner.py", line 895, in plot_dataloaders
    self.plot_dataloader(
  File "/Users/lewfish/projects/raster-vision/rastervision_pytorch_learner/rastervision/pytorch_learner/learner.py", line 886, in plot_dataloader
    x, y = next(iter(dl))
  File "/opt/homebrew/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 444, in __iter__
    return self._get_iterator()
  File "/opt/homebrew/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 390, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/opt/homebrew/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1077, in __init__
    w.start()
  File "/opt/homebrew/Cellar/python@3.9/3.9.12_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/homebrew/Cellar/python@3.9/3.9.12_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/opt/homebrew/Cellar/python@3.9/3.9.12_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/opt/homebrew/Cellar/python@3.9/3.9.12_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/homebrew/Cellar/python@3.9/3.9.12_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/homebrew/Cellar/python@3.9/3.9.12_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/homebrew/Cellar/python@3.9/3.9.12_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "stringsource", line 2, in rasterio._io.DatasetReaderBase.__reduce_cython__
TypeError: self._hds cannot be converted to a Python object for pickling

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions