Deadlock with multiprocessing (using fork) and OpenMP

Q: Deadlock with multiprocessing (using fork) and OpenMP

Problem: Python code not working properly in another process.

To reproduce the problem, please use this code:

import open3d as o3d
import numpy as np
from multiprocessing import Process, set_start_method


def test():
    all_pts = np.array([[108.46383020247207, 350.5646727751972, 116.3624462382868],
                     [112.25361397099844, 347.0114607994641, 116.26706010755152],
                     [113.18366122221072, 361.2549821304686, 116.27959668822587],
                     [111.91097601143456, 360.8370105069897, 116.30449797399342]])
    point_cloud = o3d.geometry.PointCloud()
    point_cloud.points = o3d.utility.Vector3dVector(all_pts)
    print("before")
    point_cloud.estimate_normals(o3d.geometry.KDTreeSearchParamHybrid(radius=100, max_nn=22))
    print("after")


if __name__ == '__main__':
    print('open3d version:', o3d.__version__)

    # set_start_method('spawn') # spawn, fork (default on Unix), forkserver

    process = Process(target=test, args=())
    process.start()
    process.join()

    test()

    process = Process(target=test, args=())
    process.start()
    process.join()

Output on unix:

open3d version: 0.9.0.0
before
after
before
after
before
(hangs there)

If uncomment set_start_method('spawn'), it will finish properly.

Related issue: Deadlock with multiprocessing (using fork) and OpenMP / PyTorch should warn after OMP and fork that multithreading may be broken #17199

It's caused by GNU OpenMP. Maybe compile againt Intel OpenMP?

A:

This seems to be a common issue with fork. The advice for Python multiprocessing is to use forkserver for multithreaded programs on Linux.

https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

For more detailed information, please see our Issues page at: https://github.com/isl-org/Open3D/issues/1552#issuecomment-1009478420