WIP, ENH: support parallel runtests

[tylerjereddy]

• add a runtests.py parallel option
  following Compiler: call ModuleSetup.get_main_dir() instead of method #59

Unfortunately a decent number of tests fail with i.e., python runtests.py -n 10, so this is still a WIP.

Sample traceback below, I'll try to check a bit later today

____________________________________________________________________________________________________________________________________ test_sign_1d_special_cases[in_arr0-float-float32] _____________________________________________________________________________________________________________________________________
[gw3] linux -- Python 3.10.4 /home/tyler/python_310_pykokkos_work/bin/python

in_arr = array([-5. ,  4.5,  nan], dtype=float32), pk_dtype = <class 'pykokkos.interface.data_types.float'>, numpy_dtype = <class 'numpy.float32'>

    @pytest.mark.parametrize("pk_dtype, numpy_dtype", [
            (pk.double, np.float64),
            (pk.float, np.float32),
    ])
    @pytest.mark.parametrize("in_arr", [
        np.array([-5, 4.5, np.nan]),
        np.array([np.nan, np.nan, np.nan]),
    ])
    def test_sign_1d_special_cases(in_arr, pk_dtype, numpy_dtype):
        in_arr = in_arr.astype(numpy_dtype)
        view: pk.View1D = pk.View([in_arr.size], pk_dtype)
        view[:] = in_arr
        expected = np.sign(in_arr)
>       actual = pk.sign(view=view)

tests/test_ufuncs.py:390: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pykokkos/lib/ufuncs.py:257: in sign
    pk.parallel_for(view.shape[0], sign_impl_1d_float, view=view)
pykokkos/interface/parallel_dispatch.py:167: in parallel_for
    func, args = runtime_singleton.runtime.run_workunit(
pykokkos/core/runtime.py:85: in run_workunit
    return self.execute(workunit, module_setup, members, policy=policy, name=name, **kwargs)
pykokkos/core/runtime.py:119: in execute
    module = self.import_module(module_setup.name, module_setup.path)
pykokkos/core/runtime.py:148: in import_module
    module = importlib.util.module_from_spec(spec)
<frozen importlib._bootstrap>:571: in module_from_spec
    ???
<frozen importlib._bootstrap_external>:1176: in create_module
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

f = <built-in function create_dynamic>, args = (ModuleSpec(name='pk_cpp_pk_console_ufuncs_sign_impl_1d_float_OpenMP_kernel_cpython_310_x86_64_linux_gnu_so', loader=<.../github_projects/pykokkos/pk_cpp/pk_console/ufuncs_sign_impl_1d_float/OpenMP/kernel.cpython-310-x86_64-linux-gnu.so'),), kwds = {}

>   ???
E   ImportError: /home/tyler/github_projects/pykokkos/pk_cpp/pk_console/ufuncs_sign_impl_1d_float/OpenMP/kernel.cpython-310-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

<frozen importlib._bootstrap>:241: ImportError

[tylerjereddy]

I'm still studying this a bit. Here is a plot suggesting a potential race condition on compilation vs. opening compiled objects when using pytest-xdist to run tests in parallel (for the sleeping diff below):

--- a/pykokkos/core/runtime.py
+++ b/pykokkos/core/runtime.py
@@ -1,3 +1,4 @@
+import time
 import importlib.util
 import sys
 from typing import Any, Callable, Dict, Optional, Tuple, Type, Union
@@ -145,6 +146,7 @@ class Runtime:
             return sys.modules[module_name]
 
         spec = importlib.util.spec_from_file_location(module_name, module_path)
+        time.sleep(15)
         module = importlib.util.module_from_spec(spec)
         sys.modules[module_name] = module
         spec.loader.exec_module(module)

[tylerjereddy]

A hack like this helps reduce the parallel error rate, but not completely:

--- a/pykokkos/core/runtime.py
+++ b/pykokkos/core/runtime.py
@@ -1,3 +1,5 @@
+import time
+from pathlib import Path
 import importlib.util
 import sys
 from typing import Any, Callable, Dict, Optional, Tuple, Type, Union
@@ -145,6 +147,17 @@ class Runtime:
             return sys.modules[module_name]
 
         spec = importlib.util.spec_from_file_location(module_name, module_path)
+        # poll for the compiled file
+        while not Path(spec.origin).exists():
+            # NOTE: what is a reasonable delay time
+            # here? unfortunately, short times appear
+            # to be able to cause a segfault, perhaps
+            # because the shared object is only partially
+            # written when a load attempt is made?
+            time.sleep(0.3)
+        # even if the file exists, it may not have
+        # been fully flushed?
+        time.sleep(1.0)
         module = importlib.util.module_from_spec(spec)
         sys.modules[module_name] = module
         spec.loader.exec_module(module)

[tylerjereddy]

gh-61 has a more solid explanation I think, and does allow pytest-xdist concurrency finally, with one notable drawback

[kennykos]

Is this PR fixed by #93?