Investigate use of `numba.config.DISABLE_JIT` for coverage approximations

[brandonwillard]

We're currently using a hackish mock approach to estimating coverage for our Numba implementations; however, this is cumbersome to use and extend. Let's take (another) look at the use of numba.config.DISABLE_JIT for these purposes.

For instance, instead of eval_python_only at

aesara/tests/link/numba/test_basic.py

Line 241 in 404e2d4

eval_python_only(fn_inputs, fn_outputs, inputs, mode=numba_mode)

we would—ideally—have something the following:

from numba.tests.support import override_config

with override_config('DISABLE_JIT', True):
    aesara_numba_fn(*inputs)

A change like that would need to be followed by some pytest runs with coverage enabled, and the results would need to indicate that coverage was obtained for the Python code produced for Numba by Aesara.

[Smit-create]

Looking into this I noticed that numba.config.DISABLE_JIT = True will not affect already compiled functions. To call the pure python functions i.e. without jitted, we should use numba_function.py_func. We need to figure out some way so that aesara.compile.function can recognize a new mode say -- numba_no_jit, and generate the compiled aesara function by using numba_function.py_func.

[brandonwillard]

We need to figure out some way so that aesara.compile.function can recognize a new mode say -- numba_no_jit, and generate the compiled aesara function by using numba_function.py_func.

That sounds good!

[dgerlanc]

Looking into this I noticed that numba.config.DISABLE_JIT = True will not affect already compiled functions. To call the pure python functions i.e. without jitted, we should use numba_function.py_func

So it sounds like even if you use:

from numba.tests.support import override_config

with override_config('DISABLE_JIT', True):
    aesara_numba_fn(*inputs)

So if the function has already been compiled, Numba will still use the compiled function, not the pure Python function?

If so, we can probably change the cache directory dynamically, perhaps with a pytest extension, to handle.