numba-scipy is now accepting PRs, discuss what to focus on first!

[stuartarchibald]

As title... place holder ticket for discussion over what to focus on first. scipy.special and scipy.sparse were suggested here: numba/numba#4292 (comment)

[luk-f-a]

I don't think I can help much with scipy.special (C integration is not my strong suit) but once scipy.special is available I will look into bringing in the statistical distributions

[astrojuanlu]

I would love to see scipy.integrate :)

[person142]

I can do scipy.special. @Juanlu001 what parts of integrate are you most interested in seeing? Something like quad would be a royal pain, but if we’re talking e.g. Gaussian quadrature that’s much more approachable.

[person142]

(When it comes to adaptive quadrature it would probably be good to do some work on the SciPy side first vs. reproducing quadpack here.)

[stuartarchibald]

@person142 I agree, unless there's going to be some incredible performance (or other) benefit gained by translating core numerical libraries to Numba then creating a clean API on the callee side to make it easy for Numba make make the call is preferable.

[astrojuanlu]

@person142 I was thinking more of accelerating the new Initial Value Problem solvers introduced in SciPy 1.0, which are much more feature-rich than odeint and ode but have a performance overhead because they're written in pure Python. I acknowledge though that it's not a trivial problem, but the question was open so... 😇

[stuartarchibald]

@person142 if you are happy to do scipy.special that'd be great, thanks. There's a tiny module in numba-scipy already for this, I was using it for CI and to get something uploaded to PyPI, feel free to wipe it with actual contents!

[person142]

I was thinking more of accelerating the new Initial Value Problem solvers introduced in SciPy 1.0, which are much more feature-rich than odeint and ode but have a performance overhead because they're written in pure Python

Hm that is a quite interesting case. Could be that we should work on moving some stuff into Cython on the SciPy side to reduce overhead and then make sure it integrates well with Numba jitted functions...

[stuartarchibald]

@luk-f-a "Basic" support for scipy.special is added in the now merged #13 if you are interested in writing statistical distribution implementations? Thanks.

[luk-f-a]

thanks for letting me know @stuartarchibald . I started looking at it and I have a general question: should we aim to preserve the structure of the Scipy code?

For example, for distributions I would need one auxiliary function from scipy/_lib/_util.py. Should I create a numba-scipy/_lib/_util.py file and put the jit-version in there?

[stuartarchibald]

@luk-f-a good point, perhaps we try with this approach and see how it goes ? There may be issues tracking upstream refactoring/places where it's just not reasonable to do this, however, it's probably worth a try as a predefined order might help.

[gioxc88]

Hello I am following the discussion with great interest and would like to ask how does it work exactly.

1. Do I need to import the package like import numba-scipy?
2. Does it work with scipy.stats? for example scipyp.stats.chi2.rvs() scipy.stats.t.cdf() sp.stats.t.ppf()
3. If not do you suggest any other method to speed up scipy functions (especially inversion of student t is very slow)
4. Where can I find information of how to help for integration between numba and scipy? I would like to contribute

many thanks

[luk-f-a]

hi @gioxc88 , it's great to hear that you're interested and even considering contributing. I started some work on stats, building norm.rvs as an example to test out the approach. But this has not been merged yet, so we're in a very early stage!!

to your questions:

1. the idea is that you won't need to, Numba will recognize the calls to Scipy and call numba-scipy in the same way that it does with Numpy.
2. it will eventually, but which distribution works will depend on the community interest and contributions. There are 120 distributions in total, it will take some time. We're looking into how adding new distributions could be simplified.
3. Is your problem the time that t.ppf takes for each calculation, or that you might be calling it many times?
4. Stay tuned on this issue for now, I assume that over time there'll be contributing guidelines. In particular, for stats I'd like to have a nice tutorial on adding new distributions.

ping @stuartarchibald

[gioxc88]

hello @luk-f-a
thank you for your answer. I relied on the community for too long and now I think it's time for me to contribute.

I will certainly follow this issue.

About the t.ppf I need to call it in a loop 5 million times, for an a input array that has shape [65, 13]
This is because if I call it for a 3d array of shape [65, 13, 5000000] if won't fit my RAM.
But I tried with a 64Gb RAM pc to call it one single using the big 3d array and it's still extremely slow compared do t.cdf. And when I say extremely slow I am talking about t.cdf being executed in like 1 minute while the t.ppf takes more than 30 mins.