Hello!
I came across the code the other day and I would like to ask some questions regarding parallelising the code and doing some micro optimizations.
In my fork I've implemented some functionality as external compiled code and parallelized it with OpenMP - I can get around a 2x speedup but it stops around there because some of the python functions serialize the code and the speedup decreases.
One example is the function:
if proj_to.crs.is_geographic:
geod = pyproj.Geod(ellps='WGS84')
rot_angle_vectors_rad = np.radians(
geod.inv(x2, y2, x2_delta, y2_delta)[0])We can approximate this with:
_e = 0.0818191908426215 # WGS84 first eccentricity
_e2 = _e ** 2
phi = np.radians(y2)
sin2 = np.sin(phi)**2
lon_lat_ratio = np.cos(phi) * (1 - _e2 * sin2) / (1 - _e2)
rot_angle_vectors_rad = np.arctan2(
(x2_delta - x2) * lon_lat_ratio, y2_delta - y2)But this causes some small changes in the test values (because it is an approximation) but it is a lot faster. The error is small and I don't think it should affect things much unless we're outside the range of the approximation. We can make it optional so that the pyproj.Geod can be used.
Another small one is replacing the python any with np.any and that helps a lot.
I wrote the external add-on in Fortran and used OpenMP to parallelize. I can set things up so that we can use OpenMP to access GPUs (should be portable across NVIDIA, AMD, Intel) instead of writing cuda kernels which might be complex to port/maintain.
The super advantage is that your heavy compute kernels are very well validate in Python so writing a Fortran version should be very simple to validate and maintain. If you'd like multi-GPU support it is best done with MPI we can write a small mpi4py wrapper for distributed compute and then bind each rank to a GPU. Adding the external Fortran kernels will be a big addition and there are multiple ways to do this nicely:
- inside the main repo as a fopendrift (f as in Fortran lol)
- as a git submodule
- as an external library that can live within the OpenDrift organization and will be pulled at build time or we can publish it as a conda-forge package and linked?
The numpy versions are already very good. I compared function to function the speedups were around 2-6x, with the exception of doing a full port of the vertical mixing layer code where I got around a 30x speedup but everything gets bottlenecked further by some python bits later.
So if I could get some feedback and ideas on this it'd be lovely. I'd love to contribute to the parallelization of the code and the possible GPU-ification.
Hello!
I came across the code the other day and I would like to ask some questions regarding parallelising the code and doing some micro optimizations.
In my fork I've implemented some functionality as external compiled code and parallelized it with OpenMP - I can get around a 2x speedup but it stops around there because some of the python functions serialize the code and the speedup decreases.
One example is the function:
We can approximate this with:
But this causes some small changes in the test values (because it is an approximation) but it is a lot faster. The error is small and I don't think it should affect things much unless we're outside the range of the approximation. We can make it optional so that the
pyproj.Geodcan be used.Another small one is replacing the python
anywithnp.anyand that helps a lot.I wrote the external add-on in Fortran and used OpenMP to parallelize. I can set things up so that we can use OpenMP to access GPUs (should be portable across NVIDIA, AMD, Intel) instead of writing cuda kernels which might be complex to port/maintain.
The super advantage is that your heavy compute kernels are very well validate in Python so writing a Fortran version should be very simple to validate and maintain. If you'd like multi-GPU support it is best done with MPI we can write a small
mpi4pywrapper for distributed compute and then bind each rank to a GPU. Adding the external Fortran kernels will be a big addition and there are multiple ways to do this nicely:The numpy versions are already very good. I compared function to function the speedups were around 2-6x, with the exception of doing a full port of the vertical mixing layer code where I got around a 30x speedup but everything gets bottlenecked further by some python bits later.
So if I could get some feedback and ideas on this it'd be lovely. I'd love to contribute to the parallelization of the code and the possible GPU-ification.