Multithreaded patch rendering through mplcairo backend

[andrewdnolan]

The mplcairo package reimplements a complete cairo backend for Matplotlib. Of most interest to us, it provides optional multithreaded drawing of markers and path collections, which is currently the most expensive part of mosiac.

[xylar]

@andrewdnolan, I've got linux and osx build of mplcairo working in:
https://github.com/conda-forge/staged-recipes/pull/29465/files
I'm still working on win builds but will disable win if I can't get them working (someone could always follow up).

[andrewdnolan]

Awesome! Thanks @xylar.

[xylar]

It doesn't look like mplcairo's threading is attractive for us. But even so, it is now available on conda-forge:
https://github.com/conda-forge/mplcairo-feedstock

[andrewdnolan]

Interestingly enough, I've tested on perlmutter and multithreaded cases seem to take longer.... Here's the script I'm using:

import sys
import mosaic
import matplotlib
import matplotlib.pyplot as plt
import mplcairo
import tempfile
import xarray as xr

def main():

    fp = "/global/cfs/cdirs/e3sm/inputdata/ocn/mpas-o/IcoswISC30E3r5/mpaso.IcoswISC30E3r5.20231120.nc"

    ds = xr.open_dataset(fp).squeeze()

    # create the figure and a GeoAxis
    fig, ax = plt.subplots(1, 1, )

    descriptor = mosaic.Descriptor(ds)

    # using the `Descriptor` object we just created, make a pseudocolor plot of
    # the "indexToCellID" variable, which is defined at cell centers.
    collection = mosaic.polypcolor(ax, descriptor, ds.indexToCellID, antialiaseds=True)

    fig.colorbar(collection, fraction=0.1, shrink=0.5, label="Cell Index");

    with tempfile.NamedTemporaryFile() as fp:
        fig.savefig(f"{fp.name}.png")

    plt.close()

if __name__ == '__main__':
    
    matplotlib.use("module://mplcairo.base")

    if len(sys.argv) == 2:
        N = int(sys.argv[1])
    else:
        N = 0

    with mplcairo.set_options(collection_threads=N):
        main()

[xylar]

Threads have an overhead associated with them so maybe no huge surprise. How about if you try a much larger mesh?

[xylar]

Probably a risky idea on a login node, though. Watch the memory!

[andrewdnolan]

Ahh gotcha. I've got a test running with the RRSwISC6to18E3r5 on a compute node, looping over the same number of threads as above. Will report back when that's done.

[andrewdnolan]

Unfortunately the pattern seems to be persistent on a compute node and with a larger mesh (results below are for RRSwISC6to18E3r5).

[andrewdnolan]

I think to understand things a bit better, I'll have to do some line-by-line profiling, using the mplcairo backend and see if draw_path_collection is where the bottleneck is. Cause if not, then the additional overhead of the threads definitely suggest it's not worth it.

I think still pursuing our own draw_poly_mesh (a la draw_quad_mesh) could be fruitful, since we could avoid creating paths in matplotlib/python and do all our "drawing" through the respective C code in the preferred backend (e.g. cairo).