extra conda packages kwarg fails with dask gateway

[dgergel]

When passing the extra_conda_packages kwarg, e.g.
client, cluster = rhgk.get_standard_cluster(extra_conda_packages='cfgrib'), the following error occurs:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-2-eb13a78ebd60> in <module>
----> 1 client, cluster = rhgk.get_standard_cluster(extra_conda_packages='cfgrib')
      2 # cluster.scale(10)

/srv/conda/envs/notebook/lib/python3.8/site-packages/rhg_compute_tools/kubernetes.py in get_standard_cluster(*args, **kwargs)
    542                 "trying to pass worker name, use the ``name`` kwarg."
    543             )
--> 544         return _get_cluster_dask_gateway(profile="standard", **kwargs)
    545     return _get_cluster_dask_kubernetes(*args, scaling_factor=1, **kwargs)
    546 

/srv/conda/envs/notebook/lib/python3.8/site-packages/rhg_compute_tools/kubernetes.py in _get_cluster_dask_gateway(**kwargs)
    181             }
    182         elif k not in list(default_options.keys()) + ["tag"]:
--> 183             raise KeyError(f"{k} not allowed as a kwarg when using dask-gateway")
    184 
    185     if "worker_image" in new_kwargs and "tag" in new_kwargs:

KeyError: 'extra_conda_packages not allowed as a kwarg when using dask-gateway'

[bolliger32]

@dgergel I started to try to fix this but realized the reason we took this out was that extra_pip_packages is recommended over extra_conda_packages when loading packages onto workers, b/c it can be much faster. Does extra_pip_packages meet your needs? Or do you think we want to be able to conda install things. Also, if this is a generally useful package, we could add it to the main worker images (it's already on the coastal images).

[dgergel]

@bolliger32 it would be great if we could add cfgrib to the main worker images, that would solve an issue I'm working through right now - it's a requirement for using the grib backend for xarray, which we need for pre-processing the ERA-5 grib files. My plan was to use it in a virtual env but it would be a lot easier to have it available in the default image. The reason for needing to conda install it is that the pip install isn't working at the moment, even if you pin earlier versions of eccodes. Were there environment issues to work through when it was installed on the coastal images? I found conflicts when I tried to conda install it in the main worker images...