Currently we have all optional dependencies manually specified, we should add some sort of logic to do this programatically so that new optional dependencies don't require an update to the CI image. setuptools doesn't currently seem to have a good way to do this. We could attempt parsing the setup.py ourselves, but I think it would be preferable to add a flag to setuptools to enable this if at all possible.