Update to Airflow 2.0

[wseaton]

Features like OAuth/LDAP Group Sync likely won't get back-ported to Airflow 1.10.X [1], so if we care about providing the ability for customers to deploy Airflow properly in a multi-tenant environment we should probably take a look at upgrading (or at least providing a compatible branch) for the 2.0 Airflow community image.

I've done some testing and besides changing the build process, the only glaring incompatibilities between the Puckel image (ours) and the new community image is how some of the service environment variables are injected into the container at runtime. We should be able to easily shim around this using environment variable templating.

Another thing I've noticed in testing 2.0 is that

airflow worker 

Has changed to the more specific:

airflow celery worker

So we'll need to address that in the worker code as well.

• 1: https://airflow.apache.org/docs/apache-airflow/stable/upgrading-to-2.html#support-for-airflow-1-10-x-releases

[anuj812]

Any support from opendatahub to upgrade or install airflow 2.x