airflow-db-not-allowed error when importing PostgresqlHook in a jupyter notebook and executing with PapermillOperator

[oggers]

Apache Airflow Provider(s)

papermill

Versions of Apache Airflow Providers

apache-airflow-providers-papermill==3.11.3

Apache Airflow version

3.0.6

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Other Docker-based deployment

Deployment details

No response

What happened

I have a jupyter notebook that contains:

from airflow.providers.postgres.hooks.postgres import PostgresHook

when executing this notebook with a PapermillOperator I get the following error:

---------------------------------------------------------------------------
ArgumentError                             Traceback (most recent call last)
Cell In[1], line 1
----> 1 from airflow.providers.postgres.hooks.postgres import PostgresHook

File [~/.local/lib/python3.12/site-packages/airflow/__init__.py:79](http://jupyter-new.local.zabik/user/airflow/lab/tree/bizak/dags/items_available/~/.local/lib/python3.12/site-packages/airflow/__init__.py#line=78)
     73 # Perform side-effects unless someone has explicitly opted out before import
     74 # WARNING: DO NOT USE THIS UNLESS YOU REALLY KNOW WHAT YOU'RE DOING.
     75 # This environment variable prevents proper initialization, and things like
     76 # configs, logging, the ORM, etc. will be broken. It is only useful if you only
     77 # access certain trivial constants and free functions (e.g. `__version__`).
     78 if not os.environ.get("_AIRFLOW__AS_LIBRARY", None):
---> 79     settings.initialize()
     81 # Things to lazy import in form {local_name: ('target_module', 'target_name', 'deprecated')}
     82 __lazy_imports: dict[str, tuple[str, str, bool]] = {
     83     "DAG": (".models.dag", "DAG", False),
     84     "Asset": (".assets", "Asset", False),
   (...)     89     "Dataset": (".sdk.definitions.asset", "Dataset", True),
     90 }

File [~/.local/lib/python3.12/site-packages/airflow/settings.py:618](http://jupyter-new.local.zabik/user/airflow/lab/tree/bizak/dags/items_available/~/.local/lib/python3.12/site-packages/airflow/settings.py#line=617), in initialize()
    616     is_worker = os.environ.get("_AIRFLOW__REEXECUTED_PROCESS") == "1"
    617     if not is_worker:
--> 618         configure_orm()
    619 configure_action_logging()
    621 # mask the sensitive_config_values

File [~/.local/lib/python3.12/site-packages/airflow/settings.py:359](http://jupyter-new.local.zabik/user/airflow/lab/tree/bizak/dags/items_available/~/.local/lib/python3.12/site-packages/airflow/settings.py#line=358), in configure_orm(disable_connection_pool, pool_class)
    353 if SQL_ALCHEMY_CONN.startswith("sqlite"):
    354     # FastAPI runs sync endpoints in a separate thread. SQLite does not allow
    355     # to use objects created in another threads by default. Allowing that in test
    356     # to so the `test` thread and the tested endpoints can use common objects.
    357     connect_args["check_same_thread"] = False
--> 359 engine = create_engine(SQL_ALCHEMY_CONN, connect_args=connect_args, **engine_args, future=True)
    360 async_engine = create_async_engine(SQL_ALCHEMY_CONN_ASYNC, future=True)
    361 AsyncSession = sessionmaker(
    362     bind=async_engine,
    363     autocommit=False,
   (...)    366     expire_on_commit=False,
    367 )

File <string>:2, in create_engine(url, **kwargs)

File [~/.local/lib/python3.12/site-packages/sqlalchemy/util/deprecations.py:375](http://jupyter-new.local.zabik/user/airflow/lab/tree/bizak/dags/items_available/~/.local/lib/python3.12/site-packages/sqlalchemy/util/deprecations.py#line=374), in deprecated_params.<locals>.decorate.<locals>.warned(fn, *args, **kwargs)
    368     if m in kwargs:
    369         _warn_with_version(
    370             messages[m],
    371             versions[m],
    372             version_warnings[m],
    373             stacklevel=3,
    374         )
--> 375 return fn(*args, **kwargs)

File [~/.local/lib/python3.12/site-packages/sqlalchemy/engine/create.py:514](http://jupyter-new.local.zabik/user/airflow/lab/tree/bizak/dags/items_available/~/.local/lib/python3.12/site-packages/sqlalchemy/engine/create.py#line=513), in create_engine(url, **kwargs)
    511 kwargs.pop("empty_in_strategy", None)
    513 # create url.URL object
--> 514 u = _url.make_url(url)
    516 u, plugins, kwargs = u._instantiate_plugins(kwargs)
    518 entrypoint = u._get_entrypoint()

File [~/.local/lib/python3.12/site-packages/sqlalchemy/engine/url.py:738](http://jupyter-new.local.zabik/user/airflow/lab/tree/bizak/dags/items_available/~/.local/lib/python3.12/site-packages/sqlalchemy/engine/url.py#line=737), in make_url(name_or_url)
    725 """Given a string or unicode instance, produce a new URL instance.
    726 
    727 
   (...)    734 
    735 """
    737 if isinstance(name_or_url, util.string_types):
--> 738     return _parse_url(name_or_url)
    739 else:
    740     return name_or_url

File [~/.local/lib/python3.12/site-packages/sqlalchemy/engine/url.py:799](http://jupyter-new.local.zabik/user/airflow/lab/tree/bizak/dags/items_available/~/.local/lib/python3.12/site-packages/sqlalchemy/engine/url.py#line=798), in _parse_url(name)
    796     return URL.create(name, **components)
    798 else:
--> 799     raise exc.ArgumentError(
    800         "Could not parse SQLAlchemy URL from string '%s'" % name
    801     )

ArgumentError: Could not parse SQLAlchemy URL from string 'airflow-db-not-allowed:///'

What you think should happen instead

It should allow using dbhooks and airflow connection inside a jupyter notebook.

How to reproduce

Create a notebook with only:

from airflow.providers.postgres.hooks.postgres import PostgresHook

and execute it with PapermillOperator

Other hooks like MsSqlHook also fails.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

This is expected. Airflow worker does not have access to Airflow DB any more. You seem to want to use Airfow DB by the hook of yours - if you want to do it, you need to define the connection. Apparently you want to initialize airflow from within the jupyter notebook and for some reason the notebook strips _AIRFLOW__REEXECUTED_PROCESS that should be set inside your worker - make sure it is not stripped - otherwise it will try to reinitialize airflow ORM which should not happen in worker.

[oggers]

No, I don't want to use the Airflow DB, in fact I want to use another postgresql database that has no relation with Airflow. Using another hook, like MsSqlHook it also fails.