Granular env-based solution for "connection string"-like config

[slint]

It's common practice to be able to configure services that rely on connection strings/URIs (e.g. DB, OpenSearch, RabbitMQ, Redis) by individually setting parts of the string via env variables. This allows:

• reusing config in sub-charts (e.g. passing RABBITMQ_USER to both the web/worker config, but also the RabbitMQ sub-chart)
• placing only sensitive information of the config in a secret and exposing as an env var

On the application-side building the SQLALCHEMY_DATABASE_URI config would look something like:

import os

# First check if the full value is set
if os.environ.get("SQLALCHEMY_DATABASE_URI"):
    SQLALCHEMY_DATABASE_URI = os.environ.get("SQLALCHEMY_DATABASE_URI")
else:
    # Check parts of the config
    db_host = os.environ.get("DB_HOST")
    db_password = os.environ.get("DB_PASSWORD")
    db_user = os.environ.get("DB_USER")
    db_name = os.environ.get("DB_NAME")
    db_port = os.environ.get("DB_PORT")

    if all([db_host, db_password, db_user, db_name, db_port]):
        SQLALCHEMY_DATABASE_URI = f"postgresql://{db_user}:{db_password}@{db_host}:{db_port}/{db_name}"
    else:
        SQLALCHEMY_DATABASE_URI = f"postgresql://invenio:password@localhost:5432/invenio"

Important

This requires development in the application modules that configure services like the DB, OpenSearch, RabbitMQ, etc. so that they support loading this type of config.

[lindhe]

Ah, yes! This is an important improvement, in my opinion!

An adjacent topic to this is if we can provide credentials using secrets! It would be a great improvement on security if we could set all credentials using secrets (which is not possible today, if I recall correctly). We may also want to consider using credentials from the secrets generated by our chart dependencies.

I think these topics should be part of this issue, but if you think otherwise I can create a separate issue for them.

[slint]

@lindhe exactly, I tried to (badly 😅) summarize this in:

placing only sensitive information of the config in a secret and exposing as an env var

We touched a bit on this on Discord, and I brought it up at the InvenioRDM workshop last week, where people agreed based on their experience with other Helm Charts approach to secrets.