Can someone explain more about the security/auth model of DuckLake?

[crazy-treyn]

I am using Postgres as the catalog database, and I've tried both Azure ADLS Gen2 (via fsspec) and Cloudflare R2 for storage accounts, and in both cases I had to have secrets loaded up for both Postgres and the storage locations in order to run queries on tables of data.

My question: Is this by design? From the FAQ, it sounded like all auth would be handled by the catalog database once instantiated, in this case Postgres. But in my testing, it seems that in order to query the data, you have to authenticate to both Postres and the storage location.

[djouallah]

@crazy-treyn duckdlake does not handle credential vending, it means you have to manage your own storage credential

[Tishj]

We would like to address this problem in the future, there's no reason the storage credentials can't be saved in the metadata catalog as well

[gahtan-syarif]

We would like to address this problem in the future, there's no reason the storage credentials can't be saved in the metadata catalog as well

@Tishj i wonder if this is a good idea? Having cloud storage credentials be seperated from the metadata catalog means each user can connect to ducklake using their own access keys tied to their own IAM roles. You can't do this when storage credentials are directly stored in a metadata catalog.

[gahtan-syarif]

Seperating storage credentials from the metadata also adds security. This way a compromised postgres credential does not mean a compromised cloud storage.

[dridk]

Using an external credentials service like hashicorp vault would be helpful.