Centralized Service for Auth & Credential Vending

[Yacobolo]

Hello everyone,

First off, I'm really excited about the potential of DuckLake. The concept is fantastic.

I have a question regarding the current architecture for security and access control. My understanding is that the DuckDB client requires direct credentials for both the object storage backend (e.g., S3) and the PostgreSQL metadata store. In a multi-user environment, providing every client with direct, long-lived credentials can pose security and management challenges.

This led me to wonder if there are any plans or thoughts around introducing a centralized service layer. Such a service could act as a broker:

1. A user would authenticate against this new service.
2. The service would look up the user's permissions in the PostgreSQL metadata store.
3. Based on those permissions, the service would "vend" temporary, scoped-down credentials (like AWS STS tokens, for example) that the client can then use to access the object storage directly.

I realize it's still early days for the project, but I believe this architectural pattern could be crucial for adoption in larger, security-conscious organizations.

Has this been discussed before, or what are the community's thoughts on this topic?

[staticlibs]

@Yacobolo

Hi! #291 maybe somewhat related to this (though Java-specific).

Another thought more specifically on the auth - I think that the "centralized service layer" you are describing sounds pretty similar to a Kerberos server (perhaps with a non-trivial setup).

[djouallah]

If you want credentials vending and this sort of stuff, iceberg rest Catalog is a better option for now