Investigate human-based flows behavior with distributed engines

[adutra]

The driver process is generally responsible for initializing the RESTCatalog instance, and thus interacts the most with the catalog server. Human-based flows work generally well in this setup, as long as the driver process is interacting with a human operator.

Things get more complicated though when FileIO instances, created by executors, need to interact with the catalog server. This can happen e.g. when using S3 request signing. Each signer will have its own AuthManager, but this time, there won't be any human operator available, so the flow will timeout and fail.

As of now, I think human-based flows are completely incompatible with S3 request signing and also object storage credentials refreshing. Basically any interaction between executors and the catalog server would fail.

We should investigate ways to improve this.

[adutra]

A few ideas:

• The driver node holding the catalog instance could share it's access token and refresh token with the worker nodes.
  • Pros: easy
  • Cons: sharing a refresh token is not great; would not work if refresh token rotation is enforced (that is, each refresh token can only be used once)
• The driver node holding the catalog instance could share it's access token only, then the worker nodes would exchange that token
  • Pros: less prone to failures or security concerns
  • Cons: needs additional configuration so that worker nodes can perform the token exchange appropriately. Worker nodes may need to create or obtain an actor token.

[adutra]

In any case, a change in Iceberg Core is required. More specifically, the following method needs to be modified:

https://github.com/apache/iceberg/blob/e56ec11215e7b9fec6ffd1ea435e008852454db0/core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java#L993-L1002

This method is the "entry point" for table-scoped FileIO instances, before they get distributed to worker nodes. We'd need the ability to modify the properties to "inject" the additional tokens and/or modify the configuration. Example:

  private FileIO tableFileIO(
          SessionContext context, Map<String, String> config, AuthSession tableSession, List<Credential> storageCredentials) {
    if (config.isEmpty() && ioBuilder == null && storageCredentials.isEmpty()) {
      return io; // reuse client and io since config/credentials are the same
    }

    Map<String, String> fullConf = RESTUtil.merge(properties(), config);
    fullConf = RESTUtil.merge(fullConf, tableSession.remoteFileIOProperties());
    
    return newFileIO(context, fullConf, storageCredentials);
  }

[adutra]

There might be a 3rd idea that wouldn't require changes in Iceberg Core: the table configuration is created by merging the table-specific properties onto the catalog properties:

Map<String, String> fullConf = RESTUtil.merge(properties(), config); // properties() returns the catalog properties

We can notice that session context properties, if present, are not added to the final table configuration.

This may have been done on purpose. One could leverage this peculiarity and reserve human-based flows to session contexts exclusively. Example:

Auth Session Level	Contents
Catalog	Client ID + client secret, optionally scope, etc. (implicit grant type: `client_credentials`)
Session Context	Grant type: `authorization_code` (or `device_code`)
Table	(empty)

With the above configuration, one would need to create a catalog with a non-empty Session Context. This cannot be donne with configuration only (e.g. Spark SQL wouldn't work), but I assume it can be easily done in a Spark Shell session or using a Python script.

In that case the Spark driver process holding the RESTSessionCatalog instance would use authorization_code to authenticate the user (defined by the Session Context properties), while all other Spark nodes would use client_credentials (defined by the Catalog properties, propagated to FileIO instances).

(to be tested)

[adutra]

Another way to enable human-based flows only on the driver node is to leverage environment variables or credentials files on disk, cf. #78.

E.g. the driver node could have a credentials file specifying Authorization Code grant type, while worker nodes would either inherit the default grant type from the catalog properties, or also read environment vars or credentials files, using different a grant type and/or client ID+secret.