Move REST catalog credentials to USER MAPPING, add catalogs.conf

Until now an iceberg_catalog server carried its own client_id /
client_secret on the SERVER row.  That gave every role on the
cluster the same credentials and forced operators to recreate the
server when secrets rotated.  Move credentials to per-user state
and add a platform-provided file as a middle layer.

Credential resolution (lowest to highest priority):

  1. pg_lake_iceberg.rest_catalog_* GUC defaults
  2. iceberg_catalog SERVER options (non-secret only)
  3. $PGDATA/catalogs.conf            (user-created servers only)
  4. pg_user_mapping options          (user-created servers only)

The built-in pg_lake_rest_catalog (catalog='rest') stops after step 2
on purpose: its credentials live exclusively in the GUCs so the
built-in stays a single, global, instance-wide configuration with no
hidden per-user view.  CREATE/ALTER USER MAPPING is rejected against
all three built-in long names; catalogs.conf is also ignored for the
built-in 'rest' catalog.

Notable pieces:

* iceberg_catalog_option_descs[] now carries a CATALOG_OPT_CTX_*
  bitmask per option.  The same table drives the validator, the
  per-context "Valid options are: ..." hint, and the option->struct
  applier.  client_id / client_secret are USER MAPPING-only; scope is
  accepted on both, with the USER MAPPING value winning because it is
  applied last during resolution.

* RestCatalogOptions gains a umid field; the token cache key is now
  (serverOid, umid) so different SET ROLEs in the same backend each
  get their own user mapping's credentials.  A USERMAPPINGOID syscache
  callback invalidates cached tokens on CREATE/ALTER/DROP USER MAPPING.

* ValidateRestCatalogOptions now performs an early auth-type-aware
  credentials check at resolution time:
    - client_secret is always required
    - client_id is required unless rest_auth_type='horizon'
  Missing credentials surface as "no credentials found for REST
  catalog ..." with ERRCODE_FDW_OPTION_NAME_NOT_FOUND.  The per-field
  checks inside FetchRestCatalogAccessToken are kept as defense in
  depth and now carry the same errcode.

* New ProcessUtility handler RedactRestCatalogUserMappingSecrets
  scrubs client_id / client_secret from queryString in place on
  CREATE/ALTER USER MAPPING for any iceberg_catalog server (built-in
  long names included).  Handles plain '', E'', and U&'' literal
  forms with their escape rules.  Registered after the DDL validator
  so it runs first (the handler list is prepend-LIFO), ensuring the
  failing built-in-server path never leaks secrets into the ereport
  context.  DDL itself reads option values from DefElem->arg, so
  pg_user_mapping still stores plaintext credentials -- only the
  query string surfaces (pg_stat_statements, log_min_duration_statement,
  ereport context) see the redacted form.

* New PGC_SIGHUP GUC pg_lake_iceberg.catalogs_conf_path lets
  operators point at an absolute path; the default is the relative
  'catalogs.conf' resolved against DataDir.

Tests:

* test_iceberg_catalog_server.py picks up the user-mapping DDL
  surface (per-context option lists and hints, valid/invalid options
  on each side, FOR CURRENT_USER with all three options,
  per-role mappings on the same server, dependency-driven rejections,
  built-in long-name blocks), the redaction handler (CREATE/ALTER,
  '' escape, E'' escape, scope preservation, non-iceberg FDW skip,
  built-in-rejection-after-redaction, plaintext storage preservation),
  and credential resolution via catalogs.conf (file-only success,
  USER MAPPING wins over the file, no-credentials-anywhere error,
  scope from the file, absolute catalogs_conf_path, built-in
  ignores the file).

* test_modify_iceberg_rest_table.py: client_id / client_secret are
  dropped from the server-option-overrides-GUC parametrization (they
  no longer belong on SERVER), and a new parametrized
  test_user_mapping_credential_overrides_guc covers the same
  resolution-order direction with USER MAPPING in step 4.

* test_writable_iceberg_common.py: the shared fixture for the writable
  user-created REST server now puts credentials on a PUBLIC user
  mapping, and drops the server with CASCADE to sweep up the mapping.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: sfc-gh-npuka <naisila.puka@snowflake.com>

Author: sfc-gh-npuka

pg_lake_iceberg/include/pg_lake/rest_catalog/rest_catalog.h

@@ -34,25 +34,37 @@ extern char *RestCatalogClientSecret;
 extern char *RestCatalogScope;
 extern int	RestCatalogAuthType;
 extern bool RestCatalogEnableVendedCredentials;
+extern char *CatalogsConfPath;
 
 /*
  * Resolved REST catalog connection options.  All REST catalogs --
  * built-in ('rest') and user-created (CREATE SERVER ... FOREIGN DATA
  * WRAPPER iceberg_catalog) -- are backed by a real pg_foreign_server
- * row; ApplyGUCDefaults populates the defaults, ApplyServerOptionOverrides
- * layers on any per-server options.
+ * row.
  *
- * The canonical identity of a catalog is `serverOid` (the OID of the
- * iceberg_catalog server row).  Use it for in-memory equality, token
- * cache keys, and syscache-driven invalidation.  `catalog` stores the
- * user-visible short name (e.g. 'rest', 'my_polaris') purely for error
- * messages.
+ * Resolution order, lowest to highest priority:
+ *   1. GUC defaults                         (ApplyGUCDefaults)
+ *   2. Server options                       (ApplyServerOptionOverrides)
+ *   3. $PGDATA/catalogs.conf credentials    (user-created servers only)
+ *   4. pg_user_mapping options              (user-created servers only)
+ *
+ * In-memory identity is the pair (`serverOid`, `umid`):
+ *   - serverOid is the iceberg_catalog server's OID.
+ *   - umid is the OID of the pg_user_mapping row that contributed the
+ *     credentials, or InvalidOid when no user mapping was used (built-in
+ *     pg_lake_rest_catalog, or a user-created server whose credentials
+ *     came entirely from catalogs.conf / GUCs).
+ *
+ * `catalog` is the user-visible short name (e.g. 'rest', 'my_polaris')
+ * kept purely for error messages.
  */
 typedef struct RestCatalogOptions
 {
 	Oid			serverOid;		/* iceberg_catalog server OID; canonical
 								 * identity, never InvalidOid for resolved
 								 * opts */
+	Oid			umid;			/* pg_user_mapping row OID that supplied
+								 * credentials, or InvalidOid if none */
 	char	   *catalog;		/* short user-facing name; used in error
 								 * messages, never for equality */
 	char	   *host;
@@ -138,3 +150,11 @@ extern PGDLLEXPORT RestCatalogRequest * GetRemoveSnapshotCatalogRequest(List *re
 
 /* ProcessUtility handler for iceberg_catalog server DDL validation */
 extern PGDLLEXPORT bool ValidateIcebergCatalogServerDDL(ProcessUtilityParams * processUtilityParams, void *arg);
+
+/*
+ * ProcessUtility handler that scrubs client_id / client_secret out of
+ * queryString in CREATE/ALTER USER MAPPING for iceberg_catalog
+ * servers, in place.  Register after ValidateIcebergCatalogServerDDL
+ * so it runs first (the handler list is a prepend-LIFO).
+ */
+extern PGDLLEXPORT bool RedactRestCatalogUserMappingSecrets(ProcessUtilityParams * processUtilityParams, void *arg);

pg_lake_iceberg/pg_lake_iceberg--3.3--3.4.sql

@@ -5,15 +5,36 @@
  * configurations via CREATE SERVER so that users are not limited to a
  * single global REST catalog configured through GUC settings.
  *
- * Example:
+ * Server options (non-secret): rest_endpoint, rest_auth_type,
+ *   oauth_endpoint, scope, enable_vended_credentials, location_prefix,
+ *   catalog_name.
+ * User mapping options (credentials): client_id, client_secret, scope.
+ *
+ * scope is accepted in both server and user mapping; user mapping wins.
+ *
+ * Credential resolution order:
+ *   1. CREATE USER MAPPING for the current user
+ *   2. $PGDATA/catalogs.conf (platform-provided)
+ *   3. GUC variables (backward compatibility)
+ *
+ * User-defined catalog example:
  *   CREATE SERVER my_polaris TYPE 'rest'
  *     FOREIGN DATA WRAPPER iceberg_catalog
- *     OPTIONS (rest_endpoint 'http://polaris:8181',
- *              rest_auth_type 'default',
- *              client_id '...',
- *              client_secret '...');
+ *     OPTIONS (rest_endpoint 'https://polaris.example.com');
+ *
+ *   CREATE USER MAPPING FOR user1 SERVER my_polaris
+ *     OPTIONS (client_id '...', client_secret '...');
  *
  *   CREATE TABLE t (a int) USING iceberg WITH (catalog = 'my_polaris');
+ *
+ * Platform-provided catalog example:
+ *   CREATE SERVER horizon TYPE 'rest'
+ *     FOREIGN DATA WRAPPER iceberg_catalog
+ *     OPTIONS (rest_endpoint 'https://horizon.example.com');
+ *
+ *   -- Credentials in $PGDATA/catalogs.conf:
+ *   --   horizon.client_id = 'platform_id'
+ *   --   horizon.client_secret = 'platform_secret'
  */
 CREATE FUNCTION lake_iceberg.iceberg_catalog_validator(text[], oid)
 RETURNS void

pg_lake_iceberg/src/init.c

@@ -321,6 +321,16 @@ _PG_init(void)
 							 GUC_SUPERUSER_ONLY | GUC_NO_SHOW_ALL | GUC_NOT_IN_SAMPLE,
 							 NULL, NULL, NULL);
 
+	DefineCustomStringVariable("pg_lake_iceberg.catalogs_conf_path",
+							   gettext_noop("Path to the catalog credentials file. "
+											"Defaults to $PGDATA/catalogs.conf."),
+							   NULL,
+							   &CatalogsConfPath,
+							   "catalogs.conf",
+							   PGC_SIGHUP,
+							   GUC_SUPERUSER_ONLY,
+							   NULL, NULL, NULL);
+
 	DefineCustomBoolVariable("pg_lake_iceberg.unsupported_numeric_as_double",
 							 gettext_noop("When enabled, numeric columns that cannot be represented "
 										  "as Iceberg decimals (unbounded or precision > 38) are "
@@ -335,6 +345,14 @@ _PG_init(void)
 	AvroInit();
 
 	RegisterUtilityStatementHandler(ValidateIcebergCatalogServerDDL, NULL);
+
+	/*
+	 * Register last so it runs first: RegisterUtilityStatementHandler
+	 * prepends to a linked list.  Redaction must precede the validator above
+	 * so that the failing built-in-server path never leaks client_id /
+	 * client_secret in its error context.
+	 */
+	RegisterUtilityStatementHandler(RedactRestCatalogUserMappingSecrets, NULL);
 }