feat(athena): resolve cross-account Glue catalogs in Spark Python models

[dtaniwaki]

Background

Spark Python models that use dbt.source() / dbt.ref() cannot currently read from cross-account GLUE catalogs registered as Athena Data Catalogs.

athena__py_get_spark_dbt_object's get_spark_df helper strips the catalog component from the identifier before calling spark.table(), so the Spark Glue Catalog Client always targets the local account's Glue. Attempting dbt.source("cross_account", "tbl") from a Python model fails with:

software.amazon.awssdk.services.glue.model.AccessDeniedException:
  User: arn:aws:sts::<local>:assumed-role/... is not authorized to perform:
  glue:GetTable on resource: arn:aws:glue:...:<local>:database/<db>

Today the only workarounds are (a) hard-coding the account ID via spark.table("<account_id>/schema.table") — which breaks dbt lineage (selectors, docs, source freshness) — or (b) copying all cross-account data to the local account, which is extremely heavyweight.

Change

Introduce an adapter method that discovers cross-account GLUE catalogs at compile time and injects the resolved {catalog_name: account_id} mapping into the generated Python code, so get_spark_df can translate identifiers transparently.

Adapter

AthenaAdapter.get_spark_cross_account_catalog_map()

• @available + @lru_cache() (same pattern as _get_work_group)
• Paginates athena:ListDataCatalogs
• For each non-default GLUE catalog, calls athena:GetDataCatalog and extracts Parameters.catalog-id via the existing get_catalog_id() helper
• Returns Dict[str, str] — the local awsdatacatalog is excluded (not needed) and non-GLUE catalogs are skipped (Spark's Glue Catalog Client can't read them)

Macro

athena__py_get_spark_dbt_object now:

1. Reads config.spark_cross_account_catalog (model config). When false (default), it skips the adapter call entirely and injects an empty _CROSS_ACCOUNT_CATALOGS = {} — preserving the pre-feature behavior and avoiding identifiers the Spark parser cannot handle.
2. When the model opts in, calls the new adapter method at compile time and injects the resolved mapping into the generated code as _CROSS_ACCOUNT_CATALOGS = {...} via | tojson.
3. Updates get_spark_df to:
   • Return spark.table("schema.table") for awsdatacatalog (existing behavior)
   • Return spark.table("<account_id>/schema.table") for known cross-account GLUE catalogs — Spark can resolve this when spark.hadoop.aws.glue.catalog.separator is set to /, which AthenaSparkSessionConfig does automatically when the same spark_cross_account_catalog: true model config is present
   • Fall back to the legacy identifier for unknown catalogs so existing behavior is preserved

Tests

• test_get_spark_cross_account_catalog_map_only_default — only awsdatacatalog registered → empty dict (regression guard)
• test_get_spark_cross_account_catalog_map_with_cross_account_glue — one cross-account GLUE catalog is included with its catalog-id
• test_get_spark_cross_account_catalog_map_excludes_non_glue — LAMBDA / federated catalogs are filtered out (Spark can't read them)

All 101 tests in tests/unit/test_adapter.py + tests/unit/test_python_submissions.py pass.

Required permissions

athena:ListDataCatalogs is new but is typically already granted to any role that can run Athena queries. athena:GetDataCatalog is already used by _get_data_catalog. Both are only called when the model opts in with spark_cross_account_catalog: true, so models that do not use the feature do not require the new permissions.

Runtime requirement

To activate cross-account resolution for a Python model, set spark_cross_account_catalog: true in the model config (e.g. via dbt.config(...), schema.yml, or a project-level +spark_cross_account_catalog: true). This single flag:

1. Adds spark.hadoop.aws.glue.catalog.separator: / to the Spark session properties via AthenaSparkSessionConfig, so Spark can parse <account_id>/<schema>.<table>.
2. Gates the macro on the same config: the adapter is invoked and _CROSS_ACCOUNT_CATALOGS is populated only when the flag is true.

Models that do not set the flag get an empty mapping and get_spark_df falls back to the legacy two-part <schema>.<table> form. This is important: without the gating, a model that had a cross-account catalog registered in Athena but had not enabled the separator would emit <account_id>/<schema>.<table> to spark.table() and fail with:

pyspark.sql.utils.ParseException:
  mismatched input '<account_id>' expecting {...}
  == SQL ==
  <account_id>/<schema>.<table>

Backwards compatibility

• Users with only awsdatacatalog registered get an empty map → identical to the pre-existing behavior (no regression).
• Users that were already working around this by hard-coding <account_id>/schema.table via session.table() continue to work unchanged.
• Users who have cross-account catalogs registered in Athena but do not enable spark_cross_account_catalog on their models keep the pre-feature behavior (the adapter is not called and the map is empty).
• No changes to profile schema are required; the feature is activated per-model via the spark_cross_account_catalog config.