Add catalog integration framework support to dbt-athena (enable catalogs.yml v2)

[colin-k-rogers]

Summary

dbt-athena cannot currently participate in catalogs.yml v2 (introduced for dbt-snowflake and dbt-bigquery in #1920) because it has no v1 CatalogIntegration to bridge into. Iceberg selection is hard-wired through the inline model config table_type='iceberg' and resolved directly in materialization macros.

This issue tracks the work to (1) introduce a CatalogIntegration framework on dbt-athena and (2) add the v2 bridge on top, so Athena users can declare catalogs in catalogs.yml and reference them from model config like Snowflake/BigQuery users can today.

Background

PR #1920 added a thin adapter-owned bridge: BaseAdapter.bridge_v2_catalog translates a CatalogV2 (parsed in dbt-core) into a CatalogWriteIntegrationConfig that flows into the existing v1 CatalogIntegrationClient. Adapters opt in by:

1. Declaring Capability.CatalogsV2.
2. Overriding _v2_to_v1_type, _v2_table_format, _translate_v2_properties.
3. Registering one or more CatalogIntegration subclasses via CATALOG_INTEGRATIONS.

dbt-snowflake and dbt-bigquery already had step 3 (Iceberg REST, BuiltIn, BigLake Metastore). dbt-athena does not — it has no CATALOG_INTEGRATIONS, no catalogs/ package, and its Iceberg path runs through table_type='iceberg' in the model config (see impl.py line ~130, relation.py line ~132, and the table/incremental macros).

Proposed approach

Phase 1 — Introduce v1 catalog integrations on Athena

Create dbt/adapters/athena/catalogs/ with:

• AthenaCatalogRelation dataclass implementing the CatalogRelation protocol:
  • catalog_name, table_format (hive | iceberg), external_volume (maps to external_location), file_format (parquet | orc | avro | json | textfile)
  • Adapter properties: partitioned_by, bucketed_by, bucket_count, table_properties, work_group, lf_tags_config, write_compression, ha
• GlueCatalogIntegration(CatalogIntegration) with catalog_type = "glue", allows_writes = True. One integration class covers both hive and iceberg table formats because in Athena's model both live under Glue.
• Optionally IcebergRestCatalogIntegration as a stub for Athena's emerging Iceberg REST endpoint — can land later.

Wire it up in AthenaAdapter:

CATALOG_INTEGRATIONS = [GlueCatalogIntegration]

def __init__(self, config, mp_context) -> None:
    super().__init__(config, mp_context)
    self.add_catalog_integration(DEFAULT_GLUE_CATALOG)  # implicit default

Refactor materialization macros to consult the catalog framework first, with the existing inline-config path as a back-compat fallback:

{% macro athena__create_table_as(temporary, relation, sql) %}
  {%- set catalog_relation = adapter.build_catalog_relation(config.model) -%}
  {%- set table_format = catalog_relation.table_format
                         if catalog_relation
                         else config.get('table_type', 'hive') -%}
  {% if table_format == 'iceberg' %}
    {{ athena__create_iceberg_table_as(relation, sql, catalog_relation) }}
  {% else %}
    {{ athena__create_hive_table_as(relation, sql, catalog_relation) }}
  {% endif %}
{% endmacro %}

Files affected (non-exhaustive):

• dbt-athena/src/dbt/adapters/athena/catalogs/ (new)
• dbt-athena/src/dbt/adapters/athena/impl.py — register integrations
• dbt-athena/src/dbt/include/athena/macros/materializations/models/table/{table.sql, create_table_as.sql}
• dbt-athena/src/dbt/include/athena/macros/materializations/models/incremental/incremental.sql
• dbt-athena/src/dbt/include/athena/macros/materializations/snapshots/snapshot.sql

Back-compat requirement: existing models using {{ config(table_type='iceberg', ...) }} must continue to work unchanged.

Phase 2 — Add the v2 bridge

After Phase 1 lands, mirror what BigQuery did in #1920:

# impl.py
_CATALOGS_V2_CAPABILITY = getattr(Capability, "CatalogsV2", None)

_capabilities: CapabilityDict = CapabilityDict({
    ...,
    **({_CATALOGS_V2_CAPABILITY: CapabilitySupport(support=Support.Full)}
       if _CATALOGS_V2_CAPABILITY is not None else {}),
})

_V2_TO_V1_TYPE: ClassVar[Dict[str, str]] = {
    "glue": "glue",
    "iceberg_rest": "iceberg_rest",
}

def _v2_to_v1_type(self, catalog_type: str) -> str:
    return self._V2_TO_V1_TYPE.get(catalog_type, catalog_type)

# _v2_table_format: default is fine — Athena DDL uses lowercase 'iceberg'

def _translate_v2_properties(self, catalog_type, props):
    # Map v2 keys to Athena adapter_properties where names diverge
    return props

Open question for this phase: which fields belong in the athena: platform block of catalogs.yml vs. staying on the profile (s3_staging_dir, work_group)? Suggest: catalog-level overrides allowed for external_location, work_group, lf_tags_config; profile remains the default.

Acceptance criteria

• AthenaAdapter.CATALOG_INTEGRATIONS includes at least GlueCatalogIntegration.
• adapter.build_catalog_relation(config.model) returns an AthenaCatalogRelation for any model with a catalog config, including iceberg tables.
• Existing models using table_type='iceberg' (no catalogs.yml) build identically to today — covered by the current functional test suite passing without modification.
• New unit tests under dbt-athena/tests/unit/test_catalogs.py covering Glue + hive/iceberg combinations.
• New functional test under dbt-athena/tests/functional/adapter/catalog_integrations/test_catalogs_v2.py mirroring dbt-bigquery/tests/functional/adapter/catalog_integrations/test_catalogs_v2.py.
• Capability.CatalogsV2 declared; using use_catalogs_v2: true with an athena: block produces a working iceberg table end-to-end.
• changie new entries under Features for both phases.

Out of scope

• Athena Iceberg REST catalog beyond a stub — track separately if/when AWS GA's support and customer demand warrant it.
• Removing the inline table_type='iceberg' config path. Deprecation is a follow-up after at least one minor release with both paths supported.
• Cross-adapter changes to dbt-adapters/src/dbt/adapters/catalogs/ — the existing protocol is sufficient.

Suggested PR sequence

1. PR 1: GlueCatalogIntegration + AthenaCatalogRelation + registration. No macro changes yet. Unit tests only.
2. PR 2: Macro refactor to route through build_catalog_relation with back-compat fallback. Functional tests against both code paths.
3. PR 3: Phase 2 — Capability.CatalogsV2 + the three _v2_* overrides + v2 functional test.

References

• feat: add CatalogV2 type and bridge_v2_catalog template methods for catalogs.yml v2 #1920 — feat: add CatalogV2 type and bridge_v2_catalog template methods for catalogs.yml v2
• dbt-core companion: feat: catalogs.yml v2 with adapter-owned bridge architecture dbt-core#12930
• Reference implementations: dbt-snowflake/src/dbt/adapters/snowflake/{impl.py, catalogs/}, dbt-bigquery/src/dbt/adapters/bigquery/{impl.py, catalogs/}