Skip to content

feat: add v2 catalog schema support to snowflake/bigquery adapters - #1905

Closed
aahel wants to merge 8 commits into
mainfrom
feat/catalogs-v2-registry
Closed

feat: add v2 catalog schema support to snowflake/bigquery adapters#1905
aahel wants to merge 8 commits into
mainfrom
feat/catalogs-v2-registry

Conversation

@aahel

@aahel aahel commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Problem

Implementing the catalogs.yml v2 spec requires per-(catalog_type, platform) config schemas (e.g. HorizonSnowflakeConfig, LinkedSnowflakeConfig, BiglakeMetastoreBigqueryConfig). Those schemas describe adapter-specific knowledge — Snowflake's external_volume semantics, BigQuery's gs:// requirements, etc. — and should live with the adapter that owns them, not in dbt-core. dbt-core needs a way to validate v2 catalogs.yml at parse time without knowing about adapter-specific schemas.

Solution

Each adapter declares its v2 catalog schemas as a CATALOG_V2_CONFIGS class attribute on the adapter class — mirroring the existing CATALOG_INTEGRATIONS pattern that already exists for v1 catalog integrations:

# dbt-snowflake/src/dbt/adapters/snowflake/impl.py
class SnowflakeAdapter(SQLAdapter):
    CATALOG_INTEGRATIONS = [...]                     # v1 (existing)
    CATALOG_V2_CONFIGS = {                           # v2 (new)
        "horizon": HorizonSnowflakeConfig,
        "glue": LinkedSnowflakeConfig,
        "iceberg_rest": LinkedSnowflakeConfig,
        "unity": LinkedSnowflakeConfig,
    }

dbt-core looks up via adapter_class.CATALOG_V2_CONFIGS.get(catalog_type) at parse time. Schemas are dbtClassMixin dataclasses, so cls.validate() handles structural checks (unknown keys, required fields, types) via jsonschema and __post_init__ covers semantic constraints (value ranges, enum values, cross-field rules).

Architecture

  • BaseAdapter.CATALOG_V2_CONFIGS: Dict[str, Type[dbtClassMixin]] = {} — empty default; adapter subclasses override
  • Schema definitions live with their owning adapter package (dbt-snowflake, dbt-bigquery)
  • dbt-core never imports specific platform configs; it only walks adapter_class.CATALOG_V2_CONFIGS
  • Declarative on the adapter class — no global registry, no side-effect imports, no order-sensitive registration

This is the first of three coordinated PRs:

  1. This PR: base + snowflake + bigquery
  2. dbt-core PR: v2 framework that consumes CATALOG_V2_CONFIGS for platform validation
  3. dbt-databricks PR: declare UnityDatabricksConfig, HiveMetastoreDatabricksConfig on the Databricks adapter

Files changed

  • dbt-adapters/src/dbt/adapters/base/impl.py — add CATALOG_V2_CONFIGS class attribute (empty default)
  • dbt-snowflake/src/dbt/adapters/snowflake/catalogs/_v2.py (new) — HorizonSnowflakeConfig, LinkedSnowflakeConfig
  • dbt-snowflake/src/dbt/adapters/snowflake/impl.py — declare CATALOG_V2_CONFIGS
  • dbt-bigquery/src/dbt/adapters/bigquery/catalogs/_v2.py (new) — BiglakeMetastoreBigqueryConfig
  • dbt-bigquery/src/dbt/adapters/bigquery/impl.py — declare CATALOG_V2_CONFIGS
  • 24 unit tests (16 snowflake + 8 bigquery) covering schema validation and class-attribute registration

Note on commits

The first three commits (3db4ac99, e63f0001, 68621df5) explored a global registry pattern (register_catalog_config() / get_catalog_config()); the fourth commit (18e534cf) refactors to the class-attribute pattern after recognizing it aligns better with the existing CATALOG_INTEGRATIONS convention. Final design at HEAD is cleaner; happy to squash before merge if preferred.

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

aahel added 3 commits April 29, 2026 23:18
Adds register_catalog_config()/get_catalog_config() in
dbt.adapters.catalogs._v2_registry. Adapter packages register their
platform-specific catalogs.yml v2 schemas (dbtClassMixin dataclasses)
keyed by (catalog_type, platform); dbt-core looks them up at parse
time for structural and semantic validation. The registry holds
class references only — schema definitions live with their owning
adapter package.
Adds HorizonSnowflakeConfig and LinkedSnowflakeConfig in
dbt.adapters.snowflake.catalogs._v2, registered with the v2 catalog
config registry on import. Covers four v2 catalog types on snowflake:
horizon (HorizonSnowflakeConfig) and glue/iceberg_rest/unity (all
share LinkedSnowflakeConfig since they have identical config shape
on snowflake).
Adds BiglakeMetastoreBigqueryConfig in
dbt.adapters.bigquery.catalogs._v2, registered with the v2 catalog
config registry on import. Covers the biglake_metastore catalog type
on bigquery.
@aahel
aahel requested a review from a team as a code owner April 29, 2026 17:57
@cla-bot cla-bot Bot added the cla:yes The PR author has signed the CLA label Apr 29, 2026
@cla-bot
cla-bot Bot temporarily deployed to dbt-postgres April 29, 2026 17:57 Inactive
@cla-bot
cla-bot Bot temporarily deployed to dbt-redshift April 29, 2026 17:57 Inactive
Replaces the global _v2_registry.py module with a CATALOG_V2_CONFIGS
class attribute on BaseAdapter that adapter packages override.

This mirrors the existing CATALOG_INTEGRATIONS pattern adapters already
use to declare their v1 catalog integrations - the registration is
declarative on the adapter class rather than a side-effect import,
removes global mutable state, and is what reviewers will recognize
as the idiomatic dbt pattern.

Changes:
- BaseAdapter gets a CATALOG_V2_CONFIGS: Dict[str, Type[dbtClassMixin]]
  class attribute (default empty)
- SnowflakeAdapter declares horizon/glue/iceberg_rest/unity entries
- BigQueryAdapter declares biglake_metastore entry
- _v2_registry.py and its tests are removed
- Snowflake/bigquery catalogs/__init__.py now export the config classes
  instead of relying on side-effect imports for registration

dbt-core looks up via adapter_class.CATALOG_V2_CONFIGS.get(catalog_type),
restricted to the current adapter only. Cross-platform validation (e.g.
validating a unity catalog's databricks block while running on snowflake)
is intentionally out of scope here - bad cross-platform config errors at
the moment that platform is actually used, which is when it matters.
@aahel aahel changed the title feat: add v2 catalog config registry + register snowflake/bigquery configs feat: add v2 catalog schema support to snowflake/bigquery adapters Apr 29, 2026

@QMalcolm QMalcolm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds adapter-owned CATALOG_V2_CONFIGS schema registration so Snowflake and BigQuery can expose catalogs.yml v2 validation metadata without dbt-core importing adapter-specific config classes.

Changes:

  • Adds BaseAdapter.CATALOG_V2_CONFIGS as the shared extension point for v2 catalog config schemas.
  • Introduces new Snowflake and BigQuery v2 config dataclasses and registers them on each adapter.
  • Adds unit tests plus changelog entries for the new schema-registration surface.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
dbt-snowflake/tests/unit/test_v2_catalog_configs.py Adds unit tests for Snowflake v2 config registration and validation helpers.
dbt-snowflake/src/dbt/adapters/snowflake/impl.py Registers Snowflake v2 catalog config classes on the adapter.
dbt-snowflake/src/dbt/adapters/snowflake/catalogs/_v2.py Defines new Snowflake v2 config dataclasses and semantic validation.
dbt-snowflake/src/dbt/adapters/snowflake/catalogs/__init__.py Re-exports the new Snowflake v2 config classes.
dbt-snowflake/.changes/unreleased/Features-20260429-231718.yaml Adds Snowflake changelog entry for the feature.
dbt-bigquery/tests/unit/test_v2_catalog_configs.py Adds unit tests for BigQuery v2 config registration and validation helpers.
dbt-bigquery/src/dbt/adapters/bigquery/impl.py Registers the BigQuery v2 catalog config class on the adapter.
dbt-bigquery/src/dbt/adapters/bigquery/catalogs/_v2.py Defines the BigQuery v2 config dataclass and semantic validation.
dbt-bigquery/src/dbt/adapters/bigquery/catalogs/__init__.py Re-exports the new BigQuery v2 config class.
dbt-bigquery/.changes/unreleased/Features-20260429-231718.yaml Adds BigQuery changelog entry for the feature.
dbt-adapters/src/dbt/adapters/base/impl.py Adds the base adapter hook for v2 catalog schema registration.
dbt-adapters/.changes/unreleased/Features-20260429-231718.yaml Adds dbt-adapters changelog entry for the new extension point.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +40 to +45
if self.storage_serialization_policy is not None:
_check_enum(
"storage_serialization_policy",
self.storage_serialization_policy,
{"compatible", "optimized"},
)
Comment on lines +64 to +69
if self.target_file_size is not None:
_check_enum(
"target_file_size",
self.target_file_size,
{"auto", "16mb", "32mb", "64mb", "128mb"},
)
Comment on lines +15 to +17
if not self.external_volume.strip():
raise DbtValidationError("'external_volume' must be non-empty")
if not self.external_volume.startswith("gs://"):
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla:yes The PR author has signed the CLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants