feat: add v2 catalog schema support to snowflake/bigquery adapters - #1905
Closed
aahel wants to merge 8 commits into
Closed
feat: add v2 catalog schema support to snowflake/bigquery adapters#1905aahel wants to merge 8 commits into
aahel wants to merge 8 commits into
Conversation
Adds register_catalog_config()/get_catalog_config() in dbt.adapters.catalogs._v2_registry. Adapter packages register their platform-specific catalogs.yml v2 schemas (dbtClassMixin dataclasses) keyed by (catalog_type, platform); dbt-core looks them up at parse time for structural and semantic validation. The registry holds class references only — schema definitions live with their owning adapter package.
Adds HorizonSnowflakeConfig and LinkedSnowflakeConfig in dbt.adapters.snowflake.catalogs._v2, registered with the v2 catalog config registry on import. Covers four v2 catalog types on snowflake: horizon (HorizonSnowflakeConfig) and glue/iceberg_rest/unity (all share LinkedSnowflakeConfig since they have identical config shape on snowflake).
Adds BiglakeMetastoreBigqueryConfig in dbt.adapters.bigquery.catalogs._v2, registered with the v2 catalog config registry on import. Covers the biglake_metastore catalog type on bigquery.
Replaces the global _v2_registry.py module with a CATALOG_V2_CONFIGS class attribute on BaseAdapter that adapter packages override. This mirrors the existing CATALOG_INTEGRATIONS pattern adapters already use to declare their v1 catalog integrations - the registration is declarative on the adapter class rather than a side-effect import, removes global mutable state, and is what reviewers will recognize as the idiomatic dbt pattern. Changes: - BaseAdapter gets a CATALOG_V2_CONFIGS: Dict[str, Type[dbtClassMixin]] class attribute (default empty) - SnowflakeAdapter declares horizon/glue/iceberg_rest/unity entries - BigQueryAdapter declares biglake_metastore entry - _v2_registry.py and its tests are removed - Snowflake/bigquery catalogs/__init__.py now export the config classes instead of relying on side-effect imports for registration dbt-core looks up via adapter_class.CATALOG_V2_CONFIGS.get(catalog_type), restricted to the current adapter only. Cross-platform validation (e.g. validating a unity catalog's databricks block while running on snowflake) is intentionally out of scope here - bad cross-platform config errors at the moment that platform is actually used, which is when it matters.
QMalcolm
approved these changes
Apr 30, 2026
This was referenced May 4, 2026
colin-k-rogers
temporarily deployed
to
dbt-snowflake
May 5, 2026 00:10 — with
GitHub Actions
Inactive
colin-k-rogers
temporarily deployed
to
dbt-postgres
May 5, 2026 00:10 — with
GitHub Actions
Inactive
colin-k-rogers
had a problem deploying
to
dbt-redshift
May 5, 2026 00:10 — with
GitHub Actions
Failure
colin-k-rogers
temporarily deployed
to
dbt-bigquery
May 5, 2026 00:10 — with
GitHub Actions
Inactive
Contributor
There was a problem hiding this comment.
Pull request overview
Adds adapter-owned CATALOG_V2_CONFIGS schema registration so Snowflake and BigQuery can expose catalogs.yml v2 validation metadata without dbt-core importing adapter-specific config classes.
Changes:
- Adds
BaseAdapter.CATALOG_V2_CONFIGSas the shared extension point for v2 catalog config schemas. - Introduces new Snowflake and BigQuery v2 config dataclasses and registers them on each adapter.
- Adds unit tests plus changelog entries for the new schema-registration surface.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
dbt-snowflake/tests/unit/test_v2_catalog_configs.py |
Adds unit tests for Snowflake v2 config registration and validation helpers. |
dbt-snowflake/src/dbt/adapters/snowflake/impl.py |
Registers Snowflake v2 catalog config classes on the adapter. |
dbt-snowflake/src/dbt/adapters/snowflake/catalogs/_v2.py |
Defines new Snowflake v2 config dataclasses and semantic validation. |
dbt-snowflake/src/dbt/adapters/snowflake/catalogs/__init__.py |
Re-exports the new Snowflake v2 config classes. |
dbt-snowflake/.changes/unreleased/Features-20260429-231718.yaml |
Adds Snowflake changelog entry for the feature. |
dbt-bigquery/tests/unit/test_v2_catalog_configs.py |
Adds unit tests for BigQuery v2 config registration and validation helpers. |
dbt-bigquery/src/dbt/adapters/bigquery/impl.py |
Registers the BigQuery v2 catalog config class on the adapter. |
dbt-bigquery/src/dbt/adapters/bigquery/catalogs/_v2.py |
Defines the BigQuery v2 config dataclass and semantic validation. |
dbt-bigquery/src/dbt/adapters/bigquery/catalogs/__init__.py |
Re-exports the new BigQuery v2 config class. |
dbt-bigquery/.changes/unreleased/Features-20260429-231718.yaml |
Adds BigQuery changelog entry for the feature. |
dbt-adapters/src/dbt/adapters/base/impl.py |
Adds the base adapter hook for v2 catalog schema registration. |
dbt-adapters/.changes/unreleased/Features-20260429-231718.yaml |
Adds dbt-adapters changelog entry for the new extension point. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+40
to
+45
| if self.storage_serialization_policy is not None: | ||
| _check_enum( | ||
| "storage_serialization_policy", | ||
| self.storage_serialization_policy, | ||
| {"compatible", "optimized"}, | ||
| ) |
Comment on lines
+64
to
+69
| if self.target_file_size is not None: | ||
| _check_enum( | ||
| "target_file_size", | ||
| self.target_file_size, | ||
| {"auto", "16mb", "32mb", "64mb", "128mb"}, | ||
| ) |
Comment on lines
+15
to
+17
| if not self.external_volume.strip(): | ||
| raise DbtValidationError("'external_volume' must be non-empty") | ||
| if not self.external_volume.startswith("gs://"): |
colin-k-rogers
approved these changes
May 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Implementing the catalogs.yml v2 spec requires per-(catalog_type, platform) config schemas (e.g.
HorizonSnowflakeConfig,LinkedSnowflakeConfig,BiglakeMetastoreBigqueryConfig). Those schemas describe adapter-specific knowledge — Snowflake'sexternal_volumesemantics, BigQuery'sgs://requirements, etc. — and should live with the adapter that owns them, not indbt-core.dbt-coreneeds a way to validate v2catalogs.ymlat parse time without knowing about adapter-specific schemas.Solution
Each adapter declares its v2 catalog schemas as a
CATALOG_V2_CONFIGSclass attribute on the adapter class — mirroring the existingCATALOG_INTEGRATIONSpattern that already exists for v1 catalog integrations:dbt-corelooks up viaadapter_class.CATALOG_V2_CONFIGS.get(catalog_type)at parse time. Schemas aredbtClassMixindataclasses, socls.validate()handles structural checks (unknown keys, required fields, types) via jsonschema and__post_init__covers semantic constraints (value ranges, enum values, cross-field rules).Architecture
BaseAdapter.CATALOG_V2_CONFIGS: Dict[str, Type[dbtClassMixin]] = {}— empty default; adapter subclasses overridedbt-snowflake,dbt-bigquery)dbt-corenever imports specific platform configs; it only walksadapter_class.CATALOG_V2_CONFIGSThis is the first of three coordinated PRs:
dbt-corePR: v2 framework that consumesCATALOG_V2_CONFIGSfor platform validationdbt-databricksPR: declareUnityDatabricksConfig,HiveMetastoreDatabricksConfigon the Databricks adapterFiles changed
dbt-adapters/src/dbt/adapters/base/impl.py— addCATALOG_V2_CONFIGSclass attribute (empty default)dbt-snowflake/src/dbt/adapters/snowflake/catalogs/_v2.py(new) —HorizonSnowflakeConfig,LinkedSnowflakeConfigdbt-snowflake/src/dbt/adapters/snowflake/impl.py— declareCATALOG_V2_CONFIGSdbt-bigquery/src/dbt/adapters/bigquery/catalogs/_v2.py(new) —BiglakeMetastoreBigqueryConfigdbt-bigquery/src/dbt/adapters/bigquery/impl.py— declareCATALOG_V2_CONFIGSNote on commits
The first three commits (
3db4ac99,e63f0001,68621df5) explored a global registry pattern (register_catalog_config()/get_catalog_config()); the fourth commit (18e534cf) refactors to the class-attribute pattern after recognizing it aligns better with the existingCATALOG_INTEGRATIONSconvention. Final design at HEAD is cleaner; happy to squash before merge if preferred.Checklist