fix(redshift): detect existing DWH schema via svv_all_schemas#2789
Merged
Conversation
The Diagnostics Warehouse existence check queried information_schema.schemata, which on Redshift only lists schemas the current user OWNS. A pre-provisioned DWH schema the service user merely has USAGE/CREATE on was therefore invisible, so Soda issued CREATE SCHEMA -- which needs CREATE-on-database and failed with 'permission denied for database' even though the schema already existed. Override the schemas metadata query on Redshift to use pg_catalog.svv_all_schemas (visible by access, not ownership), matching the existing svv_tables/svv_columns overrides. Base dialect behaviour is unchanged via a new build_schemas_metadata_from_clause() hook. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
CI: soda-extensions |
|
Niels-b
changed the title
nielsn
approved these changes
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On Redshift, setting up the Diagnostics Warehouse (DWH) could fail with
permission denied for database <db>when the target DWH schema already exists but the connecting user is not its owner.Root cause
The DWH schema-existence check queried
information_schema.schemata. On Redshift (Postgres-8 lineage) that view only lists schemas the current user owns —USAGE/CREATEgrants do not make a schema appear there. So a pre-created schema the service user could access came back as "missing", Soda issuedCREATE SCHEMA, which needsCREATEon the database, and failed even though the schema already existed.Because visibility depends on ownership rather than grants, the failure can look inconsistent across otherwise-identical setups: a user who owns the schema succeeds, while a user with only
USAGE/CREATEon the same schema does not.Fix
Override the schemas metadata query on Redshift to use
pg_catalog.svv_all_schemas(visible by access, not ownership), consistent with the existingsvv_tables/svv_columnsoverrides. The base dialect is unchanged (stillinformation_schema.schemata) via a newbuild_schemas_metadata_from_clause()hook — no other data source is affected.Result: the pre-existing schema is detected,
CREATE SCHEMAis skipped, and no database-levelCREATEprivilege is required.Verification
svv_all_schemas; Postgres/base output unchanged.test_schema_existsreturns correct results; reproduced the visibility gap directly (information_schema.schematalisted fewer schemas thansvv_all_schemas, missing ones the user could access, includingpublic).soda-redshift/tests— 13 passed. Core integration suite — 168 passed, 8 skipped.CI
soda-extensions
mainworkflow dispatched against this branch to confirm the DWH tests pass on Redshift:dataSource=redshift,snapshotMode=off(replay off / real DB), cross-source disabled.🤖 Generated with Claude Code