[Bug] Connecting to Redshift (datashare) external database breaks on_schema_change

[zachary-povey]

Is this a new bug?

• I believe this is a new bug
• I have searched the existing issues, and I could not find an existing issue for this bug

Which packages are affected?

• dbt-adapters
• dbt-tests-adapter
• dbt-athena
• dbt-athena-community
• dbt-bigquery
• dbt-postgres
• dbt-redshift
• dbt-snowflake
• dbt-spark

Current Behavior

If in the profiles.yml if you set up a redshift connection using an "external" (datasharing) database for dbname the on_schema_change functionality does not work:

• on_schema_change=fail fails, even if there are no changes
• on_schema_change=append_new_columns ignores new columns
• on_schema_change=ignore sidesteps the issue

Expected Behavior

I would expect either on_schema_change to work as per the docs, or a helpful error to be raised explaining the combined set of options (external db, on_schema_change != ignore) is not supported.

Steps To Reproduce

1. Create two redshift serverless namespace + workgroup pairs
2. Set up a producer -> consumer datashare between the two, enabling the consumer to create/modify tables in the producer using multi-warehouse writes
3. Connect to the consumer workgroup in your dbt profiles.yml, with dbname set to the external datashare db
4. Attempt to process incremental changes for a model with on_schema_change != ignore

Relevant log output

11:40:33  2 of 2 ERROR creating sql incremental model public.test_latest ................. [ERROR in 2.25s]
11:40:34  
11:40:34  Finished running 1 incremental model, 1 table model in 0 hours 0 minutes and 11.24 seconds (11.24s).
11:40:34  
11:40:34  Completed with 1 error, 0 partial successes, and 0 warnings:
11:40:34  
11:40:34  Failure in model test_latest (models/public/test_latest.sql)
11:40:34    Compilation Error in model test_latest (models/public/test_latest.sql)
  
                The source and target schemas on this incremental model are out of sync!
                They can be reconciled in several ways:
                  - set the `on_schema_change` config to either append_new_columns or sync_all_columns, depending on your situation.
                  - Re-run the incremental model with `full_refresh: True` to update the target schema.
                  - update the schema manually and re-run the process.
  
                Additional troubleshooting context:
                   Source columns not in target: []
                   Target columns not in source: [Column(column='id', dtype='bigint', char_size=None, numeric_precision=64, numeric_scale=0), Column(column='test_field', dtype='character varying', char_size=65535, numeric_precision=None, numeric_scale=None), Column(column='test_nested_field', dtype='character varying', char_size=65535, numeric_precision=None, numeric_scale=None), Column(column='__raw_s3_uri__', dtype='character varying', char_size=65535, numeric_precision=None, numeric_scale=None), Column(column='__processed_s3_uri__', dtype='character varying', char_size=65535, numeric_precision=None, numeric_scale=None), Column(column='__processed__timestamp__', dtype='timestamp without time zone', char_size=None, numeric_precision=None, numeric_scale=None), Column(column='schema_change', dtype='boolean', char_size=None, numeric_precision=None, numeric_scale=None), Column(column='schema_change2', dtype='boolean', char_size=None, numeric_precision=None, numeric_scale=None), Column(column='schema_change3', dtype='boolean', char_size=None, numeric_precision=None, numeric_scale=None), Column(column='schema_change4', dtype='boolean', char_size=None, numeric_precision=None, numeric_scale=None)]
                   New column types: []
            
  
  > in macro default__process_schema_changes (macros/materializations/models/incremental/on_schema_change.sql)
  > called by macro process_schema_changes (macros/materializations/models/incremental/on_schema_change.sql)
  > called by macro materialization_incremental_default (macros/materializations/models/incremental/incremental.sql)
  > called by model test_latest (models/public/test_latest.sql)

Environment

- OS: Darwin 25.4.0 Darwin Kernel Version 25.4.0: Thu Mar 19 19:31:56 PDT 2026; root:xnu-12377.101.15~1/RELEASE_ARM64_T6030 arm64
- Python: 3.13.5
- dbt-adapters: 1.23.0
- dbt-redshift: 1.11.0rc2

Additional Context

I'm using both ra3_node and datasharing in the profiles.yml:

redshift:
  outputs:
    redshift:
      type: redshift
      host: "{{ env_var('REDSHIFT_HOST') }}"
      port: "{{ env_var('REDSHIFT_PORT') | as_number }}"
      dbname: "{{ env_var('REDSHIFT_DATABASE') }}"
      # this is just the default schema, can be changed on models
      schema: "private"
      user: "{{ env_var('REDSHIFT_USERNAME') }}"
      password: "{{ env_var('REDSHIFT_PASSWORD') }}"
      threads: 4
      ra3_node: true
      datasharing: true
  target: redshift

Potential Fix

I've managed to work out what the root cause is, basically temporary tables can't be easily inspected when running in redshift connected to the external db on the consumer side, as they are created in the default db. I tried to fix using svv_columns but it doesn't bring back details for temp tables either. I managed to get it working (admittedly with a bit of help from AI) by adding a get_columns_in_temp_table method to the adapter class in the python code which simply selects from the temp table and inspects cursor.description to get the types, you can see that on my fork here (apologies my editor also added a bunch of format changes).

I'm not sure that's necessarily the best approach though and have managed to work around it for now without changing anything by connecting to the default database (dev) and overriding database at the model level.