Is this your first time submitting a feature request?
Describe the feature
Problem
dbt seed on Redshift uses INSERT statements to load CSV data. For large seed files this is extremely slow because each row is a separate round-trip through the leader node.
This also causes type-related failures on large seeds (see archived dbt-labs/dbt-redshift#347 - 'h' format requires -32768 <= number <= 32767).
Benchmarked on a 4-node ra3.large cluster (EC2 same region, 1M rows):
| Approach |
Time |
| INSERT (current dbt seed path) |
13+ hours |
| COPY via S3 (Parquet staging) |
3.33 seconds |
Proposed Solution
For Redshift targets, dbt seed could optionally use the COPY path:
- Write the seed CSV to a temporary Parquet file
- Upload to a configured S3 staging bucket
- Run
COPY table FROM 's3://...' IAM_ROLE '...' FORMAT PARQUET
- Clean up the staged file
This would require two additional profile config fields:
- staging_bucket : S3 bucket for temporary staging
- staging_iam_role : IAM role ARN that Redshift can assume for S3 access
When these are not configured, dbt seed falls back to the current INSERT behavior. No breaking changes.
Prior Art
- Redshift documentation recommends COPY over INSERT for any bulk load
- awswrangler.redshift.copy() uses this exact pattern
- The arrowjet library (https://github.com/arrowjet/arrowjet) implements this with benchmarks
Questions for Maintainers
- Is this something the dbt-redshift adapter team would consider?
- Are there architectural concerns with adding S3 staging to the seed path?
- Would a PR be welcome, or is there a preferred approach?
Happy to contribute the implementation if there's interest.
Describe alternatives you've considered
No response
Who will this benefit?
No response
Are you interested in contributing this feature?
No response
Anything else?
No response
Is this your first time submitting a feature request?
Describe the feature
Problem
dbt seedon Redshift uses INSERT statements to load CSV data. For large seed files this is extremely slow because each row is a separate round-trip through the leader node.This also causes type-related failures on large seeds (see archived dbt-labs/dbt-redshift#347 - 'h' format requires -32768 <= number <= 32767).
Benchmarked on a 4-node ra3.large cluster (EC2 same region, 1M rows):
Proposed Solution
For Redshift targets, dbt seed could optionally use the COPY path:
COPY table FROM 's3://...' IAM_ROLE '...' FORMAT PARQUETThis would require two additional profile config fields:
When these are not configured,
dbt seedfalls back to the current INSERT behavior. No breaking changes.Prior Art
Questions for Maintainers
Happy to contribute the implementation if there's interest.
Describe alternatives you've considered
No response
Who will this benefit?
No response
Are you interested in contributing this feature?
No response
Anything else?
No response