Skip to content

[Feature] COPY-based dbt seed for Redshift to replace slow INSERT path #1899

Description

@vahid110

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Problem
dbt seed on Redshift uses INSERT statements to load CSV data. For large seed files this is extremely slow because each row is a separate round-trip through the leader node.

This also causes type-related failures on large seeds (see archived dbt-labs/dbt-redshift#347 - 'h' format requires -32768 <= number <= 32767).

Benchmarked on a 4-node ra3.large cluster (EC2 same region, 1M rows):

Approach Time
INSERT (current dbt seed path) 13+ hours
COPY via S3 (Parquet staging) 3.33 seconds

Proposed Solution
For Redshift targets, dbt seed could optionally use the COPY path:

  1. Write the seed CSV to a temporary Parquet file
  2. Upload to a configured S3 staging bucket
  3. Run COPY table FROM 's3://...' IAM_ROLE '...' FORMAT PARQUET
  4. Clean up the staged file

This would require two additional profile config fields:

  • staging_bucket : S3 bucket for temporary staging
  • staging_iam_role : IAM role ARN that Redshift can assume for S3 access

When these are not configured, dbt seed falls back to the current INSERT behavior. No breaking changes.

Prior Art

  • Redshift documentation recommends COPY over INSERT for any bulk load
  • awswrangler.redshift.copy() uses this exact pattern
  • The arrowjet library (https://github.com/arrowjet/arrowjet) implements this with benchmarks

Questions for Maintainers

  1. Is this something the dbt-redshift adapter team would consider?
  2. Are there architectural concerns with adding S3 staging to the seed path?
  3. Would a PR be welcome, or is there a preferred approach?

Happy to contribute the implementation if there's interest.

Describe alternatives you've considered

No response

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions