fix(destination-redshift): use anti-join instead of correlated NOT EXISTS in dedup insert by devin-ai-integration[bot] · Pull Request #83261 · airbytehq/airbyte

devin-ai-integration · 2026-07-30T12:12:52Z

What

Dedup (*_deduped) syncs on destination-redshift 4.0.4 fail with:

com.amazon.redshift.util.RedshiftException: ERROR: This type of correlated subquery pattern is not supported due to internal error

Related to https://github.com/airbytehq/oncall/issues/13202:

https://github.com/airbytehq/oncall/issues/13202

The upsert's insert step was:

INSERT INTO target (...)
SELECT ... FROM <dedup temp table>
WHERE NOT EXISTS (
  SELECT 1 FROM target
  WHERE (target.pk = dedup.pk OR (target.pk IS NULL AND dedup.pk IS NULL)) AND ...
)

The NOT EXISTS subquery references the outer dedup table, so it is a correlated subquery. Before 4.0.4 the inner predicate was a plain equijoin; #83245 changed it to the NULL-safe OR form (to fix https://github.com/airbytehq/oncall/issues/13186), and Redshift's planner appears to reject that correlated shape — the reported failure has an 8-column compound primary key, so the predicate expands to 8 ANDed OR groups. Many other 4.0.4 dedup syncs succeed, which is consistent with the rejection depending on the predicate shape rather than on dedup in general. See AWS: correlated subquery patterns that aren't supported.

How

RedshiftSqlGenerator.insertNewRows now uses a LEFT JOIN anti-join instead of a correlated NOT EXISTS, keeping the NULL-safe PK semantics from #83245:

INSERT INTO "ns"."final" ( cols )
SELECT <dedup>.col, ...
FROM <dedup>
LEFT JOIN "ns"."final" AS "_airbyte_target"
  ON ("_airbyte_target"."pk" = <dedup>."pk" OR ("_airbyte_target"."pk" IS NULL AND <dedup>."pk" IS NULL)) AND ...
WHERE
  "_airbyte_target"."_airbyte_raw_id" IS NULL

The anti-join is detected on _airbyte_raw_id, which is declared NOT NULL in META_COLUMNS — using a PK column for the IS NULL check would be ambiguous precisely for the NULL-PK rows this predicate exists to handle. Columns in the SELECT list are now qualified with the dedup table so the join can't make them ambiguous. NOT EXISTS and the anti-join match the same set of rows, and since non-matching rows contribute exactly one all-NULL right side, the join cannot duplicate inserted rows.

Not addressed here: updateExistingRows still matches PKs with plain = (Redshift requires a pure equijoin in UPDATE ... FROM), so rows whose PK contains NULL are still never UPDATEd — the https://github.com/airbytehq/oncall/issues/13186 class of bug may only be half-fixed. That is pre-existing behavior from #83245 and would need a different formulation (e.g. delete+insert, or a NOT NULL PK-signature column); calling it out rather than expanding scope on a customer-impacting regression fix.

Review guide

airbyte-integrations/connectors/destination-redshift/src/main/kotlin/io/airbyte/integrations/destination/redshift/sql/RedshiftSqlGenerator.kt — insertNewRows rewrite
airbyte-integrations/connectors/destination-redshift/src/test/kotlin/io/airbyte/integrations/destination/redshift/sql/RedshiftSqlGeneratorTest.kt — asserts no correlated NOT EXISTS, the anti-join shape, and NULL-safe matching for single and compound PKs
metadata.yaml / docs/integrations/destinations/redshift.md — 4.0.4 → 4.0.5 + changelog

Test Coverage

./gradlew :airbyte-integrations:connectors:destination-redshift:test passes locally. The new/updated unit tests assert the absence of the correlated NOT EXISTS and the presence of the anti-join with NULL-safe compound-PK matching, so they fail against the 4.0.4 statement shape. The change has not been validated against a live Redshift cluster in this session — integration tests / a pre-release validation on the affected connection would be the confirming evidence.

Breaking Change Evaluation

Not breaking: no schema, spec, state, PK, cursor, or stream changes — only the SQL shape of the dedup insert. Non-breaking PATCH bump to 4.0.5. Progressive rollout is not enabled for this connector, so no -rc.x suffix.

User Impact

Dedup syncs to Redshift that failed on 4.0.4 with the correlated-subquery error should succeed again, with the NULL-safe PK matching from 4.0.4 preserved. Users currently blocked can pin destination-redshift to 4.0.3 as a workaround until 4.0.5 is released, at the cost of re-introducing the duplicate-rows behavior for NULL PK values described in oncall 13186.

Can this PR be safely reverted and rolled back?

YES 💚
NO ❌

Link to Devin session: https://app.devin.ai/sessions/b5ce5067355f491499b1f69e110bd765

Co-Authored-By: bot_apk <apk@cognition.ai>

devin-ai-integration · 2026-07-30T12:12:54Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment, CI, and merge conflict monitoring

github-actions · 2026-07-30T12:13:31Z

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

🛠️ Quick Fixes
- /format-fix - Fixes most formatting issues.
- /bump-version - Bumps connector versions, scraping changelog description from the PR title.
  - Bump types: patch (default), minor, major, major_rc, rc, promote.
  - The rc type is a smart default: applies minor_rc if stable, or bumps the RC number if already RC.
  - The promote type strips the RC suffix to finalize a release.
  - Example: /bump-version type=rc or /bump-version type=minor
- /bump-progressive-rollout-version - Alias for /bump-version type=rc. Bumps with an RC suffix and enables progressive rollout.
❇️ AI Testing and Review (internal link: AI-SDLC Docs):
- /ai-prove-fix - Runs prerelease readiness checks, including testing against customer connections.
- /ai-canary-prerelease - Rolls out prerelease to 5-10 connections for canary testing.
- /ai-review - AI-powered PR review for connector safety and quality gates.
📝 AI Documentation:
- /ai-docs-review - AI-powered documentation review for PRs with connector changes.
- /ai-create-docs-pr - Creates a documentation PR for connector changes, stacked on the current PR.
🚀 Connector Releases:
- /publish-connectors-prerelease - Publishes pre-release connector builds (tagged as {version}-preview.{git-sha}) for all modified connectors in the PR.
- /enable-autopilot-rollouts - Enables autopilot progressive rollouts for the modified connector(s) in the PR, remediating "autopilot rollouts not enabled for {connector-name}" auto-merge blockers. Sets defaultRolloutMode: autopilot and enableProgressiveRollout: true, preserving any existing autopilotConfig.
  - Optional args: connector=<CONNECTOR_NAME> (defaults to the modified connectors in the PR), strategy=fast|slow|default (defaults to fast).
  - Example: /enable-autopilot-rollouts or /enable-autopilot-rollouts connector=source-faker strategy=slow
☕️ JVM connectors:
- /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
  Example: /update-connector-cdk-version connector=destination-bigquery
🐍 Python connectors:
- /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
- /poe source example lock - Alias for /poe connector source-example lock.
- /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
- /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.
⚙️ Admin commands:
- /force-merge reason="<REASON>" - Force merges the PR using admin privileges, bypassing CI checks. Requires a reason.
  Example: /force-merge reason="CI is flaky, tests pass locally"

📚 Show Repo Guidance

Helpful Resources

Breaking Changes Guide - Breaking changes, migration guides, and upgrade deadlines
Developing Connectors Locally
Managing Connector Secrets
On-Demand Regression Tests
#connector-ci-issues
#connector-publish-updates
#connector-build-statuses

📝 Edit this welcome message.

Co-Authored-By: bot_apk <apk@cognition.ai>

github-actions · 2026-07-30T12:13:47Z

Note

Autopilot progressive rollouts are not enabled for the following modified connector(s):

destination-redshift

This is a courtesy heads-up only — it does not block merge or fail any check.
To enable automatic progressive rollouts for the connector(s) above, comment
/enable-autopilot-rollouts on this PR. This sets defaultRolloutMode: autopilot
and enableProgressiveRollout: true in each connector's metadata.yaml,
preserving any existing autopilotConfig.

github-actions · 2026-07-30T12:16:56Z

Deploy preview for airbyte-docs ready!

Project:	`airbyte-docs`
Status:	✅ Deploy successful!
Preview URL:	https://airbyte-docs-3z6fmpfj7-airbyte-growth.vercel.app
Latest Commit:	`1d86477`

Deployed with vercel-action

github-actions · 2026-07-30T12:20:38Z

`destination-redshift` Connector Test Results

24 files 24 suites 19m 53s ⏱️
447 tests 435 ✅ 12 💤 0 ❌
479 runs 467 ✅ 12 💤 0 ❌

Results for commit 1d86477.

♻️ This comment has been updated with latest results.

fix(destination-redshift): use anti-join for dedup inserts

842777e

Co-Authored-By: bot_apk <apk@cognition.ai>

Octavia Squidington III (octavia-squidington-iii) added the connectors/destination/redshift label Jul 30, 2026

devin-ai-integration Bot added the hyd-fix Hydra: ai-fix stage has run label Jul 30, 2026

devin-ai-integration Bot assigned Airbyte Support Bot (airbyte-support-bot) Jul 30, 2026

docs(destination-redshift): fill in changelog PR link

1d86477

Co-Authored-By: bot_apk <apk@cognition.ai>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(destination-redshift): use anti-join instead of correlated NOT EXISTS in dedup insert - #83261

fix(destination-redshift): use anti-join instead of correlated NOT EXISTS in dedup insert#83261
devin-ai-integration[bot] wants to merge 2 commits into
masterfrom
devin/1785413316-redshift-dedup-antijoin

devin-ai-integration Bot commented Jul 30, 2026

Uh oh!

devin-ai-integration Bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

PR Slash Commands

Helpful Resources

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

devin-ai-integration Bot commented Jul 30, 2026

What

How

Review guide

Test Coverage

Breaking Change Evaluation

User Impact

Can this PR be safely reverted and rolled back?

Uh oh!

devin-ai-integration Bot commented Jul 30, 2026

🤖 Devin AI Engineer

Uh oh!

github-actions Bot commented Jul 30, 2026

👋 Greetings, Airbyte Team Member!

PR Slash Commands

Helpful Resources

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

destination-redshift Connector Test Results

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jul 30, 2026 •

edited

Loading

github-actions Bot commented Jul 30, 2026 •

edited

Loading

`destination-redshift` Connector Test Results