Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add enforce_primary_key_uniqueness support to converter #507

Open
wants to merge 1 commit into
base: 2.0
Choose a base branch
from

Conversation

Zyiqin-Miranda
Copy link
Member

@Zyiqin-Miranda Zyiqin-Miranda commented Mar 14, 2025

Summary

Records with same primary key/identify fields will be deduped with this PR, the purpose is that we can enforce primary key uniqueness as well as finding primary keys to be deleted based on equality delete files within same operation(No need to download into memory same files again). All records meant to be either deduped or deleted will be output in form of position deletes.

Rationale

Explain the reasoning behind the changes and their benefits to the project.

Changes

List the major changes made in this pull request.

Impact

Discuss any potential impacts the changes may have on existing functionalities.

Testing

Describe how the changes have been tested, including both automated and manual testing strategies.
If this is a bugfix, explain how the fix has been tested to ensure the bug is resolved without introducing new issues.

Regression Risk

If this is a bugfix, assess the risk of regression caused by this fix and steps taken to mitigate it.

Checklist

  • Unit tests covering the changes have been added

    • If this is a bugfix, regression tests have been added
  • E2E testing has been performed

Additional Notes

Any additional information or context relevant to this PR.

@Zyiqin-Miranda Zyiqin-Miranda force-pushed the enforce-primary-key-uniqueness-impl branch from 693f2f0 to af1faa8 Compare March 14, 2025 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants