[WIP] deduping benchmark suite #607

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

cyruszhang wants to merge 3 commits into main from feat/cyruszhang/dedupe-benchmark

Collaborator

cyruszhang commented Mar 4, 2025

comprehensive work for deduping benchmark.

dj recipes for baseline datasets
tunable toolkit for synthetic duplicate data generation
more algos for deduping
boilerplate to run the tests

cyruszhang added 2 commits

February 27, 2025 10:11


          add initial config for c4 refine

55fa9a9


          add dupe gen version 0.1

6f8bded

cyruszhang requested review from chenyushuo and yxdyc

March 4, 2025 21:47

cyruszhang had a problem deploying to Testing

March 4, 2025 21:47

— with

GitHub Actions Failure


          remove dup files

fa4ac51

cyruszhang had a problem deploying to Testing

March 4, 2025 21:48

— with

GitHub Actions Failure

Collaborator Author

cyruszhang commented Mar 4, 2025

sending things out as we go. not ready for review yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet