Skip to content

test for multiple dead compactions #5380

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

ArbaazKhan1
Copy link
Contributor

@ArbaazKhan1 ArbaazKhan1 commented Mar 5, 2025

Closes issue #5084

Created testCleanUpMultipleDeadCompactions() starts with creating 3 compaction jobs and removes 2 from the metaData "simulating" its a dead compaction and then check coordinator.getRuning() to see if dead compactions still exist or not.

@ctubbsii ctubbsii added this to the 4.0.0 milestone Mar 25, 2025
@DomGarguilo DomGarguilo requested a review from keith-turner April 7, 2025 18:52
@DomGarguilo
Copy link
Member

As far as I can tell this test case looks like it covers the logic added in #5083. @keith-turner I requested your review as you were the one who created the mentioned PR. Maybe you could take a quick look as well.

@keith-turner
Copy link
Contributor

This unit test is not testing the metadata cleanup related to dead compactions. Would be better to do this in an IT, would be difficult to test the metadata updates in a unit test. Could do something like the following in an IT.

  1. Start 6 ExternalDoNothingCompactor processes
  2. Create a table w/ 3 tablets
  3. Limit compactions to 10 files per compaction
  4. Configure table to use the 6 ExternalDoNothingCompactor
  5. write 30 files to first tablet, want this to start 3 compactions for the tablet
  6. write 20 files to 2nd tablet, want this to start 2 compactions for the tablet
  7. write 10 files to 3rd tablet, want this to start 1 compaction for the tablet
  8. Continually read metadata table looking for the 6 compactions, when found remember their unique ids
  9. Kill the 6 ExternalDoNothingCompactor
  10. Continually read metadata table waiting for the 6 compaction ids to disappear (the compaction coordinator should detect the processes died and clean up any per tablet compaction entries).
  11. Read the running set from the coordinator until the 6 ids are gone

This test would cover what this test is attempting to cover and more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add cleanup test for multiple dead compactions
5 participants