Skip to content

HDDS-15789. OM DB size should be reduced after compaction - #11173

Open
sravani-revuri wants to merge 2 commits into
apache:masterfrom
sravani-revuri:HDDS-15789
Open

HDDS-15789. OM DB size should be reduced after compaction#11173
sravani-revuri wants to merge 2 commits into
apache:masterfrom
sravani-revuri:HDDS-15789

Conversation

@sravani-revuri

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Even after HDDS-14622, OM DB compaction test still encounters intermittent failure:

https://github.com/apache/ozone/actions/runs/28952848017/job/85906945842#step:13:663
om1-1  | 2026-07-08 15:35:19,433 [ForkJoinPool.commonPool-worker-2] INFO om.OzoneManager: Compaction request for column family "fileTable" completed successfully.
om1-1  | 2026-07-08 15:35:22,874 [ForkJoinPool.commonPool-worker-2] INFO om.OzoneManager: Compaction request for column family "deletedTable" completed successfully.
om1-1  | 2026-07-08 15:35:25,216 [ForkJoinPool.commonPool-worker-2] INFO om.OzoneManager: Compaction request for column family "deletedDirectoryTable" completed successfully.
OM DB size should be reduced after compaction. Before: 65155, After: 103393
ERROR: Test execution of ozonesecure-ha/test-repair-tools.sh is FAILED!!!!

Deletion is an asynchronous process and the old test ran the compaction as soon as deletion started . There could be a situation where deletion is not yet completed and compaction is run. In this case new sst files are written which causes a higher number even after compaction is run , causing the test to fail intermittently.
The test now waits for deletion to complete and then runs compaction.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15789

How was this patch tested?

https://github.com/sravani-revuri/ozone/actions/runs/33398347599

sample output:

OM DB SST size before compaction: 65014
OM DB SST size after compaction: 28382

@sravani-revuri
sravani-revuri marked this pull request as ready for review September 1, 2026 06:02

@sarvekshayr sarvekshayr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sravani-revuri for working on this.

Question on timeouts: Were 300s (wait_for_bucket_deletion_complete) and 180s (wait_for_om_db_size_stable) chosen from observed wait times on CI?
Did you try shorter values (e.g. 240s / 120s) to see if the test becomes flaky again?

[[ "${n:-1}" -eq 0 ]] || continue 2
done
return 0
sleep 3

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sleep 3 is unreachable because it comes after return 0. The retry loop currently has no delay between polling attempts. Let's either move the sleep into the retry check or drop it if no delay is required as the test is passing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants