Skip to content

HDDS-16357. Recon fileCount API does not normalize the fileSize query parameter - #11174

Open
henry3260 wants to merge 1 commit into
apache:masterfrom
henry3260:HDDS-16357
Open

HDDS-16357. Recon fileCount API does not normalize the fileSize query parameter#11174
henry3260 wants to merge 1 commit into
apache:masterfrom
henry3260:HDDS-16357

Conversation

@henry3260

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Recon stores file counts in a histogram keyed by (volume, bucket, fileSizeUpperBound), where the
upper bound is a power of two computed by ReconUtils.getFileSizeUpperBound(). The write path applies
this normalization in FileSizeCountTaskHelper#getFileSizeCountKey(), but the read path in
UtilizationEndpoint#getFileCounts() built the lookup key from the raw fileSize query parameter.
As a result, GET /api/v1/utilization/fileCount returned an empty list for any size that is not
an exact bin boundary:

# bucket contains two 100000-byte keys, stored in bin 131072
GET /api/v1/utilization/fileCount?volume=vol1&bucket=bucket1&fileSize=100000
-> []

GET /api/v1/utilization/fileCount?volume=vol1&bucket=bucket1&fileSize=131072
-> [{"fileSize":131072,"count":2}]

The sibling endpoint /utilization/containerCount already normalizes its parameter through
ReconUtils.getContainerSizeUpperBound(), so the two APIs behaved inconsistently.

This PR normalizes the parameter before the lookup, and sets the normalized value on the response.
The latter matters because the unfiltered branch of the same endpoint returns
key.getFileSizeUpperBound(), so without it the meaning of the fileSize response field would
depend on whether the query parameter was supplied.

Changed files:

  • hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/UtilizationEndpoint.java -
    normalize fileSize via ReconUtils.getFileSizeUpperBound() and return the bin upper bound.
  • hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestEndpoints.java -
    add a non-boundary query assertion. The existing test only queried exact bin boundaries, which is
    why this was not caught.

The write path and the on-disk layout are unchanged, so no existing Recon data is affected.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16357

How was this patch tested?

Unit tests: mvn -pl :ozone-recon test -Dtest=TestEndpoints passes (15/15).

… parameter

File counts are stored in a histogram keyed by (volume, bucket,
fileSizeUpperBound). The write path normalizes the size via
ReconUtils.getFileSizeUpperBound(), but UtilizationEndpoint#getFileCounts()
built the lookup key from the raw fileSize query parameter, so the key only
matched when the caller passed an exact bin boundary. The response also
echoed the raw size instead of the bin upper bound, unlike the unfiltered
branch of the same endpoint.
Copilot AI lite review requested due to automatic review settings August 31, 2026 16:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes an inconsistency in Recon’s /api/v1/utilization/fileCount endpoint where the read-path lookup used the raw fileSize query parameter instead of the histogram bin upper bound used by the write path. It aligns the endpoint behavior with how file counts are stored (and with the existing /utilization/containerCount behavior) by normalizing the query value and returning the normalized bin upper bound in the response.

Changes:

  • Normalize fileSize query parameter via ReconUtils.getFileSizeUpperBound() before building the RocksDB lookup key.
  • Ensure the fileSize field in the filtered response reflects the bin upper bound (consistent with the unfiltered response branch).
  • Extend unit coverage to assert correct behavior when fileSize is not an exact bin boundary.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/UtilizationEndpoint.java Normalizes fileSize to its bin upper bound for lookups and response values in getFileCounts().
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestEndpoints.java Adds a non-boundary fileSize query test to validate normalization and returned bin upper bound.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants