HDDS-11233. Ozone Storage Policy Support.

[xichen01]

What changes were proposed in this pull request?

Design storage policy for for Ozone. Please comment inline on the markdown document to ask questions and post feedback. Switch to Rich Diff mode for smoother reading.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11233

How was this patch tested?

N/A

[kerneltime]

cc @sodonnel

[ivandika3]

Nit: The table does not seem to be rendered properly.

[vtutrinov]

Is this the final list of desired/planned storage policies? Wouldn't we like to implement policies like in HDFS - https://hadoop.apache.org/docs/r3.4.1/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html ?

[xichen01]

Yes, we only support these three simple storage policies current, we can extend it in the future.

[vtutrinov]

Will we deal with the storage tier as an entry of the cluster topology?

[xichen01]

What does “deal with as a cluster topology” mean?
Storage Tier is the property of Pipeline and Container, when we create the key we will select the matched Storage Tier Pipeline and Container based on the key Storage Policy.

[vtutrinov]

I meant org.apache.hadoop.hdds.scm.net.NodeSchema. Will the Storage Tier (aka rack of specific storage volumes) become a part of the network topology

[xichen01]

Storage Tier is more like the ReplicationConfig, will be a independent fields in ContainerInfo and Pipeline

[vtutrinov]

The current implementation of the background pipeline creator creates a limited list of RATIS/THREE-pipelines per datanode. The design doc proposes to deal with specific pipelines for different storage policies. Is there any proposal doc on how we gonna deal with the limitation mentioned earlier? (How should we deal with the distribution of the pipelines of different types: 30% per storage tire or some other option? What if the pipelines of one of the storrage tire will not be used?)

[xichen01]

The count limit for current Pipeline count will be extend to each storage tier, so in the extreme condition, there will be three times Pipeline in a cluster.

But in the actual situation, there may not be so many Pipelines, because a Pipeline may support multiple tiers, such as DISK and SSD, because there may be both SSD and DISK type Volume in a Datanode machine.

And the background Pipeline creator will check the current cluster's Volume type, if a cluster has only SSD Volume, the background Pipeline will create a Pipeline with only SSD Tier type.

[greenwich]

However, on a dedicated-tier cluster (16 SSD, 16 DISK nodes), the SSD tier can only form a limited number of EC 3+2 pipelines. Does the background creator have a target count? Does it back off when the tier is saturated?

[xichen01]

Ratis' Pipeline utilizes a Background service for creation (BackgroundPipelineCreator). The number of Ratis Pipelines can be controlled through configuration.

EC-type Pipelines are created only when used. Different EC configurations generate distinct Pipelines. Pipelines from different EC configurations do not interfere with each other. (WritableECContainerProvider#getContainer) SCM selects the DN with the appropriate Storage Type to create the Pipeline.

[vtutrinov]

@xichen01 is there an understanding of the time frame for the functionality to be implemented? I'd start creating the JIRA tickets and implementing them

[vtutrinov]

@xichen01 @kerneltime @sodonnel, could you help somehow to force the review of the design doc? The feature is very needed, and I would gladly start implementation.

[ivandika3]

@vtutrinov Currently the implementation is being worked on internally for the past one year.

The basic implementation of storage policy and storage types integration on containers, pipelines, volumes, s3 storage class, and creating key / file with storage policy has been implemented but still need extensive testing. Currently we are focusing on storage policy migration implementation.

@xichen01 would know more about the approximate timestamps, but we hope to have a working implementation in the next quarter (i.e. Q3 2025) or so. We are currently quite busy with our internal works so there might not be any bandwidth to backport all these patches in the short term.

[vtutrinov]

@ivandika3 thanks for the response!
Can we glance at the implementation as the first phase (maybe in a custom feature branch)? Or are there too many private details?

[ivandika3]

@vtutrinov the fastest way I think we can do is to provide you with the diffs. However these diffs won't apply cleanly on the master branch since our branch is based on 1.4.1 version with some of our internal specific changes. I probably can provide some of it this weekend.

Feature branch in community fork might take a while since we need to resolve the conflicts.

[vtutrinov]

@ivandika3 it would be great!

[vtutrinov]

@ivandika3 I don't want to rush, but is there any news about the mentioned diff?

[ivandika3]

@vtutrinov Thanks for the reminder. I have attached https://issues.apache.org/jira/secure/attachment/13077025/storage-policy-diff.tar.gz for the list of diffs of the storage policy integration.

Please be reminded to attribute @xichen01 for any commits generated from these diffs.

[vtutrinov]

@ivandika3 @xichen01 the diff above has references to the following non-existent files (relative to ozone-1.4.1):

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementDataCenterAware.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementDataRecovery.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelinePlacementDataCenterAware.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementDataCenterAware.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementDataCenterAwareSpecialCase.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementDataRecovery.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementDcFlow.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestPipelinePlacementDataCenterAware.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/diskbalancer/TestDiskBalancerService.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementStorageTier.java
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/StorageTierUtil.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeUtils.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSpecialCloseContainerEventHandler.java
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestPeriodicContainerCloser.java
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/AbstractRootedOzoneFileSystemTest.java
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/UpdateBucketOptions.java
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/StorageTypeUtils.java
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneStoragePolicy.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/balancer/TestContainerBalancerTaskDcFlow.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerUtils.java
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/client/StorageTierUtilTest.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/dcflow/ContainerBalancerSelectionCriteriaDcFlow.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/AbstractStorageTypeChoosingPolicy.java
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/storagepolicy/StoragePolicyCommands.java
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/storagepolicy/UsageInfoSubCommand.java

Could you provide them too, or point me to the commit where I can fetch them?

[github-actions]

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

[github-actions]

Thank you for your contribution. This PR is being closed due to inactivity. If needed, feel free to reopen it.

[greenwich]

Hi, do you know if any work is planned for this ticket? AFAIK, the patch diff wasn't added to the branch and is probably out of date right now. What are the next steps here?

[errose28]

Hi @greenwich I see you have also opened #9807. If you would like to continue work on this, we should start by reaching agreement on a design doc. I'm not sure we finished that process yet. @ivandika3 @xichen01 does this doc need more updates/review? Should we continue work on it here or open a new PR?

[greenwich]

@errose28 Thanks for looking! I am very sorry for the noise my PR caused; it wasn't intended to be public (My bad - I haven't set it to Draft when I created it).

Let me explain my motivation.

1. My team needs storage policy and tiering support in Ozone this year (ideally in H1). This capability is becoming increasingly important for us, especially as there are parallel evaluations happening within the company. I would strongly prefer to continue building on Ozone, which is why I’m investing time in helping move this forward. We are planning for this cluster to grow beyond 20-30PB, and at that scale storage tiering becomes operationally critical rather than just a nice-to-have capability.

2. The patch attached to this pull request doesn’t seem to be up to date with master or even the 2.0 or 2.1 releases.

3. I might be missing something, but I didn’t see recent updates on this PR since Nov 2025.

4. I may be mistaken, but I have the impression that before starting implementation, we aim for a comprehensive and fully generic design that covers all possible use cases. I completely understand and appreciate that approach — it leads to a robust and future-proof solution. At the same time, I’m mindful of the time constraints on my side (as mentioned in point 1), so I’m hoping we can find a pragmatic way to make incremental progress while the broader design continues to evolve.

5. My approach is slightly different — I would like to move forward incrementally. My goal is to design and implement storage tiering in small, self-contained steps (MVP-1, MVP-2, etc.), where each stage delivers a complete end-to-end capability.

This would allow my team to start using the functionality early, gather real-world feedback, and better understand practical requirements. I believe this iterative approach could also help inform and refine the broader design discussion.

I am currently working on MVP-1, and MVP-2 already includes many of the most important features we need. My plan is to validate these internally in our TEST environment (and eventually PROD if stable), and then share the results and learnings with the Ozone community for discussion and alignment. Of course, I’m very open to feedback on this approach and happy to adjust direction based on community input.

6. I haven’t created any Jira tickets yet because my initial intention was to implement and validate the first MVP internally. However, each independent feature is pushed as a separate commit, so it can later be aligned with and retrofitted into ASF Jira tickets if needed.

7. Finally, regarding why I created a PR at this stage: the GitHub PR workflow is very useful for triggering CI, viewing diffs against master, and keeping my branch aligned. I also hope that parts of this work may eventually be contributed back to master.

[xichen01]

@errose28 Thanks for noticing this PR.

@greenwich Thank you for the update.

Regarding the current status of this PR

The current PR hasn't been updated for a while, mainly because there doesn't seem to be a strong demand for this feature from other members of the community, so reviews have stagnated.
However, this feature has been fully implemented internally, including support for StoragePolicy across all S3 and Filesystem write interfaces, as well as support for StoragePolicy in ReplicationManager and ContainerBalancer.
We've basically implemented it according to this design document (some parts of the document need updating; I can update the document if needed).

We also support S3 Lifecycle (https://issues.apache.org/jira/browse/HDDS-8342), allowing you to set a Lifecycle for a Bucket to migrate specified keys to a specified StoragePolicy at a specified time (including from SSD migrate to DISK, and also from THREE Replication migrate to EC) or similar HDFS SPS (Storage Policy Satisfaction) functionality.

Follow-up

If you or others in the community are willing, we can continue to move forward with this PR, of course, you can also move forward with your own proposals, we can cooperate if you need it.

[greenwich]

@xichen01, Thanks for the update. That's great news indeed! In that case, there is no point for me to invent a wheel. I had a quick look at the patch you submitted earlier, and it looks comprehensive. I will have a detailed look today.

Would it be possible to move forward with that PR (and merge it, or at least create a working branch and sync it with master)? Also, as I am very interested in that feature, I can provide assistance on my end.

cc: @ivandika3 @chungen0126
Please let me know.

[ivandika3]

@xichen01 Let's reopen this patch and move this forward. I am willing spend some time on the storage policy support development. Hopefully other community members (@errose28 @chungen0126) are also able to help to push the review process and commit to our implementation.

@greenwich As mentioned in the #8811 (reply in thread) discussion last year. There are mainly two phases of Ozone Storage Policy Support

1. Supporting storage policy and storage types on Ozone
2. Storage Policy Migration support

Phase 1 goal is to allow client to upload keys with different storage policy and integrate storage policy and types to all Ozone pipeline, containers, etc. This will be our short term focus now since we are not introducing any new subsystem. @xichen01 Let's start by backporting this to the community fork and ensure that everything works normally, we can discuss on which patch we will do.

Phase 2 goal is to support Storage Policy Migration. However, our current implementation requires a separate job worker subsystem. This will be longer term since we are introducing some new subsystem. If you don't require a separate job worker subsystem, you might need to write your own implementation of "Storage Policy Satisfier".

That said, hope you understand that all contributions to Ozone are purely voluntary and made by members with other higher level priorities and therefore we cannot 100% guarantee that this will be done in a timely manner.

[greenwich]

@greenwich Sorry for the late reply. Let me try to use an AI agent to backport the storage policy to my fork.

@ivandika3, I managed to do a similar thing because I wanted to prepare everything up-front to make things go faster for this PR.

• I picked the patch attached to this PR (that was branched out from ozone 1.4)
• Forked the upstream master to my own branch around two weeks ago
• Merged that patch into that fork, fixed all the conflicts and broken tests
• Yesterday I synced upstream's master with that branch again to be up to date.
• Please have a look, maybe it already has what you were planning to do: https://github.com/greenwich/ozone/tree/refs/heads/HDDS-11233_patch_merged
• All tests are green: https://github.com/greenwich/ozone/actions/runs/24915662451
  cc: @xichen01

[ivandika3]

Thanks @greenwich , @xichen01 would you mind take a look at that?

I'm currently backporting it in my fork, but the test are still failing (https://github.com/ivandika3/ozone/tree/refs/heads/backport-storage-policy-storage-class)

[ivandika3]

@greenwich @xichen01 FYI, test in https://github.com/ivandika3/ozone/tree/refs/heads/backport-storage-policy-storage-class passed. You can refer to the diff in ivandika3#4

[ivandika3]

Regarding Phase 2, just thinking ahead — would it make sense to initially target a simpler mover-style implementation (similar in spirit to HDFS Mover) before introducing a separate job worker subsystem? That might allow basic Storage Policy Migration functionality to be delivered earlier and iterated on over time.

@greenwich Yes, we can first implement a standalone StoragePolicySatisfier (similar to HDFS Mover or HDFS StoragePolicySatisfier) without needing to implement a separate subsystem.

[xichen01]

@greenwich @xichen01 FYI, test in https://github.com/ivandika3/ozone/tree/refs/heads/backport-storage-policy-storage-class passed. You can refer to the diff in ivandika3#4

@ivandika3 We can create corresponding sub tasks for these commits and then merge them into the HDDS-8342 branch via MR

[ivandika3]

@xichen01 How about we cut a new HDDS-11233 branch? The current branch can be done without the transition patch (HDDS-8342). We can add the transition after HDDS-8342 is merged to master.

[xichen01]

@xichen01 How about we cut a new HDDS-11233 branch? The current branch can be done without the transition patch (HDDS-8342). We can add the transition after HDDS-8342 is merged to master.

I think we can create a new HDDS-11233 branch on the Apache Ozone repositories, just like https://github.com/apache/ozone/tree/HDDS-8342. Then we can merge related Commit to this branch.

[ivandika3]

@xichen01 Could you help to take over? I'm currently quite busy and my GH account was recently blocked from GH actions (still waiting for support) so no CI can be triggered.

[xichen01]

@xichen01 Could you help to take over? I'm currently quite busy and my GH account was recently blocked from GH actions (still waiting for support) so no CI can be triggered.

OK, I will

[xichen01]

The new HDDS-11233 branch https://github.com/apache/ozone/tree/HDDS-11233

[xichen01]

@ivandika3 @greenwich @chungen0126
#10191 Please help to review

[chungen0126]

Overall, looks good to me.

[greenwich]

@ivandika3 @greenwich @chungen0126 #10191 Please help to review

Looks good!

[greenwich]

@ivandika3, @xichen01, as we have HDDS-15239 merged and a new branch created. Are we ready to move forward?

[xichen01]

@ivandika3, @xichen01, as we have HDDS-15239 merged and a new branch created. Are we ready to move forward?

Yes, we will continue to submit new PRs after the merge

[github-actions]

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

[github-actions]

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.