HDFS versus Ozone performance

[frensjan]

Hi!

I'm evaluating Apache Ozone to replace HDFS for storing blobs. I've created a small 4 node cluster with SSD's. Each node has 2 volumes for metadata and 2 to 3 volumes for data. All nodes are data-nodes and run the S3 gateway. The first three nodes run the OM and SCM in HA mode. Some initial benchmarking using warp uncovered some lower than expected performance.

I performed a small comparison on the same cluster between Ozone and a simple HDFS cluster with distcp. I used another HDFS cluster as source / target. Both reading from and writing to the Ozone cluster was 2-3x faster when using HDFS vs when using Ozone. Note that the HDFS cluster was not in HA mode; just a single stand-alone name-node.

The non-HA mode of HDFS obviously creates a benefit for the HDFS option. But the difference was still a lot larger then what I expected. I expected similar performance, just with better scalability for Ozone in the number of files and node density.

Is there anything I can do to debug this performance difference?

[kerneltime]

Yes there is a bit of tuning that is needed to improve default Ozone performance. Cloudera does document what they recommend https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/ozone-performance-tuning/topics/ozone-performance-tuning-for-ozone.html

ozone.scm.pipeline.owner.container.count is the main config change which increases the number of pipelines that accept writes.

[frensjan]

Thanks @kerneltime, for the quick response. I had these recommendations in sight. I have 10 pipelines with RF=THREE active for the 4 nodes with in total 10 data disks (marked as DISK, but they're actually on SSD).

I seem to be mainly hitting CPU limits. The nodes are on a VM with some noisy neighbours. Each node has 3 CPU's. They are hitting 85-95% CPU during a write workload and 65-75% on a read workload. Around 1/3 of it is system time.

[kerneltime]

Which release are you one? There many performance fixes in master. 1.4 is really old at this time and the community is working on 2.0 release.

[frensjan]

I'm using 1.4.1.

I noticed a Jira about finishing up the 2.0 release; would love to test-drive. Are you recommending to create a custom build from master? I usually steer away from that sort of stuff. 🙃