Severe performance degradation due to file read hang with single-core max-out

[g0715158]

Search before reporting

• I searched in the issues and found nothing similar.

Read release policy

• I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

User environment

version:3.0.10
jdk 17
os arm

Issue Description

When I split the snapshot and ledgers directories of the bookie from one into four, a problem arises during stress tests with 500KB large messages: file reading gets stuck, resulting in extremely low performance. At this time, IO, network, and overall CPU utilization are all low, but a single CPU core becomes fully saturated. Thread stack analysis shows that the process is blocked in the file reading phase.

Error messages

Reproducing the issue

When I split the snapshot and ledgers directories of the bookie from one into four, a problem arises during stress tests with 500KB large messages: file reading gets stuck, resulting in extremely low performance. At this time, IO, network, and overall CPU utilization are all low, but a single CPU core becomes fully saturated. Thread stack analysis shows that the process is blocked in the file reading phase.

Additional information

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

btw. From the screenshot I can see that you are using Java 17.0.14 . You should upgrade to at least 17.0.17 since it includes an important fix, https://bugs.openjdk.org/browse/JDK-8351933 . It's not related to this issue, but if you want to avoid hitting known issues, it's recommended to use latest released Java 17 for 3.0.x . For 3.3.x and newer, latest Java 21 is recommended at runtime.

[lhotari]

When I split the snapshot and ledgers directories of the bookie from one into four, a problem arises during stress tests with 500KB large messages: file reading gets stuck, resulting in extremely low performance. At this time, IO, network, and overall CPU utilization are all low, but a single CPU core becomes fully saturated. Thread stack analysis shows that the process is blocked in the file reading phase.

There are multiple ways to tune Pulsar and BookKeeper. I don't think that it makes sense that you report this as a bug. I'll convert this to a discussion instead.