GH-6372: Expose client request metrics.

[chickenchickenlove]

Motivation:

Because of #6372 .

Modifications:

• Add new class ClientMetrics.
• HttpClientFactory has ClientMetrics as its field.
• HttpChannelPool has ClientMetrics as its field.
• HttpChannelPool calls ClientMetrics whenever calling setPendingAcquisition(...) and removePendingAcquisition(...).
• HttpSessionHandler calls ClientMetrics to increment count for active request and reserve to decrement count of active request.

Result:

• Closes Expose client request metrics #6372

[github-actions]

🔍 Build Scan® (commit: 6be255d)

Job name	Status	Build Scan®
build-ubicloud-standard-16-jdk-8	✅	https://ge.armeria.dev/s/pjsenp6os2law
build-ubicloud-standard-16-jdk-21-snapshot-blockhound	✅	https://ge.armeria.dev/s/c74g3djmby2lk
build-ubicloud-standard-16-jdk-17-min-java-17-coverage	❌ (failure)	https://ge.armeria.dev/s/klkopj4ss6gw6
build-ubicloud-standard-16-jdk-17-min-java-11	✅	https://ge.armeria.dev/s/iht5argexjpha
build-ubicloud-standard-16-jdk-17-leak	✅	https://ge.armeria.dev/s/qosskl6uggjk6
build-ubicloud-standard-16-jdk-11	✅	https://ge.armeria.dev/s/mhfikxw3horqg
build-macos-latest-jdk-21	✅	https://ge.armeria.dev/s/rywv5lqg5sby2

[jrhee17]

Question) If the objective is to determine the optimal number of event loops depending on pending requests for each endpoint group/protocol, would it be enough to check the duration instead?

final EndpointGroup endpointGroup = ctx.endpointGroup();
ctx.log().whenComplete().thenAccept(log -> {
    final long pendingDuration = log.connectionTimings().pendingAcquisitionDurationNanos();
    final SessionProtocol sessionProtocol = log.sessionProtocol();
});

[chickenchickenlove]

@jrhee17 thanks for your comments!
Your comment also make sense to me!
However, IMHO, it may not be enough or enough. 😄
So I would like to vote for counting the number of pending requests.

The pendingAcquisitionDurationNanos seems to be set after request get channel.
In that case, pendingAcquisitionDurationNanos can be -1 or large value.

If the acquisition of a channel is delayed due to a sudden surge in requests, the value of pendingAcquisitionDurationNanos will continuously remain -1. Or, if the channel is acquired after a very long time, the value of pendingAcquisitionDurationNanos will change from -1 to a very large value. Therefore, the responsiveness of the operation that increases the EventLoop is likely to be degraded.

Additionally, when the Channel is Busy, we need to consider two states:

1. The value is -1.
2. The value is an enormously large number.

Consequently, the ClientMetrics object would need to contain code to account for this, which potentially implies that ClientMetrics is tightly coupled with the logic of ConnectionTimings.

For these reasons, I vote for counting the number of pending requests.

What do you think?

Also, @ikhoon , please give your opinion when you have time!

[ikhoon]

I think using pending duration is also a good idea. However, adding ClientMetrics would be worthwhile because:

• ClientMetrics itself seems to provide useful information at the ClientFactory level.
• Using ConnectionTimings inside maxNumEventLoopsFunction method doesn't look straightforward as it may require additional processing.

[jrhee17]

I see - I don't think I can meaningfully review this PR at this point since I'm not sure from the issue/PR description how ClientMetrics is expected to be used.

Do you envision ClientMetrics as a general metric collector that collects metrics on the overall Client (or ClientFactory)? What would be the relationship between this metric collector and other metrics that are already being exported?

Otherwise, it might help if the class name were more specific.

[ikhoon]

Most APIs that are exposed at the request level will primarily be used for logging or collected by metric collectors such as Prometheus. Since they contain per-request details, it does not seem easy to use them directly to control the server’s runtime behavior.

ServerMetrics was first exposed as a Java API because it was difficult to obtain information about in-flight requests at runtime when implementing a custom graceful shutdown. Similarly, ClientMetrics will expose the information needed to control runtime behavior at the client or client-factory level. I expect these values to be exposed mostly as simple counters rather than histograms.

Otherwise, it might help if the class name were more specific.

I’m open to changing it if you think there’s a better name.

[ikhoon]

Question) Couldn't we use Endpoint instead of EndpointGroup as the key to collect metrics?

[chickenchickenlove]

@ikhoon nim, thanks for your comments.
Addressed it!

[chickenchickenlove]

@ikhoon , @jrhee17
I made a new commit base on comments!
When you have time, please take another look. 🙇‍♂️