Add worker threads related metrics to Grafana

[natanasow]

Problem

We added several metrics tracking the worker threads via this PR.

Solution

We need add these metrics to Grafana as well:

• We need to design a chart to be included in the dasboard via an ADR, approve it and then add it. added to this we need to define alerts and implement them
• We need to update the reference community dashboard and hasio dashboard

Added metrics:

• rpc_relay_worker_task_duration_seconds
• rpc_relay_worker_tasks_completed_total
• rpc_relay_worker_task_failures_total
• rpc_relay_worker_queue_wait_time_seconds
• rpc_relay_worker_pool_utilization
• rpc_relay_worker_pool_active_threads
• rpc_relay_worker_pool_queue_size

Alternatives

No response

[jasuwienas]

ADR: Grafana dashboard for worker threads pool metrics

Status

Proposed

Date

2026-01-23

Context

The Hedera JSON-RPC Relay already collects and exports a comprehensive set of Prometheus metrics describing the behavior of the worker threads pool, including task execution latency, queueing, throughput, failures, and pool saturation.

Worker thread metrics are not included in the recently deployed v0.74.0 tag, but are part of the main branch (and docker image ghcr.io/hiero-ledger/hiero-json-rpc-relay:main).

These metrics are currently available via the /metrics endpoint and are being scraped by Prometheus. However, they are not surfaced in Grafana yet, which makes it difficult to:

• Understand worker pool health (at a glance),
• Detect that the wrokers are getting bussy early
• Compare performance and error rates across task types (json rpc methods)..

Decision / Options

We will visualize the existing worker pool metrics in Grafana using one of the following options:

• Option A: Create a separate dashboard (e.g. “RPC Relay - Worker Pool”) to keep worker metrics isolated and easier to reason about. (name to be decided)
• Option B: Add worker thread panels to the existing HTTP Server and WS Server dashboards

Metrics Included (Existing)

Worker Pool Core Metrics

Metric Name	Type	Labels	Description
rpc_relay_worker_task_duration_seconds	Histogram	function	Task execution time from start to completion.
rpc_relay_worker_queue_wait_time_seconds	Histogram	(none)	Time tasks spend waiting in the worker queue before execution.
rpc_relay_worker_tasks_completed_total	Counter	function	Total number of successfully completed tasks.
rpc_relay_worker_task_failures_total	Counter	function, error_type	Total failed tasks by type and error category.
rpc_relay_worker_pool_utilization	Gauge	(none)	Worker pool utilization ratio (0–1).
rpc_relay_worker_pool_active_threads	Gauge	(none)	Number of currently active worker threads.
rpc_relay_worker_pool_queue_size	Gauge	(none)	Number of tasks currently waiting in the queue.

All metrics listed above are already implemented and exported by the relay.
This ADR only defines how they should be visualized together in Grafana.

Dashboard design principles

The dashboard must clearly answer:

• Are workers fully utilized?
• Is work queueing up?

This can be achieved via:

• Worker utilization (time series + stat)
• Queue size (time series + stat)
• Queue wait time p95/p99

We should also be able to monitor latency, accomplished by:

• Average execution time over short windows (e.g. last 1m)
• Overall p50 / p95 / p99 execution time
• p95 execution time by function

This allows fast identification of which task type is slow.

We have to monitor overal throughtput and how many requests are processed in a time.

We must track and monitor the number of errors which are occuring.,...

---

Derived Dashboard Signals)

The dashboard will derive higher-level signals using PromQL only:

• Average task duration (last 1m)
  sum(rate(..._sum[1m])) / sum(rate(..._count[1m]))

• Request rate (req/s)
  sum(rate(rpc_relay_worker_tasks_completed_total[1m]))

• Failure ratio
  failures / completed

No additional instrumentation is required.

Alerting

The dashboard is designed to support alerting such as:

Signal	Condition	How to handle
Pool saturation, critical situation	utilization > 0.9	Restart relay or explicitly shut down the worker queue (dropping queued tasks); then consider increasing worker pool size when the problem repeats itself. If it's recurring consider optimising code as well.
Pool saturation, first warning	utilization > 0.8	Monitor closely; check which task types are slow; consider increasing worker pool size if sustained. If it's recurring consider optimising code as well.
Queued for too long	queue wait time > threshold	Shut down the relay to drop waiting tasks; reassess worker pool size. If it's recurring consider optimising code as well.
Backpressure	queue size > 0 AND utilization > 0.9	Restart relay or explicitly shut down the worker queue (dropping queued tasks); then consider increasing worker pool size when the problem repeats itself.
Latency regression	p95 task duration > baseline	Identify slow rpc method; decide whether more workers are needed or restart relay if tasks appear stuck. Consider optimising code...
Elevated failures	failure ratio > threshold	Inspect failing task types and error categories; restart relay if failures indicate stuck or unhealthy workers. Fix the code otherwise

Proposed solution

worker-pool-metrics.json

Proposed solution (metric-by-metric)

• • rpc_relay_worker_pool_utilization (Gauge, 0–1)
    Shows how busy the worker pool is. Close to 1.0 means workers are fully busy. If this stays high and the queue grows, the relay cannot keep up.
• • rpc_relay_worker_pool_active_threads (Gauge)
    Shows the number of worker threads currently active. Helps confirm the pool is running and whether thread count changes correlate with load.
• • rpc_relay_worker_pool_queue_size (Gauge)
    Shows how many tasks are waiting to be picked up by workers. A rising value means work is piling up.
• • rpc_relay_worker_tasks_completed_total (Counter, label: function)
    Counts how many worker tasks completed successfully. Used as the base signal for a simple “is work happening” indicator.
    Dashboard uses:
  • • Total processed (all time): sum(rpc_relay_worker_tasks_completed_total)
  • • Request rate (total): sum(rate(rpc_relay_worker_tasks_completed_total[...]))
  • • Request rate by function: sum by (function) (rate(rpc_relay_worker_tasks_completed_total{function=~"$function"}[1m]))
• • rpc_relay_worker_task_failures_total (Counter, labels: function, error_type)
    Counts failed worker tasks. Grouped by error_type to quickly see what kind of failures are happening.
    Dashboard uses:
  • • Failures/sec by error_type: sum by (error_type) (rate(...))
  • • Failure ratio: failures / completed to track the error percentage under load.
• • rpc_relay_worker_task_duration_seconds (Histogram, label: function)
    Tracks how long a worker task runs (execution time). Used to monitor latency and find slow task types.
    Dashboard uses:
  • • Quantiles overall (p50/p95/p99): shows typical vs worst-case latency.
  • • p95 by function: quickly identifies which task type is slow (How to read these metric : 95 out of 100 tasks finished faster than this, only 5 were slower.)
  • • Average duration (last 1m): sum(rate(_sum[1m])) / sum(rate(_count[1m])) for a stable “current average”.
• • rpc_relay_worker_queue_wait_time_seconds (Histogram)
    Tracks how long tasks wait in the queue before they start running. This is the clearest “work is waiting” signal.
    Dashboard uses:
  • • Queue wait quantiles (p50/p95/p99): shows how bad waiting gets during load.
  • • Average queue wait (last 1m): sum(rate(_sum[1m])) / sum(rate(_count[1m])) to track the current average delay.
• • [SOMETHING ELSE, PROPOSALS TO BE DISCUSSED...]

How to read it quickly

• If utilization ~ 1.0 and queue size > 0 / growing, workers are too busy and tasks are waiting.
• If queue wait p95 rises, users will feel slowdown even if average looks fine.
• If p95 task duration rises for a specific function, that task type is the main cause.
• If failure ratio rises, the system is not only slow but also failing more under load.

[silkshjasuwienas]

Alerts: queued for too long;l 0.8, 0.9
Add to alerting part
What to do: