Worker heartbeat: New in-memory metrics mechism, plumb rest of heartbeat data by yuandrew · Pull Request #1023 · temporalio/sdk-rust

yuandrew · 2025-09-29T22:14:44Z

What was changed

NOTE: targeting worker-heartbeat feature branch.

New in memory mechanism to keep track of certain metrics for worker heartbeating.

Why?

Checklist

Closes
How was this tested:

Any docs updates needed?

Note

Adds in‑memory metric tracking and plumbs full worker heartbeat data (pollers, slots, cpu/mem, plugins), plus new DescribeWorker/SetWorkerDeploymentManager APIs and supporting refactors.

Worker Heartbeat (core/worker):
- Capture full heartbeat payload: poller counts (with last-successful-poll timestamps), slot usage (available/used, totals, last-interval deltas), status, start/heartbeat times, host CPU/mem, deployment version, plugins.
- Introduce SlotSupplierKind (Fixed/ResourceBased/Custom) and report per-slot supplier kind.
- Track last successful poll times in PollScaler/LongPollBuffer and feed into heartbeats.
- Add worker_instance_key() to Worker API; export via heartbeats and tests.
- Support skipping duplicate client worker checks (skip_client_worker_set_check).
Telemetry Metrics (core-api/core):
- New in-memory metrics layer (HeartbeatMetricType, WorkerHeartbeatMetrics) + *_with_in_memory constructors for Counter/Gauge/HistogramDuration and attribute updates.
- Wire key SDK metrics to in-memory counters/gauges/histograms for heartbeat reporting; enhance NoOpCoreMeter attributes.
Resource/Tuner:
- RealSysInfo now refreshes on a background thread; expose SystemResourceInfo via TunerBuilder; report CPU/mem.
Client/Registry:
- HeartbeatCallback now Arc; remove re-register callback path; register(worker, skip_check) and test updates.
- Client fills heartbeat client fields and computes last-interval slot deltas.
APIs and Protos:
- Add RPCs: DescribeWorker, SetWorkerDeploymentManager; wire through raw client and C-bridge.
- OpenAPI/Protos updated (new fields: allowNoPollers, managerIdentity, workerHeartbeats capability, cluster failover versions, doc tweaks).
Config & Tests:
- WorkerConfig gains plugins and skip_client_worker_set_check.
- Extensive new integ tests for heartbeats/metrics; enable feature flags in test runner.
- Minor fix: comment/typo, UUID dep added.

^{Written by Cursor Bugbot for commit 8f1ef5a. This will update automatically on new commits. Configure here.}

…g, counter_with_in_mem and and_then()

…, remove stale metric previously added

…ing integ test

Sushisource · 2025-10-07T22:58:56Z

+impl WorkerHeartbeatMetrics {
+    pub fn get_metric(&self, name: &str) -> Option<HeartbeatMetricType> {
+        match name {
+            "sticky_cache_size" => Some(HeartbeatMetricType::Individual(


All these names are duplicative of some existing metric name in core/src/telemetry/metrics.rs, which is calling through into this anyway. Rather than having all the fields be pub, I think it'd be safer to make getters for each, individually, that do the same thing this match does.

that do the same thing this match does.

Not sure I follow this part, happy to make getters so all fields aren't pub, but it sounds like you're suggesting a change to the match itself? We need some way to map the str name to the struct field. Are you suggesting moving this whole fn over to MetricParameters?

…ers directly, WithLabel API improvement

…ing, more tests

Sushisource

Looks like some of the tests are failing too

Sushisource · 2025-10-16T17:17:46Z

-                        ..Default::default()
-                    }),
+    worker.register_wf(wf_name.to_string(), move |ctx: WfContext| async move {
+        COUNT.store(COUNT.load(Ordering::Relaxed) + 1, Ordering::Relaxed);


This is just a slightly broken version of fetch_add 😅

Sushisource

🎉

…nt level (#1038) * Runtime/namespace/client wide worker heartbeat (#983) * worker heartbeat * Address Spencer's comments * wip use client_identity_override as part of key, added test * Refactor almost complete, need to plumb through telemetry to SharedNamespaceWorker * Verified client replacement works, need to update tests and cleanup * formating * clean up * forgot to remove new() now that using builder pattern * Switch to worker_set_key * Replace client test passes, need to write unit tests in worker_registry * cargo test-lint * limit nexus to 1 poller, add tests for worker_registry for heartbeat * PR comments * new test helper * Return error on multi worker register for same namespace and task queue on same client * cargo fmt * Fix registration order, unique task queue for test worker * Remove TEST_Q variable * Missing quotes * CI lint and docker test fix, rename worker_set_key to worker_grouping_key * clippy bug * Worker heartbeat: New in-memory metrics mechism, plumb rest of heartbeat data (#1023) * plumb in memory metrics * simplify worker::new(), fix some heartbeat metrics, new test file * CounterImpl, final_heartbeat, more specific metric label dbg_panic msg, counter_with_in_mem and and_then() * Support in-mem metrics when metrics aren't configured * Move sys_info refresh to dedicated thread, use tuner's existing sys info * Format, AtomicCell * Fix unit test * Set dynamic config for WorkerHeartbeatsEnabled and ListWorkersEnabled, remove stale metric previously added * Should not expect heartbeat nexus worker in metrics for non-heartbeating integ test * recv_timeout instead of thread::sleep, use WorkflowService::list_workers directly, WithLabel API improvement * MetricAttributes::NoOp, add mechanism to ignore dupe workers for testing, more tests * More tests, sticky cache miss, plugins * Formatting, fix skip_client_worker_set_check * Cursor found a bug * Lower sleep time, add print for debugging * more prints * use semaphores for worker_heartbeat_failure_metrics * skip_client_worker_set_check for all integ workers * Can't use tokio semaphore in workflow code * use signal to test workflow_slots.last_interval_failure_tasks * Use Notify instead of semaphores, fix test flake * Use eventually() instead of a manual sleep * max_outstanding_workflow_tasks 2 * merge * Forgot to commit format fixes * Fix test

yuandrew added 2 commits September 29, 2025 15:13

plumb in memory metrics

c32d6c9

simplify worker::new(), fix some heartbeat metrics, new test file

0ad6be2

Sushisource reviewed Sep 30, 2025

View reviewed changes

Sushisource mentioned this pull request Sep 30, 2025

[Feature Request] Support CPU Cgroup Limits in Resource-based tuner / Confirm memory functionality #903

Closed

yuandrew added 5 commits October 1, 2025 11:31

CounterImpl, final_heartbeat, more specific metric label dbg_panic ms…

d314a35

…g, counter_with_in_mem and and_then()

Support in-mem metrics when metrics aren't configured

b00b80e

Move sys_info refresh to dedicated thread, use tuner's existing sys info

894d916

Format, AtomicCell

ef96616

Fix unit test

0350f91

yuandrew marked this pull request as ready for review October 5, 2025 03:34

yuandrew requested a review from a team as a code owner October 5, 2025 03:34

This comment was marked as outdated.

Sign in to view

Set dynamic config for WorkerHeartbeatsEnabled and ListWorkersEnabled…

fe3956c

…, remove stale metric previously added

This comment was marked as outdated.

Sign in to view

Should not expect heartbeat nexus worker in metrics for non-heartbeat…

d956784

…ing integ test

This comment was marked as outdated.

Sign in to view

Sushisource reviewed Oct 7, 2025

View reviewed changes

recv_timeout instead of thread::sleep, use WorkflowService::list_work…

98e778d

…ers directly, WithLabel API improvement

Sushisource reviewed Oct 8, 2025

View reviewed changes

Comment thread core/src/worker/mod.rs Outdated

MetricAttributes::NoOp, add mechanism to ignore dupe workers for test…

96e0186

…ing, more tests

Sushisource reviewed Oct 10, 2025

View reviewed changes

Comment thread client/src/worker_registry/mod.rs Outdated

More tests, sticky cache miss, plugins

e1ad0ae

Sushisource reviewed Oct 14, 2025

View reviewed changes

Comment thread core-api/src/worker.rs

Formatting, fix skip_client_worker_set_check

8f42387

cursor Bot reviewed Oct 14, 2025

View reviewed changes

Comment thread core/src/worker/mod.rs

yuandrew added 2 commits October 14, 2025 15:44

Cursor found a bug

d54a3eb

Lower sleep time, add print for debugging

13d981c

cursor Bot reviewed Oct 14, 2025

View reviewed changes

Comment thread core/src/worker/mod.rs

yuandrew force-pushed the worker-heartbeat-telemetry branch from 398b5ce to 33fd78d Compare October 14, 2025 23:24

cursor Bot reviewed Oct 14, 2025

View reviewed changes

Comment thread core/src/worker/client.rs Outdated

more prints

f1a3634

yuandrew force-pushed the worker-heartbeat-telemetry branch from 33fd78d to f1a3634 Compare October 14, 2025 23:53

cursor Bot reviewed Oct 14, 2025

View reviewed changes

Comment thread core/src/worker/client.rs Outdated

use semaphores for worker_heartbeat_failure_metrics

6d3a909

cursor Bot reviewed Oct 15, 2025

View reviewed changes

Comment thread core/src/worker/client.rs Outdated

skip_client_worker_set_check for all integ workers

a29d2df

cursor Bot reviewed Oct 15, 2025

View reviewed changes

Comment thread core/src/worker/client.rs Outdated

Can't use tokio semaphore in workflow code

fc7f839

Sushisource reviewed Oct 16, 2025

View reviewed changes

Comment thread tests/integ_tests/worker_heartbeat_tests.rs Outdated

Sushisource reviewed Oct 16, 2025

View reviewed changes

yuandrew added 4 commits October 16, 2025 12:05

use signal to test workflow_slots.last_interval_failure_tasks

c73d7b8

Use Notify instead of semaphores, fix test flake

f52345e

Use eventually() instead of a manual sleep

1601f0e

max_outstanding_workflow_tasks 2

8f1ef5a

Sushisource approved these changes Oct 17, 2025

View reviewed changes

yuandrew merged commit 4bc7383 into temporalio:worker-heartbeat Oct 17, 2025
17 checks passed

yuandrew mentioned this pull request Oct 17, 2025

Worker Heartbeat: Plumb metrics and migrate to runtime/namespace/client level #1038

Merged

Conversation

yuandrew commented Sep 29, 2025 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was changed

Why?

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Sushisource Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

yuandrew Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Sushisource left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Sushisource Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

Sushisource left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yuandrew commented Sep 29, 2025 •

edited by cursor Bot

Loading