Skip to content

feat(worker): expose task completion rate in worker heartbeat#16865

Merged
fhussonnois merged 1 commit into
developfrom
feat/worker-group-throughput
Jun 17, 2026
Merged

feat(worker): expose task completion rate in worker heartbeat#16865
fhussonnois merged 1 commit into
developfrom
feat/worker-group-throughput

Conversation

@fhussonnois

Copy link
Copy Markdown
Member

Add a per-Worker RateMeter that derives the task completion rate (tasks/sec) from the existing worker.ended.count counter — EWMA-smoothed (ALPHA=0.3), sampled on each heartbeat and emitted as the worker.tasks.rate metric in the heartbeat set. No new Micrometer meter is registered; the rate is an operational figure for the Worker Group UI only.

Add a per-Worker RateMeter that derives the task completion rate (tasks/sec)
from the existing worker.ended.count counter — EWMA-smoothed (ALPHA=0.3),
sampled on each heartbeat and emitted as the worker.tasks.rate metric in the
heartbeat set. No new Micrometer meter is registered; the rate is an
operational figure for the Worker Group UI only.
@github-actions

github-actions Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

📄 OpenAPI Spec Changes

Spec generated with EE branch feat/worker-group-throughput. Diff vs client-sdk:

24572a24573,24575
>         endedRate:
>           type: number
>           format: double
24621a24625,24627
>         totalEndedRate:
>           type: number
>           format: double

🐋 Docker image

ghcr.io/kestra-io/kestra-pr:16865
docker run --pull=always --rm -it -p 8080:8080 --user=root -v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/tmp ghcr.io/kestra-io/kestra-pr:16865 server local

🧪 Java Unit Tests

TestsPassed ☑️Skipped ⚠️Failed ❌️Time ⏱
Java Tests Report6945 ran6924 ✅20 ⚠️1 ❌52m 38s 962ms
TestResultTime ⏱
Java Tests Report
GrpcWorkerIOSenderTest.shouldRequeueAndRedeliverWhenSendFailsWithRetryableError()❌ failure6s 192ms

@github-actions

github-actions Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Tests report quick summary:

failed ❌ > tests: 6945, success: 6924, skipped: 20, failed: 1 (🔄 6945 executed, 📦 0 from cache)

Project Status Success Skipped Failed
cli success ✅ 🔄 55 1 0
core success ✅ 🔄 2686 6 0
executor success ✅ 🔄 70 3 0
jdbc success ✅ 🔄 30 6 0
jdbc-h2 success ✅ 🔄 912 1 0
jdbc-mysql success ✅ 🔄 903 2 0
jdbc-postgres success ✅ 🔄 917 0 0
processor success ✅ 🔄 7 0 0
queue success ✅ 🔄 39 0 0
runner-memory success ✅ 🔄 1 0 0
scheduler success ✅ 🔄 99 0 0
script success ✅ 🔄 37 0 0
storage-local success ✅ 🔄 66 0 0
tests success ✅ 🔄 2 0 0
webserver success ✅ 🔄 901 1 0
worker failed ❌ 🔄 64 0 1
worker-controller success ✅ 🔄 135 0 0

Failed tests:

worker > io.kestra.worker.senders.GrpcWorkerIOSenderTest > shouldRequeueAndRedeliverWhenSendFailsWithRetryableError() failed ❌ in 6.192
org.mockito.exceptions.verification.TooFewActualInvocations: &#10;grpcWorkerControllerService.sendWorkerTaskResults(&#10;    <Capturing argument: OpaqueData>,&#10;    <any>&#10;);&#10;Wanted *at least* 2 times:&#10;-> at io.kestra.controller.grpc.services.GrpcWorkerControllerService.sendWorkerTaskResults(GrpcWorkerControllerService.java:193)&#10;But was 1 time:&#10;-> at io.kestra.worker.senders.$GrpcWorkerIOSenderTest$GrpcWorkerControllerService0$Definition$Exec.dispatch(Unknown Source)&#10;

org.mockito.exceptions.verification.TooFewActualInvocations: 
grpcWorkerControllerService.sendWorkerTaskResults(
    <Capturing argument: OpaqueData>,
    <any>
);
Wanted *at least* 2 times:
-> at io.kestra.controller.grpc.services.GrpcWorkerControllerService.sendWorkerTaskResults(GrpcWorkerControllerService.java:193)
But was 1 time:
-> at io.kestra.worker.senders.$GrpcWorkerIOSenderTest$GrpcWorkerControllerService0$Definition$Exec.dispatch(Unknown Source)

	at io.kestra.controller.grpc.services.GrpcWorkerControllerService.sendWorkerTaskResults(GrpcWorkerControllerService.java:193)
	at io.kestra.worker.senders.GrpcWorkerIOSenderTest.lambda$shouldRequeueAndRedeliverWhenSendFailsWithRetryableError$1(GrpcWorkerIOSenderTest.java:192)
	at org.awaitility.core.AssertionCondition.lambda$new$0(AssertionCondition.java:53)
	at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:248)
	at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:235)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
	at java.base/java.lang.Thread.run(Thread.java:1474)


Develocity build scan: https://develocity.kestra.io/s/pwwepivrfmvde


Flaky tests report quick summary:

success ✅ > tests: 13, success: 13, skipped: 0, failed: 0

unfold for details
Project Status Success Skipped Failed
cli success ✅ 2 0 0
jdbc-h2 success ✅ 1 0 0
jdbc-mysql success ✅ 2 0 0
jdbc-postgres success ✅ 1 0 0
script success ✅ 2 0 0
webserver success ✅ 5 0 0

@ammeek ammeek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@fhussonnois fhussonnois merged commit 4c85dcf into develop Jun 17, 2026
9 of 12 checks passed
@github-project-automation github-project-automation Bot moved this from To review to Done in Pull Requests Jun 17, 2026
@fhussonnois fhussonnois deleted the feat/worker-group-throughput branch June 17, 2026 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants