[SLI Metrics] kuberay_job_execution_duration_seconds by troychiu · Pull Request #3488 · ray-project/kuberay

troychiu · 2025-04-25T07:09:24Z

Why are these changes needed?

refactor rayJobMetricsCollector, which allows
- easier unit testing
Implement metric kuberay_job_execution_duration_seconds proposed in 1.4.0.

Metric Name	Type	Description	Labels
`kuberay_job_execution_duration_seconds`	Gauge	Duration from when the RayJob CR’s JobDeploymentStatus transitions from Initializing to either the Retrying state or a terminal state, such as Complete or Failed. The Retrying state indicates that the CR previously failed and that spec.backoffLimit is enabled.	`name` `namespace` `job_deployment_status` (Complete/Failed/Retrying) `retry_count`

End-to-end test

Create ray jobs and check actual metrics. Below is an example result

# HELP kuberay_job_execution_duration_seconds Duration from when the RayJob CR’s JobDeploymentStatus transitions from Initializing to either the Retrying state or a terminal state, such as Complete or Failed. The Retrying state indicates that the CR previously failed and that spec.backoffLimit is enabled.
# TYPE kuberay_job_execution_duration_seconds gauge
kuberay_job_execution_duration_seconds{job_deployment_status="Complete",name="rayjob-fail-sample",namespace="test",retry_count="1"} 35.219411124
kuberay_job_execution_duration_seconds{job_deployment_status="Retrying",name="rayjob-fail-sample",namespace="test",retry_count="0"} 36.091843603

Related issue number

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

…#3386)

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

win5923 · 2025-04-26T03:16:28Z

ray-operator/controllers/ray/metrics/ray_job_metrics.go

+		rayJobExecutionDurationSeconds: prometheus.NewGaugeVec(
+			prometheus.GaugeOpts{
+				Name: "kuberay_job_execution_duration_seconds",
+				Help: "Duration from RayJob CR initialization to reaching a terminal state.",


I think the description should also include retrying state

Sure! Updated. Thank you

win5923

LGTM, thanks!

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

troychiu · 2025-04-26T19:42:38Z

@kevin85421 PTAL. Thank you.

ray-operator/controllers/ray/metrics/ray_job_metrics.go

ray-operator/controllers/ray/rayjob_controller.go

kevin85421 · 2025-04-28T18:26:28Z

ray-operator/controllers/ray/metrics/ray_job_metrics.go

+				Name: "kuberay_job_execution_duration_seconds",
+				Help: "Duration from RayJob CR initialization to reaching a terminal state or retrying state, where retrying state indicates the CR was previously failed and backoff is enabled.",
+			},
+			[]string{"name", "namespace", "result", "retry_count"},


is there a better name for "result"?

change to job_deployment_result. Wdyt?

ray-operator/apis/ray/v1/rayjob_types.go

ray-operator/controllers/ray/rayjob_controller.go

ray-operator/controllers/ray/metrics/ray_job_metrics.go

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

ray-operator/controllers/ray/metrics/ray_job_metrics.go

ray-operator/controllers/ray/rayjob_controller.go

ray-operator/controllers/ray/metrics/ray_job_metrics.go

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

kevin85421

left nit comments. Others LGTM

kevin85421 · 2025-04-30T02:23:53Z

ray-operator/main.go


 	rayJobOptions := ray.RayJobReconcilerOptions{
-		RayJobMetricsCollector: rayJobMetricsCollector,
+		RayJobMetricsObserver: rayJobMetricsManager,


Suggested change

RayJobMetricsObserver: rayJobMetricsManager,

RayJobMetricsManager: rayJobMetricsManager,

I did this intentionally since ray job controller only call methods in RayJobMetricsObserver. Do you think using RayJobMetricsManager would be better?

I’d prefer to use RayJobMetricsManager: rayJobMetricsManager. The inconsistency between RayJobMetricsObserver and rayJobMetricsManager looks odd and may confuse future readers.

That makes sense. Fixed. Thank you

I only keep emitRayJobExecutionDuration to accept observer so that it can be unit tested

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

troychiu and others added 9 commits April 20, 2025 22:03

refactor ray job metrics

6696724

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

merge main

37127a7

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

[TEST] Improve test coverage for apiserver pkg/manager (ray-project…

3b228ef

…#3386)

[3/N] [Lint] Group imports by sections (ray-project#3430)

2e6aeab

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

job execution duration

fa117cc

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

merge main

c72c9e9

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

refactor and add unit test

a1ed979

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

refine description

f34a3fb

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

refine comment

68fc3ba

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

troychiu changed the title ~~Kuberay job execution duration seconds~~ [SLI Metrics] kuberay_job_execution_duration_seconds Apr 25, 2025

fix test

92e177e

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

win5923 reviewed Apr 26, 2025

View reviewed changes

This comment was marked as duplicate.

Sign in to view

win5923 approved these changes Apr 26, 2025

View reviewed changes

refine metric description

7ed3761

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

win5923 mentioned this pull request Apr 27, 2025

[Prometheus] Refactor kuberay_cluster_provisioned_duration_seconds #3497

Merged

4 tasks

kevin85421 reviewed Apr 28, 2025

View reviewed changes

troychiu added 2 commits April 28, 2025 17:39

code review

6005a72

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

nit

f10ad94

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

kevin85421 reviewed Apr 29, 2025

View reviewed changes

ray-operator/controllers/ray/metrics/ray_job_metrics.go Outdated Show resolved Hide resolved

ray-operator/controllers/ray/metrics/ray_job_metrics.go Outdated Show resolved Hide resolved

code review

6012415

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

kevin85421 reviewed Apr 30, 2025

View reviewed changes

troychiu added 2 commits April 30, 2025 00:24

code review

26dcf56

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

fix test

5b16829

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

kevin85421 approved these changes Apr 30, 2025

View reviewed changes

kevin85421 merged commit 62302d8 into ray-project:master Apr 30, 2025
23 checks passed

laurafitzgerald pushed a commit to laurafitzgerald/kuberay that referenced this pull request May 7, 2025

[SLI Metrics] kuberay_job_execution_duration_seconds (ray-project#3488)

1818205

laurafitzgerald pushed a commit to laurafitzgerald/kuberay that referenced this pull request May 8, 2025

[SLI Metrics] kuberay_job_execution_duration_seconds (ray-project#3488)

78c7756

win5923 mentioned this pull request May 22, 2025

[Epic][Feature] KubeRay v1.4.0 - Operator SLI Tracking #3171

Closed

9 tasks

kevin85421 mentioned this pull request May 24, 2025

[Feature][metrics] ray_job_execution_duration_seconds #3181

Closed

1 task

	RayJobMetricsObserver: rayJobMetricsManager,
	RayJobMetricsManager: rayJobMetricsManager,

Conversation

troychiu commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

End-to-end test

Related issue number

Checks

Uh oh!

win5923 Apr 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as duplicate.

Uh oh!

win5923 left a comment

Choose a reason for hiding this comment

Uh oh!

troychiu commented Apr 26, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevin85421 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

troychiu commented Apr 25, 2025 •

edited

Loading

win5923 Apr 26, 2025 •

edited

Loading