[Feat] [history server] Add node endpoint by JiangJiaWei1103 · Pull Request #4436 · ray-project/kuberay

JiangJiaWei1103 · 2026-01-24T06:19:21Z

Why are these changes needed?

For full context, please refer to this design_doc.

This PR adds support for the /nodes and /nodes/{node_id} endpoints to the history server.

With these endpoints, the history server can reconstruct node state for terminated (dead) clusters, enabling post-mortem analysis through historical event replay. This includes node summaries and snapshots of node-level resources at different time points.

Change Summary

At a high level, this PR introduces changes across two main layers:

History Server Layer

Implement the core logic for the /nodes?view=summary and /nodes/{node_id} endpoints
- Retrieve the target node map from the event server layer
- Construct and format node summaries and node-level resource snapshots for API responses

Event Server Layer

Define node-related data structures and public interfaces
Introduce a reusable state transition interface for merging node state transitions
Process NODE_DEFINITION_EVENT and NODE_LIFECYCLE_EVENT
Build and maintain a ClusterNodeMap, and expose a GetNodeMap helper for consumption by the history server layer

Test Result

Related issue number

Closes #4376.

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

historyserver/pkg/historyserver/router.go

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

JiangJiaWei1103 · 2026-01-25T15:17:07Z

historyserver/pkg/eventserver/state_transition.go

@@ -0,0 +1,52 @@
+package eventserver


This file defines the interface and processing helpers (e.g., MergeStateTransitions) that can be reused across different events' state transitions.

JiangJiaWei1103 · 2026-01-25T15:18:12Z

historyserver/test/support/raycluster.go

+// injectCollectorRayClusterID injects the ray-cluster-id argument into all collector containers.
+func injectCollectorRayClusterID(containers []corev1.Container, rayClusterID string) {
+	for i := range containers {
+		if containers[i].Name == "collector" {
+			containers[i].Command = append(
+				containers[i].Command,
+				fmt.Sprintf("--ray-cluster-id=%s", rayClusterID),
+			)
+		}
+	}
+}
+


This patch is proposed by @machichima in this PR.

JiangJiaWei1103 · 2026-01-25T15:19:03Z

historyserver/test/e2e/historyserver_test.go

 			body, err := io.ReadAll(resp.Body)
 			gg.Expect(err).NotTo(HaveOccurred())
-			gg.Expect(resp.StatusCode).To(Equal(200),
+			gg.Expect(resp.StatusCode).To(Equal(http.StatusOK),


Use HTTP status code to preserve semantics without hardcoding the code number.

JiangJiaWei1103 · 2026-01-25T15:23:13Z

historyserver/test/e2e/historyserver_test.go

+// isLive indicates whether the response is from a live cluster or a dead cluster:
+//   - isLive: true for a live cluster (current snapshot)
+//   - isLive: false for a dead cluster (historical replay)
+func verifyNodesRespSchema(test Test, g *WithT, nodesResp map[string]any, isLive bool) {


Considering that different endpoints will have exclusive schema verification, we should extract them to separate files to avoid messing up the main test file.

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

JiangJiaWei1103 · 2026-01-26T03:02:57Z

historyserver/pkg/utils/utils.go

+func ConvertBase64ToHex(input string) (string, error) {
+	bytes, err := base64.StdEncoding.DecodeString(input)
+	if err != nil {
+		return "", err
+	}
+
+	hexStr := hex.EncodeToString(bytes)
+
+	return hexStr, nil
+}


This conversion is implemented by @chiayi in this PR.

This reverts commit 8d7ff9a.

historyserver/pkg/eventserver/eventserver.go

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

historyserver/pkg/eventserver/types/node.go

historyserver/pkg/historyserver/router.go

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

historyserver/pkg/historyserver/router.go

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

historyserver/.env

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

…ay-project#4464) Signed-off-by: Future-Outlier <eric901201@gmail.com>

…e-endpoint

historyserver/pkg/eventserver/eventserver.go

Signed-off-by: Future-Outlier <eric901201@gmail.com>

historyserver/test/e2e/historyserver_test.go

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

Signed-off-by: Future-Outlier <eric901201@gmail.com>

historyserver/test/e2e/historyserver_test.go

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

historyserver/pkg/historyserver/router.go

historyserver/test/e2e/historyserver_test.go

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

historyserver/pkg/historyserver/router.go

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

historyserver/pkg/historyserver/router.go

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

historyserver/pkg/historyserver/router.go

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

JiangJiaWei1103 added 7 commits January 24, 2026 14:15

feat: Add node endpoint /nodes POC

8b84e0e

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

docs: Clarify router nodes

fd5e8ff

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

feat: Support dashboard API-compatible response

7dc3f85

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

refactor: Clean event processing logic

d093b88

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

refactor: Extract lifecycle dedup and sorting helper

25f87bf

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

feat: Support more node fields

7f5a5b6

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

Support local dev outside cluster (should be reverted)

8d7ff9a

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

JiangJiaWei1103 commented Jan 25, 2026

View reviewed changes

historyserver/pkg/historyserver/router.go Outdated Show resolved Hide resolved

test: Add e2e tests for /nodes endpoint for both live and dead

0a1a558

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

JiangJiaWei1103 moved this to In review in My Kuberay & Ray Jan 25, 2026

JiangJiaWei1103 added this to My Kuberay & Ray Jan 25, 2026

JiangJiaWei1103 commented Jan 25, 2026

View reviewed changes

JiangJiaWei1103 added 4 commits January 26, 2026 09:22

feat: Support base64 to hex conversion

73809f4

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

feat: Support /nodes/node_id endpoint with shared formatters

46bee4e

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

docs: Remove redundant comments

bbbccd8

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

test: Add e2e tests for /nodes/node_id endpoint for both live and dead

eb542fb

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

JiangJiaWei1103 commented Jan 26, 2026

View reviewed changes

JiangJiaWei1103 marked this pull request as ready for review January 26, 2026 03:51

Revert "Support local dev outside cluster (should be reverted)"

147da88

This reverts commit 8d7ff9a.

cursor bot reviewed Jan 26, 2026

View reviewed changes

historyserver/pkg/eventserver/eventserver.go Outdated Show resolved Hide resolved

historyserver/pkg/eventserver/eventserver.go Outdated Show resolved Hide resolved

fix: Use test namespace instead of default

492a011

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

cursor bot reviewed Jan 26, 2026

View reviewed changes

historyserver/pkg/eventserver/types/node.go Show resolved Hide resolved

historyserver/pkg/historyserver/router.go Show resolved Hide resolved

JiangJiaWei1103 added 2 commits January 26, 2026 14:58

fix: Check empty node ID before conversion

06240b6

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

fix: Fix deepcopy to prevent race conditions

ae4a75a

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

cursor bot reviewed Jan 26, 2026

View reviewed changes

historyserver/pkg/historyserver/router.go Outdated Show resolved Hide resolved

JiangJiaWei1103 added 5 commits January 27, 2026 08:33

Merge branch 'my-master' into epic-4374/add-node-endpoint

f9a39a2

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

Support local dev outside cluster (should be reverted)

2589818

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

feat: Add more fields

05926f1

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

test: Add new test fields

450c50d

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

feat: Add storeStats

6d2a660

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

JiangJiaWei1103 added 2 commits January 30, 2026 09:08

Merge branch 'my-master' into epic-4374/add-node-endpoint

6d2dc71

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

fix: Fix merging def conflicts

721a4d7

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

cursor bot reviewed Jan 30, 2026

View reviewed changes

historyserver/.env Outdated Show resolved Hide resolved

JiangJiaWei1103 and others added 5 commits January 30, 2026 09:26

refactor: Add a helper to get node by node id

f8574cd

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

Remove .env

00fa74e

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

fix: Construct resource string only for alive

325beea

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

[history server] use sessionName in task, actor, and job endpoints (r…

bb7b067

…ay-project#4464) Signed-off-by: Future-Outlier <eric901201@gmail.com>

Merge remote-tracking branch 'upstream/master' into epic-4374/add-nod…

72101a2

…e-endpoint

cursor bot reviewed Jan 30, 2026

View reviewed changes

historyserver/pkg/eventserver/eventserver.go Outdated Show resolved Hide resolved

fix merge bug

5e2afa7

Signed-off-by: Future-Outlier <eric901201@gmail.com>

cursor bot reviewed Jan 30, 2026

View reviewed changes

historyserver/test/e2e/historyserver_test.go Outdated Show resolved Hide resolved

Future-Outlier and others added 3 commits January 30, 2026 23:15

fix node endpoints

63e4be6

Signed-off-by: Future-Outlier <eric901201@gmail.com>

fix: Fix get one node summary API schema

ce06cdf

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

update

9f1cb93

Signed-off-by: Future-Outlier <eric901201@gmail.com>

cursor bot reviewed Jan 30, 2026

View reviewed changes

historyserver/test/e2e/historyserver_test.go Outdated Show resolved Hide resolved

Future-Outlier and others added 2 commits January 30, 2026 23:43

fix TODO

83b474a

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Merge branch 'my-master' into epic-4374/add-node-endpoint

4ba9f9e

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

cursor bot reviewed Feb 3, 2026

View reviewed changes

historyserver/pkg/historyserver/router.go Outdated Show resolved Hide resolved

historyserver/test/e2e/historyserver_test.go Outdated Show resolved Hide resolved

JiangJiaWei1103 mentioned this pull request Feb 3, 2026

[Test][HistoryServer] E2E test for dead cluster actor endpoint #4461

Open

4 tasks

Copilot AI mentioned this pull request Feb 5, 2026

Review all open pull requests #4482

Closed

4 tasks

Merge branch 'my-master' into epic-4374/add-node-endpoint

3788bc8

cursor bot reviewed Feb 5, 2026

View reviewed changes

historyserver/pkg/historyserver/router.go Show resolved Hide resolved

JiangJiaWei1103 added 4 commits February 5, 2026 21:43

fix: Check rsc str for non-empty string

c9a4050

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

fix: Avoid missing key

79684af

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

fix: Sort resource keys first

7fcd1db

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

test: Verify hostnamelist view and clean code

9210d0d

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

cursor bot reviewed Feb 5, 2026

View reviewed changes

historyserver/pkg/historyserver/router.go Outdated Show resolved Hide resolved

refactor: Use existing helper to get cluster session key

0847170

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

cursor bot reviewed Feb 6, 2026

View reviewed changes

historyserver/pkg/historyserver/router.go Outdated Show resolved Hide resolved

refactor: Use existing helper to get cluster session key

21b1d58

Signed-off-by: JiangJiaWei1103 <waynechuang97@gmail.com>

Conversation

JiangJiaWei1103 commented Jan 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Change Summary

History Server Layer

Event Server Layer

Test Result

Related issue number

Checks

Uh oh!

Uh oh!

JiangJiaWei1103 Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

JiangJiaWei1103 Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

JiangJiaWei1103 Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

JiangJiaWei1103 Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

JiangJiaWei1103 Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JiangJiaWei1103 commented Jan 24, 2026 •

edited

Loading