[Feature][History Server] Support "running" clusters by restoring state from storage

[win5923]

Search before asking

• I had searched in the issues and found no similar feature requirement.

KubeRay Component

historyserver

Description

Follow-up to #5199, based on community feedback in the Ray Slack.

Today the history server has two modes:

RayCluster CR	Behavior
Deleted	Replay the session from object storage
Running	Reverse-proxy to the live head's Ray Dashboard

SessionProcessor.isDead() treats "CR exists" as live, so ProcessSession() short-circuits with SessionStatusLive and never reads storage.

kuberay/historyserver/pkg/historyserver/session_processor.go

Lines 81 to 97 in ffec815

	// isDead determines if the RayCluster CR is absent.
	//
	// Known limit: An old session of a still-running RayCluster will be misclassified as live.
	func (p *SessionProcessor) isDead(ctx context.Context, session utils.ClusterInfo) (bool, error) {
	rc := &rayv1.RayCluster{}
	err := p.k8sClient.Get(ctx, k8stypes.NamespacedName{
	Namespace: session.Namespace,
	Name: session.Name,
	}, rc)
	if apierrors.IsNotFound(err) {
	return true, nil
	}
	if err != nil {
	return false, err
	}
	return false, nil
	}

We should add a "running" mode that restores state from storage even while the cluster is still running:

• "running" is enabled by default, "live" (proxy) is opt-in via --enable-live-clusters
• The snapshot refresh interval is configurable

Use case

Long-running RayClusters risk head node OOM. Users want to offload observability data to storage and read it from the history server, so the head node can keep far less. That only works if a running cluster can be served from storage instead of proxied to its head.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[wuallen57730]

Hi @win5923, could I take this?

[win5923]

This need further discussions.

1. Periodic log upload
   Active-session logs are currently uploaded only when the collector stops. Supporting periodic uploads requires decisions on whole-file
   replacement versus chunked uploads, task-log offset compatibility, upload cost, and whether the feature should be enabled by default.
   Related to [Feature] Collect rotated logs #4830.

2. Caching and incremental loading
   Cached snapshots currently assume completed, immutable sessions. Running sessions require a refresh model: periodically rebuild the full snapshot or incrementally process newly uploaded files.

3. Session liveness detection
   isDead currently treats a session as live whenever its RayCluster CR still exists, which is incorrect for older sessions from the same cluster. Lifecycle status should be tracked per session, with a clear source of truth—preferably storage metadata or collector heartbeats, with the Kubernetes API used only as optional context.

4. How we surface data freshness
   Each node uploads on its own schedule, so the view is always a slightly inconsistent stitch. That can't be eliminated.

5. Observability Metrics

I think we can address this issue once periodic log upload is supported. cc @chiayi, WDYT?