[RayService][e2e] fix flaky test TestAutoscalingRayService

[AndySung320]

Why are these changes needed?

There is a flaky test in Test E2E rayservice (TestAutoscalingRayService).
The dashboard's MetricsHead module times out during startup with 500m CPU limit, causing the dashboard to crash and RayService to never become ready.

This triggers a chain of failures:

1. Dashboard crashes -> port 8265 is not listening -> Serve API is unavailable
2. No Serve API -> serve applications cannot be deployed
3. No serve applications -> no resource demands are reported to the autoscaler
4. No resource demands -> autoscaler does not scale up workers
5. No workers -> RayService never becomes ready -> test times out

Thus, increase to 1 CPU.
Once we increase to 1 CPU, test can pass successfully.

note: I also had to pin gevent==24.11.1 locally to run the test on my ARM Mac, since gevent 26.4.0 doesn't provide a prebuilt wheel for linux/aarch64 and the Ray image lacks a C compiler to build from source. This change is local-only and not included in this PR.

Reproduction Step

First run the e2e test

 go test ./test/e2erayservice/  -run TestAutoscalingRayService -v -timeout 30m

Wait for 30 seconds and open another terminal

NS=test-ns-xxxx

kubectl exec -n $NS -c ray-head $(kubectl get pods -n $NS -l ray.io/node-type=head -o name) -- bash -c "cat /tmp/ray/session_latest/logs/dashboard.log 2>/dev/null | tail -20"

You will see RuntimeError: Module MetricsHead failed to start. Timeout after 30.0 seconds, confirming the dashboard process crashed due to insufficient CPU during startup.

kubectl exec -n $NS -c ray-head \
  $(kubectl get pods -n $NS -l ray.io/node-type=head -o name) \
  -- python -c "import socket; s=socket.socket(); s.settimeout(2); s.connect(('localhost',8265)); print('8265 OPEN'); s.close()"

You will see ConnectionRefusedError, confirming the Serve API is unreachable because the dashboard is dead.

kubectl exec -n $NS -c ray-head \
  $(kubectl get pods -n $NS -l ray.io/node-type=head -o name) \
  -- ray status

It shows Pending Demands: (no resource demands), confirming no serve application was deployed so the autoscaler has no reason to scale up.

kubectl get pods -n $NS

It shows only the head pod at 1/2 Running with no worker pods, confirming the autoscaler never triggered a scale-up

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[Future-Outlier]

how do you observe that "The dashboard's MetricsHead module times out during startup with 500m CPU limit, causing the dashboard to crash and RayService to never become ready."?

is there any proof?

[AndySung320]

Hi, I've updated in the reproduction step for more details.

[Future-Outlier]

reasons I approved:

1. our example in https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.sample.yaml#L34-L40 use 1 CPU for ray head container too
2. before after comparison

  Comparison

  ┌────────────────────┬───────────────────────┬─────────────────────────┐
  │                    │ Build #14465 (before) │  Build #14478 (after)   │
  ├────────────────────┼───────────────────────┼─────────────────────────┤
  │ Head CPU limit     │ 500m                  │ 1                       │
  ├────────────────────┼───────────────────────┼─────────────────────────┤
  │ Dashboard response │ ~34s + 3 timeouts     │ < 46s (entire pipeline) │
  ├────────────────────┼───────────────────────┼─────────────────────────┤
  │ Worker pod         │ Never created         │ Created successfully    │
  ├────────────────────┼───────────────────────┼─────────────────────────┤
  │ RayService Ready   │ ❌ Timed out at 300s  │ ✅ ~46s                 │
  └────────────────────┴───────────────────────┴─────────────────────────┘

[Future-Outlier]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Swish!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Future-Outlier]

cc @andrewsykim @rueian to merge

[andrewsykim]

Can we also consider removing limits entirely? Or does the autoscaling test require it?

[Future-Outlier]

cc @AndySung320 to do the investigation, and update the yaml manifest if needed