[Feature] Support per-pod FQDNs & inject to Ray node ip - #5191
Open
machichima wants to merge 7 commits into
Open
[Feature] Support per-pod FQDNs & inject to Ray node ip#5191machichima wants to merge 7 commits into
machichima wants to merge 7 commits into
Conversation
Collaborator
Author
Test
|
Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: machichima <nary12321@gmail.com>
machichima
force-pushed
the
fqdn-hostname
branch
from
August 23, 2026 11:18
dd7975c to
14d72b3
Compare
Contributor
|
Hey @machichima apologies for the delay in looking at this. This is neat but I think it still hits the same race when it comes to mTLS. It'll handle stable identity but I don't think it will stop Ray from resolving to a pod IP on the Python side. |
Signed-off-by: machichima <nary12321@gmail.com>
machichima
commented
Sep 3, 2026
| podTemplate := workerSpec.Template | ||
| // If the replica of workers is more than 1, `ObjectMeta.Name` may cause name conflict errors. | ||
| // Hence, we set `ObjectMeta.Name` to an empty string, and use GenerateName to prevent name conflicts. | ||
| podTemplate.ObjectMeta.Name = "" |
Collaborator
Author
There was a problem hiding this comment.
Just move it closer to podTemplate.GenerateName for readability. No behavior change
Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: machichima <nary12321@gmail.com>
Collaborator
Author
Yes, I looked into it, I think there are still two places that would need change:
Did you see more places that would still need changes? I'll first change the two I listed above and test the mTLS behavior first |
machichima
force-pushed
the
fqdn-hostname
branch
from
September 3, 2026 13:33
59df251 to
57575db
Compare
Collaborator
Author
Test with mTLSWithout Ray code changeRay autoscaler will use node ip to do health check (here), this requires Ray code change. Tests in this section are done without Ray code change, which head certificate still need it's IP in the certificate hence need the Basic - without Ray change - check certificate set correctlyMainly check if we include worker wildcard in the certificate rather than pod IPs, and check:
❯ export CLUSTER=raycluster-mtls
export HEAD=$(kubectl get pod -l ray.io/cluster=$CLUSTER,ray.io/node-type=head -o jsonpath='{.items[0].metadata.name}')
export HEADSVC=$CLUSTER-head-svc.default.svc.cluster.local
export WORKER=$(kubectl get pod -l ray.io/cluster=$CLUSTER,ray.io/node-type=worker -o jsonpath='{.items[0].metadata.name}')
# Run a new pod (without certificate), cannot connect to GCS -> ensure mTLS is working
❯ kubectl run tls-probe --rm -it --restart=Never --image=rayproject/ray:2.55.1 -- \
ray health-check --address $HEADSVC:6379 --skip-version-check
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
[2026-09-03 04:58:01,506 W 1 1] rpc_client.h:153: Failed to connect to GCS at address raycluster-mtls-head-svc.default.svc.cluster.local:6379 within 5 seconds.
[2026-09-03 04:58:31,568 W 1 1] gcs_client.cc:205: Failed to get cluster ID from GCS server: TimedOut: Timed out while waiting for GCS to become available.
Aborted!
pod "tls-probe" deleted from default namespace
pod default/tls-probe terminated (Error)
# Call from head pod
❯ kubectl exec $HEAD -c ray-head -- ray health-check --address $HEADSVC:6379 --skip-version-check && echo OK
OK
# Call from worker pod
❯ kubectl exec $WORKER -c ray-worker -- ray health-check --address $HEADSVC:6379 --skip-version-check && echo OK
# Get DNS and IP set in the certificate
❯ kubectl get certificate -o custom-columns='NAME:.metadata.name,DNS:.spec.dnsNames,IP:.spec.ipAddresses'
NAME DNS IP
ray-ca-certificate-raycluster-mtls <none> <none>
ray-head-cert-raycluster-mtls [localhost raycluster-mtls-head-svc.default.svc.cluster.local] [10.244.0.11 127.0.0.1]
ray-worker-cert-raycluster-mtls [*.raycluster-mtls-headless.default.svc.cluster.local localhost] [127.0.0.1]
# Get secret of worker
❯ kubectl get secret ray-worker-secret-$CLUSTER -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -ext subjectAltName
X509v3 Subject Alternative Name: critical
DNS:*.raycluster-mtls-headless.default.svc.cluster.local, DNS:localhost, IP Address:127.0.0.1
# Run ray.init in head pod and do get_node_ip_address() -> we will get the fqdn of the head pod
❯ kubectl exec $HEAD -c ray-head -- python -c "import ray; ray.init(); print('gcs:', ray.get_runtime_context().gcs_address); print('node_ip:', ray.util.get_node_ip_address())"
2026-09-03 04:59:13,204 INFO worker.py:1672 -- Using address raycluster-mtls-head-svc.default.svc.cluster.local:6379 set in the environment variable RAY_ADDRESS
2026-09-03 04:59:13,210 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: raycluster-mtls-head-svc.default.svc.cluster.local:6379...
2026-09-03 04:59:13,221 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at raycluster-mtls-head-svc.default.svc.cluster.local:8265
/home/ray/anaconda3/lib/python3.10/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
gcs: raycluster-mtls-head-svc.default.svc.cluster.local:6379
node_ip: raycluster-mtls-head-svc.default.svc.cluster.local
# See containers in the pod, head still have wait-for-tls-ip-san init container
❯ kubectl get pod -l ray.io/cluster=raycluster-mtls \
-o custom-columns='POD:.metadata.name,INIT:.spec.initContainers[*].name,CONTAINERS:.spec.containers[*].name'
POD INIT CONTAINERS
raycluster-mtls-head-zsmgx wait-for-tls-ip-san ray-head
raycluster-mtls-small-group-worker-hsslt wait-gcs-ready ray-worker
Trigger scaling manually
❯ k apply -f ray-operator/config/samples/ray-cluster.mtls.yaml
raycluster.ray.io/raycluster-mtls created
# Trigger scale up
❯ kubectl patch raycluster $CLUSTER --type json -p '[{"op":"replace","path":"/spec/workerGroupSpecs/0/replicas","value":2},{"op":"replace","path":"/spec/workerGroupSpecs/0/maxReplicas","value":2}]'
raycluster.ray.io/raycluster-mtls patched
❯ kubectl get pod -l ray.io/cluster=$CLUSTER,ray.io/node-type=worker
NAME READY STATUS RESTARTS AGE
raycluster-mtls-small-group-worker-6wjcc 1/1 Running 0 54s
raycluster-mtls-small-group-worker-hsslt 1/1 Running 0 27s
# check pod name = hostname
❯ kubectl get pod -l ray.io/cluster=$CLUSTER,ray.io/node-type=worker -o custom-columns='NAME:.metadata.name,HOSTNAME:.spec.hostname,SUBDOMAIN:.spec.subdomain'
NAME HOSTNAME SUBDOMAIN
raycluster-mtls-small-group-worker-6wjcc raycluster-mtls-small-group-worker-6wjcc raycluster-mtls-headless
raycluster-mtls-small-group-worker-hsslt raycluster-mtls-small-group-worker-hsslt raycluster-mtls-headless
# List nodes, we can see two worker nodes
❯ kubectl exec $HEAD -c ray-head -- ray list nodes --format json | jq -r '.[] | [.node_ip, .state, (if .is_head_node then "head" else "worker" end)] | @tsv'
raycluster-mtls-small-group-worker-6wjcc.raycluster-mtls-headless.default.svc.cluster.local ALIVE worker
raycluster-mtls-head-svc.default.svc.cluster.local ALIVE head
raycluster-mtls-small-group-worker-hsslt.raycluster-mtls-headless.default.svc.cluster.local ALIVE worker
# Trigger scale down
❯ kubectl patch raycluster $CLUSTER --type json -p '[{"op":"replace","path":"/spec/workerGroupSpecs/0/replicas","value":1}]'
raycluster.ray.io/raycluster-mtls patched
# One worker pod being removed
❯ kubectl get pod -l ray.io/cluster=$CLUSTER
NAME READY STATUS RESTARTS AGE
raycluster-mtls-head-zsmgx 1/1 Running 0 3m14s
raycluster-mtls-small-group-worker-hsslt 1/1 Running 0 2m47s
# One worker becomes dead
❯ kubectl exec $HEAD -c ray-head -- ray list nodes --format json | jq -r '.[] | [.node_ip, .state, (if .is_head_node then "head" else "worker" end)] | @tsv'
raycluster-mtls-small-group-worker-6wjcc.raycluster-mtls-headless.default.svc.cluster.local DEAD worker
raycluster-mtls-head-svc.default.svc.cluster.local ALIVE head
raycluster-mtls-small-group-worker-hsslt.raycluster-mtls-headless.default.svc.cluster.local ALIVE workerAutoscaling
❯ kubectl apply -f ray-operator/config/samples/ray-cluster.mtls.yaml
raycluster.ray.io/raycluster-mtls created
configmap/ray-example created
❯ kubectl get pods -l ray.io/cluster=raycluster-mtls
NAME READY STATUS RESTARTS AGE
raycluster-mtls-head-5v6js 2/2 Running 0 54s
❯ export CLUSTER=raycluster-mtls
export HEAD=$(kubectl get pod -l ray.io/cluster=$CLUSTER,ray.io/node-type=head -o jsonpath='{.items[0].metadata.name}')
❯ kubectl logs $HEAD -c autoscaler | grep -E "The Ray head is ready|Starting the autoscaler|error|TLS" | head
2026-09-03 05:37:57,356 INFO run_autoscaler.py:65 -- The Ray head is ready. Starting the autoscaler.
# Scale up
❯ kubectl exec $HEAD -c ray-head -- python /home/ray/samples/detached_actor.py a1
kubectl exec $HEAD -c ray-head -- python /home/ray/samples/detached_actor.py a2
2026-09-03 05:39:12,578 INFO worker.py:1672 -- Using address raycluster-mtls-head-svc.default.svc.cluster.local:6379 set in the environment variable RAY_ADDRESS
2026-09-03 05:39:12,584 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: raycluster-mtls-head-svc.default.svc.cluster.local:6379...
2026-09-03 05:39:12,593 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at raycluster-mtls-head-svc.default.svc.cluster.local:8265
/home/ray/anaconda3/lib/python3.10/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
2026-09-03 05:39:13,885 INFO worker.py:1672 -- Using address raycluster-mtls-head-svc.default.svc.cluster.local:6379 set in the environment variable RAY_ADDRESS
2026-09-03 05:39:13,890 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: raycluster-mtls-head-svc.default.svc.cluster.local:6379...
2026-09-03 05:39:13,898 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at raycluster-mtls-head-svc.default.svc.cluster.local:8265
/home/ray/anaconda3/lib/python3.10/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(autoscaler +0s) Tip: use `ray status` to view detailed cluster status. To disable these messages, set RAY_SCHEDULER_EVENTS=0.
(autoscaler +0s) Adding 1 node(s) of type small-group.
(autoscaler +0s) Resized to 1 CPUs.
# See new worker pods
❯ kubectl get pod -l ray.io/cluster=$CLUSTER,ray.io/node-type=worker
NAME READY STATUS RESTARTS AGE
raycluster-mtls-small-group-worker-c7bxl 0/1 Init:0/1 0 6s
raycluster-mtls-small-group-worker-xrjwm 0/1 Running 0 11s
# Check their name = hostname
❯ kubectl get pod -l ray.io/cluster=$CLUSTER,ray.io/node-type=worker -o custom-columns='NAME:.metadata.name,HOSTNAME:.spec.hostname,SUBDOMAIN:.spec.subdomain'
NAME HOSTNAME SUBDOMAIN
raycluster-mtls-small-group-worker-c7bxl raycluster-mtls-small-group-worker-c7bxl raycluster-mtls-headless
raycluster-mtls-small-group-worker-xrjwm raycluster-mtls-small-group-worker-xrjwm raycluster-mtls-headless
# Check ray list node shows fqdn
❯ kubectl exec $HEAD -c ray-head -- ray list nodes --format json | jq -r '.[] | [.node_ip, .state] | @tsv'
raycluster-mtls-head-svc.default.svc.cluster.local ALIVE
raycluster-mtls-small-group-worker-c7bxl.raycluster-mtls-headless.default.svc.cluster.local ALIVE
raycluster-mtls-small-group-worker-xrjwm.raycluster-mtls-headless.default.svc.cluster.local ALIVE
❯ kubectl logs $HEAD -c autoscaler | grep "Submitting a scale request"
2026-09-03 05:39:14,442 - INFO - Submitting a scale request: KubeRayProvider.ScaleRequest(desired_num_workers=defaultdict(<class 'int'>, {'small-group': 1}), workers_to_delete=defaultdict(<class 'list'>, {}), worker_groups_without_pending_deletes=set(), worker_groups_with_pending_deletes=set())
2026-09-03 05:39:14,442 INFO cloud_provider.py:365 -- Submitting a scale request: KubeRayProvider.ScaleRequest(desired_num_workers=defaultdict(<class 'int'>, {'small-group': 1}), workers_to_delete=defaultdict(<class 'list'>, {}), worker_groups_without_pending_deletes=set(), worker_groups_with_pending_deletes=set())
2026-09-03 05:39:19,513 - INFO - Submitting a scale request: KubeRayProvider.ScaleRequest(desired_num_workers=defaultdict(<class 'int'>, {'small-group': 2}), workers_to_delete=defaultdict(<class 'list'>, {}), worker_groups_without_pending_deletes=set(), worker_groups_with_pending_deletes=set())
2026-09-03 05:39:19,513 INFO cloud_provider.py:365 -- Submitting a scale request: KubeRayProvider.ScaleRequest(desired_num_workers=defaultdict(<class 'int'>, {'small-group': 2}), workers_to_delete=defaultdict(<class 'list'>, {}), worker_groups_without_pending_deletes=set(), worker_groups_with_pending_deletes=set())
# Scale down
❯ kubectl exec $HEAD -c ray-head -- python /home/ray/samples/terminate_detached_actor.py a1
kubectl exec $HEAD -c ray-head -- python /home/ray/samples/terminate_detached_actor.py a2
2026-09-03 05:40:44,050 INFO worker.py:1672 -- Using address raycluster-mtls-head-svc.default.svc.cluster.local:6379 set in the environment variable RAY_ADDRESS
2026-09-03 05:40:44,056 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: raycluster-mtls-head-svc.default.svc.cluster.local:6379...
2026-09-03 05:40:44,064 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at raycluster-mtls-head-svc.default.svc.cluster.local:8265
/home/ray/anaconda3/lib/python3.10/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
2026-09-03 05:40:45,199 INFO worker.py:1672 -- Using address raycluster-mtls-head-svc.default.svc.cluster.local:6379 set in the environment variable RAY_ADDRESS
2026-09-03 05:40:45,205 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: raycluster-mtls-head-svc.default.svc.cluster.local:6379...
2026-09-03 05:40:45,213 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at raycluster-mtls-head-svc.default.svc.cluster.local:8265
/home/ray/anaconda3/lib/python3.10/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
# Ensure we scale down
❯ kubectl get pod -l ray.io/cluster=$CLUSTER
NAME READY STATUS RESTARTS AGE
raycluster-mtls-head-5v6js 2/2 Running 0 4m14s
❯ kubectl logs $HEAD -c autoscaler | grep "Submitting a scale request" | tail -1
2026-09-03 05:41:51,214 INFO cloud_provider.py:365 -- Submitting a scale request: KubeRayProvider.ScaleRequest(desired_num_workers=defaultdict(<class 'int'>, {'small-group': 0}), workers_to_delete=defaultdict(<class 'list'>, {'small-group': [CloudInstance(cloud_instance_id='raycluster-mtls-small-group-worker-xrjwm', node_type='small-group', node_kind=2, is_running=True, request_id=None), CloudInstance(cloud_instance_id='raycluster-mtls-small-group-worker-c7bxl', node_type='small-group', node_kind=2, is_running=True, request_id=None)]}), worker_groups_without_pending_deletes=set(), worker_groups_with_pending_deletes=set()) |
Signed-off-by: machichima <nary12321@gmail.com>
Collaborator
Author
Autoscaling v2 with Ray code change
Update Ray Image with my python change
git diff master -- python/ray > autoscaler.patch
# create docker file
cat > Dockerfile <<'EOF'
FROM rayproject/ray:nightly-py311-cpu
COPY autoscaler.patch /tmp/autoscaler.patch
RUN cd $(python -c "import ray,os; print(os.path.dirname(os.path.dirname(ray.__file__)))") \
&& patch -p2 < /tmp/autoscaler.patch
EOF
# Build and load to kind
docker build -t ray-autoscaler-hostname:dev .
kind load docker-image ray-autoscaler-hostname:dev --name <your-kind-cluster>Update YAMLUpdate autoscalerOptions:
version: v2
idleTimeoutSeconds: 60
# TODO(local test only, remove before commit): Ray image with `ray kuberay-autoscaler --gcs-address`.
image: ray-autoscaler-hostname:dev
imagePullPolicy: IfNotPresentResult❯ kubectl apply -f ray-operator/config/samples/ray-cluster.mtls.yaml
raycluster.ray.io/raycluster-mtls created
configmap/ray-example unchanged
❯ kubectl get pods -l ray.io/cluster=raycluster-mtls
NAME READY STATUS RESTARTS AGE
raycluster-mtls-head-5pl62 1/2 Running 0 8s
❯ export HEAD=$(kubectl get pod -l ray.io/cluster=raycluster-mtls,ray.io/node-type=head -o jsonpath='{.items[0].metadata.name}')
# ** Ensure there's no init container for head pod
❯ kubectl get pod $HEAD -o jsonpath='init={.spec.initContainers[*].name}{"\n"}'
init=
# Head pod certificate IP only include localhost
❯ kubectl get certificate ray-head-cert-raycluster-mtls -o jsonpath='{.spec.ipAddresses}{"\n"}'
["127.0.0.1"]
# Ful certificates
❯ kubectl get certificate -o custom-columns='NAME:.metadata.name,DNS:.spec.dnsNames,IP:.spec.ipAddresses'
NAME DNS IP
ray-ca-certificate-raycluster-mtls <none> <none>
ray-head-cert-raycluster-mtls [localhost raycluster-mtls-head-svc.default.svc.cluster.local] [127.0.0.1]
ray-worker-cert-raycluster-mtls [*.raycluster-mtls-headless.default.svc.cluster.local localhost] [127.0.0.1]
# Check the autoscaler command include --gcs-address
❯ kubectl exec $HEAD -c autoscaler -- sh -c 'tr "\0" " " < /proc/1/cmdline; echo' | grep -o -- '--gcs-address=[^ ]*'
--gcs-address=raycluster-mtls-head-svc.default.svc.cluster.local:6379
❯ kubectl logs $HEAD -c autoscaler | grep -E "The Ray head is ready|rror" | head -3
2026-09-03 06:51:44,646 INFO run_autoscaler.py:89 -- The Ray head is ready. Starting the autoscaler.
# Scale up
❯ kubectl exec $HEAD -c ray-head -- python /home/ray/samples/detached_actor.py a1
kubectl exec $HEAD -c ray-head -- python /home/ray/samples/detached_actor.py a2
2026-09-03 06:54:15,209 INFO worker.py:1672 -- Using address raycluster-mtls-head-svc.default.svc.cluster.local:6379 set in the environment variable RAY_ADDRESS
2026-09-03 06:54:15,215 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: raycluster-mtls-head-svc.default.svc.cluster.local:6379...
2026-09-03 06:54:15,225 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at raycluster-mtls-head-svc.default.svc.cluster.local:8265
/home/ray/anaconda3/lib/python3.10/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
2026-09-03 06:54:16,546 INFO worker.py:1672 -- Using address raycluster-mtls-head-svc.default.svc.cluster.local:6379 set in the environment variable RAY_ADDRESS
2026-09-03 06:54:16,552 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: raycluster-mtls-head-svc.default.svc.cluster.local:6379...
2026-09-03 06:54:16,561 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at raycluster-mtls-head-svc.default.svc.cluster.local:8265
/home/ray/anaconda3/lib/python3.10/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
# Ensure new worker pods being created
❯ kubectl get pod -l ray.io/cluster=$CLUSTER,ray.io/node-type=worker
NAME READY STATUS RESTARTS AGE
raycluster-mtls-small-group-worker-5wlkg 0/1 Running 0 15s
raycluster-mtls-small-group-worker-ntr49 0/1 Running 0 15s
# Check pod name = hostname
❯ kubectl get pod -l ray.io/cluster=$CLUSTER,ray.io/node-type=worker -o custom-columns='NAME:.metadata.name,HOSTNAME:.spec.hostname,SUBDOMAIN:.spec.subdomain'
NAME HOSTNAME SUBDOMAIN
raycluster-mtls-small-group-worker-5wlkg raycluster-mtls-small-group-worker-5wlkg raycluster-mtls-headless
raycluster-mtls-small-group-worker-ntr49 raycluster-mtls-small-group-worker-ntr49 raycluster-mtls-headless
# ray list nodes will use hostname as node ip
❯ kubectl exec $HEAD -c ray-head -- ray list nodes --format json | jq -r '.[] | [.node_ip, .state] | @tsv'
raycluster-mtls-small-group-worker-ntr49.raycluster-mtls-headless.default.svc.cluster.local ALIVE
raycluster-mtls-head-svc.default.svc.cluster.local ALIVE
raycluster-mtls-small-group-worker-5wlkg.raycluster-mtls-headless.default.svc.cluster.local ALIVE
❯ kubectl logs $HEAD -c autoscaler | grep "Submitting a scale request"
2026-09-03 06:54:20,577 INFO cloud_provider.py:443 -- Submitting a scale request: KubeRayProvider.ScaleRequest(desired_num_workers=defaultdict(<class 'int'>, {'small-group': 2}), workers_to_delete=defaultdict(<class 'list'>, {}), worker_groups_without_pending_deletes=set(), worker_groups_with_pending_deletes=set())
# Scale down
❯ kubectl exec $HEAD -c ray-head -- python /home/ray/samples/terminate_detached_actor.py a1
kubectl exec $HEAD -c ray-head -- python /home/ray/samples/terminate_detached_actor.py a2
2026-09-03 06:55:33,611 INFO worker.py:1672 -- Using address raycluster-mtls-head-svc.default.svc.cluster.local:6379 set in the environment variable RAY_ADDRESS
2026-09-03 06:55:33,617 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: raycluster-mtls-head-svc.default.svc.cluster.local:6379...
2026-09-03 06:55:33,626 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at raycluster-mtls-head-svc.default.svc.cluster.local:8265
/home/ray/anaconda3/lib/python3.10/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
2026-09-03 06:55:34,729 INFO worker.py:1672 -- Using address raycluster-mtls-head-svc.default.svc.cluster.local:6379 set in the environment variable RAY_ADDRESS
2026-09-03 06:55:34,736 INFO worker.py:1814 -- Connecting to existing Ray cluster at address: raycluster-mtls-head-svc.default.svc.cluster.local:6379...
2026-09-03 06:55:34,745 INFO worker.py:2003 -- Connected to Ray cluster. View the dashboard at raycluster-mtls-head-svc.default.svc.cluster.local:8265
/home/ray/anaconda3/lib/python3.10/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
# Ensure worker pods being terminated
❯ kubectl get pod -l ray.io/cluster=$CLUSTER
NAME READY STATUS RESTARTS AGE
raycluster-mtls-head-5pl62 2/2 Running 0 9m37s
|
Collaborator
Author
Quick Summary
|
machichima
marked this pull request as ready for review
September 4, 2026 02:10
machichima
requested review from
MortalHappiness,
andrewsykim,
kevin85421 and
rueian
as code owners
September 4, 2026 02:11
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
Reviewed by Cursor Bugbot for commit 2fff093. Configure here.
| workerSpec.RayStartParams["node-ip-address"] = fmt.Sprintf("%s.%s.%s.svc.%s", | ||
| podTemplate.Name, podTemplate.Spec.Subdomain, instance.Namespace, utils.GetClusterDomainName()) | ||
| } | ||
| } |
There was a problem hiding this comment.
Preset subdomain breaks mTLS identity
Medium Severity
When a worker template already has subdomain set, the new FQDN path is skipped entirely, including --node-ip-address injection. Those workers still register with pod IPs, but this change also removes IP SANs and wait-for-tls-ip-san. mTLS handshakes then fail for TPU multi-host and any other preset-subdomain group.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 2fff093. Configure here.
| podTemplate.Name = podName + rand.String(5) // podName ends with "-", <= 63 chars total | ||
| podTemplate.GenerateName = "" | ||
| podTemplate.Spec.Hostname = podTemplate.Name | ||
| podTemplate.Spec.Subdomain = instance.Name + utils.DashSymbol + utils.HeadlessServiceSuffix |
There was a problem hiding this comment.
Headless subdomain exceeds DNS limit
Medium Severity
Worker subdomain is set to {cluster.Name}-headless with no truncation. Service names and pod subdomains must be valid DNS labels (≤63 characters). A cluster name of 55+ characters yields an invalid subdomain, so Kubernetes rejects the pods (and the headless Service). The new long-name test only checks the pod name length.
Reviewed by Cursor Bugbot for commit 2fff093. Configure here.
| ip := utils.LOCAL_HOST | ||
| if utils.EnvVarExists(utils.RAY_USE_TLS, container.Env) { | ||
| ip = fqdnRayIP | ||
| } |
There was a problem hiding this comment.
TLS env check is too broad
Medium Severity
Head RAY_ADDRESS switches from localhost to the head Service FQDN whenever RAY_USE_TLS is present on the container, not when KubeRay mTLS is enabled. Users who already set RAY_USE_TLS for Ray’s native file-based TLS will get a new head address, which can break ray.init() if their certificates only cover loopback or the pod IP.
Reviewed by Cursor Bugbot for commit 2fff093. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Ray pods are only addressable by pod IP, which changes on every recreation. Also when using mTLS feature, when scale up a worker, we will need to update the certificate to include the worker pod IP. This PR adds
spec.podFQDNto RayCluster, giving each pod a resolvable DNS name:<hostname>.<cluster>-headless.<ns>.svc.<cluster-domain>, and optionally inject this to Ray node's--node-ip-address.Two modes:
DNSOnly(default): create per-pod DNS records only (podhostname/subdomain+ the headless worker Service). Ray keeps registering with pod IPs; FQDNs are for external consumers.RegisterAsNodeAddress: additionally inject the FQDN as--node-ip-address, so Ray nodes register and address each other by DNS name. This enables static DNS-SAN certificates for mTLS instead of re-issuing certs on every pod IP change.Notes:
subdomain(e.g. TPU multi-host webhook).Manual test:
RegisterAsNodeAddressmode: [Feature] Support per-pod FQDNs & inject to Ray node ip #5191 (comment)TODO left:
Related issue number
Related to: #5000 and #5048
Labels
doc-updates-requiredlabel.breaking-changelabel.Checks
Manual test instructions