Skip to content

Commit b65acbc

Browse files
chipspeakDavidAdaRHNolbag
committed
[doc][KubeRay] Add NetworkPolicy user guide + kuberay namespace note
Signed-off-by: Pat O'Connor <paoconno@redhat.com> Co-authored-by: David Adamache <dadamach@redhat.com> Co-authored-by: Paul Nolan <pnolan@redhat.com>
1 parent e8bd0a2 commit b65acbc

3 files changed

Lines changed: 229 additions & 0 deletions

File tree

‎doc/source/cluster/kubernetes/getting-started/kuberay-operator-installation.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,13 @@ helm repo update
2121
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.0
2222
```
2323

24+
For an optimal security posture, install the operator in a dedicated namespace such as `ray-system` rather than `default`. A dedicated namespace reduces the security surface by isolating the operator's service account and any generated NetworkPolicy selectors from workload pods in `default`.
25+
26+
```sh
27+
kubectl create namespace ray-system
28+
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.0 -n ray-system
29+
```
30+
2431
### Method 2: Kustomize
2532

2633
```sh

‎doc/source/cluster/kubernetes/user-guides.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ user-guides/tpu
2929
user-guides/pod-command
3030
user-guides/helm-chart-rbac
3131
user-guides/tls
32+
user-guides/network-policy
3233
user-guides/k8s-autoscaler
3334
user-guides/kubectl-plugin
3435
user-guides/kuberay-auth
@@ -66,6 +67,7 @@ To learn the basics of Ray on Kubernetes, we recommend taking a look at the {ref
6667
* {ref}`kuberay-pod-command`
6768
* {ref}`kuberay-helm-chart-rbac`
6869
* {ref}`kuberay-tls`
70+
* {ref}`kuberay-network-policy`
6971
* {ref}`kuberay-gke-bucket`
7072
* {ref}`ray-k8s-autoscaler-comparison`
7173
* {ref}`kubectl-plugin`

‎doc/source/cluster/kubernetes/user-guides/network-policy.md‎

Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
(kuberay-network-policy)=
2+
3+
# NetworkPolicy
4+
5+
The KubeRay operator can generate Kubernetes `NetworkPolicy` resources for each `RayCluster`. When enabled, the operator creates separate policies for the head pod and worker pods, and removes them when the cluster is deleted.
6+
7+
**NOTE** NetworkPolicy support is alpha and disabled by default. Enable the `RayClusterNetworkPolicy` feature gate before using `spec.networkPolicy`.
8+
9+
## Enable the feature gate
10+
11+
### Helm
12+
13+
Pass the feature gate when installing the operator. The default `values.yaml` lists six gates at indices 0–5, so append at index 6:
14+
15+
```bash
16+
helm install kuberay-operator kuberay/kuberay-operator \
17+
--set 'featureGates[6].name=RayClusterNetworkPolicy' \
18+
--set 'featureGates[6].enabled=true'
19+
```
20+
21+
To enable it on an existing installation, use `helm upgrade` with the same flags.
22+
23+
### Kustomize
24+
25+
Add a strategic merge patch to the operator `Deployment` that appends `RayClusterNetworkPolicy=true` to the `--feature-gates` argument:
26+
27+
```yaml
28+
apiVersion: apps/v1
29+
kind: Deployment
30+
metadata:
31+
name: kuberay-operator
32+
spec:
33+
template:
34+
spec:
35+
containers:
36+
- name: kuberay-operator
37+
args:
38+
- --feature-gates=RayClusterNetworkPolicy=true
39+
```
40+
41+
## Enable NetworkPolicy
42+
43+
Add a `networkPolicy` field to `spec` and choose a mode:
44+
45+
```yaml
46+
spec:
47+
networkPolicy:
48+
mode: DenyAll
49+
```
50+
51+
Three modes are available:
52+
53+
- **`DenyAll`**: deny all ingress and egress except intra-cluster pod traffic.
54+
- **`DenyAllIngress`**: deny inbound traffic only; outbound is unrestricted.
55+
- **`DenyAllEgress`**: deny outbound traffic only; inbound is unrestricted.
56+
57+
In all modes, the operator always permits pod-to-pod traffic within the same `RayCluster` on all ports. For `DenyAll` and `DenyAllIngress`, the head pod policy also allows the submitter pod to reach the dashboard port when a `RayJob` running in `K8sJobMode` owns the cluster.
58+
59+
## Apply a namespace default-deny policy first
60+
61+
Ray cluster pods and their `NetworkPolicy` resources are created by different controllers asynchronously. A pod can start before its policy is applied, leaving a brief window where all traffic is permitted.
62+
63+
Pre-apply a namespace-level deny policy to close that window. The rules the operator generates are additive on top of it:
64+
65+
```yaml
66+
apiVersion: networking.k8s.io/v1
67+
kind: NetworkPolicy
68+
metadata:
69+
name: default-deny-all
70+
namespace: <your-ray-namespace>
71+
spec:
72+
podSelector: {}
73+
policyTypes:
74+
- Ingress
75+
- Egress
76+
```
77+
78+
## Required rules under DenyAll and DenyAllEgress
79+
80+
The operator doesn't add DNS or API server egress rules. Add both when using `DenyAll` or `DenyAllEgress`.
81+
82+
### DNS egress
83+
84+
Workers reach the head node through its service FQDN. Without a DNS egress rule, workers can't resolve the head address and the cluster won't start. Add this rule to both `head.egressRules` and `worker.egressRules`:
85+
86+
```yaml
87+
spec:
88+
networkPolicy:
89+
mode: DenyAll
90+
head:
91+
egressRules:
92+
- to:
93+
- namespaceSelector:
94+
matchLabels:
95+
kubernetes.io/metadata.name: kube-system
96+
podSelector:
97+
matchLabels:
98+
k8s-app: kube-dns
99+
ports:
100+
- port: 53
101+
protocol: UDP
102+
- port: 53
103+
protocol: TCP
104+
worker:
105+
egressRules:
106+
- to:
107+
- namespaceSelector:
108+
matchLabels:
109+
kubernetes.io/metadata.name: kube-system
110+
podSelector:
111+
matchLabels:
112+
k8s-app: kube-dns
113+
ports:
114+
- port: 53
115+
protocol: UDP
116+
- port: 53
117+
protocol: TCP
118+
```
119+
120+
### API server egress
121+
122+
When `enableInTreeAutoscaling: true`, the Ray autoscaler on the head pod must reach the Kubernetes API server to scale the cluster. The operator doesn't add this rule because the correct form depends on the CNI:
123+
124+
| CNI | Required form |
125+
|---|---|
126+
| kindnet, Calico, Antrea | `ipBlock` with the endpoint IP (post-DNAT) |
127+
| Cilium | `CiliumNetworkPolicy` with `toEntities: [kube-apiserver]`; standard `ipBlock` doesn't work |
128+
| EKS (Amazon VPC CNI) | `ipBlock` with the `kubernetes` Service ClusterIP |
129+
130+
Find the endpoint IP with:
131+
132+
```bash
133+
kubectl get endpointslices -n default -l kubernetes.io/service-name=kubernetes
134+
```
135+
136+
For kindnet, Calico, and Antrea, add this rule to `head.egressRules`:
137+
138+
```yaml
139+
- to:
140+
- ipBlock:
141+
cidr: <endpoint-ip>/32
142+
ports:
143+
- port: 6443
144+
protocol: TCP
145+
```
146+
147+
For EKS, replace `<endpoint-ip>` with the ClusterIP from `kubectl get svc kubernetes`. For Cilium, create a `CiliumNetworkPolicy` instead.
148+
149+
## RayJob patterns
150+
151+
### RayJob-owned clusters
152+
153+
When a `RayJob` in `K8sJobMode` creates and owns the `RayCluster`, the operator automatically injects an ingress rule allowing the submitter pod to reach the head dashboard port. No additional configuration is needed.
154+
155+
For `HTTPMode` and `SidecarMode`, there is no standalone submitter pod, so no additional ingress rule is required.
156+
157+
### Standalone clusters with clusterSelector
158+
159+
When a `RayJob` targets a pre-existing cluster with `clusterSelector`, the cluster has no `RayJob` owner reference. The operator can't automatically add a submitter ingress rule. In `K8sJobMode`, KubeRay stamps the submitter pod with stable identity labels, so you can match them directly:
160+
161+
```yaml
162+
spec:
163+
networkPolicy:
164+
mode: DenyAll
165+
head:
166+
ingressRules:
167+
- from:
168+
- podSelector:
169+
matchLabels:
170+
ray.io/originated-from-cr-name: <rayjob-name>
171+
ray.io/originated-from-crd: RayJob
172+
ports:
173+
- port: 8265
174+
protocol: TCP
175+
```
176+
177+
Replace `<rayjob-name>` with the name of the submitting `RayJob`.
178+
179+
## Custom rules
180+
181+
Add per-role rules under `head` and `worker`. The operator appends them to the base policy for that role and doesn't apply head rules to workers or vice versa.
182+
183+
### Allow Prometheus to scrape metrics
184+
185+
```yaml
186+
spec:
187+
networkPolicy:
188+
mode: DenyAll
189+
head:
190+
ingressRules:
191+
- from:
192+
- namespaceSelector:
193+
matchLabels:
194+
kubernetes.io/metadata.name: <prometheus-namespace>
195+
ports:
196+
- port: 8080
197+
protocol: TCP
198+
```
199+
200+
### Allow worker egress to external services
201+
202+
To allow workers to pull packages or reach external APIs, add an egress rule:
203+
204+
```yaml
205+
spec:
206+
networkPolicy:
207+
mode: DenyAll
208+
worker:
209+
egressRules:
210+
- to:
211+
- ipBlock:
212+
cidr: 0.0.0.0/0
213+
ports:
214+
- port: 443
215+
protocol: TCP
216+
```
217+
218+
## Full example
219+
220+
See [`ray-cluster.network-policy-deny-all.yaml`](https://raw.githubusercontent.com/ray-project/kuberay/refs/heads/master/ray-operator/config/samples/ray-cluster.network-policy-deny-all.yaml) for a complete annotated example covering `DenyAll` mode, DNS egress, API server egress (commented out with per-CNI notes), Prometheus scraping, and the clusterSelector `RayJob` pattern.

0 commit comments

Comments
 (0)