docs: add cross-cluster & authentication docs (#336)

Author: danielgafni

docs/api/kuberay.md

@@ -2,6 +2,12 @@
 
 KubeRay integration components for running Ray on Kubernetes.  Learn how to use it [here](../tutorial/kuberay.md).
 
+!!! tip "Authentication"
+    For connecting to auth-protected Ray clusters, see [Authentication docs](../tutorial/auth.md). Client-mode resources support Ray's [token authentication](https://docs.ray.io/en/latest/ray-core/internals/token-authentication.html) via environment variables. Pipes clients (`PipesKubeRayJobClient`) additionally accept `headers`, `cookies`, and `verify` parameters for HTTP-level authentication.
+
+!!! tip "Custom host resolution"
+    Override [`BaseKubeRayResource.resolve_hostname`][dagster_ray.kuberay.resources.base.BaseKubeRayResource.resolve_hostname] to customize how the Ray head service address is constructed (e.g. `name.namespace.company.com` instead of the default in-cluster FQDN).
+
 ---
 
 ## Client Mode Resources

docs/tutorial/auth.md

@@ -0,0 +1,26 @@
+# Authentication
+
+## Ray Token Authentication
+
+Ray supports [token authentication](https://docs.ray.io/en/latest/ray-core/internals/token-authentication.html) (v2.52.0+) which protects both gRPC (used by `ray.init()`) and HTTP (used by job submission) endpoints.
+
+The client process needs two environment variables to be set:
+
+- `RAY_AUTH_MODE=token`
+- `RAY_AUTH_TOKEN=<token>` (or `RAY_AUTH_TOKEN_PATH`)
+
+### Authenticating Against Proxies
+
+Pipes clients such as [`PipesKubeRayJobClient`][dagster_ray.kuberay.PipesKubeRayJobClient] or [`PipesRayJobClient`][dagster_ray.core.pipes.PipesRayJobClient] communicate with the Ray dashboard over HTTP. They support `headers`, `cookies`, and `verify` parameters for authentication:
+
+```python
+PipesKubeRayJobClient(
+    headers={"Authorization": "Bearer <token>"},
+)
+```
+
+!!! tip
+
+    These parameters can also be overridden per-invocation in the [`PipesKubeRayJobClient.run`][dagster_ray.kuberay.PipesKubeRayJobClient.run] method.
+
+This is useful when submitting jobs across Kubernetes clusters, where the traffic may be flowing through a proxy requiring additional authentication.

docs/tutorial/index.md

@@ -5,3 +5,6 @@ Check out [External Ray Clusters tutorial](./external.md) if you want to run Ray
 
 ## Dagster + KubeRay
 See [KubeRay tutorial](./kuberay.md) if you want to use KubeRay's `RayJob` and `RayCluster` managed from Dagster
+
+## Authentication
+See [Authentication](./auth.md) for setting up Ray authentication with `dagster-ray`

docs/tutorial/kuberay.md

@@ -288,3 +288,16 @@ definitions = dg.Definitions(
 1. :bulb: `dagster-pipes` has to be installed in the remote environment!
 
 When materializing the asset, `PipesKubeRayJobClient` will submit the script as a `RayJob` custom resource, monitor its status, and stream back logs and Dagster metadata.
+
+## Custom Host Resolution
+
+By default, KubeRay resources resolve the Ray head address as an in-cluster FQDN (`<service>.<namespace>.svc.cluster.local`). Override the [`resolve_hostname`][dagster_ray.kuberay.resources.base.BaseKubeRayResource.resolve_hostname] method to customize this, for example when connecting across clusters or through a custom DNS:
+
+```python
+from dagster_ray.kuberay import KubeRayInteractiveJob
+
+
+class MyKubeRayInteractiveJob(KubeRayInteractiveJob):
+    def resolve_hostname(self, service_name: str, namespace: str) -> str:
+        return f"{service_name}.{namespace}.company.com"
+```

mkdocs.yml

@@ -10,6 +10,7 @@ nav:
     - Overview: tutorial/index.md
     - External Ray Clusters: tutorial/external.md
     - KubeRay: tutorial/kuberay.md
+    - Authentication: tutorial/auth.md
   - API Reference:
     - Overview: api.md
     - Core: api/core.md