📖 simplify README.md, improve docs (#238)

danielgafni · web-flow · commit 5d0c98d8e41f · 2025-09-19T23:22:05.000+03:00
diff --git a/README.md b/README.md
@@ -12,7 +12,7 @@
 
 **Ray integration for Dagster.**
 
-`dagster-ray` enables you to orchestrate distributed Ray compute from Dagster pipelines, providing seamless integration between Dagster's orchestration capabilities and Ray's distributed computing power.
+`dagster-ray` enables working with distributed Ray compute from Dagster pipelines, combining Dagster's excellent orchestration capabilities and Ray's distributed computing power together.
 
 > [!NOTE]
 > This project is ready for production use, but some APIs may change between minor releases.
@@ -21,79 +21,31 @@ Learn more in the [docs](https://danielgafni.github.io/dagster-ray)
 
 ## 🚀 Key Features
 
-- **Run Launchers & Executors**: Submit Dagster runs or individual ops as Ray jobs
-- **Ray Resources**: Manage Ray clusters with Kubernetes (KubeRay) or local backends, connect to them in client mode
-- **Dagster Pipes**: Execute external Ray scripts with rich logging and metadata
-- **Production Ready**: Tested against a matrix of core dependencies and platform versions, integrated with Dagster+
+- **🎯 Run Launchers & Executors**: Submit Dagster runs or individual steps by submitting Ray jobs
+- **🔧 Ray Resources**: Automatically create and destroy ephemeral Ray clusters and connect to them in client mode
+- **📡 Dagster Pipes Integration**: Submit external scripts as Ray jobs, stream back logs and rich Dagster metadata
+- **☸️ KubeRay Support**: Utilize `RayJob` and `RayCluster` custom resources in client or job submission mode ([tutorial](tutorial/kuberay.md))
+- **🏭 Production Ready**: Tested against a matrix of core dependencies, integrated with Dagster+
 
-## 📦 Quick Start
-
-### Installation
+## Installation
 
 ```shell
 pip install dagster-ray
 ```
 
-### Example
-
-Define a Dagster asset that uses Ray in client mode
-```python
-import dagster as dg
-from dagster_ray import RayResource
-import ray
-
-
-@ray.remote
-def compute_square(x: int) -> int:
-    return x**2
-
-
-@dg.asset
-def my_distributed_computation(ray_cluster: RayResource) -> int:
-    futures = [compute_square.remote(i) for i in range(10)]
-    return sum(ray.get(futures))
-```
-
-Now use `LocalRay` for local development and swap it with a thick cluster in Kubernetes!
-
-```python
-from dagster_ray.kuberay import in_k8s, KubeRayInteractiveJob
-
-ray_cluster = LocalRay() if not in_k8s else KubeRayInteractiveJob()
-
-definitions = dg.Definitions(
-    assets=[my_distributed_computation],
-    resources={"ray_cluster": ray_cluster},
-)
-```
-
-Learn more by reading the [tutorials](https://danielgafni.github.io/dagster-ray/tutorial).
-
 ## 📚 Docs
 
 **📖 [Full Documentation](https://danielgafni.github.io/dagster-ray)**
 
 - **[Tutorial](https://danielgafni.github.io/dagster-ray/tutorial/)**: Step-by-step guide with examples
-- **[API Reference](https://danielgafni.github.io/dagster-ray/api/)**: Complete API documentation
-
-## 🛠️ Integration Options
-
-| Component | Use Case | Cluster Management | Ray Mode |
-|-----------|----------|-------------------|------|
-| `RayRunLauncher` | Deployment-wide Ray runtime | External | Job Mode |
-| `ray_executor` | Ray runtime scoped to a Code Location | External | Job Mode |
-| `PipesRayJobClient` | Submit external scripts as Ray jobs | External | Job Mode |
-| `PipesKubeRayJobClient` | Submit an external script as a `RayJob`, forward logs and Dagster metadata | Automatic | Job Mode |
-| `KubeRayInteractiveJob` | Create a `RayJob`, connect in Client mode without an external script  | Automatic | Client Mode |
+- **[API Reference](https://danielgafni.github.io/dagster-ray/api/)**: Complete API reference
 
 ## 🤝 Contributing
 
 Contributions are very welcome! To get started:
 
 ```bash
-git clone https://github.com/danielgafni/dagster-ray.git
-cd dagster-ray
-uv sync --all-extras
+uv sync --all-extras --all-groups
 uv run pre-commit install
 ```
 
@@ -104,7 +56,16 @@ uv run pytest
 ```
 
 Running KubeRay tests requires the following tools to be present:
-- `docker`, `kubectl`, `helm`, `minikube`
+- `docker`
+- `kubectl`
+- `helm`
+- `minikube`
+
+❄️ Nix users will find them provided in the dev shell:
+
+```
+nix develop
+```
 
 ### Documentation
 
@@ -113,9 +74,4 @@ To build and serve the documentation locally:
 ```bash
 # Serve documentation locally
 uv run --group docs mkdocs serve
-
-# Build documentation
-uv run--group docs mkdocs build
 ```
-
-The documentation is automatically deployed to GitHub Pages.
diff --git a/docs/index.md b/docs/index.md
@@ -1,21 +1,30 @@
-# Welcome to dagster-ray
-
-**Ray integration for Dagster** - Orchestrate distributed Ray compute from Dagster pipelines with seamless integration between Dagster's orchestration capabilities and Ray's distributed computing power.
+# Welcome to `dagster-ray` docs
 
 [![PyPI version](https://img.shields.io/pypi/v/dagster-ray.svg)](https://pypi.python.org/pypi/dagster-ray)
 [![License](https://img.shields.io/pypi/l/dagster-ray.svg)](https://pypi.python.org/pypi/dagster-ray)
 [![Python versions](https://img.shields.io/pypi/pyversions/dagster-ray.svg)](https://pypi.python.org/pypi/dagster-ray)
 [![CI](https://github.com/danielgafni/dagster-ray/actions/workflows/CI.yml/badge.svg)](https://github.com/danielgafni/dagster-ray/actions/workflows/CI.yml)
+[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)
+[![basedpyright - checked](https://img.shields.io/badge/basedpyright-checked-42b983)](https://docs.basedpyright.com)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+
+---
+
+**Ray integration for Dagster.**
+
+
+`dagster-ray` enables working with distributed Ray compute from Dagster pipelines, combining Dagster's excellent orchestration capabilities and Ray's distributed computing power together.
+
 
 > [!NOTE]
 > This project is ready for production use, but some APIs may change between minor releases.
 
 ## 🚀 Key Features
 
-- **🎯 Run Launchers & Executors**: Submit Dagster runs or individual steps as Ray jobs
-- **🔧 Ray Resources**: Connect to Ray cluster in client mode and manage their lifecycle
-- **📡 Dagster Pipes Integration**: Execute external Ray scripts with rich logging and metadata
-- **☸️ KubeRay Support**: Utilize `RayJob` and `RayCluster` custom resources in client or job mode ([tutorial](tutorial/kuberay.md))
+- **🎯 Run Launchers & Executors**: Submit Dagster runs or individual steps by submitting Ray jobs
+- **🔧 Ray Resources**: Automatically create and destroy ephemeral Ray clusters and connect to them in client mode
+- **📡 Dagster Pipes Integration**: Submit external scripts as Ray jobs, stream back logs and rich Dagster metadata
+- **☸️ KubeRay Support**: Utilize `RayJob` and `RayCluster` custom resources in client or job submission mode ([tutorial](tutorial/kuberay.md))
 - **🏭 Production Ready**: Tested against a matrix of core dependencies, integrated with Dagster+
 
 ## ⚡ Quick Start
@@ -27,14 +36,31 @@
     pip install dagster-ray
     ```
 
+    !!! tip
+        See [external cluster tutorial](tutorial/external.md)
+
 === "With KubeRay"
     ```bash
     pip install 'dagster-ray[kuberay]'
     ```
 
+    !!! tip
+        See [KubeRay tutorial](tutorial/kuberay.md)
+
 ### Basic Usage
+#### Execute Dagster steps on an existing Ray cluster
+!!! example
+    ```python
+    import dagster as dg
+    from dagster_ray import ray_executor
+
+    defs = dg.Definitions(..., executor=ray_executor)
+    ```
 
+#### Execute an asset on Ray in client mode
 !!! example
+
+
     Define a Dagster asset that uses Ray in client mode
     ```python
     import dagster as dg
@@ -56,9 +82,10 @@
     1. :zap: I am already running in Ray!
     2. :bulb: `RayResource` is a type annotation that provides a common interface for Ray resources
 
-    Now use `LocalRay` for local development and swap it with a thick cluster in Kubernetes!
+    Now use [`LocalRay`](api/core.md#dagster_ray.core.resources.LocalRay) for development and swap it with a thick cluster in Kubernetes!
 
     ```python
+    from dagster_ray import LocalRay
     from dagster_ray.kuberay import in_k8s, KubeRayInteractiveJob
 
     ray_cluster = LocalRay() if not in_k8s else KubeRayInteractiveJob()
@@ -69,6 +96,8 @@
     )
     ```
 
+    [`KubeRayInteractiveJob`](api/kuberay.md#dagster_ray.kuberay.KubeRayInteractiveJob) will create a `RayJob`, connect to it, and optionally perform cleanup according to the configured policy.
+
 Learn more by reading the [tutorials](tutorial/index.md).
 
 ## 🛠️ Choosing Your Integration
@@ -118,6 +147,7 @@ Learn more by reading the [tutorials](tutorial/index.md).
 
     !!! tip
         See [KubeRay tutorial](tutorial/kuberay.md)
+
 ## 📚 What's Next?
 
 <div class="grid cards" markdown>
diff --git a/flake.nix b/flake.nix
@@ -35,7 +35,7 @@
         UV_PYTHON = "${python}/bin/python";
         shellHook = ''
           uv venv --allow-existing
-          uv sync --frozen --all-extras
+          uv sync --frozen --all-extras --all-groups
         '';
       };
     };
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -177,3 +177,6 @@ validation:
   absolute_links: warn  # Or 'relative_to_docs' - new in MkDocs 1.6
   unrecognized_links: warn
   anchors: warn  # New in MkDocs 1.6
+
+watch:
+  - src
diff --git a/src/dagster_ray/_base/resources.py b/src/dagster_ray/_base/resources.py
@@ -82,19 +82,19 @@ def my_asset(ray_cluster: RayResource):
     )
     enable_tracing: bool = Field(
         default=False,
-        description="Enable tracing: inject `RAY_PROFILING=1` and `RAY_task_events_report_interval_ms=0` into the Ray cluster configuration. This allows using `ray.timeline()` to fetch recorded task events. Learn more: https://docs.ray.io/en/latest/ray-core/api/doc/ray.timeline.html#ray-timeline",
+        description="Enable tracing: inject `RAY_PROFILING=1` and `RAY_task_events_report_interval_ms=0` into the Ray cluster configuration. This allows using `ray.timeline()` to fetch recorded task events. Learn more: [KubeRay docs](https://docs.ray.io/en/latest/ray-core/api/doc/ray.timeline.html#ray-timeline)",
     )
     enable_actor_task_logging: bool = Field(
         default=False,
         description="Enable actor task logging: inject `RAY_ENABLE_RECORD_ACTOR_TASK_LOGGING=1` into the Ray cluster configuration.",
     )
     enable_debug_post_mortem: bool = Field(
         default=False,
-        description="Enable post-mortem debugging: inject `RAY_DEBUG_POST_MORTEM=1` into the Ray cluster configuration. Learn more: https://docs.ray.io/en/latest/ray-observability/ray-distributed-debugger.html",
+        description="Enable post-mortem debugging: inject `RAY_DEBUG_POST_MORTEM=1` into the Ray cluster configuration. Learn more: [KubeRay docs](https://docs.ray.io/en/latest/ray-observability/ray-distributed-debugger.html)",
     )
     enable_legacy_debugger: bool = Field(
         default=False,
-        description="Enable legacy debugger: inject `RAY_DEBUG=legacy` into the Ray cluster configuration. Learn more: https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/ray-debugging.html#using-the-ray-debugger",
+        description="Enable legacy debugger: inject `RAY_DEBUG=legacy` into the Ray cluster configuration. Learn more: [KubeRay docs](https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/ray-debugging.html#using-the-ray-debugger)",
     )
 
     _context: RayBaseContext | None = PrivateAttr()
diff --git a/src/dagster_ray/core/executor.py b/src/dagster_ray/core/executor.py
@@ -63,7 +63,7 @@ def ray_executor(init_context: InitExecutorContext) -> Executor:
     """Executes steps by submitting them as Ray jobs.
 
     The steps are started inside the Ray cluster directly.
-    When used together with the `RayRunLauncher`, the executor can inherit the job submission client configuration.
+    When used together with [`RayRunLauncher`][dagster_ray.core.run_launcher.RayRunLauncher], the executor can inherit the job submission client configuration.
     This behavior can be disabled by setting `inherit_job_submission_client_from_ray_run_launcher` to `False`.
 
     Example:
diff --git a/src/dagster_ray/core/pipes.py b/src/dagster_ray/core/pipes.py
@@ -223,10 +223,10 @@ class PipesRayJobClient(dg.PipesClient, TreatAsResourceParam):
     Args:
         client (JobSubmissionClient): The Ray job submission client
         context_injector (Optional[PipesContextInjector]): A context injector to use to inject
-            context into the Ray job. Defaults to `PipesEnvContextInjector`.
+            context into the Ray job. Defaults to [`PipesEnvContextInjector`][dagster.PipesEnvContextInjector].
         message_reader (Optional[PipesMessageReader]): A message reader to use to read messages
-            from the glue job run. Defaults to `PipesRayJobMessageReader`.
-        forward_termination (bool): Whether to cancel the `RayJob` job run when the Dagster process receives a termination signal.
+            from the glue job run. Defaults to [`PipesRayJobMessageReader`][dagster_ray.core.pipes.PipesRayJobMessageReader].
+        forward_termination (bool): Whether to cancel the Ray job run when the Dagster process receives a termination signal.
         timeout (int): Timeout for various internal interactions with the Kubernetes RayJob.
         poll_interval (int): Interval at which to poll Kubernetes for status updates.
             Is useful when running in a local environment.
diff --git a/src/dagster_ray/kuberay/pipes.py b/src/dagster_ray/kuberay/pipes.py
@@ -38,9 +38,9 @@ class PipesKubeRayJobClient(dg.PipesClient, TreatAsResourceParam):
 
     Args:
         context_injector (Optional[PipesContextInjector]): A context injector to use to inject
-            context into the ``RayJob``. Defaults to `PipesEnvContextInjector`.
+            context into the ``RayJob``. Defaults to [`PipesEnvContextInjector`][dagster.PipesEnvContextInjector].
         message_reader (Optional[PipesMessageReader]): A message reader to use to read messages
-            from the glue job run. Defaults to `PipesRayJobMessageReader`.
+            from the glue job run. Defaults to [`PipesRayJobMessageReader`][dagster_ray.core.pipes.PipesRayJobMessageReader].
         client (Optional[boto3.client]): The Kubernetes API client.
         forward_termination (bool): Whether to terminate the Ray job when the Dagster process receives a termination signal,
             or if the startup timeout is reached. Defaults to ``True``.
diff --git a/src/dagster_ray/kuberay/resources/raycluster.py b/src/dagster_ray/kuberay/resources/raycluster.py
@@ -26,9 +26,9 @@ def create_resource(self, context: dg.InitResourceContext) -> RayClusterClient:
 
 class KubeRayCluster(BaseKubeRayResourceConfig, RayResource):
     """
-    Provides a `RayCluster` for Dagster steps.
+    Provides a Ray Cluster for Dagster steps.
 
-    It is advised to use `dagster_ray.kuberay.KubeRayInteractiveJob` with KubeRay >= 1.3.0 instead.
+    It is advised to use [`KubeRayInteractiveJob`][dagster_ray.kuberay.resources.KubeRayInteractiveJob] with KubeRay >= 1.3.0 instead.
 
     Info:
         Image defaults to `dagster/image` run tag.
@@ -49,7 +49,7 @@ class KubeRayCluster(BaseKubeRayResourceConfig, RayResource):
     )
     log_cluster_conditions: bool = Field(
         default=True,
-        description="Whether to log RayCluster conditions while waiting for the RayCluster to become ready. For more information, see https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/observability.html#raycluster-status-conditions.",
+        description="Whether to log RayCluster conditions while waiting for the RayCluster to become ready. Learn more: [KubeRay docs](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/observability.html#raycluster-status-conditions).",
     )
 
     _name: str = PrivateAttr()
diff --git a/src/dagster_ray/kuberay/resources/rayjob.py b/src/dagster_ray/kuberay/resources/rayjob.py
@@ -32,22 +32,22 @@ def create_resource(self, context: dg.InitResourceContext):
 
 
 class InteractiveRayJobSpec(RayJobSpec):
-    """Same as `dagster_ray.kuberay.resources.rayjob.RayJobSpec`, but submission mode has to be `InteractiveMode`"""
+    """Same as [`RayJobSpec`][dagster_ray.kuberay.configs.RayJobSpec], but `mode` has to be `InteractiveMode`"""
 
     submission_mode: Literal["InteractiveMode"] = "InteractiveMode"  # pyright: ignore[reportIncompatibleVariableOverride]
 
 
 class InteractiveRayJobConfig(RayJobConfig):
-    """Same as `dagster_ray.kuberay.resources.rayjob.RayJobConfig`, but `spec.submission_mode` mode has to be `InteractiveMode`"""
+    """Same as [`RayJobConfig`][dagster_ray.kuberay.configs.RayJobConfig], but `spec.submission_mode` mode has to be `InteractiveMode`"""
 
     spec: InteractiveRayJobSpec = Field(default_factory=InteractiveRayJobSpec)  # pyright: ignore[reportIncompatibleVariableOverride]
 
 
 class KubeRayInteractiveJob(RayResource, BaseKubeRayResourceConfig):
     """
-    Provides a `RayJob` for Dagster steps.
+    Provides a Ray Job for Dagster steps.
 
-    Is the recommended way to run Ray workloads with automatic cluster management. It creates a `RayJob`, connects to it in client mode and sets the `jobId` field. Cleanup is handled by the KubeRay controller or by the resource lifecycle logic.
+    Is the recommended way to run Ray workloads with automatic cluster management. It creates a Ray Job, connects to it in client mode and sets the `jobId` field. Cleanup is handled by the KubeRay controller or by the resource lifecycle logic.
 
     Info:
         Image defaults to `dagster/image` run tag.
@@ -72,7 +72,7 @@ class KubeRayInteractiveJob(RayResource, BaseKubeRayResourceConfig):
 
     log_cluster_conditions: bool = Field(
         default=True,
-        description="Whether to log `RayCluster` conditions while waiting for the RayCluster to become ready. For more information, see https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/observability.html#raycluster-status-conditions.",
+        description="Whether to log `RayCluster` conditions while waiting for the RayCluster to become ready. Learn more: [KubeRay docs](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/observability.html#raycluster-status-conditions).",
     )
 
     _name: str = PrivateAttr()

Original file line number	Diff line number	Diff line change
`@@ -82,19 +82,19 @@ def my_asset(ray_cluster: RayResource):`
`82`	`82`	`)`
`83`	`83`	`enable_tracing: bool = Field(`
`84`	`84`	`default=False,`
`85`		- description="Enable tracing: inject `RAY_PROFILING=1` and `RAY_task_events_report_interval_ms=0` into the Ray cluster configuration. This allows using `ray.timeline()` to fetch recorded task events. Learn more: https://docs.ray.io/en/latest/ray-core/api/doc/ray.timeline.html#ray-timeline",
	`85`	+ description="Enable tracing: inject `RAY_PROFILING=1` and `RAY_task_events_report_interval_ms=0` into the Ray cluster configuration. This allows using `ray.timeline()` to fetch recorded task events. Learn more: [KubeRay docs](https://docs.ray.io/en/latest/ray-core/api/doc/ray.timeline.html#ray-timeline)",
`86`	`86`	`)`
`87`	`87`	`enable_actor_task_logging: bool = Field(`
`88`	`88`	`default=False,`
`89`	`89`	description="Enable actor task logging: inject `RAY_ENABLE_RECORD_ACTOR_TASK_LOGGING=1` into the Ray cluster configuration.",
`90`	`90`	`)`
`91`	`91`	`enable_debug_post_mortem: bool = Field(`
`92`	`92`	`default=False,`
`93`		- description="Enable post-mortem debugging: inject `RAY_DEBUG_POST_MORTEM=1` into the Ray cluster configuration. Learn more: https://docs.ray.io/en/latest/ray-observability/ray-distributed-debugger.html",
	`93`	+ description="Enable post-mortem debugging: inject `RAY_DEBUG_POST_MORTEM=1` into the Ray cluster configuration. Learn more: [KubeRay docs](https://docs.ray.io/en/latest/ray-observability/ray-distributed-debugger.html)",
`94`	`94`	`)`
`95`	`95`	`enable_legacy_debugger: bool = Field(`
`96`	`96`	`default=False,`
`97`		- description="Enable legacy debugger: inject `RAY_DEBUG=legacy` into the Ray cluster configuration. Learn more: https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/ray-debugging.html#using-the-ray-debugger",
	`97`	+ description="Enable legacy debugger: inject `RAY_DEBUG=legacy` into the Ray cluster configuration. Learn more: [KubeRay docs](https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/ray-debugging.html#using-the-ray-debugger)",
`98`	`98`	`)`
`99`	`99`
`100`	`100`	`_context: RayBaseContext \| None = PrivateAttr()`