dagster-ray

---

Ray integration for Dagster.

dagster-ray enables working with distributed Ray compute from Dagster pipelines, combining Dagster's excellent orchestration capabilities and Ray's distributed computing power together.

Note

This project is ready for production use, but some APIs may change between minor releases.

Learn more in the docs

🚀 Key Features

• 🎯 Run Launchers & Executors: Submit Dagster runs or individual steps by submitting Ray jobs
• 🔧 Ray Resources: Automatically create and destroy ephemeral Ray clusters and connect to them in client mode
• 📡 Dagster Pipes Integration: Submit external scripts as Ray jobs, stream back logs and rich Dagster metadata
• ☸️ KubeRay Support: Utilize RayJob and RayCluster custom resources in client or job submission mode (tutorial)
• 🏭 Production Ready: Tested against a matrix of core dependencies, integrated with Dagster+
• ⚡ Instant Startup: Leverage RayCluster with Cluster Sharing for lightning-fast development cycles with zero cold start times (intended for development environments)

Installation

pip install dagster-ray

📚 Docs

📖 Full Documentation

• Tutorial: Step-by-step guide with examples
• API Reference: Complete API reference

🤝 Contributing

Contributions are very welcome! To get started:

uv sync --all-extras --all-groups
uv run pre-commit install

🧪 Testing

uv run pytest

Running KubeRay tests requires the following tools to be present:

• docker
• kubectl
• helm
• minikube

❄️ Nix users will find them provided in the dev shell:

nix develop

Documentation

To build and serve the documentation locally:

# Serve documentation locally
uv run --group docs mkdocs serve