✨ implement cluster sharing for KubeRayCluster
#251
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This stack of pull requests is managed by Graphite. Learn more about stacking. |
danielgafni
force-pushed
the
10-08-_sparkles_implement_cluster_sharing_for_kuberay_s_raycluster
branch
7 times, most recently
from
7e6224e to
2ccfb8d
Compare
danielgafni
changed the title
KubeRayCluster
danielgafni
force-pushed
the
10-08-_sparkles_implement_cluster_sharing_for_kuberay_s_raycluster
branch
4 times, most recently
from
2b6fffb to
64fe3a4
Compare
Merged
danielgafni
force-pushed
the
10-08-_sparkles_implement_cluster_sharing_for_kuberay_s_raycluster
branch
8 times, most recently
from
4765c04 to
a2e2f9b
Compare
This was referenced Oct 9, 2025
danielgafni
force-pushed
the
10-08-_sparkles_implement_cluster_sharing_for_kuberay_s_raycluster
branch
from
a2e2f9b to
5872d8f
Compare
danielgafni
force-pushed
the
10-08-_sparkles_implement_cluster_sharing_for_kuberay_s_raycluster
branch
from
5872d8f to
d504e74
Compare
Merge activity
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements cluster sharing for
KubeRayCluster Dagster resource. Cluster sharing allows reusing the sameRayClustercreated by one of a previously executed Dagster steps across subsequent Dagster steps. It can dramatically speed up step setup, making if effectively instant in the presence of existing clusters.Because KubeRay doesn't currently provide a TTL mechanism for
RayCluster, this PR does 2 things:KubeRayCluster.cluster_sharing config is responsible for matching on existingRayCluster based on system (generated by Dagster anddagster-ray) and user-provided Kubernetes labelsKubeRayCluster placesdagster/lock-<run-id>-<step-key> annotations onRayCluster resources targeted by the current Dagster step. The annotation value is a serializedClusterSharingLock object with creation time and ttl set.dagster_ray.kuberay.cleanup_expired_rayclusters sensor monitorsRayClsuter resources and submits run requests for expired clusters usingdagster_ray.kuberay.delete_kuberay_rayclusters job.Resolve #131