Enable persistent Ray cluster state via external Redis GCS fault tolerance

[kramaranya]

Issue link

RHOAIENG-11115

What changes have been made

Provided Ray cluster head pod persistency through GCS fault tolerance.
Added new config options: enable_gcs_ft, redis_address, redis_password_secret and external_storage_namespace
Added unit tests to cover gcs fault tolerance

Verification steps

• Provision a cluster
• Deploy Redis in your cluster ( you can use this example)
• Deploy KubeRay operator (1.3.0 or newer, RHOAI currently provides an older version)
• Create RayCluster with GCS fault tolerance enabled, for example:

cluster = Cluster(ClusterConfiguration(
        name='raycluster',
        head_cpu_requests='500m',
        head_cpu_limits='500m',
        head_memory_requests=2,
        head_memory_limits=2,
        num_workers=1,
        worker_cpu_requests='250m',
        worker_cpu_limits=1,
        worker_memory_requests=4,
        worker_memory_limits=4,
        enable_gcs_ft=True,
        redis_address="redis:6379",
        redis_password_secret={
            "name": "redis-password-secret",
            "key": "password"
        }
    ))

• Make sure all pods are running
• Create a detached actor
• Check if the actor is visible - ray list actors from RayCluster head pod
• Restart the head pod
• Run ray list actors - you should see previously created actor

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[kryanbeane]

great work! lgtm

[codecov]

Codecov Report

Attention: Patch coverage is 95.65217% with 1 line in your changes missing coverage. Please review.

Project coverage is 92.45%. Comparing base (82fa656) to head (24a8950).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/codeflare_sdk/ray/cluster/build_ray_cluster.py	88.88%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #821      +/-   ##
==========================================
+ Coverage   92.41%   92.45%   +0.04%     
==========================================
  Files          24       24              
  Lines        1397     1419      +22     
==========================================
+ Hits         1291     1312      +21     
- Misses        106      107       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kryanbeane]

thanks for adding the tests!

[chipspeak]

/lgtm

[laurafitzgerald]

/override kubernetes

[openshift-ci]

@laurafitzgerald: Overrode contexts on behalf of laurafitzgerald: kubernetes

In response to this:

/override kubernetes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kryanbeane]

/lgtm
/approve
/override kubernetes

[openshift-ci]

@kryanbeane: /override requires failed status contexts, check run or a prowjob name to operate on.
The following unknown contexts/checkruns were given:

• kubernetes

Only the following failed contexts/checkruns were expected:

• add-approve-lgtm-label
• tide

If you are trying to override a checkrun that has a space in it, you must put a double quote on the context.

In response to this:

/lgtm
/approve
/override kubernetes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chipspeak, kryanbeane

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [chipspeak,kryanbeane]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment