What scheduling mechanisms does skypilot use?

[zhenrong-wang]

Hi all, I am new to skypilot. I am curious that, skypilot supports multi-task/job scheduling in a cluster. Does it use any external schedulers such as SLURM? If not, what are the scheduling strategies skypilot support? Any priority management?

Thanks a lot.

[Michaelvll]

Hi @zhenrong-wang, thanks for the question! We currently just use a very simple FIFO scheduler to schedule the job, i.e., jobs will be in PENDING state if a job submitted earlier is still PENDING. Please see our FIFOScheduler for more details:

skypilot/sky/skylet/job_lib.py

Line 266 in 5e6b39c

class FIFOScheduler(JobScheduler):

[zhenrong-wang]

Thanks a lot for the reply. From the code, I'd guess the job management is based on the Ray (maybe I am totally wrong because I an pretty new to skypilot). Given the fact that Ray doesn't fully support multi-tenancy, I'd dive deep to see how skypilot solve the requirements for multi-job & multi-user, such as resoure isolation and permission management.

Does skypilot support these features? I'm thinking about implementing a for-production AI environment based on skypilot, which is really a fantastic "anycloud" solution.

I may also consider connect skypilot with other clouds (such as Linode, Aliyun, or others). I'd be happy to contribute.

[Michaelvll]

We do have Ray for resource book keeping, but not using it for scheduling, as ray does not guarantee the order for job to start.

For multi-tenancy, we actually just added some preliminary support in #4660, where we use the user id for distinguishing different users. We don’t have isolation support at the moment.

Really happy to discuss more on potential collaboration. Please feel free to join our slack channel: https://slack.skypilot.co

[zhenrong-wang]

Good to know the fresh update.

So, by "We don’t have isolation support at the moment", does it mean that any users can see others' resources in a shared cluster? E.g. user_a is able to check user_b's directory and workloads when connecting (e.g. ssh) to the cluster?

Thanks.

[Michaelvll]

Yep, for a same cluster, we assume it is a trusted env.

…

________________________________ From: zhenrong-wang ***@***.***> Sent: Thursday, February 6, 2025 9:08:57 PM To: skypilot-org/skypilot ***@***.***> Cc: Zhanghao Wu ***@***.***>; Comment ***@***.***> Subject: Re: [skypilot-org/skypilot] What scheduling mechanisms does skypilot use? (Discussion #4666) Good to know the fresh update. So, by "We don’t have isolation support at the moment", does it mean that any users can see others' resources in a shared cluster? E.g. user_a is able to check user_b's directory and workloads when connecting (e.g. ssh) to the cluster? Thanks. — Reply to this email directly, view it on GitHub<#4666 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABTQXJMV2KXQISVG4DSLUMD2OQ5WTAVCNFSM6AAAAABWVBD2RWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBYHE4DOMY>. You are receiving this because you commented.Message ID: ***@***.***>

[zhenrong-wang]

Good to know. Thanks!