[Docs] Add API server tuning guide #5176

aylei · 2025-04-10T16:25:41Z

Setups tested:

4c8g (our internal API server also use this setup)
8c16g
16c32g
128c256g

The concurrency of other resource level is calculated based on our code. All the setups not seeing issue with 1000 concurrent jobs launch requests (10 clients x 100 jobs launch per client) and 10 * 10K status requests.

128c256g encountered the ulimit issue due to high executor numbers #5174, addressed after tuning, this should be fixed in another PR so we don't have to mention this in our doc here.

Tested (run the relevant ones):

Code formatting: install pre-commit (auto-check on commit) or bash format.sh
Any manual or new tests for this PR (please specify below)
All smoke tests: /smoke-test (CI) or pytest tests/test_smoke.py (local)
Relevant individual tests: /smoke-test -k test_name (CI) or pytest tests/test_smoke.py::test_name (local)
Backward compatibility: /quicktest-core (CI) or pytest tests/smoke_tests/test_backward_compat.py (local)

Signed-off-by: Aylei <[email protected]>

aylei · 2025-04-11T10:54:42Z

docs/source/reference/api-server/api-server-tunning.rst

+Queuing requests and polling status asynchronously
+--------------------------------------------------
+
+There is no limit on the number of queued requests. So in addition to increasing the allocated resources to improve the maximum concurrency, you can also submit requests with ``--async`` flag and poll the status asynchronously to avoid blocking. For example:


The queue length is verified in #5175

aylei · 2025-04-11T10:55:03Z

docs/source/reference/api-server/api-server-tunning.rst

+
+    sky api cancel <requst_id>
+
+Avoid concurrent logs requests


We can remove this chapter after log optimization completed

Michaelvll · 2025-04-17T05:48:58Z

Preview: https://docs.skypilot.co/en/apiserver-tuning/docs/index.html

Michaelvll

Thanks @aylei! I like the doc. Left some comments.

Michaelvll · 2025-04-17T05:33:23Z

charts/skypilot/values.yaml

+    limits:
+      cpu: "4"
+      memory: "8Gi"


Do we have to set it, just want to reduce the chance that API server is killed by k8s due to slight out of memory?

Explained in the latter dropdown

docs/source/reference/api-server/api-server.rst

docs/source/reference/api-server/api-server-tunning.rst

Michaelvll · 2025-04-17T07:16:02Z

docs/source/reference/api-server/api-server-tunning.rst

+    d9c95c7e-d248-4a7f-b72e-636511405357  alice            sky.jobs.launch  a few secs ago  PENDING
+    767182fd-0202-4ae5-b2d7-ddfabea5c821  alice            sky.jobs.launch  a few secs ago  PENDING
+    5667cff2-e953-4b80-9e5f-546cea83dc59  alice            sky.jobs.launch  a few secs ago  RUNNING
+


Add a title: Check logs for a request

Michaelvll · 2025-04-17T07:16:13Z

docs/source/reference/api-server/api-server-tunning.rst

+    $ sky api logs <request_id>
+
+If the request is stuck according to the log, e.g. retrying to launch VMs that is out of stock, you can cancel the request with:
+


Add a title: Cancel a request

docs/source/reference/api-server/api-server-tunning.rst

Co-authored-by: Zhanghao Wu <[email protected]>

Signed-off-by: Aylei <[email protected]>

aylei · 2025-04-21T09:06:35Z

@Michaelvll @concretevitamin ping for another look, thanks!

concretevitamin

Thanks @aylei, some comments.

concretevitamin · 2025-04-23T23:05:16Z

docs/source/reference/api-server/api-server-tunning.rst

+* ``Long-running request``: request that takes long time and more resources to run, including ``launch``, ``exec``, ``jobs.launch``, etc.
+* ``Short-running request``: request that takes short time or less resources to run, including ``status``, ``logs``, etc.


Suggested change

* ``Long-running request``: request that takes long time and more resources to run, including ``launch``, ``exec``, ``jobs.launch``, etc.

* ``Short-running request``: request that takes short time or less resources to run, including ``status``, ``logs``, etc.

* ``Long-running requests``: requests that take longer time and more resources to run, including ``launch``, ``exec``, ``jobs.launch``, etc.

* ``Short-running requests``: requests that take shorter time or less resources to run, including ``status``, ``logs``, etc.

concretevitamin · 2025-04-23T23:13:51Z

docs/source/reference/api-server/api-server-tunning.rst

+.. note::
+
+    Though a task (or job) can run for any length of time, concurrent tasks does not occupy the concurrency. Because once a task is submitted to the cluster, it will be detached and no longer takes any resources off the API server.


I think we should rephrase this somewhat, and pull it out of note box. How about something like:

Requests are queued and processed by the API server. Therefore, they only take resources off the API server when they are in queue or being processed. Once requests are processed and remote clusters start doing real work, they no longer require API server's resources or count against its concurrency limit.

For example, long-running requests for launch and exec no longer take resources off the API server once a cluster has been provisioned, or once a job has been submitted to a cluster, respectively.

Please check accuracy.

concretevitamin · 2025-04-23T23:26:42Z

docs/source/reference/api-server/api-server-tunning.rst

+
+        .. note:: 
+
+            If you specify a resources that is lower than the minimum recommended resources (4 CPUs with 8GB of memory) for team usage, an error will be raised on ``helm upgrade``. You can specify ``--set apiService.skipResourcesCheck=true`` to skip the check if performance and stability is not an issue for you scenario.


It's a bit confusing that " (4 CPUs with 8GB of memory)" is mentioned as our rec settings, but snippet above doesn't reflect this?

We should mention the rec setting in a 1-sentence paragraph.

concretevitamin · 2025-04-23T23:54:53Z

docs/source/reference/api-server/api-server-tunning.rst

+Queuing requests and polling status asynchronously
+--------------------------------------------------
+
+There is no limit on the number of queued requests, i.e. despite increasing the allocated resources to improve the maximum concurrency, you can also submit requests with :ref:`async<async>` (``--async``) and poll the status asynchronously to avoid blocking. For example:


Suggested change

There is no limit on the number of queued requests, i.e. despite increasing the allocated resources to improve the maximum concurrency, you can also submit requests with :ref:`async<async>` (``--async``) and poll the status asynchronously to avoid blocking. For example:

There is no limit on the number of queued requests. To avoid request blocking, you can either (1) allocate more resources to increase the maximum concurrency (described above), or (2) :ref:`submit requests asynchronously <async>` (``--async``) and poll the status asynchronously.

For example:

Do we mean this?

I still find this revised section & the sec title confusing. What do we really want to say in this section? Lmk and I can try to rephrase.

It feels like "Use asynchronous requests as much as possible"?

concretevitamin · 2025-04-23T23:57:58Z

docs/source/reference/api-server/api-server-tunning.rst

+    767182fd-0202-4ae5-b2d7-ddfabea5c821  alice sky.jobs.launch  a few secs ago  PENDING
+    5667cff2-e953-4b80-9e5f-546cea83dc59  alice sky.jobs.launch  a few secs ago  RUNNING
+
+Check logs for a request


Suggested change

Check logs for a request

Checking the logs of a request

concretevitamin · 2025-04-23T23:58:03Z

docs/source/reference/api-server/api-server-tunning.rst

+    # Replace <request_id> with the actual request id from the ID column
+    $ sky api logs <request_id>
+
+Cancel a request


Suggested change

Cancel a request

Canceling a request

concretevitamin · 2025-04-23T23:58:57Z

docs/source/reference/api-server/api-server-tunning.rst

+Avoid concurrent logs requests
+------------------------------
+
+If you run ``sky logs`` to tail the logs of a task, the log tailing will keep taking off the resources of the API server as long as the task being tailed is running. So concurrent log requests will occupy the concurrency and make other requests to be delayed.


Suggested change

If you run ``sky logs`` to tail the logs of a task, the log tailing will keep taking off the resources of the API server as long as the task being tailed is running. So concurrent log requests will occupy the concurrency and make other requests to be delayed.

If you run ``sky logs`` to tail the logs of a task, the log tailing will keep taking resources off the API server as long as the task being tailed is still running. Thus, concurrent log requests will occupy the concurrency limit and potentially delay other requests.

concretevitamin · 2025-04-23T23:59:43Z

docs/source/reference/api-server/api-server-tunning.rst

+Avoid concurrent logs requests
+------------------------------
+
+If you run ``sky logs`` to tail the logs of a task, the log tailing will keep taking off the resources of the API server as long as the task being tailed is running. So concurrent log requests will occupy the concurrency and make other requests to be delayed.


In this section, both 'task' and 'job' are used; can we keep one?

aylei added 4 commits April 10, 2025 23:58

API server tuning guide

4af9cc8

Signed-off-by: Aylei <[email protected]>

Refinement

7f1b770

Signed-off-by: Aylei <[email protected]>

Update QoS explaination

519faf3

Signed-off-by: Aylei <[email protected]>

Update QoS explaination

e19f3bf

Signed-off-by: Aylei <[email protected]>

aylei requested review from romilbhardwaj, Michaelvll and concretevitamin April 11, 2025 10:48

aylei commented Apr 11, 2025

View reviewed changes

aylei marked this pull request as ready for review April 11, 2025 10:55

Michaelvll removed the request for review from romilbhardwaj April 17, 2025 02:17

Michaelvll approved these changes Apr 17, 2025

View reviewed changes

aylei and others added 3 commits April 18, 2025 16:28

Apply suggestions from code review

3c3cdfc

Co-authored-by: Zhanghao Wu <[email protected]>

Update docs

eb56c7a

Signed-off-by: Aylei <[email protected]>

Refine table look

68a4678

Signed-off-by: Aylei <[email protected]>

aylei requested a review from Michaelvll April 18, 2025 10:16

Fix merge conflicts

fce9775

Signed-off-by: Aylei <[email protected]>

concretevitamin reviewed Apr 23, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Docs] Add API server tuning guide #5176

[Docs] Add API server tuning guide #5176

aylei commented Apr 10, 2025 •

edited

Loading

aylei Apr 11, 2025 •

edited

Loading

aylei Apr 11, 2025

Michaelvll commented Apr 17, 2025

Michaelvll left a comment

Michaelvll Apr 17, 2025

aylei Apr 18, 2025

Michaelvll Apr 17, 2025

Michaelvll Apr 17, 2025

aylei commented Apr 21, 2025

concretevitamin left a comment

concretevitamin Apr 23, 2025

concretevitamin Apr 23, 2025

concretevitamin Apr 23, 2025

concretevitamin Apr 23, 2025

concretevitamin Apr 23, 2025

concretevitamin Apr 23, 2025

concretevitamin Apr 23, 2025

concretevitamin Apr 23, 2025

concretevitamin Apr 23, 2025

		$ sky api logs <request_id>

		If the request is stuck according to the log, e.g. retrying to launch VMs that is out of stock, you can cancel the request with:

		* ``Long-running request``: request that takes long time and more resources to run, including ``launch``, ``exec``, ``jobs.launch``, etc.
		* ``Short-running request``: request that takes short time or less resources to run, including ``status``, ``logs``, etc.

		.. note::

		Though a task (or job) can run for any length of time, concurrent tasks does not occupy the concurrency. Because once a task is submitted to the cluster, it will be detached and no longer takes any resources off the API server.


		.. note::

		If you specify a resources that is lower than the minimum recommended resources (4 CPUs with 8GB of memory) for team usage, an error will be raised on ``helm upgrade``. You can specify ``--set apiService.skipResourcesCheck=true`` to skip the check if performance and stability is not an issue for you scenario.

	If you run ``sky logs`` to tail the logs of a task, the log tailing will keep taking off the resources of the API server as long as the task being tailed is running. So concurrent log requests will occupy the concurrency and make other requests to be delayed.
	If you run ``sky logs`` to tail the logs of a task, the log tailing will keep taking resources off the API server as long as the task being tailed is still running. Thus, concurrent log requests will occupy the concurrency limit and potentially delay other requests.

[Docs] Add API server tuning guide #5176

Are you sure you want to change the base?

[Docs] Add API server tuning guide #5176

Conversation

aylei commented Apr 10, 2025 • edited Loading

aylei Apr 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Michaelvll commented Apr 17, 2025

Michaelvll left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aylei commented Apr 21, 2025

concretevitamin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aylei commented Apr 10, 2025 •

edited

Loading

aylei Apr 11, 2025 •

edited

Loading