[Core/Docs] Add Warning for Lost File Mount on RollingUpdate w/ Consolidation Mode

[lloyd-brown]

This PR is a follow up to #8333 adding a warning to users when they have a file mount or workdir when running with consolidation mode and rolling update without a cloud bucket configured.

WARNING: File mounts or workdir detected with rolling update strategy enabled and no jobs bucket configured. These files will be stored locally and may be lost during API server updates. To persist files across updates, configure a cloud storage bucket in your SkyPilot config under `jobs.bucket`.'

It also adds a warning to the docs https://docs.skypilot.co/en/rollingupdate-docs/reference/api-server/api-server-upgrade.html#upgrade-strategy.

Tested (run the relevant ones):

• Code formatting: install pre-commit (auto-check on commit) or bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: /smoke-test (CI) or pytest tests/test_smoke.py (local)
• Relevant individual tests: /smoke-test -k test_name (CI) or pytest tests/test_smoke.py::test_name (local)
• Backward compatibility: /quicktest-core (CI) or pytest tests/smoke_tests/test_backward_compat.py (local)

[gemini-code-assist]

Summary of Changes

Hello @lloyd-brown, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical data persistence concern for users leveraging SkyPilot's API server with rolling updates and consolidation mode. It introduces a proactive warning system, both at runtime and within the official documentation, to alert users about the potential loss of local file mounts and workdirs if a cloud storage bucket is not configured. The changes aim to prevent unexpected data loss and guide users toward best practices for data persistence during API server upgrades.

Highlights

• New Warning Mechanism: A new warning function, _warn_file_mounts_rolling_update, has been implemented and integrated into the job launch process. This function checks for specific conditions (rolling update enabled, consolidation mode active, no jobs bucket configured, and presence of file mounts or workdirs) and issues a prominent warning to the user if data loss is a risk.
• Documentation Update: The documentation for API server upgrades has been updated with a new warning section. This section explicitly informs users about the potential loss of locally stored file mounts and workdirs during a rolling update when consolidation mode is enabled and no cloud storage bucket is configured, advising them to configure jobs.bucket for persistence.
• Environment Variable for Rolling Update: A new environment variable, SKYPILOT_ROLLING_UPDATE_ENABLED, has been introduced and is set to 'true' in the API server deployment. This variable serves as a flag to indicate when the rolling update strategy is active, which is then used by the new warning mechanism.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request effectively addresses a potential data loss scenario by adding a warning when using file mounts or workdirs with rolling updates in consolidation mode without a configured cloud bucket. The changes are well-implemented across the Helm chart, documentation, and core logic. The new warning is clear and provides a direct solution for the user. My review includes a minor suggestion to improve code conciseness.

[lloyd-brown]

/build-docs

[github-actions]

✅ ReadTheDocs build triggered for branch rollingupdate-docs

The documentation will be available at: https://docs.skypilot.co/en/rollingupdate-docs/

[Michaelvll]

Thanks @lloyd-brown! See the comments : )

[Michaelvll]

Thanks @lloyd-brown! Some minor edits.

[Michaelvll]

Thanks @lloyd-brown! LGTM

[Michaelvll]

Suggested change

	When using consolidation mode with a remote :ref:`SkyPilot API server with RollingUpdate upgrade strategy <sky-api-server-upgrade-strategy>`, any file mounts or workdirs that uploads local files/folders of the managed jobs will be lost during a rolling update. To address that, configure a cloud bucket via :ref:`config-yaml-jobs-bucket` in your :ref:`SkyPilot config <config-yaml>` to persist them.
	When using consolidation mode with a remote :ref:`SkyPilot API server with RollingUpdate upgrade strategy <sky-api-server-upgrade-strategy>`, any file mounts or workdirs that uploads local files/folders of the managed jobs will be lost during a rolling update. To address that, use :ref:`bucket <todo-link>`, :ref:`volume <todo-link>`, or :ref:`git <todo-link>`; or, configure a cloud bucket for all local files via :ref:`config-yaml-jobs-bucket` in your :ref:`SkyPilot config <config-yaml>` to persist them.
	.. code-block::
	jobs:
	bucket: s3://xxx

[Michaelvll]

Suggested change

	f'{colorama.Fore.YELLOW}WARNING: Local file mounts or workdir detected '
	'with rolling update enabled for API server. '
	'To persist files across API server restarts/update, configure a cloud '
	f'bucket in your SkyPilot config under `jobs.bucket`.'
	f'{colorama.Style.RESET_ALL}')
	f'{colorama.Fore.YELLOW}WARNING: Local file mounts or workdir detected '
	'with rolling update enabled for API server. '
	'To persist files across API server restarts/update, use buckets, volumes, or git '
	'for your file mounts; or, configure a bucket in your SkyPilot config under '
	f'`jobs.bucket`. {colorama.Style.RESET_ALL}')