Skip to content

[Core] Add provision.conda_auto_activate config option#9944

Open
lloyd-brown wants to merge 5 commits into
masterfrom
lloyd/conda-auto-activate-config
Open

[Core] Add provision.conda_auto_activate config option#9944
lloyd-brown wants to merge 5 commits into
masterfrom
lloyd/conda-auto-activate-config

Conversation

@lloyd-brown

Copy link
Copy Markdown
Collaborator

Summary

SkyPilot auto-activates the conda base environment on clusters that do not use a custom Docker image. This is convenient for conda-based workflows, but it makes base the implicit target of uv/pip commands run in setup/run scripts and interactive shells. For users who manage their own environments (e.g. uv-first projects), this is surprising: a uv pip install from a directory without a .venv silently installs into conda base instead of erroring or using the intended environment.

This PR adds a provision.conda_auto_activate config option so users can keep conda installed but not auto-activated.

  • sky/backends/backend_utils.py: conda_auto_activate now honors provision.conda_auto_activate when set, falling back to the existing default (true, except false for custom Docker images).
  • sky/skylet/constants.py: the post-install conda deactivate guard now also fires when conda_auto_activate is false. This is required because the install commands explicitly run conda activate base; without deactivating, base stays active in the chained provisioning shell and leaks into the Ray runtime (and therefore into user setup/run jobs), even though auto_activate_base is set to false for new shells.
  • sky/utils/schemas.py: schema entry for the new option.
  • Docs (config.rst) + a unit test.

Default behavior is unchanged.

Test plan

  • Added tests/unit_tests/test_backend_utils.py::test_conda_installation_deactivates_base_when_auto_activate_disabled: asserts the rendered conda commands set auto_activate_base false and deactivate base when the option is false, while preserving the existing custom-Docker deactivate behavior.
  • Manual: launched a Kubernetes cluster with provision.conda_auto_activate: false. Verified CONDA_PREFIX is unset and uv resolves cleanly (no longer binds to conda base) in both run commands and interactive SSH, and that a user-created .venv is used correctly. Verified that with the option unset, behavior is unchanged (base still auto-activated).

🤖 Generated with Claude Code

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new configuration option, provision.conda_auto_activate, allowing users to disable the automatic activation of the conda base environment on remote clusters. This is particularly useful for workflows that manage their own environments (e.g., uv-first projects). The changes include documentation updates, schema validation, backend logic to parse the config, and updates to the conda installation commands to deactivate the base environment when requested. Additionally, unit tests were added to verify this behavior. Feedback was provided to simplify the logic for determining the conda_auto_activate value in sky/backends/backend_utils.py to improve readability.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread sky/backends/backend_utils.py Outdated
@lloyd-brown lloyd-brown marked this pull request as ready for review June 25, 2026 17:16

@kevinmingtarja kevinmingtarja left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question, otherwise LGTM.

Comment thread sky/skylet/constants.py
Comment on lines +276 to +277
'if [ "{is_custom_docker}" = "true" ] || '
'[ "{conda_auto_activate}" = "false" ]; then '

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should conda_auto_activate: false also take effect on images that already have conda on PATH?

This whole block — including conda config --set auto_activate_base false — only runs in the which conda short-circuit:

'which conda > /dev/null 2>&1 || '

So on an image where conda is already initialized and on PATH (e.g. the AWS DLAMI), which conda succeeds, the block is skipped, and auto_activate_base is never set to false. New interactive/SSH shells would then still auto-activate base, so the option silently has no effect there. (The in-shell→Ray leak isn't hit in that case, since we also never run our own conda activate base — it's the SSH/interactive path that's left exposed.) Worth handling that path, or at least noting the limitation in the docs?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, fixed in c79ce49. I moved the auto_activate_base setting out of the which conda short-circuit into a stanza that runs whenever conda is available (command -v conda), gated on the user having explicitly set provision.conda_auto_activate so we don't override conda config baked into custom images.

Validated live on a conda-already-on-PATH image (so the install block is skipped): the stanza fires and conda config --show auto_activate_base ends up false. Re-verified the default-image path for no regression.

One caveat I hit while testing, now noted in the docs: images that bake CONDA_PREFIX into the container ENV itself (e.g. continuumio/miniconda3) keep base as the implicit target regardless, since that's the image's environment rather than conda auto-activation. The DLAMI case you named activates base via conda init/auto_activate_base, which this handles.

lloyd-brown and others added 2 commits June 25, 2026 12:01
SkyPilot auto-activates the conda `base` environment on clusters that do
not use a custom Docker image. This is convenient for conda-based
workflows, but it makes `base` the implicit target of `uv`/`pip` run in
setup/run scripts and interactive shells, which is undesirable for
workflows that manage their own environments (e.g. uv-first projects).

Add a `provision.conda_auto_activate` config option (default unchanged)
so users can keep conda installed but not auto-activated. When set to
false, the conda installation commands both set `auto_activate_base
false` and deactivate `base` after setup, so it does not stay active in
the chained provisioning shell and leak into the Ray runtime / user jobs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address review: resolve the boolean once (config value, falling back to
the no-docker-image default) and stringify a single time, instead of
duplicating the 'true'/'false' conversion across branches. No behavior
change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@lloyd-brown lloyd-brown force-pushed the lloyd/conda-auto-activate-config branch from 0d69dc3 to 76f518b Compare June 25, 2026 19:03
lloyd-brown and others added 3 commits June 25, 2026 12:51
Address review: `conda config --set auto_activate_base` previously ran
only inside the `which conda` short-circuit, so on images that already
have conda on PATH (e.g. the AWS DLAMI) the block is skipped and
`provision.conda_auto_activate: false` had no effect on interactive/SSH
shells.

Apply the preference outside the install block too (guarded by
`command -v conda`), but only when the user explicitly set the option,
so we don't override conda config baked into custom images.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Images that set CONDA_PREFIX in the container environment itself (rather
than activating base via conda init) keep base as the implicit target
regardless of the option, since that is not conda auto-activation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants