conda: propagate remote micromamba path to Runner-spawned child processes#94
Merged
Merged
Conversation
added 2 commits
June 18, 2026 21:16
…sses When a step bootstraps via _install_remote_conda(), micromamba lands at ../conda_env/micromamba — a path not on PATH. If Runner() is then called inside that step (flow-of-flows pattern), the spawned inner flow creates a new Conda(mode="local") instance that calls _ensure_local_conda(), which searches only PATH and ~/.local/bin via shutil.which(). It never finds the binary that was just installed, producing "No micromamba binary found". Fix: set _METAFLOW_CONDA_MICROMAMBA in os.environ after _install_remote_conda() completes. _ensure_local_conda() now checks this env var first; if the file exists, it reuses the remote binary directly (setting conda_executable_type and is_non_conda_exec accordingly) without PATH lookup. Child processes inherit the env var automatically, so multiple Runner calls in the same step all work without any user-side workaround. Fixes #87
The previous commit incorrectly used os.environ to advertise the micromamba path from _install_remote_conda(). That runs inside the remote_bootstrap.py subprocess; os.environ changes in a child process die when that process exits and are never visible to the step's Python process. Switch to Option 3: directly probe the two paths where _install_remote_conda() always places the binary (../conda_env/micromamba, ./conda_env/micromamba). These paths are deterministic from _install_remote_conda() and the step process shares the same CWD. No cross-process coordination needed.
npow
commented
Jun 18, 2026
npow
left a comment
npow
left a comment
Collaborator
Author
There was a problem hiding this comment.
LGTM. The updated Option 3 implementation is correct: path probing with isfile+X_OK guards avoids false positives on local runs, the fast-path bins match what _install_remote_conda() sets directly (only conda+micromamba, consistent with _install_remote_conda behavior), and there's no env-var propagation dependency. Fixes the process boundary issue from the first version.
added 4 commits
June 18, 2026 21:25
Covers the flow-of-flows regression: _ensure_local_conda() should find micromamba installed by _install_remote_conda() at ../conda_env/ or ./conda_env/ without it being on PATH. Three cases: - Binary found at ../conda_env/micromamba (parent dir writable, normal case) - Binary found at ./conda_env/micromamba (fallback) - Non-executable file in conda_env/ is skipped and falls through to which()
On macOS, tempfile.TemporaryDirectory() returns /var/folders/... but os.getcwd() after os.chdir() resolves the symlink to /private/var/folders/... causing path comparison assertions to fail. Resolve base with os.path.realpath() so expected paths match what _ensure_local_conda() stores.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #87 — conda does not find installed micromamba when using
Runnerinside aFlowSpec.Root cause:
_install_remote_conda()(run inside theremote_bootstrap.pysubprocess) installs micromamba to../conda_env/micromamba(not onPATH). WhenRunner(...).run(...)is called inside the same step, the step's Python process creates a newConda(mode="local")instance._ensure_local_conda()searches onlyPATHand~/.local/binviashutil.which()— it never finds the binary that the bootstrap installed.Fix (Option 3):
_ensure_local_conda()now probes the two deterministic paths that_install_remote_conda()always uses (../conda_env/micromamba,./conda_env/micromamba) before falling back towhich(). Both the bootstrap subprocess and the step process share the same CWD, so the probe is reliable. If the binary is found and executable, it's used directly withconda_executable_type="micromamba"andis_non_conda_exec=True— matching the state that_install_remote_conda()establishes. All other cases fall through to the originalwhich()logic unchanged.Why not an env var? Setting
os.environin_install_remote_conda()only affects theremote_bootstrap.pysubprocess; those changes die when that process exits and are never visible to the step's Python process. The env-var-via-file approach (write path → bashexport→ child inherits) would work but requires touchingremote_bootstrap.pyandconda_environment.py:bootstrap_commands. Path probing is simpler and equally correct since_install_remote_conda()always uses the same two candidate paths.Test plan
@condaon the inner flow, outer step on Kubernetes (no micromamba on image PATH) — no longer fails with "No micromamba binary found"../conda_env/micromambawon't exist locally so path probing falls through to the originalwhich()pathRunnercalls in the same step all find the binary via the path probe