Figure out how to enable Conda environments inside the shell

[certik]

At first at least some workaround to be able to load an environment.

[certik]

I can launch cmd, activate conda, load an lf environment, then launch shell. This is the $PATH:

C:\Users\ondrejcertik\AppData\Local\miniforge3\envs\lf;
C:\Users\ondrejcertik\AppData\Local\miniforge3\envs\lf\Library\mingw-w64\bin;
C:\Users\ondrejcertik\AppData\Local\miniforge3\envs\lf\Library\usr\bin;
C:\Users\ondrejcertik\AppData\Local\miniforge3\envs\lf\Library\bin;
C:\Users\ondrejcertik\AppData\Local\miniforge3\envs\lf\Scripts;
C:\Users\ondrejcertik\AppData\Local\miniforge3\envs\lf\bin;
C:\Users\ondrejcertik\AppData\Local\miniforge3\condabin;
C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;
C:\Windows\system32;
C:\Windows;
C:\Windows\System32\Wbem;
C:\Windows\System32\WindowsPowerShell\v1.0;
C:\Windows\System32\OpenSSH;
C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;
C:\Program Files\dotnet;
C:\Program Files\Git\cmd;
C:\Users\ondrejcertik\.cargo\bin;
C:\Users\ondrejcertik\.pixi\bin;
C:\Users\ondrejcertik\AppData\Local\Microsoft\WindowsApps;
C:\Users\ondrejcertik\AppData\Local\Programs\Microsoft VS Code\bin;
C:\Users\ondrejcertik\.dotnet\tools;
C:\Users\ondrejcertik\AppData\Local\Programs\AzureAuth\0.8.4

vim from Pixi does not work anymore, I suspect it segfault but I can't see it due to #89.

This is the path in the top-most shell without conda:

C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;
C:\Windows\system32;
C:\Windows;
C:\Windows\System32\Wbem;
C:\Windows\System32\WindowsPowerShell\v1.0\;
C:\Windows\System32\OpenSSH\;
C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;
C:\Program Files\dotnet\;
C:\Program Files\Git\cmd;
C:\Users\ondrejcertik\.cargo\bin;
C:\Users\ondrejcertik\.pixi\bin;
C:\Users\ondrejcertik\AppData\Local\Microsoft\WindowsApps;
C:\Users\ondrejcertik\AppData\Local\Programs\Microsoft VS Code\bin;
C:\Users\ondrejcertik\.dotnet\tools;
C:\Users\ondrejcertik\AppData\Local\Programs\AzureAuth\0.8.4

This can then be enabled via #90 (comment) and #94 (comment).

[certik]

The workaround to load environments is to run cmd.exe, load a conda environment, then run shell and save $PATH to a file. Then explicitly list all paths, and set CONDA_PREFIX. Then manually load this file via source. That works very well, can even do mamba install / uninstall.

[certik]

Specifically:

~$ cmd
Microsoft Windows [Version 10.0.22631.4169]
(c) Microsoft Corporation. All rights reserved.

C:\Users\ondrejcertik>AppData\Local\miniforge3\Scripts\activate.bat

(base) C:\Users\ondrejcertik>bin\shell
~$ echo $PATH
C:\Users\ondrejcertik\AppData\Local\miniforge3;C:\Users\ondrejcertik\AppData\Local\miniforge3\Library\mingw-w64\bin;C:\Users\ondrejcertik\AppData\Local\miniforge3\Library\usr\bin;C:\Users\ondrejcertik\AppData\Local\miniforge3\Library\bin;C:\Users\ondrejcertik\AppData\Local\miniforge3\Scripts;C:\Users\ondrejcertik\AppData\Local\miniforge3\bin;C:\Users\ondrejcertik\AppData\Local\miniforge3\condabin;C:\Users\ondrejcertik\bin;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Program Files\dotnet;C:\Program Files\Git\cmd;C:\Users\ondrejcertik\.cargo\bin;C:\Users\ondrejcertik\.pixi\bin;C:\Users\ondrejcertik\AppData\Local\Microsoft\WindowsApps;C:\Users\ondrejcertik\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\ondrejcertik\.dotnet\tools;C:\Users\ondrejcertik\AppData\Local\Programs\AzureAuth\0.8.4

Then we create a shell script ~/.shell-conda like this:

export PATH='C:\Users\ondrejcertik\AppData\Local\miniforge3;C:\Users\ondrejcertik\AppData\Local\miniforge3\Library\mingw-w64\bin;C:\Users\ondrejcertik\AppData\Local\miniforge3\Library\usr\bin;C:\Users\ondrejcertik\AppData\Local\miniforge3\Library\bin;C:\Users\ondrejcertik\AppData\Local\miniforge3\Scripts;C:\Users\ondrejcertik\AppData\Local\miniforge3\bin;C:\Users\ondrejcertik\AppData\Local\miniforge3\condabin;C:\Users\ondrejcertik\bin;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Program Files\dotnet;C:\Program Files\Git\cmd;C:\Users\ondrejcertik\.cargo\bin;C:\Users\ondrejcertik\.pixi\bin;C:\Users\ondrejcertik\AppData\Local\Microsoft\WindowsApps;C:\Users\ondrejcertik\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\ondrejcertik\.dotnet\tool%;C:\Users\ondrejcertik\AppData\Local\Programs\AzureAuth\0.8.4'

And source it like:

source ~/.shell-conda

In order to make such "path" update scripts additive, we need to fix #130 first.

[certik]

Here is a workaround how to get conda working in the shell: #69 (comment).

[certik]

On macOS, here is how to activate a conda environment using zsh:

~$ which myst
myst not found
~$ zsh -c 'source ~/miniforge3/bin/activate && conda activate myst && ~/bin/shell'
~$ which myst
/Users/ondrej/miniforge3/envs/myst/bin/myst

[traversaro]

I have been using prefix-dev/shell on Windows for the last week, and it has been working quite nicely, thanks a lot for all the work on it! I see why conda activate does not work, but just as a feedback not being able to do conda activate is what I think is one of the main limitations in suggesting prefix-dev/shell to existing Windows users.

fyi @S-Dafarra

[certik]

@traversaro I agree, we need a solution. I started using shell on all platforms, and this is the number one issue everywhere.

I got the conda working in the shell yesterday, but I am not on my Windows machine right now, but roughly I created a file ~/bin/activate_myst.bat, I have my ~/bin in my $PATH, and inside the file I have:

call \path\to\conda\Scripts\activate
call conda activate myst
\path\to\shell --norc

and I launch it simply by calling activate_myst. It works well. I have a similar script to activate MSVC.

I think we just need to ship these scripts.

Alternatively, I am not 100% sure what exactly "conda activate" does, but it might be as simple as setting a few environment variables, in which case we can figure out an equivalent solution for the shell.

[certik]

Let's figure out how to get conda working, and the same approach will work for other such tools.

Conda is using the following bash script that gets sourced

__conda_activate() {
    if [ -n "${CONDA_PS1_BACKUP:+x}" ]; then
        # Handle transition from shell activated with conda <= 4.3 to a subseque
nt activation
        # after conda updated to >= 4.4. See issue #6173.
        PS1="$CONDA_PS1_BACKUP"
        \unset CONDA_PS1_BACKUP
    fi
    \local ask_conda
    ask_conda="$(PS1="${PS1:-}" __conda_exe shell.posix "$@")" || \return
    \eval "$ask_conda"
    __conda_hashr
}

This effectively just does the following:

$ miniforge3/bin/conda shell.posix activate lf
PS1='(lf) '
export PATH='/Users/ondrej/miniforge3/envs/lf/bin:/Users/ondrej/miniforge3/condabin:/Users/ondrej/.pixi/bin:/Users/ondrej/.cargo/bin:/Users/ondrej/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Ghostty.app/Contents/MacOS'
export CONDA_PREFIX='/Users/ondrej/miniforge3/envs/lf'
export CONDA_SHLVL='1'
export CONDA_DEFAULT_ENV='lf'
export CONDA_PROMPT_MODIFIER='(lf) '
export CONDA_EXE='/Users/ondrej/miniforge3/bin/conda'
export _CE_M=''
export _CE_CONDA=''
export CONDA_PYTHON_EXE='/Users/ondrej/miniforge3/bin/python'

We can now set it into a variable:

$ ask_conda="$(miniforge3/bin/conda shell.posix activate lf)"
~$ echo $ask_conda
PS1='(lf) ' export PATH='/Users/ondrej/miniforge3/envs/lf/bin:/Users/ondrej/miniforge3/condabin:/Users/ondrej/.pixi/bin:/Users/ondrej/.cargo/bin:/Users/ondrej/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Ghostty.app/Contents/MacOS' export CONDA_PREFIX='/Users/ondrej/miniforge3/envs/lf' export CONDA_SHLVL='1' export CONDA_DEFAULT_ENV='lf' export CONDA_PROMPT_MODIFIER='(lf) ' export CONDA_EXE='/Users/ondrej/miniforge3/bin/conda' export _CE_M='' export _CE_CONDA='' export CONDA_PYTHON_EXE='/Users/ondrej/miniforge3/bin/python'

But there is a bug that it concatenates the lines. Another issue is that we don't have an eval yet.

So as a workaround for both of these bugs we can do:

$ miniforge3/bin/conda shell.posix activate lf > /tmp/tmp.sh && source /tmp/tmp.sh

That seems to do the trick! It works for me.

The prompt does not update, since we don't support the PS1 mechanism yet, but everything else seems to work.

To deactivate, follow the same approach:

$ conda shell.posix deactivate > /tmp/tmp.sh && source /tmp/tmp.sh

On Windows we need to see if shell.posix sets the Windows paths correctly. If not, we either have to postprocess the paths, or extract them from:

$ conda shell.powershell activate lf
$Env:PATH = "/Users/ondrej/miniforge3/envs/lf/bin:/Users/ondrej/miniforge3/condabin:/Users/ondrej/.pixi/bin:/Users/ondrej/.cargo/bin:/Users/ondrej/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Ghostty.app/Contents/MacOS"
$Env:CONDA_PREFIX = "/Users/ondrej/miniforge3/envs/lf"
$Env:CONDA_SHLVL = "1"
$Env:CONDA_DEFAULT_ENV = "lf"
$Env:CONDA_PROMPT_MODIFIER = "(lf) "
$Env:CONDA_EXE = "/Users/ondrej/miniforge3/bin/conda"
$Env:_CE_M = ""
$Env:_CE_CONDA = ""
$Env:CONDA_PYTHON_EXE = "/Users/ondrej/miniforge3/bin/python"

One can also use shell.fish, shell.xonsh. There is also shell.cmd.exe but it doesn't seem to print anything on macOS.

To automate this, we need "aliases with arguments", so that I can make an alias as "alias conda_activate=miniforge3/bin/conda shell.posix activate $1 > /tmp/tmp.sh && source /tmp/tmp.sh". However, it seems that in Bash one uses functions for that. So we might need to implement functions after all (#207). The tricky part is the "source" at the end, which forces us to do this programming in the shell itself. The alternative would be to implement some magics in the shell that would allow 3rd party programs (say in Python) to leave around the above tmp.sh file which the shell would then source afterwards automatically on some signal (error code?). Then we can implement a simple Python or Rust script to extract what needs to happen from conda, and leave around the tmp.sh file.

@wolfv, @prsabahrami any ideas what we should do?

[certik]

Ok, here is the best way so far that I got working, only tested on macOS so far: conda-incubator/conda-spawn#9 (comment). Works like a charm!

[certik]

@traversaro I figured it out on both macOS and Windows, screenshots here: conda-incubator/conda-spawn#9 (comment). I am using conda spawn -n lf and it just works on both platforms in the shell. You must put export SHELL=shell into your ~/.shellrc.

[certik]

Things are greatly simplified now. Setup:

Merge these PRs (check them out, install them into your base conda environment):

• Add support for (prefix) shell sarugaku/shellingham#92
• Add support for the (prefix) shell conda-incubator/conda-spawn#9

Hopefully they'll get merged upstream soon.

On Windows put this into your ~/bin:

$ cat ~/bin/conda.bat
@shell -c "C:\Users\ondrejcertik\AppData\Local\miniforge3\Scripts\conda.exe %*"

Then:

~/repos/conda-spawn(shell)$ conda spawn myst
(myst) ~/repos/conda-spawn(shell)$ which myst
C:\Users\ondrejcertik\AppData\Local\miniforge3\envs\myst\Scripts\myst.exe

Notice the automatic prompt modification with the environment name, etc. Everything just works.

You can also launch programs directly from the environment:

~/repos/conda-spawn(shell)$ conda spawn myst -- myst --help