docker-image does not take the python tools from the moose envs

[mcacace]

Check these boxes if you have followed the posting rules.

• I have consulted the Posting Guidelines.
• I have searched the Discussions Forum and MOOSE Framework Troubleshooting and have not found what I was looking for
• Q&A Getting Started is the most appropriate category for my question (trouble installing, beginner user, ...)

Issue or question about MOOSE

dear all,
while trying to set up a CI/CD pipeline in git for a moose-based application, we rely on the docker image installation (moose-dev). However we noticed that all python tools and libraries are not those from the moose env, but those from the conda base env (which lacks all additional libraries like numpy, pandas etc).
In order to compile and test both moose and our moose-based app we have to create a dockerfile and built an image based on moose-dev, and install all additional python tools/libraries in the base env, otherwise when we run the container those are not available.
Are we doing anything wrong or rather is something not properly working with the current docker image?
Thanks for any feedback,
mauro

(Optional) code in question / simulation log / errors

No response

Encountering Errors? Please include diagnostic output

No response

[loganharbour]

Our docker image is a bit odd. We build everything with apptainer and then convert to docker, which is a bit of a manual process. Part of this includes the environment setup. I do plan on transitioning everything to a docker build at some point, but we're not there yet.

In the meantime, at the beginning of each step you should be able to run:

source /environment

which should set everything you need.

Let me know if this doesn't work.

[mcacace]

Thanks @loganharbour for your quick reply.
We tried to source the environment, but it does not work properly, still the compiler hits the base python (guess it has to do with setting the PATH in the source script). If we then update the PATH to have the target python from the moose env, we run into a conflict between different python versions.
This said, I think our best choice would be to go for our manual installation of the python tools inside the image.
Mauro

[loganharbour]

Where are you doing this? Gitlab? GitHub? Can you share an excerpt of the workflow?

I have examples of other external collaborators using both GitLab CI and GitHub Actions without issue, using the moose-dev docker images for testing.

[somayehabdi216]

Thanks @loganharbour for your quick reply.

We are running the pipeline in GitLab, and I have attached the pipeline configuration to this discussion.
Inside the container, we need to run the following command to ensure the required Python packages are available. Otherwise, the Python environment from the moose Conda environment is not used correctly:
python3 -m pip install numpy scipy matplotlib pandas pyyaml Jinja2

Best,
Somayeh
gitlab-ci (1).yml

[loganharbour]

I've done some digging here. The issue is that we're not in bash at start up which is screwing up conda's activation scripts. I should be able to fix this in the released container soon.

The following worked for me (using the container myself):

$ docker run --rm -it idaholab/moose-dev:latest /bin/bash
(moose) bash-4.4# conda deactivate
bash-4.4# source /opt/miniforge3/etc/profile.d/conda.sh
bash-4.4# conda activate moose
(moose) bash-4.4# echo $PATH
/opt/miniforge3/envs/moose/bin:/opt/wasp/bin:/opt/code-server/bin:/opt/miniforge3/bin:/opt/openmpi/bin:/opt/rh/gcc-toolset-13/root/usr/bin:/opt/wasp/bin:/opt/code-server/bin:/opt/miniforge3/condabin:/opt/miniforge3/bin:/opt/openmpi/bin:/opt/rh/gcc-toolset-13/root/usr/bin:/usr/share/Modules/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/bin
(moose) bash-4.4# which python
/opt/miniforge3/envs/moose/bin/python

With that, can you please try adding those commands at the beginning of your step:

conda deactivate
source /opt/miniforge3/etc/profile.d/conda.sh
conda activate moose

and report back?

With this: python should be correct, pip should be correct, and conda (for conda install) should also be correct.

[somayehabdi216]

Many thanks for your support.

I ran the suggested commands, and the Python tools are now correctly coming from the moose environment. However, the issue is that the Python version used during compilation does not match the Python version at runtime.

I have included the error message below for your reference:

:491: RuntimeWarning: compile time Python version 3.13 of module 'hit' does not match runtime version 3.14
Segmentation fault (core dumped)

[loganharbour]

That happens when moose is built with a different python than it is ran with. Thus, the environment during the build stage is different than the run stage. Did you also run the commands above during the build step?

[somayehabdi216]

Yes, that happened because of the issue you mentioned. Now, it works for our moose-based app.

[loganharbour]

I've made an issue: #32803

If you subscribe to it, you'll get a notification when it's closed and complete. When it's complete, you'll no longer need to perform those extra steps for environment activation.

I anticipate we will close this within the next week to two weeks.