Support for file creation from conda environments (including conda packed ones).

[cpelley]

I've created a script called copy_conda_env.sh that simplifies the process of copying both packaged and standard Conda environments from a source to a destination. The destination can also be either packed or unpacked, and both source and destination paths can be local or remote.
When a standard Conda environment is provided as the source, the script creates a temporary environment containing conda-pack, to pack the source (on the source host), which is then transferred and unpacked at the destination as needed.
This script is already demonstrated in one of our workflows and helps ensure consistent environments usage across platforms. It also centralises environment management, making it easier to maintain and update.

Usage: copy_conda_env.sh <remote_source_host>:<source_env_path_or_tarball> [<remote_target_host>:<dest_env_path_or_tarball>]

Would there be appetite for me to turn such capability into rose file-creation capability?

[cpelley]

@oliver-sanders, @benfitzpatrick you may have interest here?

[oliver-sanders]

An interesting idea, thanks for reaching out.

I think it would be reasonable to add this functionality as a Rose scheme hander. It might be advisable to wrap the existing rsync handler for the remote transfer step. For an example of what this would involve, see #2754

Mid/long term, I suspect that more comprehensive Conda environment management at the Cylc level might be desirable (i.e, you give us a conda env file, we provision the environment for you), as conda-pack requires you to have already created the env locally for installation onto the remote platform which pushes infrastructure into the workflow / account setup.

[cpelley]

Thanks for your reply @oliver-sanders.

...we provision the environment for you...

Well in so far as conda environment creation via an env file, I can add this to the script easily enough.

This reminds me of thoughts I have had concerning mechanisms for caching support for rose (perhaps I'll raise a separate issue for this if I can gather my thoughts on this).

......which pushes infrastructure into the workflow / account setup.

Ah, sounds like you are thinking in terms of environment creation and deployment via a cylc mechanism before the workflow is running is that right?

It might be advisable to wrap the existing rsync handler for the remote transfer step.

What are your thoughts regarding drawing the line between external script responsibilities VS rose responsibilities?
I'm thinking that such capability would be useful both via a Rose scheme hander as well as via standalone usage.

Cheers

[oliver-sanders]

I can add this to the script easily enough.

Would suggest keeping any Rose PR to the conda-pack transfer functionality. If env management is going to be introduced, it would be best achieved from the Cylc side and would require more thought from the team.

Ah, sounds like you are thinking in terms of environment creation and deployment via a cylc mechanism before the workflow is running is that right?

More or less (though it would likely be done in auto-generated R1 tasks whilst the workflow is running). We haven't explored or agreed this concept yet, my reason for raising it is that if we were going to encapsulate environment management, it would be best done in Cylc (where envs can be tethered to workflow lifespan or Cylc managed caches). Cylc environment management could potentially look something along the lines of this:

[environments]
  [[foo]]
    type = conda
    file = etc/envs/foo.yml
  [[bar]]
    type = docker
    file = etc/envs/bar/Dockerfile
    # url = dockerhub...

[runtime]
  [[FOO]]
    script = rose task-run  # CMD to run inside Conda env
    environment = foo
  [[BAR]]
    script = ...  # args to provide to container entry point
    environment = bar

caching support

There would be various options for this, global share dirs, package manager level caching (e.g micromamba can symlink packages from a cache), etc.

What are your thoughts regarding drawing the line between external script responsibilities VS rose responsibilities?

Rose doesn't want to get involved in application specifics so should only work with generic concepts. However, environments are a generic concept, Rose already configures environment variables (Conda envs are just a lump of filesystem activated by setting a couple of env vars).

I'm thinking that such capability would be useful both via a Rose scheme hander as well as via standalone usage.

Sounds good. In fact it may actually be a functional requirement to implement this behind the Rose CLI anyway (CLI handles the application packaging making it easier to call remotely, etc).