Testing Anaconda local installation instructions on Cloud-Sandbox head node

[jduckerOWP]

What can be improved? What is missing?

This issue was created here for tracking PR 104 and looping in discussions about the method to construct a local Anaconda environment on the Cloud Sandbox head node. The goal here is to ensure that this method is noninvasive on the head node and will not present any conflicts to other ec2-users if they follow this general method for local Python installations. If issues arise, this issue is intended on discussing further details on this GitHub issue and proposed changes to the PR.

Testing Instructions for the new local Python environment to ensure its compatibility with the master branch of the Cloud Sandbox

1. cd /save/ec2-user/OWP/
2. mkdir UW_test_case; cd UW_test_case
3. git clone https://github.com/ioos/Cloud-Sandbox.git
4. cd Cloud-Sandbox/cloudflow/
5. vi ./job/jobs/roms.template (This template is where a user specifies their model setup/run directory on the Cloud-Sandbox or some s3 bucket mount, the pathway to the model executable, and the required master file specifically for ROMS. The "jobtype" and "ofs" should be left as their defaults for the model template workflow since ROMS has been integrated already)
6. vi ./cluster/configs/OWP/roms.ioos (This template is where a user would specify the computational resources needed for their own given ROMS model run in this case. The "nodetype" is the AWS node instance requested by a user and the "nodecount" is the number of nodes a user will request (granted full resources for each node). All the rest of the configurations here are left as default credentials required for the Cloud-Sandbox AWS account to access resources.
7. vi ./workflows/workflow_template.py (This is the model template workflow driver script to run the entire Cloud-Sandbox process. In this case, the user will want to modify the variable "template_conf" pointing to the pathway of the same file here they've modified in step # 6 (roms.ioos).
8. Finally, the user will now execute the workflow driver script USING THEIR OWN LOCALLY INSTALLED PYTHON ENVIRONMENT: nohup /pathway/to/FirstName_LastName_Anaconda_Installation/anaconda3/bin/python3 ./workflows/workflow_template.py ./job/jobs/roms.template > roms_test.out &
9. To monitor the progress of the Cloud-Sandbox execution, execute the following command: tail -f roms_test.out
10. All done!

[patrick-tripp]

Suggestion: create this in a conda environment and activate it. It won't require the long path to python3.
"/pathway/to/FirstName_LastName_Anaconda_Installation/anaconda3/bin/python3"

https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html

example:
conda activate myenvironment
python3 somepythonscript.py

[jduckerOWP]

@patrick-tripp thanks for the information here you've presented. However, the problem related to this general method is that conda expects you to execute conda init in this case before executing conda activate since the installation method presented here did not modify the ~/.bashrc file to link your anaconda environment. This conda method alone requires a modification to the ec2-user ~/.bashrc file to activate your anaconda environment, which is what we're trying to avoid here in this case for ec2-users altogether (avoid conflicts of overlapping Anaconda environments). @mykelalvis is testing out these methods here right now, but I guess what I'm trying to get at is that users will always have to use the entire pathway of the Python environment (or some alias command) to link their exact Python environment for execution if we are to avoid overlapping Python environments by ec2-users.

[KatherinePowell-NOAA]

PR 104 merged and closed