Merge pull request #6975 from mvdbeek/ansible-cvmfs-update

Update CVMFS training material

Author: bgruening

topics/admin/tutorials/apptainer/tutorial.md

@@ -204,17 +204,19 @@ Now, we will configure Galaxy to run tools using Apptainer containers, which wil
 >    ```diff
 >    --- a/group_vars/galaxyservers.yml
 >    +++ b/group_vars/galaxyservers.yml
->    @@ -78,6 +78,9 @@ galaxy_config:
+>    @@ -78,8 +78,11 @@ galaxy_config:
 >         tus_upload_store: "{{ galaxy_tus_upload_store }}"
 >         # CVMFS
->         tool_data_table_config_path: /cvmfs/data.galaxyproject.org/byhand/location/tool_data_table_conf.xml,/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml
+>         tool_data_table_config_path:
+>           - /cvmfs/data.galaxyproject.org/byhand/location/tool_data_table_conf.xml
+>           - /cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml
 >    +    # Tool Dependencies
 >    +    dependency_resolvers_config_file: "{{ galaxy_config_dir }}/dependency_resolvers_conf.xml"
 >    +    container_resolvers_config_file: "{{ galaxy_config_dir }}/container_resolvers_conf.yml"
 >       gravity:
 >         process_manager: systemd
 >         galaxy_root: "{{ galaxy_root }}/server"
->    @@ -117,6 +120,12 @@ galaxy_config_files:
+>    @@ -118,5 +121,11 @@ galaxy_config_files:
 >       - src: files/galaxy/themes.yml
 >         dest: "{{ galaxy_config.galaxy.themes_config_file }}"
 >
@@ -226,7 +228,6 @@ Now, we will configure Galaxy to run tools using Apptainer containers, which wil
 >    +
 >     galaxy_extra_dirs:
 >       - /data
->
 >    {% endraw %}
 >    ```
 >    {: data-commit="Configure the container and dependency resolvers"}

topics/admin/tutorials/cvmfs-manual/tutorial.md

@@ -101,19 +101,31 @@ The configuration is not complex for CVMFS:
 >
 >    This tells CVMFS to mount the Galaxy reference data repository and use a specific location for the cache which is limited to 500MB in size and to use the instance's geo-location to choose the best CVMFS repo server to connect to. You can use the `cvmfs_quota_limit` role variable to control this setting.
 >
+>    If you also want to mount the BRC Analytics pathogen data and Vertebrate Genomes Project repositories, include them in `CVMFS_REPOSITORIES`:
+>
+>    ```
+>    CVMFS_REPOSITORIES="data.galaxyproject.org,brc.galaxyproject.org,vgp.galaxyproject.org"
+>    ```
+>
 > > <tip-title>What is a good size for CVMFS_QUOTA_LIMIT?</tip-title>
 > > In production UseGalaxy.org.au uses 100GB, different sites have different needs and you can make your cache smaller depending on your usage. E.g. if your users only use one dataset from the reference data (e.g. just hg38) then perhaps you don't need such a large cache.
 > {: .tip}
 >
 > 2. Create a `/etc/cvmfs/domain.d/galaxyproject.org.conf` file with the following contents:
 >
 >    ```
->    CVMFS_SERVER_URL="http://cvmfs1-tacc0.galaxyproject.org/cvmfs/@fqrn@;http://cvmfs1-iu0.galaxyproject.org/cvmfs/@fqrn@;http://cvmfs1-psu0.galaxyproject.org/cvmfs/@fqrn@;http://galaxy.jrc.ec.europa.eu:8008/cvmfs/@fqrn@;http://cvmfs1-mel0.gvl.org.au/cvmfs/@fqrn@;http://cvmfs1-ufr0.galaxyproject.eu/cvmfs/@fqrn@"
+>    CVMFS_SERVER_URL="http://cvmfs1-psu0.galaxyproject.org/cvmfs/@fqrn@;http://cvmfs1-iu0.galaxyproject.org/cvmfs/@fqrn@;http://cvmfs1-tacc0.galaxyproject.org/cvmfs/@fqrn@;http://cvmfs1-mel0.gvl.org.au/cvmfs/@fqrn@;http://cvmfs1-ufr0.galaxyproject.eu/cvmfs/@fqrn@"
 >    ```
 >
 >    This is a list of the available stratum 1 servers that have this repo.
 >
-> 3. Create a `/etc/cvmfs/keys/data.galaxyproject.org.pub` file with the following contents:
+> 3. Create the key directory:
+>
+>    ```
+>    sudo mkdir -p /etc/cvmfs/keys/galaxyproject.org
+>    ```
+>
+> 4. Create a `/etc/cvmfs/keys/galaxyproject.org/data.galaxyproject.org.pub` file with the following contents:
 >
 >    ```
 >    -----BEGIN PUBLIC KEY-----
@@ -127,7 +139,23 @@ The configuration is not complex for CVMFS:
 >    -----END PUBLIC KEY-----
 >    ```
 >
-> 4. Make a directory for the cache files
+> 5. Create a `/etc/cvmfs/keys/galaxyproject.org/galaxyproject.org.pub` file with the following contents:
+>
+>    ```
+>    -----BEGIN PUBLIC KEY-----
+>    MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuJZTWTY3/dBfspFKifv8
+>    TWuuT2Zzoo1cAskKpKu5gsUAyDFbZfYBEy91qbLPC3TuUm2zdPNsjCQbbq1Liufk
+>    uNPZJ8Ubn5PR6kndwrdD13NVHZpXVml1+ooTSF5CL3x/KUkYiyRz94sAr9trVoSx
+>    THW2buV7ADUYivX7ofCvBu5T6YngbPZNIxDB4mh7cEal/UDtxV683A/5RL4wIYvt
+>    S5SVemmu6Yb8GkGwLGmMVLYXutuaHdMFyKzWm+qFlG5JRz4okUWERvtJ2QAJPOzL
+>    mAG1ceyBFowj/r3iJTa+Jcif2uAmZxg+cHkZG5KzATykF82UH1ojUzREMMDcPJi2
+>    dQIDAQAB
+>    -----END PUBLIC KEY-----
+>    ```
+>
+>    The BRC Analytics and VGP repositories use this common Galaxy Project key. The `singularity.galaxyproject.org` repository also uses it, and the data repository is expected to transition to it eventually.
+>
+> 6. Make a directory for the cache files
 >
 >    ```
 >    sudo mkdir /srv/cvmfs
@@ -206,4 +234,3 @@ Probe the connection.
 ## Look at the repository
 
 Now to configure Galaxy to use the CVMFS references we have just installed, see [the Ansible tutorial.]({% link topics/admin/tutorials/cvmfs/tutorial.md %}#configuring-galaxy-to-use-the-cvmfs-references)
-

topics/admin/tutorials/cvmfs/tutorial.md

@@ -143,6 +143,8 @@ The Galaxy project supports a few CVMFS repositories.
 | Repository                 | Repository Address              | Contents                                                                         |
 | ----------                 | ------------------              | --------                                                                         |
 | Reference Data and Indices | `data.galaxyproject.org`        | Genome sequences and their tool indices, Galaxy `.loc` files for them as well    |
+| BRC Analytics Data         | `brc.galaxyproject.org`         | Pathogen sequences and indexes available in Galaxy for BRC Analytics             |
+| Vertebrate Genomes Project | `vgp.galaxyproject.org`         | Assemblies and indexes from the Vertebrate Genomes Project                       |
 | Singularity Containers     | `singularity.galaxyproject.org` | Singularity containers for everything in Biocontainers for use in Galaxy systems |
 | Galaxy Main Configuration  | `main.galaxyproject.org`        | The configuration files etc for Galaxy Main (usegalaxy.org)                      |
 
@@ -522,29 +524,40 @@ Now all we need to do is tell Galaxy how to find it! This tutorial assumes that
 >    ```diff
 >    --- a/group_vars/galaxyservers.yml
 >    +++ b/group_vars/galaxyservers.yml
->    @@ -76,6 +76,8 @@ galaxy_config:
+>    @@ -76,6 +76,10 @@ galaxy_config:
 >         # TUS
 >         galaxy_infrastructure_url: "https://{{ inventory_hostname }}"
 >         tus_upload_store: "{{ galaxy_tus_upload_store }}"
 >    +    # CVMFS
->    +    tool_data_table_config_path: /cvmfs/data.galaxyproject.org/byhand/location/tool_data_table_conf.xml,/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml
+>    +    tool_data_table_config_path:
+>    +      - /cvmfs/data.galaxyproject.org/byhand/location/tool_data_table_conf.xml
+>    +      - /cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml
 >       gravity:
 >         process_manager: systemd
 >         galaxy_root: "{{ galaxy_root }}/server"
 >    {% endraw %}
 >    ```
 >    {: data-commit="Add tool_data_table_config_path to group variables" data-ref="gvconf"}
 >
+>    If you have also enabled the BRC Analytics or VGP CVMFS repositories, add their data table configuration files to the same list:
+>
+>    ```yaml
+>    tool_data_table_config_path:
+>      - /cvmfs/data.galaxyproject.org/byhand/location/tool_data_table_conf.xml
+>      - /cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml
+>      - /cvmfs/brc.galaxyproject.org/config/tool_data_table_conf.xml
+>      - /cvmfs/vgp.galaxyproject.org/config/tool_data_table_conf.xml
+>    ```
+>
 >    > This time I'm going to edit the groupvars galaxyservers.yml file. And in
 >    > our Galaxy section. Which is here. Which is here. At the bottom of that
 >    > I'm going to add a variable called tool_data_table_config_path. And I'm
 >    > going to point it to the locations that we um I showed you before. um I
 >    > can't remember what they are off the top of my head but luckily they're
 >    > inside this solution box and so I will just copy them. And paste. And as
 >    > you can see pointing to /cvmfs/data.galaxyproject.org/byhand/location
->    > and then that tool data table conf xml file. And then we have a list
->    > here and we separate it by commas and then we point it to the second
->    > one. Right, so we save this file.
+>    > and then that tool data table conf xml file. And then we have a YAML list
+>    > here and we point it to the second one. Right, so we save this file.
 >    {: .spoken data-visual="terminal" data-ref="gvconf"}
 >
 > 2. Re-run the playbook