Remove Nextstrain profiles

This is a comprehensive shift from --profile to --configfile,
standardizing the preference started in "Remove my_profiles/" (2b88070)
and already used by the "100k" workflow (which was misleadingly under
"nextstrain_profiles").

Workflow-specific options are translated into command line arguments in
the GitHub Actions workflow. Default options are retained in a default
profile config, a practice used across other pathogen workflows.

Author: victorlin

.github/workflows/ci.yaml

@@ -11,7 +11,7 @@ jobs:
   test-build:
     uses: nextstrain/.github/.github/workflows/pathogen-repo-ci.yaml@v0
     with:
-      build-args: all_regions -j 2 --profile nextstrain_profiles/nextstrain-ci
+      build-args: all_regions -j 2 --configfile nextstrain_configs/gisaid/builds.yaml nextstrain_configs/ci/builds.yaml
 
   test-cram:
     runs-on: ubuntu-latest

.github/workflows/rebuild-100k.yml

@@ -27,7 +27,7 @@ jobs:
           --memory 12GiB \
           . \
             upload \
-            --configfile nextstrain_profiles/100k/config-open.yaml \
+            --configfile nextstrain_configs/100k/config-open.yaml \
             --config "${config[@]}" \
             --set-threads tree=2
       artifact-name: open-build-output

.github/workflows/rebuild-gisaid.yml

@@ -51,5 +51,6 @@ jobs:
             deploy \
             upload \
             --config "${config[@]}" \
-            --profile nextstrain_profiles/nextstrain-gisaid \
+            --configfile nextstrain_configs/gisaid/builds.yaml \
+            --restart-times 2 \
             --set-threads tree=2

.github/workflows/rebuild-open.yml

@@ -52,5 +52,6 @@ jobs:
             deploy \
             upload \
             --config "${config[@]}" \
-            --profile nextstrain_profiles/nextstrain-open \
+            --configfile nextstrain_configs/open/builds.yaml \
+            --restart-times 2 \
             --set-threads tree=2

docs/dev_docs.md

@@ -29,15 +29,15 @@ We do not release new minor versions for new features, but you should document n
 The "core" nextstrain builds consist of a global analysis and six regional analyses, performed independently for GISAID data and open data (currently open data is GenBank data).
 Stepping back, the process can be broken into three steps:
 1. Ingest and curation of raw data. This is performed by the [ncov-ingest](https://github.com/nextstrain/ncov-ingest/) repo and resulting files are uploaded to S3 buckets.
-2. Phylogenetic builds, which start from the files produced by the previous step. This is performed by the profiles `nextstrain_profiles/nextstrain-open` and `nextstrain_profiles/nextstrain-gisaid`. The resulting files are uploaded to S3 buckets by the `upload` rule. 
+2. Phylogenetic builds, which start from the files produced by the previous step. These are configured by files under `nextstrain_configs`. Shared Snakemake execution defaults are defined in `profiles/default/config.yaml`. The resulting files are uploaded to S3 buckets by the `upload` rule. 
 
 
 ### Manually running phylogenetic builds
 
 To run these pipelines locally, without uploading the results:
 ```sh
-nextstrain build . all --profile nextstrain_profiles/nextstrain-open
-nextstrain build . all --profile nextstrain_profiles/nextstrain-gisaid
+nextstrain build . all --configfile nextstrain_configs/open/builds.yaml
+nextstrain build . all --configfile nextstrain_configs/gisaid/builds.yaml
 ```
 You can replace `all` with, for instance, `auspice/ncov_open_global.json` to avoid building all regions.
 The resulting dataset(s) can be visualised in the browser by running `auspice view --datasetDir auspice`.
@@ -49,13 +49,13 @@ You may wish to overwrite these parameters for your local runs to avoid overwrit
 For instance, here are the commands used by the trial builds action (see below):
 ```sh
 nextstrain build . upload deploy \
-    --profile nextstrain_profiles/nextstrain-open \
+    --configfile nextstrain_configs/open/builds.yaml \
     --config \
         S3_DST_BUCKET=nextstrain-staging/files/ncov/open/trial/TRIAL_NAME \
         deploy_url=s3://nextstrain-staging/ \
         auspice_json_prefix=ncov_open_trial_TRIAL_NAME
 nextstrain build . upload deploy \
-    --profile nextstrain_profiles/nextstrain-gisaid \
+    --configfile nextstrain_configs/gisaid/builds.yaml \
     --config \
         S3_DST_BUCKET=nextstrain-ncov-private/trial/TRIAL_NAME \
         deploy_url=s3://nextstrain-staging/ \

docs/src/reference/files.rst

@@ -75,4 +75,4 @@ These files are used to generate the `workflow documentation <https://docs.nexts
 Nextstrain user files
 ~~~~~~~~~~~~~~~~~~~~~
 
-The Nextstrain team maintains user files in the ``ncov/`` repo, under ``nextstrain_profiles/``.
+The Nextstrain team maintains user files in the ``ncov/`` repo, under ``nextstrain_configs/``.

nextstrain_profiles/100k/README.md nextstrain_configs/100k/README.mdnextstrain_profiles/100k/README.md renamed to nextstrain_configs/100k/README.md

@@ -9,19 +9,19 @@ It will upload these files:
 * `s3://nextstrain-ncov-private/100k/metadata.tsv.xz`
 * `s3://nextstrain-ncov-private/100k/sequences.fasta.xz`
 
-While this profile is not recommended to be run locally, you can see what rules would be run via:
+While this workflow configuration is not recommended to be run locally, you can see what rules would be run via:
 
 ```
-snakemake --cores 1 --configfile nextstrain_profiles/100k/config-gisaid.yaml -npf upload --dag | dot -Tpdf > dag-100k-gisaid.pdf
-snakemake --cores 1 --configfile nextstrain_profiles/100k/config-open.yaml -npf upload --dag | dot -Tpdf > dag-100k-open.pdf
+snakemake --cores 1 --configfile nextstrain_configs/100k/config-gisaid.yaml -npf upload --dag | dot -Tpdf > dag-100k-gisaid.pdf
+snakemake --cores 1 --configfile nextstrain_configs/100k/config-open.yaml -npf upload --dag | dot -Tpdf > dag-100k-open.pdf
 ```
 
 To run manually you can trigger the GitHub action (recommended) or run the jobs locally via:
 ```
 nextstrain build --aws-batch --cpus 16 --memory 31GiB --detach . \
-  --configfile nextstrain_profiles/100k/config-gisaid.yaml \
+  --configfile nextstrain_configs/100k/config-gisaid.yaml \
   -f upload
 nextstrain build --aws-batch --cpus 16 --memory 31GiB --detach . \
-  --configfile nextstrain_profiles/100k/config-open.yaml \
+  --configfile nextstrain_configs/100k/config-open.yaml \
   -f upload
 ```