Merge pull request #324 from xylar/improve-sync-docs

Add docs for `mache sync diags`

Author: xylar

docs/index.md

@@ -11,6 +11,7 @@
 
 users_guide/quick_start
 users_guide/spack/build
+users_guide/sync/diags
 ```
 
 ```{toctree}

docs/users_guide/sync/diags.md

@@ -0,0 +1,179 @@
+# Synchronize diagnostics between machines (`mache sync diags`)
+
+This command copies precomputed E3SM diagnostics (both public and private)
+between supported HPC systems using rsync. A common use is to pull diagnostics
+stored on the LCRC filesystem (Chrysalis) down to another site so local
+post-processing and plotting tools can find them.
+
+The command supports two directions:
+
+- `from <other>`: Pull diagnostics from an LCRC machine (e.g., `chrysalis`) to
+  your current machine
+- `to <other>`: Push diagnostics from your current LCRC machine to another
+  machine
+
+Important constraints:
+
+- LCRC machines are `anvil` and `chrysalis`
+- You may only:
+  - run `to` when you are currently on an LCRC machine; and
+  - run `from` when the other machine is an LCRC machine.
+- If you try to sync between two different LCRC machines, you'll be told to
+  sync with the same machine instead, since the files are local/shared.
+- It is *highly* recommended that you sync `from` an LCRC machine to another
+  HPC system because this allows permissions to be updated after the sync.
+
+---
+
+## Prerequisites
+
+- A valid LCRC/CELS account and username (used below as `<cels_username>`)
+- SSH key-based access configured for each machine from which you will run the
+  sync
+- `mache` installed and configured on the machine where you run the command
+
+### 1) Generate an SSH key (if you don’t already have one)
+
+On each HPC machine where you plan to run the sync:
+
+```bash
+ssh-keygen -t ed25519
+```
+
+Accept the default path (`~/.ssh/id_ed25519`) unless you have a reason to use a
+different one. Don’t share your private key.
+
+### 2) Add your public key to your CELS account
+
+- Copy the content of your public key (typically `~/.ssh/id_ed25519.pub`).
+- Visit https://accounts.cels.anl.gov/ and add it under your account’s SSH
+  keys.
+- Give it a descriptive name (e.g., `andes`, `frontier`, `compy`).
+- Allow a few minutes for the new key to propagate.
+
+### 3) Configure your `~/.ssh/config`
+
+We recommend a control connection and a short host alias for Chrysalis:
+
+```ini
+Host *
+    ControlMaster auto
+    ControlPath ~/.ssh/connections/%r@%h:%p
+    ServerAliveInterval 300
+    ServerAliveCountMax 3
+
+Host chrys
+    Hostname chrysalis.lcrc.anl.gov
+    User <cels_username>
+    ProxyJump <cels_username>@logins.lcrc.anl.gov
+```
+
+Also create the connections directory if it doesn’t exist:
+
+```bash
+mkdir -p ~/.ssh/connections
+chmod 700 ~/.ssh ~/.ssh/connections
+```
+
+#### OLCF (Andes, Frontier) extra settings
+
+Some OLCF systems require explicit auth options. Add these lines to the
+`chrys` host in your SSH config if you’re on Andes/Frontier:
+
+```ini
+Host chrys
+    Hostname chrysalis.lcrc.anl.gov
+    User <cels_username>
+    ProxyJump <cels_username>@logins.lcrc.anl.gov
+    IdentityFile ~/.ssh/id_ed25519
+    PreferredAuthentications publickey,keyboard-interactive
+    PasswordAuthentication no
+```
+
+---
+
+## Recommended workflow
+
+1) Start a background control connection to Chrysalis (you’ll be prompted for
+   Duo):
+
+```bash
+ssh -MNf chrys
+```
+
+You should be returned to your original login shell after MFA.
+
+2) Run the sync. For example, to pull diagnostics from Chrysalis to your
+   current machine:
+
+```bash
+mache sync diags from chrysalis -u <cels_username>
+```
+
+If the control connection is active, you shouldn’t be prompted for Duo again.
+You’ll see `rsync` output similar to:
+
+```
+running: rsync --verbose --recursive --times --links --compress --progress --update --no-perms --omit-dir-times <cels_username>@chrysalis.lcrc.anl.gov:/lcrc/group/e3sm/public_html/diagnostics/ /path/to/local/diagnostics
+receiving incremental file list
+grids/ocean.RRSwISC6to18E3r5.mask.scrip.20240327.nc
+    633,767,353 100%   16.58MB/s    0:00:36 (xfr#1, ir-chk=1293/1488)
+grids/ocean.RRSwISC6to18E3r5.nomask.scrip.20240327.nc
+    633,767,353 100%   26.88MB/s    0:00:22 (xfr#2, ir-chk=1292/1488)
+...
+```
+
+3) When you’re done, close the control connection:
+
+```bash
+ssh -O exit chrys
+```
+
+Notes:
+- When pulling data (`from`), `mache` will automatically fix permissions on
+  the local destination according to machine settings.
+- Destination paths are derived from your machine configuration (diagnostics
+  base path), and source paths from the LCRC machine configuration.
+
+---
+
+## Command reference
+
+Basic usage:
+
+```text
+mache sync diags to <other>   [-u <username>] [-m <this_machine>] [-f <config_file>]
+mache sync diags from <other> [-u <username>] [-m <this_machine>] [-f <config_file>]
+```
+
+- `to | from` — direction of sync
+- `<other>` — the other machine name (e.g., `chrysalis`)
+- `-u, --username` — the username to use on the other machine (required in
+  practice)
+- `-m, --machine` — explicitly set the name of the current machine
+  (auto-detected if omitted)
+- `-f, --config_file` — path to a config file that overrides defaults for the
+  current machine
+
+Constraints enforced by the command:
+
+- Only `anvil`/`chrysalis` are considered LCRC machines
+- `to` is only allowed when you are on an LCRC machine
+- `from` is only allowed when the other machine is an LCRC machine
+- Do not attempt to sync between two different LCRC machines (there is no need
+  and this wastes bandwidth)
+
+---
+
+## Troubleshooting
+
+- You get Duo prompts during rsync:
+  - Ensure the control connection is active (`ssh -MNf chrys`) and your
+    `Host chrys` alias matches the command you used to connect.
+- Permission errors on the destination:
+  - Verify your local diagnostics base path exists and that you have write
+    access; `mache` adjusts group/world permissions on pull, but can’t create
+    parent paths that don’t exist.
+- Connection fails through the login proxy:
+  - Double-check `ProxyJump <cels_username>@logins.lcrc.anl.gov` and that your
+    public key is present at https://accounts.cels.anl.gov/.