Add orcd skill for MIT ORCD remote execution

[satra]

Connects orcd-login.mit.edu (Slurm cluster eofe7, Engaging) as a remote execution environment and packages what it takes to use it reliably from Claude Code / Cowork / Science.

Why a skill rather than a README

Three things about ORCD are easy to get wrong and cost hours each time:

1. SSH auth is two-factor even with a key, and the failure message points at the wrong cause.
2. Entitlements are per-person and invisible. Partition access is gated by Unix group membership, so two people in the same lab get different answers and nothing can be hardcoded.
3. Storage tiers differ by ~10x in speed, and the slowest one ($HOME) is the default working directory.

Discovery, not hardcoding

Partition access comes from AllowGroups; per-user ceilings come from a QOS each partition attaches automatically. Neither is guessable, so the scripts ask the cluster at runtime:

• scontrol show partition for partitions and their gates — note PrivateData=none here, so this lists partitions regardless of access and cannot be used as an entitlement list.
• sbatch --test-only as the authoritative yes/no. It queues nothing, and it also reports estimated start time, which turns partition choice into a measurement.
• sacctmgr show qos for the ceilings that actually bind, distinguishing MaxTRESPU (yours alone) from GrpTRES (one pool shared with the whole group, so a colleague's job can block yours).
• GPU inventory from each partition's TRES string. Summing sinfo output instead undercounts badly, because sinfo collapses identically configured nodes onto one line.

Findings encoded in the scripts and docs

SSH requires publickey AND keyboard-interactive (Duo). The key alone leaves the session at "partial success"; Duo then answers the second stage, usually with zero prompts. So setting BatchMode=yes always fails with Permission denied (keyboard-interactive) — which reads like a rejected key, and sends people off replacing keys that were fine. Connection multiplexing (ControlMaster + ControlPersist 12h) is what keeps later calls non-interactive.

Never pass --qos. Most associations permit only normal, so naming a partition's own QOS fails with Invalid qos specification. Choose the partition; the QOS follows.

Storage tier is encoded in group names (orcd_rg_<server>_<owner>): fstor* is flash over NFS-over-RDMA, hstor* capacity disk, core* archive. Measured on an H100 node (1 GiB dd, plus 500 small-file creation):

Tier	Write	Read	500 files
/orcd/scratch/bcs/001-002 (flash)	~1000 MB/s	~2900 MB/s	0.13 s
/orcd/compute/bcs/001 (flash)	232 MB/s	3300 MB/s	3.19 s
/home	222 MB/s	1300 MB/s	0.78 s

Two non-obvious results: /orcd/compute/bcs/001 has excellent reads but ~25x worse metadata than scratch, so it is for reading shared datasets rather than writing many small files; and node-local /scratch (3.5 TB) exists on some nodes but not on the bcs H100 nodes, while Slurm reports TmpDisk=0 everywhere — so it must be probed, not assumed.

These are single samples on a busy shared cluster. The docs present them as indicative snapshots, not constants, and say so.

Two ways to hang a script here: bare df -h blocks on unresponsive network mounts (it hit a 120s timeout during development), so the scripts read /proc/mounts and size paths individually under timeout; and /orcd is autofs, so listing a parent is not an inventory — project trees are found by enumerating the LDAP automount maps.

Contents

Script	Purpose
orcd_doctor.py	Verify access; route to the right remedy on failure. --fix writes SSH config
orcd_resources.py	Usable partitions, GPU models, QOS ceilings, live free capacity
orcd_storage.py	Writable storage by tier and speed. --setup creates per-user dirs
orcd_submit.py	Plan, submit, track; auto-selects partition by start time
orcd_common.py	Shared SSH plumbing (multiplexing, base64 payloads, error mapping)

Plus references/setup.md, references/slurm.md, references/storage.md. All scripts are stdlib-only Python 3, run locally over one multiplexed SSH connection, and accept --host; the data-gathering ones accept --json.

Onboarding a new group member

ORCD accepts no password over SSH, so the key must be installed via the OnDemand portal (which does support Duo): sign in at https://orcd-ood.mit.edu/, use Clusters → Shell Access, append the public key to authorized_keys, then orcd_doctor.py --fix locally. The doctor prints this walkthrough itself when it detects the key is missing, and reports a missing Slurm association or orcd_rg_* groups as warnings — those need an email to orcd-help@mit.edu, not a config change.

Verification

• Repo validator passes (9/9 skills); all 76 unit tests pass.
• orcd_doctor.py passes against the live cluster, and correctly fails with the right remedy on bad-DNS and bad-alias paths (exit 1).
• Real GPU job submitted end-to-end: ran on node1702, got an H100 80GB, exit 0.
• orcd_storage.py --setup is idempotent.
• Installed copy matches the repo byte-for-byte; all probe artifacts cleaned off the cluster.

🤖 Generated with Claude Code