Skip to content

Commit e5f3fc0

Browse files
wettowelreactorwettowelreactor
committed
docs: comprehensive SLURM integration documentation with caveats and requirements
1 parent 87ed3f5 commit e5f3fc0

2 files changed

Lines changed: 298 additions & 0 deletions

File tree

‎docs/mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -165,6 +165,7 @@ nav:
165165
- Supported APIs: hostfactory/supported-apis.md
166166
- SLURM Integration:
167167
- Integration Guide: slurm/integration_guide.md
168+
- Operations Guide: slurm/operations_guide.md
168169
- Deployment Scenarios: slurm/deployment_scenarios.md
169170
- Feature Mapping: slurm/feature_mapping.md
170171
- Supported APIs: slurm/supported-apis.md

‎docs/root/slurm/operations_guide.md‎

Lines changed: 297 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,297 @@
1+
# SLURM Operations Guide
2+
3+
## Template Generation
4+
5+
ORB generates templates directly from your `slurm.conf` partition definitions, ensuring that provisioned instances match SLURM's expected resource specs.
6+
7+
### Basic Usage
8+
9+
```bash
10+
orb templates generate --slurm-conf /etc/slurm/slurm.conf
11+
```
12+
13+
### How It Works
14+
15+
1. Parses `NodeName=` lines for CPUs and RealMemory declarations
16+
2. Parses `PartitionName=` lines to associate partitions with node specs
17+
3. Maps each partition's CPU/memory requirements to an appropriate AWS instance type
18+
4. Generates one template per partition with `template_id` = partition name
19+
20+
Example: given this `slurm.conf`:
21+
22+
```ini
23+
NodeName=compute-[001-050] CPUs=4 RealMemory=16000 State=CLOUD
24+
NodeName=gpu-[001-010] CPUs=8 RealMemory=32000 State=CLOUD
25+
26+
PartitionName=batch Nodes=compute-[001-050] Default=YES MaxTime=INFINITE State=UP
27+
PartitionName=gpu Nodes=gpu-[001-010] MaxTime=INFINITE State=UP
28+
```
29+
30+
ORB generates:
31+
32+
```json
33+
[
34+
{"template_id": "batch", "machine_types": {"t3.xlarge": 1}, "provider_api": "EC2Fleet"},
35+
{"template_id": "gpu", "machine_types": {"t3.2xlarge": 1}, "provider_api": "EC2Fleet"}
36+
]
37+
```
38+
39+
### slurm.conf Path Resolution
40+
41+
The path to `slurm.conf` is resolved in this order:
42+
43+
1. `--slurm-conf /path` CLI flag (highest priority)
44+
2. `scheduler.slurm.config_path` in config.json
45+
3. `SLURM_CONF` environment variable
46+
4. Default paths: `/etc/slurm/slurm.conf`, `/usr/local/etc/slurm.conf`, `$ORB_ROOT_DIR/slurm.conf`
47+
48+
### User-Specified Instance Type Preferences
49+
50+
For partitions where you want to control instance selection (e.g., for cost optimization or spot diversity), add preferences in `config.json`:
51+
52+
```json
53+
{
54+
"scheduler": {
55+
"type": "slurm",
56+
"slurm": {
57+
"config_path": "/etc/slurm/slurm.conf",
58+
"partitions": {
59+
"batch": {
60+
"instance_types": ["t3.medium", "t3.large", "m5.large"],
61+
"allocation_strategy": "capacityOptimized"
62+
},
63+
"gpu": {
64+
"instance_types": ["g4dn.xlarge", "g4dn.2xlarge"],
65+
"allocation_strategy": "lowestPrice"
66+
}
67+
}
68+
}
69+
}
70+
}
71+
```
72+
73+
Rules:
74+
75+
- `instance_types` is an ordered list (first = highest priority)
76+
- All listed types should have CPU >= partition's declared CPUs and Memory >= partition's declared RealMemory
77+
- Partitions without a config entry fall back to auto-mapping from slurm.conf
78+
- Multiple instance types are used as EC2 Fleet overrides for capacity diversification
79+
80+
Generated template output with preferences:
81+
82+
```json
83+
{
84+
"template_id": "batch",
85+
"machine_types": {"t3.medium": 1, "t3.large": 1, "m5.large": 1},
86+
"allocation_strategy": "capacityOptimized",
87+
"provider_api": "EC2Fleet",
88+
"fleet_type": "instant"
89+
}
90+
```
91+
92+
### Validation
93+
94+
At generation time, ORB validates that all specified instance types meet the partition's resource requirements. If validation fails, template generation is refused:
95+
96+
```
97+
ERROR: Template 'batch' validation failed:
98+
- Instance type 't3.micro' (1 vCPU, 1024MB) does not meet partition requirements (CPUs=2, RealMemory=3800)
99+
Fix: Remove undersized instance types from scheduler.slurm.partitions.batch.instance_types
100+
or reduce partition resource requirements in slurm.conf
101+
```
102+
103+
Use `--force` to bypass validation (for advanced users).
104+
105+
### Overwriting Existing Templates
106+
107+
```bash
108+
orb templates generate --slurm-conf /etc/slurm/slurm.conf --force
109+
```
110+
111+
The `--force` flag both overwrites existing template files and skips instance type validation.
112+
113+
---
114+
115+
## Per-Instance Tagging (orb:node-name)
116+
117+
### Mechanism
118+
119+
When machines are provisioned with `--nodes` (or via SLURM ResumeProgram):
120+
121+
1. Application layer assigns each instance a SLURM node name from the request
122+
2. After provisioning, ORB dispatches a `TAG_INSTANCES` operation to the provider
123+
3. Provider calls `ec2:CreateTags` with `orb:node-name=<node_name>` on each instance
124+
4. Compute nodes read this tag on boot to start slurmd with the correct identity
125+
126+
### Why It Exists
127+
128+
SLURM requires each compute node to register with the name declared in `slurm.conf`. Since ORB provisions generic instances, the node name assignment happens post-launch via instance tags. The compute AMI's boot script reads the tag and starts `slurmd -N <name>`.
129+
130+
### Provider Requirements
131+
132+
- Must support `TAG_INSTANCES` operation type
133+
- Must be able to apply per-instance tags after launch
134+
- AWS: requires `ec2:CreateTags` permission on the controller's IAM role
135+
- Compute nodes: require `ec2:DescribeTags` permission to read their own tags
136+
137+
### Error Handling
138+
139+
Tag failures are logged at ERROR level but do NOT fail the overall provisioning request (fire-and-forget). The `TAG_INSTANCES` operation returns `error_result` on failure so monitoring can alert on it.
140+
141+
---
142+
143+
## Caveats and Known Limitations
144+
145+
### Environment
146+
147+
| Issue | Detail | Mitigation |
148+
|-------|--------|------------|
149+
| POSIX locale | AL2023 SSM sessions default to `LANG=C` | `PYTHONUTF8=1` is set automatically in hook scripts and CLI entry point |
150+
| Hook env isolation | systemd env vars don't propagate to ResumeProgram | All config goes in `$ORB_ROOT_DIR/slurm_hooks.env` |
151+
152+
### Networking
153+
154+
| Requirement | Reason |
155+
|-------------|--------|
156+
| Security groups: controller ↔ compute all TCP | srun uses ephemeral ports for I/O forwarding |
157+
| Controller on fixed IP/hostname | slurm.conf `SlurmctldHost` must be reachable from compute |
158+
| DNS resolution | Compute nodes must resolve controller hostname |
159+
160+
### Timing
161+
162+
| Parameter | Recommended | Reason |
163+
|-----------|-------------|--------|
164+
| `ResumeTimeout` | ≥ 300s | Instance boot (~60s) + tag propagation (~5s) + tag read with retries (~30s) + slurmd registration |
165+
| `SuspendTime` | 30-300s | How long idle nodes wait before termination; lower = faster cost savings |
166+
| `SuspendTimeout` | 120s | Time for SuspendProgram to complete termination |
167+
| Tag retry window | 30s (15 retries × 2s) | Built into boot script; covers EC2 tag propagation delay |
168+
169+
### IAM Requirements
170+
171+
**Controller node (slurmctld) IAM role:**
172+
173+
- `ec2:RunInstances`, `ec2:CreateFleet`, `ec2:TerminateInstances`
174+
- `ec2:CreateLaunchTemplate`, `ec2:CreateLaunchTemplateVersion`
175+
- `ec2:CreateTags` (for per-instance tagging)
176+
- `ec2:DescribeInstances`, `ec2:DescribeFleets`
177+
- `iam:PassRole` (to attach instance profile to compute nodes)
178+
- `ssm:GetParameter` (if using SSM for AMI resolution)
179+
180+
**Compute node IAM role (instance profile):**
181+
182+
- `ec2:DescribeTags` (read own `orb:node-name` tag)
183+
- `ssm:GetParameter` (if fetching munge key / slurm.conf from SSM)
184+
185+
The instance profile must be specified in config:
186+
```json
187+
"provider_defaults": {
188+
"aws": {
189+
"template_defaults": {
190+
"iam_instance_profile": "orb-compute-instance-profile"
191+
}
192+
}
193+
}
194+
```
195+
196+
### SLURM Resource Matching
197+
198+
- SLURM requires all nodes in a partition to match the declared `CPUs` and `RealMemory`
199+
- Instances with MORE resources than declared are accepted (SLURM uses declared values for scheduling)
200+
- Instances with FEWER resources will fail `slurmd` registration
201+
- Generated templates should use a single instance type per partition, OR multiple types that ALL exceed the partition's declared resources
202+
- `orb templates generate` validates this automatically
203+
204+
### Tag Race Condition
205+
206+
EC2 tags are eventually consistent. Between `CreateTags` and the instance boot script reading the tag, there is a propagation window (typically < 5s, worst case ~30s).
207+
208+
The recommended boot script pattern:
209+
210+
```bash
211+
# Retry loop for tag propagation
212+
for i in $(seq 1 15); do
213+
NODE_NAME=$(aws ec2 describe-tags --region "${REGION}" \
214+
--filters "Name=resource-id,Values=${INSTANCE_ID}" "Name=key,Values=orb:node-name" \
215+
--query 'Tags[0].Value' --output text 2>/dev/null)
216+
if [ -n "${NODE_NAME}" ] && [ "${NODE_NAME}" != "None" ]; then
217+
break
218+
fi
219+
sleep 2
220+
done
221+
```
222+
223+
---
224+
225+
## Compute AMI Requirements
226+
227+
### Base Packages
228+
229+
- Amazon Linux 2023 (al2023) or Ubuntu 22.04+
230+
- SLURM 24.05.5+ compiled with `--with-systemd` for cgroup/v2 support
231+
- Build dependencies: `systemd-devel`, `dbus-devel`, `munge-devel`
232+
- Runtime: `munge`, `slurm-slurmd`
233+
234+
### Configuration Files
235+
236+
| File | Purpose |
237+
|------|---------|
238+
| `/etc/slurm/slurm.conf` | Must match controller's config (SlurmctldHost, partitions) |
239+
| `/etc/slurm/cgroup.conf` | cgroup/v2 configuration for resource enforcement |
240+
| `/etc/munge/munge.key` | Shared authentication key (must match controller) |
241+
242+
### Boot Script
243+
244+
The AMI must include a boot script (via cloud-init or systemd oneshot) that:
245+
246+
1. Reads `orb:node-name` EC2 tag (with retry loop for propagation)
247+
2. Sets hostname to the SLURM node name
248+
3. Optionally fetches `slurm.conf` and `munge.key` from SSM Parameter Store
249+
4. Starts munge
250+
5. Starts slurmd with `-N <node_name>`
251+
252+
Example minimal boot script:
253+
254+
```bash
255+
#!/bin/bash
256+
set -euo pipefail
257+
258+
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
259+
REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region)
260+
261+
# Wait for orb:node-name tag
262+
for i in $(seq 1 15); do
263+
NODE_NAME=$(aws ec2 describe-tags --region "${REGION}" \
264+
--filters "Name=resource-id,Values=${INSTANCE_ID}" "Name=key,Values=orb:node-name" \
265+
--query 'Tags[0].Value' --output text 2>/dev/null)
266+
[ -n "${NODE_NAME}" ] && [ "${NODE_NAME}" != "None" ] && break
267+
sleep 2
268+
done
269+
270+
if [ -z "${NODE_NAME}" ] || [ "${NODE_NAME}" = "None" ]; then
271+
echo "ERROR: Could not read orb:node-name tag after 30s" >&2
272+
exit 1
273+
fi
274+
275+
# Set hostname and start SLURM
276+
hostnamectl set-hostname "${NODE_NAME}"
277+
systemctl start munge
278+
slurmd -N "${NODE_NAME}"
279+
```
280+
281+
### cgroup.conf Example
282+
283+
```ini
284+
CgroupAutomount=yes
285+
ConstrainCores=yes
286+
ConstrainRAMSpace=yes
287+
ConstrainSwapSpace=yes
288+
```
289+
290+
### Verification Checklist
291+
292+
- [ ] `slurmd -C` reports correct CPUs/RealMemory matching slurm.conf
293+
- [ ] munge auth succeeds: `munge -n | ssh controller unmunge`
294+
- [ ] slurmd can reach slurmctld on port 6817
295+
- [ ] Instance profile has `ec2:DescribeTags` permission
296+
- [ ] Boot script handles tag propagation delay (retry loop)
297+
- [ ] Security groups allow controller ↔ compute traffic (all TCP)

0 commit comments

Comments
 (0)