daylily-ephemeral-cluster/README.md.bland at main · Daylily-Informatics/daylily-ephemeral-cluster · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
# Daylily Ephemeral Cluster

DayEC creates short-lived AWS ParallelCluster environments for Daylily analysis work. Current clusters use a DRA-backed FSx for Lustre filesystem:

- references: `/fsx/data`
- run folders: `/fsx/run_dir_mounts/<mount_id>`
- workflow outputs: `/fsx/analysis_results/...`
- direct export source: `/fsx/analysis_results/ubuntu/<analysis_dir>`

The cluster is disposable. S3 is durable. Export and verify `fsx_export.yaml` before delete.

## Happy Path

```bash
source ./activate

export AWS_PROFILE=daylily-service-lsmc
export REGION=us-west-2
export REGION_AZ=us-west-2d
export CLUSTER_NAME=day-demo-$(date +%Y%m%d%H%M%S)
export DAY_EX_CFG="$HOME/.config/daylily/daylily_ephemeral_cluster.yaml"
export REF_BUCKET=s3://lsmc-dayoa-omics-analysis-us-west-2
export ANALYSIS_BUCKET=s3://lsmc-dayoa-analysis-results-us-west-2
export ANALYSIS_DIR=dayoa
export ANALYSIS_SAMPLES=etc/analysis_samples_template.tsv
export STAGE_CFG_DIR="$PWD/tmp-stage-config/$CLUSTER_NAME"
export EXPORT_DIR="$PWD/tmp-export/$ANALYSIS_DIR"
export EXPORT_S3_URI="$ANALYSIS_BUCKET/analysis_results/ubuntu/$ANALYSIS_DIR/"

dyec preflight --profile "$AWS_PROFILE" --region-az "$REGION_AZ" --config "$DAY_EX_CFG"
dyec create --profile "$AWS_PROFILE" --region-az "$REGION_AZ" --config "$DAY_EX_CFG"
dyec headnode connect --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME"

dyec samples stage "$ANALYSIS_SAMPLES" \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --reference-bucket "$REF_BUCKET" \
  --config-dir "$STAGE_CFG_DIR"

dyec workflow launch \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --stage-dir "/fsx/data/staged_sample_data/remote_stage_<timestamp>" \
  --destination "$ANALYSIS_DIR" \
  --git-tag 1.0.16

dyec export \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --source-path "/fsx/analysis_results/ubuntu/$ANALYSIS_DIR" \
  --destination-s3-uri "$EXPORT_S3_URI" \
  --output-dir "$EXPORT_DIR"

cat "$EXPORT_DIR/fsx_export.yaml"

dyec delete --dry-run --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME"
dyec delete --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME"
```

## Run Folder Inputs

Attach only the run prefixes needed for the current work:

```bash
dyec --json mounts create \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --s3-uri "s3://sequencer-run-bucket/runs/RUN123/" \
  --mount-id RUN123 \
  --run-id RUN123 \
  --platform ILMN \
  --read-only \
  --wait

dyec --json mounts verify \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --mount-id RUN123
```

Run mounts are input-oriented. Do not write outputs into `/fsx/run_dir_mounts`. Export completed analysis directories from `/fsx/analysis_results/ubuntu/<analysis_dir>` instead.

## Read More

- [docs/dra_fsx_strategy.md](docs/dra_fsx_strategy.md)
- [docs/quickest_start.md](docs/quickest_start.md)
- [docs/operations.md](docs/operations.md)
- [docs/cli_reference.md](docs/cli_reference.md)
- [docs/archive/README.md](docs/archive/README.md)