Skip to content

operations.md

Latest commit

 

History

History
229 lines (178 loc) · 5.59 KB

operations.md

File metadata and controls

229 lines (178 loc) · 5.59 KB

Operations

This is the day-2 runbook for current DayEC clusters.

Connect

dyec headnode connect \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME"

Expected on the headnode:

whoami
pwd
command -v day-clone
command -v tmux

The supported user is ubuntu and the supported working directory is /home/ubuntu.

Re-run Headnode Configuration

dyec headnode configure \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME"

Use this after a cluster exists but the DayEC headnode tools, catalog, or login shell need repair.

Inspect Cluster And Jobs

dyec cluster list --profile "$AWS_PROFILE" --region "$REGION" --verbose
dyec --json cluster describe --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME"
dyec headnode jobs --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME"

Stage Sample Inputs

Use samples stage for sample-manifest workflows:

dyec samples stage "$ANALYSIS_SAMPLES" \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --reference-s3-uri "$REF_S3_URI" \
  --control-data-s3-uri "$CONTROL_DATA_S3_URI" \
  --stage-s3-uri "$STAGE_S3_URI" \
  --config-dir "$STAGE_CFG_DIR"

The helper writes local staged config files and prints a remote stage directory under /fsx/staging/staged_external_sequencing_data/....

For catalog-driven sample launches:

dyec samples run "$ANALYSIS_SAMPLES" \
  --command-id complete_genomics_mgi_snv_concordance \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --reference-s3-uri "$REF_S3_URI" \
  --control-data-s3-uri "$CONTROL_DATA_S3_URI" \
  --stage-s3-uri "$STAGE_S3_URI" \
  --analysis-id dayoa \
  --executing-entity "${EXECUTING_ENTITY:-ubuntu}" \
  --dry-run

The catalog pin for DayOA commands is 2.0.8.

Attach Run Folders

Use run DRAs for raw run folders that should stay in S3 until read by the workflow:

dyec --json mounts create "s3://sequencer-run-bucket/runs/RUN123/" \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --platform ILMN \
  --read-only \
  --batch-import-metadata-on-create \
  --auto-import NEW,CHANGED \
  --wait

Rules:

  • FSx API path is /run_dir_mounts/<mount_id>/.
  • Headnode path is /fsx/run_dir_mounts/<mount_id>/.
  • AutoExport is not configured by default.
  • Source prefixes and FSx paths must not overlap active DRAs.
  • Run mounts are input paths, not result paths.

Verify on the headnode:

dyec --json mounts verify \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --mount-id RUN123

Detach when done:

dyec --json mounts delete \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --mount-id RUN123 \
  --wait

Deletion detaches the DRA with DeleteDataInFileSystem=False; it does not delete S3 objects.

Launch Workflows

Sample-manifest workflow:

dyec workflow launch \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --stage-dir "/fsx/staging/staged_external_sequencing_data/remote_stage_<timestamp>" \
  --analysis-id dayoa \
  --executing-entity "${EXECUTING_ENTITY:-ubuntu}" \
  --git-tag 2.0.8

Run-folder workflow:

dyec workflow launch \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --run-context-file ./runs.tsv \
  --analysis-id run-qc \
  --executing-entity "${EXECUTING_ENTITY:-ubuntu}" \
  --git-tag 2.0.8 \
  --dy-command "bin/day_run produce_illumina_run_qc --config run_context_file=config/runs.tsv -p -j 5 -k"

The launcher creates /home/ubuntu/daylily-runs/<session>/ with launch.sh, tmux.log, and status.json.

Monitor

dyec --json workflow status \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --session <session>

dyec workflow logs \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --session <session> \
  --lines 100

Inside the headnode shell:

tmux ls
tmux attach -t <session>
squeue

Export Results

Export is a separate output DRA flow. It does not write back through the reference DRA or run-input DRA. It attaches a temporary DRA directly to one completed analysis directory.

From the operator machine:

dyec export \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --source-path "/fsx/analysis_results/$EXECUTING_ENTITY/$ANALYSIS_ID" \
  --destination-s3-uri "$EXPORT_S3_URI" \
  --output-dir "$EXPORT_DIR"

Verify:

cat "$EXPORT_DIR/fsx_export.yaml"

Success means status: success, task_lifecycle: SUCCEEDED, detached: true, delete_data_in_file_system: false, a completed FSx export task id, and an explicit fsx_root to s3_root mapping. The destination must be an explicit S3 URI ending in <executing_entity>/<analysis_id>/.

For DayOA commands with catalog artifact_registration enabled, pass --artifact-registration-command-id, --dewey-url, and --dewey-token-env. DYEC then loads the exported DayOA evidence manifest, maps selected relative paths to S3 URIs through fsx_export.yaml, posts to Dewey, and writes dewey_registration_receipt.json. Missing policy, manifest, Dewey URL, token, or invalid Dewey response is a hard failure.

Delete

dyec delete --dry-run \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME"

dyec delete \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME"

Use --yes only after the exact delete has already been approved.