Skip to content

cli_reference.md

Latest commit

 

History

History
327 lines (256 loc) · 9.85 KB

cli_reference.md

File metadata and controls

327 lines (256 loc) · 9.85 KB

CLI Reference

This reference is grounded in the current dyec / daylily-ec command surface. Both executable names use the same entrypoint.

Root Commands

dyec --help

Current commands:

  • version
  • info
  • create
  • preflight
  • drift
  • cluster-info
  • export
  • delete
  • resources-dir
  • env
  • runtime
  • pricing
  • aws
  • cluster
  • headnode
  • samples
  • workflow
  • repositories
  • exports
  • mounts
  • mount
  • state

Use global --json for machine-readable output where supported.

Create And Preflight

dyec preflight \
  --profile "$AWS_PROFILE" \
  --region-az "$REGION_AZ" \
  --config "$DAY_EX_CFG"

dyec create \
  --profile "$AWS_PROFILE" \
  --region-az "$REGION_AZ" \
  --config "$DAY_EX_CFG"

create runs preflight, renders the ParallelCluster YAML, creates the cluster, waits for the headnode, configures DayEC on the headnode over SSM, and validates the supported ubuntu login shell.

When storage URI config values are blank and create is run interactively, DYEC discovers contract-valid candidates in the selected profile and region, including the export destination. It auto-selects a URI only when exactly one candidate is valid, presents numbered choices when multiple candidates are valid, and asks for an explicit S3 URI when no valid candidates are found. --non-interactive does not discover or prompt; the config must provide explicit URIs.

Important options:

  • --region-az
  • --profile
  • --config
  • --pass-on-warn
  • --debug
  • --non-interactive

Cluster

dyec cluster list --profile "$AWS_PROFILE" --region "$REGION" --verbose
dyec --json cluster describe --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME"
dyec cluster wait --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME"

cluster-info remains available, but cluster list is the preferred current operator surface.

Headnode

dyec headnode connect --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME"
dyec headnode configure --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME"
dyec headnode info --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME"
dyec headnode jobs --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME"

Supported headnode command payloads run as ubuntu. Interactive sessions use SSM-SessionManagerRunShell and must land in /home/ubuntu in a bash login shell.

Samples

samples stage translates analysis_samples.tsv into workflow-ready staged manifests:

dyec samples stage "$ANALYSIS_SAMPLES" \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --reference-s3-uri "$REF_S3_URI" \
  --control-data-s3-uri "$CONTROL_DATA_S3_URI" \
  --stage-s3-uri "$STAGE_S3_URI" \
  --config-dir "$STAGE_CFG_DIR"

samples run stages the manifest, validates it against a catalog command, and launches the workflow:

dyec samples run "$ANALYSIS_SAMPLES" \
  --command-id complete_genomics_mgi_snv_concordance \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --reference-s3-uri "$REF_S3_URI" \
  --control-data-s3-uri "$CONTROL_DATA_S3_URI" \
  --stage-s3-uri "$STAGE_S3_URI" \
  --analysis-id dayoa \
  --executing-entity "${EXECUTING_ENTITY:-ubuntu}" \
  --dry-run

Important options:

  • --reference-s3-uri
  • --control-data-s3-uri
  • --stage-s3-uri
  • --config-dir
  • --stage-target
  • --run-metric-staging RUN_UID:PLATFORM:FOFN
  • --command-id
  • --analysis-id
  • --executing-entity
  • --export-destination-s3-uri
  • --export-trigger
  • --delete-on-export-success
  • --artifact-registration-command-id
  • --dewey-url
  • --dewey-token-env
  • --git-tag

Run Mounts

Run mounts are FSx Data Repository Associations from selected S3 run prefixes to /run_dir_mounts/<mount_id>/, visible on the headnode as /fsx/run_dir_mounts/<mount_id>/. The mount id defaults to the final folder in the S3 URI.

dyec --json mounts create "s3://sequencer-run-bucket/runs/RUN123/" \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --platform ILMN \
  --read-only \
  --batch-import-metadata-on-create \
  --auto-import NEW,CHANGED \
  --wait

dyec --json mounts list \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME"

dyec --json mounts describe \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --mount-id RUN123

dyec --json mounts verify \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --mount-id RUN123

dyec --json mounts delete \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --mount-id RUN123 \
  --wait

mounts verify is currently headnode-only. It returns exit code 0 when the /fsx/... path is usable and nonzero when it is not.

dyec mount rundir is an alias for dyec mounts create.

Default behavior is read-oriented:

  • no AutoExport policy
  • no source S3 writeback
  • no deletion of S3 objects on detach
  • overlapping active FSx paths or S3 prefixes are rejected

Workflow

Sample-manifest launch:

dyec workflow launch \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --stage-dir "/fsx/staging/staged_external_sequencing_data/remote_stage_<timestamp>" \
  --analysis-id dayoa \
  --executing-entity "${EXECUTING_ENTITY:-ubuntu}" \
  --git-tag 2.0.19

Run-context launch:

dyec workflow launch \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --run-context-file ./runs.tsv \
  --analysis-id run-qc \
  --executing-entity "${EXECUTING_ENTITY:-ubuntu}" \
  --git-tag 2.0.19 \
  --dy-command "bin/day_run produce_illumina_run_qc --config run_context_file=config/runs.tsv -p -j 5 -k"

Workflow launch requires --analysis-id and --executing-entity; both must be safe path segments. The headnode checkout root is /fsx/analysis_results/<executing_entity>/<analysis_id>/, and the repository checkout sits below it.

Auto-export options:

  • --export-destination-s3-uri: full S3 destination prefix ending in <executing_entity>/<analysis_id>/
  • --export-trigger: one of none, on-success, on-fail, or all; default none
  • --delete-on-export-success: deletes only the FSx analysis directory after a successful requested export
  • --artifact-registration-command-id: command-catalog policy to apply after successful export
  • --dewey-url: Dewey base URL for DYEC registration requests
  • --dewey-token-env: environment variable containing the Dewey bearer token on the headnode

Artifact registration requires auto-export and a command with an explicit catalog policy. DayOA writes only local evidence artifacts; DYEC owns Dewey requests after export.

Inspect:

dyec --json workflow status --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME" --session <session>
dyec workflow logs --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME" --session <session> --lines 100

Repository Catalog

dyec repositories commands
dyec repositories commands --config config/daylily_available_repositories.yaml
dyec repositories commands --command-id illumina_run_qc

The catalog is version 2. DayOA repository and command pins are 2.0.19.

Command classes:

  • sample_analysis: uses analysis_samples.tsv, staging, samples.tsv, and units.tsv
  • run_analysis: uses runs.tsv and requires a run mount

Export

Root export runs the complete explicit output-DRA workflow on one completed analysis directory:

dyec export \
  --profile "$AWS_PROFILE" \
  --region "$REGION" \
  --cluster "$CLUSTER_NAME" \
  --source-path "/fsx/analysis_results/$EXECUTING_ENTITY/$ANALYSIS_ID" \
  --destination-s3-uri "$EXPORT_S3_URI" \
  --output-dir "$EXPORT_DIR"

Required:

  • --cluster or --fsx-file-system-id
  • --source-path
  • --destination-s3-uri
  • --region
  • --output-dir

Optional artifact registration:

  • --artifact-registration-command-id
  • --repository-catalog
  • --dewey-url
  • --dewey-token-env

The source path must be /fsx/analysis_results/<executing_entity>/<analysis_id> or /analysis_results/<executing_entity>/<analysis_id>. The destination must be an explicit S3 URI ending in <executing_entity>/<analysis_id>/. Run mounts, reference data, nested paths, old export staging paths, unsafe path segments, and non-empty destination prefixes are rejected.

Lower-level helpers:

dyec --json exports attach --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME" --source-path "/fsx/analysis_results/$EXECUTING_ENTITY/$ANALYSIS_ID" --destination-s3-uri "$EXPORT_S3_URI"
dyec --json exports run --profile "$AWS_PROFILE" --region "$REGION" --source-path "/fsx/analysis_results/$EXECUTING_ENTITY/$ANALYSIS_ID" --destination-s3-uri "$EXPORT_S3_URI" --fsx-file-system-id "$FSX_FILE_SYSTEM_ID"
dyec --json exports detach --profile "$AWS_PROFILE" --region "$REGION" --association-id "$EXPORT_DRA_ID"

Delete

dyec delete --dry-run --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME"
dyec delete --profile "$AWS_PROFILE" --region "$REGION" --cluster "$CLUSTER_NAME"

Use --yes only when the destructive delete has already been approved.

AWS Validation

dyec aws validate permissions --profile "$AWS_PROFILE" --region-az "$REGION_AZ" --gap-analysis aws_permissions_gap.md
dyec aws validate quotas --profile "$AWS_PROFILE" --region-az "$REGION_AZ" --config "$DAY_EX_CFG" --gap-analysis aws_quota_gap.md
dyec --json aws validate all --profile "$AWS_PROFILE" --region-az "$REGION_AZ" --config "$DAY_EX_CFG"

Validation is read-only.

Runtime, Environment, State, And Pricing

dyec runtime status
dyec runtime check
dyec runtime explain
dyec env status
dyec resources-dir
dyec --json state list
dyec state show --cluster "$CLUSTER_NAME"
dyec pricing snapshot --profile "$AWS_PROFILE" --region "$REGION"