Merge pull request #252 from Daylily-Informatics/codex/fsx-export-report-path

[codex] Fix FSx export report path

Author: iamh2o

daylily_ec/workflow/export_data.py

@@ -287,12 +287,10 @@ def run_export_task(
         destination_s3_uri,
         source_path=normalized_source,
     )
-    destination_parts = urlparse(destination)
-    analysis_dir = analysis_dir_from_source_path(normalized_source)
     timestamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
     report_path = (
-        f"s3://{destination_parts.netloc}/daylily-monitor/fsx-export/"
-        f"{analysis_dir}/{timestamp}/export-report/"
+        f"{destination.rstrip('/')}/_daylily_monitor/fsx-export/"
+        f"{timestamp}/export-report/"
     )
     try:
         response = fsx_client.create_data_repository_task(

docs/plans/dra_test4_run_qc_ont_export_ledger.md

@@ -0,0 +1,40 @@
+# dra-test4 run_qc_ont_all FSx Export Ledger
+
+Date: 2026-05-18
+
+## Objective
+
+Export the dra-test4 headnode path `/fsx/analysis_results/ubuntu/run_qc_ont_all` to the active DayOA us-west-2 bucket at `s3://lsmc-dayoa-omics-analysis-us-west-2/analysis_results/ubuntu/run_qc_ont_all/` using the explicit `dyec export` DRA workflow.
+
+## Gate 0: Inventory Freeze
+
+- Controlling SOP: `/Users/jmajor/.codex/docs/plan-ledger-workflow.md`
+- Ledger path: `docs/plans/dra_test4_run_qc_ont_export_ledger.md`
+- Repo path: `/Users/jmajor/.codex/worktrees/dyec-fsx-dra-mounts/daylily-ephemeral-cluster`
+- Branch/status: `main...origin/main`; pre-existing dirty files: `README.md`, `daylily_ec/cli.py`, `docs/LSMC_run_directory_mounting_run_analysis_spec.md`, `docs/cli_reference.md`, `docs/dra_fsx_strategy.md`, `docs/operations.md`, `docs/overview.md`, `docs/quickest_start.md`, `tests/test_run_mounts.py`
+- Activation: `source ./activate` resolved `CONDA_DEFAULT_ENV=DAY-EC`, `dyec=/Users/jmajor/miniconda3/envs/DAY-EC/bin/dyec`
+- Cluster inventory: `AWS_PROFILE=lsmc AWS_REGION=us-west-2 pcluster list-clusters --region us-west-2` showed `dra-test4` as `CREATE_COMPLETE`
+- Headnode info: `dyec --json headnode info --profile lsmc --region us-west-2 --cluster dra-test4` showed headnode `i-0d85f5c6569c7e475`, state `running`, compute fleet `RUNNING`
+- FSx inventory: `aws fsx describe-file-systems` for `parallelcluster:cluster-name=dra-test4` resolved `fs-0bcf5a60d9b48f03e`, lifecycle `AVAILABLE`, `SCRATCH_2`, 2400 GiB
+- Bucket inventory: `aws s3api head-bucket --bucket lsmc-dayoa-omics-analysis-us-west-2` succeeded with bucket region `us-west-2`
+- Existing destination inventory: `aws s3api list-objects-v2 --bucket lsmc-dayoa-omics-analysis-us-west-2 --prefix analysis_results/ubuntu/run_qc_ont_all/ --max-items 10` returned no contents
+- Existing DRA inventory for `fs-0bcf5a60d9b48f03e`: active run mounts under `/run_dir_mounts/`, active `/data/`, active `/analysis_results/ubuntu/run_qc_illumina_all/`, and misconfigured `/analysis_results/ubuntu/illumina_run_qc/`; no overlap with `/analysis_results/ubuntu/run_qc_ont_all/`
+- Source path proof: SSM via `daylily_ec.aws.ssm.run_shell` as `ubuntu` verified `/fsx/analysis_results/ubuntu/run_qc_ont_all` exists, owner `ubuntu:ubuntu`, mtime `2026-05-18 08:49:42 +0000`, 6 top-level entries, 192 files and 23 directories at max depth 4, with `/fsx` at 42G used of 2.2T
+
+## Rows
+
+| ID | Area | Requirement | Status | Category | Approval Gate | Owner | Evidence | Root Cause | Terminal Note |
+|---|---|---|---|---|---|---|---|---|---|
+| G0-001 | Baseline | Record repo, live cluster, FSx, bucket, destination, DRA, and source-path inventory before export. | `SUCCESS` | `legitimate_safety_handling` | Gate 0 | orchestrator | Gate 0 section above. |  | Baseline complete before live export. |
+| PRE-001 | Source | Prove the requested headnode path exists on dra-test4 as `ubuntu`. | `SUCCESS` | `legitimate_safety_handling` | Gate 0 | orchestrator | SSM command `a8352ab0-16f6-49bc-b54e-81927466aa0d`; source directory exists with 6 top-level entries and 192 files at max depth 4. |  | Requested source path is present. |
+| PRE-002 | Destination | Resolve and validate the explicit DayOA us-west-2 destination prefix. | `SUCCESS` | `legitimate_safety_handling` | Gate 0 | orchestrator | `s3://lsmc-dayoa-omics-analysis-us-west-2/analysis_results/ubuntu/run_qc_ont_all/`; bucket region `us-west-2`; no listed existing objects under the prefix. |  | Destination is explicit and empty at preflight. |
+| EXP-001 | Export | Run explicit `dyec export` from source path to destination S3 prefix and preserve `fsx_export.yaml`. | `SUCCESS` | `feature_implementation` | Gate 1 | orchestrator | First `dyec export` wrote failure receipt `reports/fsx_exports/dra-test4_run_qc_ont_all_20260518T085756Z/fsx_export.yaml`; focused fix moved report path under the export destination; `pytest tests/test_export.py -q -> 17 passed`; retry wrote success receipt `reports/fsx_exports/dra-test4_run_qc_ont_all_20260518T090219Z/fsx_export.yaml`. |  | Export task `task-09dd75221119a5311` completed `SUCCEEDED`; source `/analysis_results/ubuntu/run_qc_ont_all/` exported to `s3://lsmc-dayoa-omics-analysis-us-west-2/analysis_results/ubuntu/run_qc_ont_all/`. |
+| VER-001 | Verification | Verify export receipt success, DRA detach, and S3 object presence after export. | `SUCCESS` | `legitimate_safety_handling` | Gate 5 | orchestrator | Receipt status `success`, `task_lifecycle: SUCCEEDED`, `detached: true`, `detach_lifecycle: DELETED`; `describe-data-repository-associations` for `dra-004d5751b19bf2eae` and for `/analysis_results/ubuntu/run_qc_ont_all/` returned no active associations; S3 prefix contains 214 objects totaling 13,784,915,480 bytes; headnode source regular files total 13,784,915,480 bytes; report prefix contains 0 failed-file report objects. |  | Post-export verification complete; all rows are terminal and the objective is complete. |
+
+## Final Status
+
+- Rows terminal: 5/5
+- Objective complete: yes
+- Export receipt: `reports/fsx_exports/dra-test4_run_qc_ont_all_20260518T090219Z/fsx_export.yaml`
+- S3 destination: `s3://lsmc-dayoa-omics-analysis-us-west-2/analysis_results/ubuntu/run_qc_ont_all/`
+- Focused test: `pytest tests/test_export.py -q -> 17 passed`

reports/fsx_exports/dra-test4_run_qc_ont_all_20260518T085756Z/fsx_export.yaml

@@ -0,0 +1,21 @@
+fsx_export:
+  schema_version: 3
+  status: error
+  phase: detach
+  cluster_name: dra-test4
+  region: us-west-2
+  analysis_dir: run_qc_ont_all
+  source_path: /analysis_results/ubuntu/run_qc_ont_all/
+  headnode_path: /fsx/analysis_results/ubuntu/run_qc_ont_all/
+  destination_s3_uri: s3://lsmc-dayoa-omics-analysis-us-west-2/analysis_results/ubuntu/run_qc_ont_all/
+  detached: true
+  delete_data_in_file_system: false
+  failure_details:
+    message: 'Unable to start FSx export task: An error occurred (BadRequest) when
+      calling the CreateDataRepositoryTask operation: The report path provided is
+      not associated with a data repository linked to this file system.'
+  fsx_file_system_id: fs-0bcf5a60d9b48f03e
+  file_system_path: /analysis_results/ubuntu/run_qc_ont_all/
+  association_id: dra-013f59a28f338ae60
+  lifecycle: AVAILABLE
+  detach_lifecycle: DELETED

reports/fsx_exports/dra-test4_run_qc_ont_all_20260518T090219Z/fsx_export.yaml

@@ -0,0 +1,21 @@
+fsx_export:
+  schema_version: 3
+  status: success
+  phase: complete
+  cluster_name: dra-test4
+  region: us-west-2
+  analysis_dir: run_qc_ont_all
+  source_path: /analysis_results/ubuntu/run_qc_ont_all/
+  headnode_path: /fsx/analysis_results/ubuntu/run_qc_ont_all/
+  destination_s3_uri: s3://lsmc-dayoa-omics-analysis-us-west-2/analysis_results/ubuntu/run_qc_ont_all/
+  detached: true
+  delete_data_in_file_system: false
+  failure_details: {}
+  fsx_file_system_id: fs-0bcf5a60d9b48f03e
+  file_system_path: /analysis_results/ubuntu/run_qc_ont_all/
+  association_id: dra-004d5751b19bf2eae
+  lifecycle: AVAILABLE
+  task_id: task-09dd75221119a5311
+  task_lifecycle: SUCCEEDED
+  report_path: s3://lsmc-dayoa-omics-analysis-us-west-2/analysis_results/ubuntu/run_qc_ont_all/_daylily_monitor/fsx-export/20260518T090424Z/export-report/
+  detach_lifecycle: DELETED

tests/test_export.py

@@ -242,9 +242,8 @@ def test_run_export_task_starts_exact_analysis_path_and_report() -> None:
     assert fake.created_task["Type"] == "EXPORT_TO_REPOSITORY"
     assert fake.created_task["Paths"] == ["/analysis_results/ubuntu/illumina_run_qc/"]
     assert fake.created_task["Report"]["Path"].startswith(
-        "s3://bucket/daylily-monitor/fsx-export/illumina_run_qc/"
+        "s3://bucket/analysis_results/ubuntu/illumina_run_qc/_daylily_monitor/fsx-export/"
     )
-    assert "/analysis_results/ubuntu/illumina_run_qc/" not in fake.created_task["Report"]["Path"]
 
 
 def test_run_export_workflow_success_writes_v3_receipt(tmp_path, monkeypatch) -> None: