Merge branch 'dev' into edmundmiller/pass-or-fail-stack

Author: FloWuenne

.github/workflows/linting.yml

@@ -1,7 +1,5 @@
-name: nf-core linting
-# This workflow is triggered on pushes and PRs to the repository.
-# It runs the `nf-core pipelines lint` and markdown lint tests to ensure
-# that the code meets the nf-core guidelines.
+name: Pipeline linting
+# Runs pre-commit (Prettier, editorconfig, etc.) on pushes to dev and on pull requests.
 on:
   push:
     branches:
@@ -26,58 +24,3 @@ jobs:
 
       - name: Run pre-commit
         run: pre-commit run --all-files
-
-  nf-core:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Check out pipeline code
-        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
-
-      - name: Install Nextflow
-        uses: nf-core/setup-nextflow@v2
-
-      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-          architecture: "x64"
-
-      - name: read .nf-core.yml
-        uses: pietrobolcato/action-read-yaml@1.1.0
-        id: read_yml
-        with:
-          config: ${{ github.workspace }}/.nf-core.yml
-
-      - name: Install dependencies
-        run: |
-          python -m pip install --upgrade pip
-          pip install nf-core==${{ steps.read_yml.outputs['nf_core_version'] }}
-
-      - name: Run nf-core pipelines lint
-        if: ${{ github.base_ref != 'master' }}
-        env:
-          GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }}
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }}
-        run: nf-core -l lint_log.txt pipelines lint --dir ${GITHUB_WORKSPACE} --markdown lint_results.md
-
-      - name: Run nf-core pipelines lint --release
-        if: ${{ github.base_ref == 'master' }}
-        env:
-          GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }}
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }}
-        run: nf-core -l lint_log.txt pipelines lint --release --dir ${GITHUB_WORKSPACE} --markdown lint_results.md
-
-      - name: Save PR number
-        if: ${{ always() }}
-        run: echo ${{ github.event.pull_request.number }} > PR_number.txt
-
-      - name: Upload linting log file artifact
-        if: ${{ always() }}
-        uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4
-        with:
-          name: linting-logs
-          path: |
-            lint_log.txt
-            lint_results.md
-            PR_number.txt

.github/workflows/nf-test.yml

@@ -48,7 +48,7 @@ jobs:
           rm -rf ./* || true
           rm -rf ./.??* || true
           ls -la ./
-      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
         with:
           fetch-depth: 0
 
@@ -91,7 +91,7 @@ jobs:
       TOTAL_SHARDS: ${{ needs.nf-test-changes.outputs.total_shards }}
 
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
         with:
           fetch-depth: 0
 

.seqera/context/ERRORS.md

@@ -0,0 +1,63 @@
+# ERRORS.md — Known Errors and Failure Modes
+
+## Current Runs — No Errors Observed
+
+Both runs `1cF65l2PDvxNd5` (maniac_celsius) and `5VGC0gjEmghOaz` (clever_kalman) are **RUNNING** with all observed tasks COMPLETED with exit code 0. No task failures, retries, or OOM kills detected in the first 20 tasks of each run.
+
+## Historical / Known Failure Modes
+
+### 1. TOWER_ACCESS_TOKEN Missing
+
+**When:** API-mode runs (non-external workspace entries) without the token set.
+
+**Error:** `RuntimeException: Environment variable 'TOWER_ACCESS_TOKEN' is not set (required for run <id>)`
+
+**Fix:** Export `TOWER_ACCESS_TOKEN` before running the pipeline. The env var name can be overridden per-row via the `token_env` CSV column.
+
+### 2. Workspace Resolution Failure
+
+**When:** The `workspace` column value (e.g., `org/workspace`) doesn't match any accessible workspace.
+
+**Error:** `RuntimeException: Organization '<name>' not found` or `Workspace '<name>' not found in org '<name>'`
+
+**Fix:** Verify the org/workspace names in the input CSV match the Platform exactly (case-sensitive).
+
+### 3. API Rate Limiting / Transient Failures
+
+**When:** Fetching data from busy Platform instances with many concurrent runs.
+
+**Mitigation:** Built-in retry with exponential backoff (3 attempts, 1s/2s/4s delays). After 3 failures, the pipeline aborts with the original error.
+
+### 4. Wave Container Build Failures
+
+**When:** Using `wave` profile with `spack` strategy.
+
+**Error:** Wave freeze builds fail with spack.
+
+**Fix:** The wave profile explicitly uses `strategy = ['conda', 'container', 'dockerfile']` — no spack. Don't add spack to the strategy list.
+
+### 5. CGROUPv2 Docker Failures (Cloud VM / Firecracker)
+
+**When:** Running with Docker in Cloud VMs where cgroup resource delegation is restricted.
+
+**Error:** `cannot enter cgroupv2 ... with domain controllers`
+
+**Fix:** Apply the runc wrapper documented in AGENTS.md that strips `linux.resources` from the OCI spec.
+
+### 6. Empty Benchmark Report
+
+**When:** API runs are provided but `--generate_benchmark_report` is not set.
+
+**Warning:** `Found N API run(s) but --generate_benchmark_report is not enabled. API runs will not produce any output.`
+
+**Fix:** Add `--generate_benchmark_report` to the run command.
+
+### 7. Scheduling Overhead (Observed Pattern)
+
+**Not an error per se**, but both current runs show 4–8 minute gaps between task submit and task start times. This is expected with AWS Batch spot instances — EC2 instances must be provisioned and containers pulled before execution begins. Not actionable unless overhead exceeds ~15 minutes consistently.
+
+## Data Gaps
+
+- **No failed runs analysed:** Both Platform runs are currently successful. To populate this section with real failure analysis, re-run context generation after a failure occurs or provide historical failed run IDs.
+- **No STAR/alignment task data yet:** Both runs are still in early stages (QC/trim). Resource pressure and potential OOM failures during STAR genome generation and alignment are not yet observable.
+- **No cost data:** Neither run has completed, so total run cost is unavailable. Individual task costs are in the $0.001–$0.007 range for early QC tasks.

.seqera/context/HISTORY.md

@@ -0,0 +1,69 @@
+# HISTORY.md — Recent Runs Analysed
+
+## Seqera Platform Runs (unified-compute/sched-testing)
+
+### Run 1: `maniac_celsius` (1cF65l2PDvxNd5)
+
+- **Pipeline:** nf-core/rnaseq
+- **Status:** RUNNING (as of 2026-04-17T18:45)
+- **Submitted:** 2026-04-17T17:51:26Z
+- **Started:** 2026-04-17T17:53:20Z
+- **User:** edmund-miller
+- **Executor:** seqera/aws (AWS Batch, eu-west-1)
+- **Price model:** spot instances
+- **Workspace:** unified-compute/sched-testing (ID: 224344613053615)
+- **Work dir:** s3://nextflow-ci-dev/scratch/1cF65l2PDvxNd5/
+
+**Task Summary (30 tasks captured from API fetch):**
+
+- 21 tasks COMPLETED, 9 tasks RUNNING, 0 FAILED
+- Completed early-stage tasks include `SAMTOOLS_FAIDX`, `GTF_FILTER`, `FQ_LINT`, and `GTF2BED`
+- Active work is mostly `TRIMGALORE` fan-out across samples plus `SALMON_INDEX`
+- `STAR_GENOMEGENERATE_IGENOMES` is defined in the run graph but was not yet observed as running/completed in the fetched task set
+
+**Scheduling overhead:** Tasks show 4–8 min gap between submit and start (spot instance spin-up).
+
+### Run 2: `clever_kalman` (5VGC0gjEmghOaz)
+
+- **Pipeline:** nf-core/rnaseq
+- **Status:** RUNNING (as of 2026-04-17T18:45)
+- **Submitted:** 2026-04-17T17:50:14Z
+- **Started:** 2026-04-17T17:52:09Z
+- **User:** edmund-miller
+- **Executor:** seqera/aws (AWS Batch, eu-west-1)
+- **Price model:** spot instances
+- **Workspace:** unified-compute/sched-testing (ID: 224344613053615)
+- **Work dir:** s3://nextflow-ci-dev/scratch/5VGC0gjEmghOaz/
+
+**Task Summary (30 tasks captured from API fetch):**
+
+- 22 tasks COMPLETED, 8 tasks RUNNING, 0 FAILED
+- Completed early-stage tasks include `SAMTOOLS_FAIDX`, `GTF_FILTER`, `FQ_LINT`, and `GTF2BED`
+- Active work is mostly `TRIMGALORE` fan-out across samples
+- `STAR_GENOMEGENERATE_IGENOMES` is defined in the run graph but was not yet observed as running/completed in the fetched task set
+
+**Scheduling overhead:** Similar 5–6 min submit-to-start gaps.
+
+## Comparison Notes
+
+Both runs are concurrent nf-core/rnaseq executions on the same workspace with identical test data (GM12878_REP1, GM12878_REP2). Key differences:
+
+| Metric                  | 1cF65l2PDvxNd5                       | 5VGC0gjEmghOaz             |
+| ----------------------- | ------------------------------------ | -------------------------- |
+| Submit time             | 17:51:26Z                            | 17:50:14Z                  |
+| Machine types           | mix of m5d.large + c5d.large         | mostly c5d.large           |
+| SAMTOOLS_FAIDX realtime | 28s                                  | 62s                        |
+| GTF_FILTER realtime     | 58s                                  | 109s                       |
+| FQ_LINT avg realtime    | ~605s                                | ~613s                      |
+| Cost pattern            | Slightly higher per-task (m5d.large) | Lower per-task (c5d.large) |
+
+The timing variance in short tasks (SAMTOOLS_FAIDX: 28s vs 62s) likely reflects spot instance variability and S3 staging latency rather than algorithmic differences. Both runs are still in progress — STAR genome generation and alignment phases are pending.
+
+## Local Development History (git log)
+
+Recent development has focused on the **benchmark report rendering pipeline**:
+
+1. **Split architecture** (df2232e): Refactored monolithic benchmark report into 3 stages — normalize JSONL → aggregate → render
+2. **Combined runtime charts** (18a64f4 → a49a9a4): Added combined task runtime visualisation with scheduling overhead, per-pipeline breakdowns, and ECharts integration
+3. **CUR cost streaming** (b96ecfd): Optimised AWS CUR parquet processing with batch streaming
+4. **Test colocation** (96b8b81): Moved benchmark tests to live alongside their modules

.seqera/context/PATTERNS.md

@@ -0,0 +1,111 @@
+# PATTERNS.md — Codebase Conventions and Run Patterns
+
+## Pipeline Architecture Patterns
+
+### 1. Branch-and-Merge Input Strategy
+
+The workflow uses `.branch {}` to split input into API vs external paths, then merges all JSON outputs via `.mix().collect()` before the benchmark pipeline. This pattern allows mixed-source runs in a single CSV.
+
+```
+ids.branch { api: ...; external: ... }
+  → separate processing paths
+  → ch_api_jsons.mix(ch_tarball_jsons).mix(ch_external_dir_jsons).collect()
+```
+
+### 2. In-Process API Fetch (No Task)
+
+`SeqeraApi.fetchRunData()` runs inside the Nextflow process (Groovy), NOT as a Nextflow task. This means:
+
+- No container, no retry directive, no task monitoring
+- Retry logic is manually coded with exponential backoff (3 attempts)
+- Token is read from environment at execution time
+- Results are written to temp files and emitted as channel values
+
+### 3. Three-Stage Python Pipeline
+
+The benchmark report follows a strict data pipeline:
+
+1. **Normalize** (raw JSON → JSONL bundle) — streaming, handles CUR parquet
+2. **Aggregate** (JSONL → report_data.json) — statistical rollups
+3. **Render** (report_data.json → HTML) — Jinja2 template + ECharts
+
+Each stage is a separate Nextflow process with the same container image. The boundary between stages is explicit (JSONL files, then JSON), making debugging straightforward.
+
+### 4. External Test Fixtures
+
+Tests use pre-exported tarball fixtures (`workflows/nf_aggregate/assets/log_dirs/`, referenced from `test_benchmark.csv`) rather than live API calls. This enables fully offline CI.
+
+## Coding Conventions
+
+### Nextflow
+
+- **DSL2** with `include {}` for all modules
+- **No strict syntax yet** — uses `Channel.empty()`, implicit closures in places
+- Processes use `conda` + `container` directives (Wave-compatible)
+- All processes emit `versions.yml` for software version tracking
+- `publishDir` configured per-process in `workflows/nf_aggregate/nextflow.config`
+
+### Python (bin/)
+
+- **Typer CLI** with subcommands for each pipeline stage
+- Modular: `benchmark_report.py` is a thin dispatcher; logic lives in `benchmark_report_normalize.py`, `benchmark_report_aggregate.py`, `benchmark_report_render.py`, `benchmark_report_fetch.py`
+- **JSONL** as intermediate format (streaming-friendly, Fusion-compatible)
+- **Jinja2** for HTML templating with ECharts for charts
+- **PyArrow** for CUR parquet reading (batched streaming for memory efficiency)
+- Test runner: `pytest` with tests colocated under each module's `tests/` directory
+
+### Testing
+
+- **nf-test** for pipeline integration (4 test suites: benchmark-tarball, benchmark-directory, api-only, mixed-no-benchmark)
+- **pytest** for Python unit tests (normalize, aggregate, render, fetch)
+- **[nft-utils@0.0.4](mailto:nft-utils@0.0.4)** plugin for snapshot assertions
+- Test config: `tests/nextflow.config` (intentionally empty — relies on profile `test`)
+- Snapshots verify task counts, output file lists, and software versions
+
+### Configuration
+
+- Container registry: `quay.io` (default for all runtimes)
+- Docker runs as current user: `-u $(id -u):$(id -g)`
+- Process defaults: 1 CPU, 6 GB memory, 4h time
+- Error strategy: retry on exit codes 130–145 and 104 (OOM/signal kills)
+
+## Run Patterns (from Platform observations)
+
+### Scheduling Overhead
+
+Both observed runs show consistent 4–8 minute submit→start latency. This is a characteristic of AWS Batch with spot instances:
+
+- Instance provisioning
+- Container image pull (Wave-built images)
+- S3 staging of inputs
+
+### Instance Type Selection
+
+AWS Batch auto-selects instance types based on resource requests:
+
+- 1 CPU / 6 GB → m5d.large or c5d.large
+- 2 CPU / 2 GB → c5d.large (compute-optimised)
+- All spot pricing
+
+### Cost Efficiency
+
+Early QC tasks cost $0.001–$0.007 each. The dominant cost will come from STAR genome generation and alignment tasks (not yet observed in these in-progress runs).
+
+## File Naming Conventions
+
+| Pattern                | Convention                                  |
+| ---------------------- | ------------------------------------------- |
+| Modules                | `modules/local/<name>/main.nf`              |
+| Module tests (Python)  | `modules/local/<name>/tests/test_<name>.py` |
+| Module docs            | `modules/local/<name>/AGENTS.md`            |
+| Pipeline tests         | `tests/<scenario>/main.nf.test`             |
+| Pipeline test fixtures | `workflows/nf_aggregate/assets/`            |
+| Bin scripts            | `bin/benchmark_report_<stage>.py`           |
+| Agent docs             | `AGENTS.md` at each directory level         |
+
+## Git Workflow
+
+- Feature branches named `<user>/<description>`
+- Conventional commits: `feat()`, `fix()`, `test()`, `refactor()`, `docs()`, `perf()`
+- Commit signing required (GPG via SSH/1Password)
+- Pre-commit hooks for linting