Skip to content

Commit 929f688

Browse files
FloWuenneFloWuenne
authored
Merge pull request #128 from seqeralabs/codex/fix-combined-runtime-blank
fix: combined runtime
2 parents 051a381 + 461aac7 commit 929f688

14 files changed

Lines changed: 6568 additions & 4 deletions

File tree

‎.seqera/context/ERRORS.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# ERRORS.md — Known Errors and Failure Modes
2+
3+
## Current Runs — No Errors Observed
4+
5+
Both runs `1cF65l2PDvxNd5` (maniac_celsius) and `5VGC0gjEmghOaz` (clever_kalman) are **RUNNING** with all observed tasks COMPLETED with exit code 0. No task failures, retries, or OOM kills detected in the first 20 tasks of each run.
6+
7+
## Historical / Known Failure Modes
8+
9+
### 1. TOWER_ACCESS_TOKEN Missing
10+
11+
**When:** API-mode runs (non-external workspace entries) without the token set.
12+
13+
**Error:** `RuntimeException: Environment variable 'TOWER_ACCESS_TOKEN' is not set (required for run <id>)`
14+
15+
**Fix:** Export `TOWER_ACCESS_TOKEN` before running the pipeline. The env var name can be overridden per-row via the `token_env` CSV column.
16+
17+
### 2. Workspace Resolution Failure
18+
19+
**When:** The `workspace` column value (e.g., `org/workspace`) doesn't match any accessible workspace.
20+
21+
**Error:** `RuntimeException: Organization '<name>' not found` or `Workspace '<name>' not found in org '<name>'`
22+
23+
**Fix:** Verify the org/workspace names in the input CSV match the Platform exactly (case-sensitive).
24+
25+
### 3. API Rate Limiting / Transient Failures
26+
27+
**When:** Fetching data from busy Platform instances with many concurrent runs.
28+
29+
**Mitigation:** Built-in retry with exponential backoff (3 attempts, 1s/2s/4s delays). After 3 failures, the pipeline aborts with the original error.
30+
31+
### 4. Wave Container Build Failures
32+
33+
**When:** Using `wave` profile with `spack` strategy.
34+
35+
**Error:** Wave freeze builds fail with spack.
36+
37+
**Fix:** The wave profile explicitly uses `strategy = ['conda', 'container', 'dockerfile']` — no spack. Don't add spack to the strategy list.
38+
39+
### 5. CGROUPv2 Docker Failures (Cloud VM / Firecracker)
40+
41+
**When:** Running with Docker in Cloud VMs where cgroup resource delegation is restricted.
42+
43+
**Error:** `cannot enter cgroupv2 ... with domain controllers`
44+
45+
**Fix:** Apply the runc wrapper documented in AGENTS.md that strips `linux.resources` from the OCI spec.
46+
47+
### 6. Empty Benchmark Report
48+
49+
**When:** API runs are provided but `--generate_benchmark_report` is not set.
50+
51+
**Warning:** `Found N API run(s) but --generate_benchmark_report is not enabled. API runs will not produce any output.`
52+
53+
**Fix:** Add `--generate_benchmark_report` to the run command.
54+
55+
### 7. Scheduling Overhead (Observed Pattern)
56+
57+
**Not an error per se**, but both current runs show 4–8 minute gaps between task submit and task start times. This is expected with AWS Batch spot instances — EC2 instances must be provisioned and containers pulled before execution begins. Not actionable unless overhead exceeds ~15 minutes consistently.
58+
59+
## Data Gaps
60+
61+
- **No failed runs analysed:** Both Platform runs are currently successful. To populate this section with real failure analysis, re-run context generation after a failure occurs or provide historical failed run IDs.
62+
- **No STAR/alignment task data yet:** Both runs are still in early stages (QC/trim). Resource pressure and potential OOM failures during STAR genome generation and alignment are not yet observable.
63+
- **No cost data:** Neither run has completed, so total run cost is unavailable. Individual task costs are in the $0.001–$0.007 range for early QC tasks.

‎.seqera/context/HISTORY.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# HISTORY.md — Recent Runs Analysed
2+
3+
## Seqera Platform Runs (unified-compute/sched-testing)
4+
5+
### Run 1: `maniac_celsius` (1cF65l2PDvxNd5)
6+
7+
- **Pipeline:** nf-core/rnaseq
8+
- **Status:** RUNNING (as of 2026-04-17T18:45)
9+
- **Submitted:** 2026-04-17T17:51:26Z
10+
- **Started:** 2026-04-17T17:53:20Z
11+
- **User:** edmund-miller
12+
- **Executor:** seqera/aws (AWS Batch, eu-west-1)
13+
- **Price model:** spot instances
14+
- **Workspace:** unified-compute/sched-testing (ID: 224344613053615)
15+
- **Work dir:** s3://nextflow-ci-dev/scratch/1cF65l2PDvxNd5/
16+
17+
**Task Summary (30 tasks captured from API fetch):**
18+
19+
- 21 tasks COMPLETED, 9 tasks RUNNING, 0 FAILED
20+
- Completed early-stage tasks include `SAMTOOLS_FAIDX`, `GTF_FILTER`, `FQ_LINT`, and `GTF2BED`
21+
- Active work is mostly `TRIMGALORE` fan-out across samples plus `SALMON_INDEX`
22+
- `STAR_GENOMEGENERATE_IGENOMES` is defined in the run graph but was not yet observed as running/completed in the fetched task set
23+
24+
**Scheduling overhead:** Tasks show 4–8 min gap between submit and start (spot instance spin-up).
25+
26+
### Run 2: `clever_kalman` (5VGC0gjEmghOaz)
27+
28+
- **Pipeline:** nf-core/rnaseq
29+
- **Status:** RUNNING (as of 2026-04-17T18:45)
30+
- **Submitted:** 2026-04-17T17:50:14Z
31+
- **Started:** 2026-04-17T17:52:09Z
32+
- **User:** edmund-miller
33+
- **Executor:** seqera/aws (AWS Batch, eu-west-1)
34+
- **Price model:** spot instances
35+
- **Workspace:** unified-compute/sched-testing (ID: 224344613053615)
36+
- **Work dir:** s3://nextflow-ci-dev/scratch/5VGC0gjEmghOaz/
37+
38+
**Task Summary (30 tasks captured from API fetch):**
39+
40+
- 22 tasks COMPLETED, 8 tasks RUNNING, 0 FAILED
41+
- Completed early-stage tasks include `SAMTOOLS_FAIDX`, `GTF_FILTER`, `FQ_LINT`, and `GTF2BED`
42+
- Active work is mostly `TRIMGALORE` fan-out across samples
43+
- `STAR_GENOMEGENERATE_IGENOMES` is defined in the run graph but was not yet observed as running/completed in the fetched task set
44+
45+
**Scheduling overhead:** Similar 5–6 min submit-to-start gaps.
46+
47+
## Comparison Notes
48+
49+
Both runs are concurrent nf-core/rnaseq executions on the same workspace with identical test data (GM12878_REP1, GM12878_REP2). Key differences:
50+
51+
| Metric | 1cF65l2PDvxNd5 | 5VGC0gjEmghOaz |
52+
| ----------------------- | ------------------------------------ | -------------------------- |
53+
| Submit time | 17:51:26Z | 17:50:14Z |
54+
| Machine types | mix of m5d.large + c5d.large | mostly c5d.large |
55+
| SAMTOOLS_FAIDX realtime | 28s | 62s |
56+
| GTF_FILTER realtime | 58s | 109s |
57+
| FQ_LINT avg realtime | ~605s | ~613s |
58+
| Cost pattern | Slightly higher per-task (m5d.large) | Lower per-task (c5d.large) |
59+
60+
The timing variance in short tasks (SAMTOOLS_FAIDX: 28s vs 62s) likely reflects spot instance variability and S3 staging latency rather than algorithmic differences. Both runs are still in progress — STAR genome generation and alignment phases are pending.
61+
62+
## Local Development History (git log)
63+
64+
Recent development has focused on the **benchmark report rendering pipeline**:
65+
66+
1. **Split architecture** (df2232e): Refactored monolithic benchmark report into 3 stages — normalize JSONL → aggregate → render
67+
2. **Combined runtime charts** (18a64f4 → a49a9a4): Added combined task runtime visualisation with scheduling overhead, per-pipeline breakdowns, and ECharts integration
68+
3. **CUR cost streaming** (b96ecfd): Optimised AWS CUR parquet processing with batch streaming
69+
4. **Test colocation** (96b8b81): Moved benchmark tests to live alongside their modules

‎.seqera/context/PATTERNS.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# PATTERNS.md — Codebase Conventions and Run Patterns
2+
3+
## Pipeline Architecture Patterns
4+
5+
### 1. Branch-and-Merge Input Strategy
6+
7+
The workflow uses `.branch {}` to split input into API vs external paths, then merges all JSON outputs via `.mix().collect()` before the benchmark pipeline. This pattern allows mixed-source runs in a single CSV.
8+
9+
```
10+
ids.branch { api: ...; external: ... }
11+
→ separate processing paths
12+
→ ch_api_jsons.mix(ch_tarball_jsons).mix(ch_external_dir_jsons).collect()
13+
```
14+
15+
### 2. In-Process API Fetch (No Task)
16+
17+
`SeqeraApi.fetchRunData()` runs inside the Nextflow process (Groovy), NOT as a Nextflow task. This means:
18+
19+
- No container, no retry directive, no task monitoring
20+
- Retry logic is manually coded with exponential backoff (3 attempts)
21+
- Token is read from environment at execution time
22+
- Results are written to temp files and emitted as channel values
23+
24+
### 3. Three-Stage Python Pipeline
25+
26+
The benchmark report follows a strict data pipeline:
27+
28+
1. **Normalize** (raw JSON → JSONL bundle) — streaming, handles CUR parquet
29+
2. **Aggregate** (JSONL → report_data.json) — statistical rollups
30+
3. **Render** (report_data.json → HTML) — Jinja2 template + ECharts
31+
32+
Each stage is a separate Nextflow process with the same container image. The boundary between stages is explicit (JSONL files, then JSON), making debugging straightforward.
33+
34+
### 4. External Test Fixtures
35+
36+
Tests use pre-exported tarball fixtures (`workflows/nf_aggregate/assets/log_dirs/`, referenced from `test_benchmark.csv`) rather than live API calls. This enables fully offline CI.
37+
38+
## Coding Conventions
39+
40+
### Nextflow
41+
42+
- **DSL2** with `include {}` for all modules
43+
- **No strict syntax yet** — uses `Channel.empty()`, implicit closures in places
44+
- Processes use `conda` + `container` directives (Wave-compatible)
45+
- All processes emit `versions.yml` for software version tracking
46+
- `publishDir` configured per-process in `workflows/nf_aggregate/nextflow.config`
47+
48+
### Python (bin/)
49+
50+
- **Typer CLI** with subcommands for each pipeline stage
51+
- Modular: `benchmark_report.py` is a thin dispatcher; logic lives in `benchmark_report_normalize.py`, `benchmark_report_aggregate.py`, `benchmark_report_render.py`, `benchmark_report_fetch.py`
52+
- **JSONL** as intermediate format (streaming-friendly, Fusion-compatible)
53+
- **Jinja2** for HTML templating with ECharts for charts
54+
- **PyArrow** for CUR parquet reading (batched streaming for memory efficiency)
55+
- Test runner: `pytest` with tests colocated under each module's `tests/` directory
56+
57+
### Testing
58+
59+
- **nf-test** for pipeline integration (4 test suites: benchmark-tarball, benchmark-directory, api-only, mixed-no-benchmark)
60+
- **pytest** for Python unit tests (normalize, aggregate, render, fetch)
61+
- **[nft-utils@0.0.4](mailto:nft-utils@0.0.4)** plugin for snapshot assertions
62+
- Test config: `tests/nextflow.config` (intentionally empty — relies on profile `test`)
63+
- Snapshots verify task counts, output file lists, and software versions
64+
65+
### Configuration
66+
67+
- Container registry: `quay.io` (default for all runtimes)
68+
- Docker runs as current user: `-u $(id -u):$(id -g)`
69+
- Process defaults: 1 CPU, 6 GB memory, 4h time
70+
- Error strategy: retry on exit codes 130–145 and 104 (OOM/signal kills)
71+
72+
## Run Patterns (from Platform observations)
73+
74+
### Scheduling Overhead
75+
76+
Both observed runs show consistent 4–8 minute submit→start latency. This is a characteristic of AWS Batch with spot instances:
77+
78+
- Instance provisioning
79+
- Container image pull (Wave-built images)
80+
- S3 staging of inputs
81+
82+
### Instance Type Selection
83+
84+
AWS Batch auto-selects instance types based on resource requests:
85+
86+
- 1 CPU / 6 GB → m5d.large or c5d.large
87+
- 2 CPU / 2 GB → c5d.large (compute-optimised)
88+
- All spot pricing
89+
90+
### Cost Efficiency
91+
92+
Early QC tasks cost $0.001–$0.007 each. The dominant cost will come from STAR genome generation and alignment tasks (not yet observed in these in-progress runs).
93+
94+
## File Naming Conventions
95+
96+
| Pattern | Convention |
97+
| ---------------------- | ------------------------------------------- |
98+
| Modules | `modules/local/<name>/main.nf` |
99+
| Module tests (Python) | `modules/local/<name>/tests/test_<name>.py` |
100+
| Module docs | `modules/local/<name>/AGENTS.md` |
101+
| Pipeline tests | `tests/<scenario>/main.nf.test` |
102+
| Pipeline test fixtures | `workflows/nf_aggregate/assets/` |
103+
| Bin scripts | `bin/benchmark_report_<stage>.py` |
104+
| Agent docs | `AGENTS.md` at each directory level |
105+
106+
## Git Workflow
107+
108+
- Feature branches named `<user>/<description>`
109+
- Conventional commits: `feat()`, `fix()`, `test()`, `refactor()`, `docs()`, `perf()`
110+
- Commit signing required (GPG via SSH/1Password)
111+
- Pre-commit hooks for linting

‎.seqera/context/PIPELINE.md‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# PIPELINE.md — seqeralabs/nf-aggregate
2+
3+
## Overview
4+
5+
**nf-aggregate** aggregates task-level metrics from Seqera Platform pipeline runs and produces benchmark HTML reports. It fetches run data via the Platform API (or from pre-exported tarballs/directories), normalises it to JSONL, aggregates statistics, and renders a branded interactive HTML report with ECharts visualisations.
6+
7+
- **Repository:** [https://github.com/seqeralabs/nf-aggregate](https://github.com/seqeralabs/nf-aggregate)
8+
- **Latest released version:** 0.7.0 (2025-05-05)
9+
- **Current branch:** `edmundmiller/seqera-context` (active development)
10+
- **Plugins:** [nf-schema@2.3.0](mailto:nf-schema@2.3.0) (param validation, samplesheet parsing), [nf-boost@0.6.0](mailto:nf-boost@0.6.0) (request(), JSON helpers)
11+
- **Required env:** `TOWER_ACCESS_TOKEN` for API-mode runs
12+
13+
## Architecture
14+
15+
```
16+
Input CSV (id, workspace, group, logs, platform, token_env)
17+
│
18+
├─ branch: api → SeqeraApi.fetchRunData() [Groovy, in-process]
19+
│ └── JSON file per run
20+
├─ branch: external (tarball) → EXTRACT_TARBALL
21+
│ └── JSON files from tar.gz
22+
└─ branch: external (dir) → direct JSON collection
23+
│
24+
▼
25+
ch_data_dir (all JSON files merged)
26+
│
27+
▼
28+
NORMALIZE_BENCHMARK_JSONL
29+
│ raw JSON → jsonl_bundle/ (runs.jsonl, tasks.jsonl, metrics.jsonl)
30+
▼
31+
AGGREGATE_BENCHMARK_REPORT_DATA
32+
│ jsonl_bundle/ → report_data.json
33+
▼
34+
RENDER_BENCHMARK_REPORT
35+
│ report_data.json + brand.yml + logo.svg → benchmark_report.html
36+
▼
37+
results/benchmark_report/
38+
```
39+
40+
## Entry Points
41+
42+
| File | Role |
43+
| -------------------------------- | -------------------------------------------------------------------------------- |
44+
| `main.nf` | Pipeline entry — validates params, parses samplesheet, delegates to NF_AGGREGATE |
45+
| `workflows/nf_aggregate/main.nf` | Core workflow — branching, API fetch, process chaining |
46+
| `lib/SeqeraApi.groovy` | Groovy API client — paginated task fetch, workspace resolution |
47+
| `bin/benchmark_report.py` | Typer CLI — normalize-jsonl, aggregate-report-data, render-html, fetch |
48+
49+
## Processes (4 local modules)
50+
51+
| Process | Container | Input | Output |
52+
| --------------------------------- | ------------------------------- | ------------------------------------------- | --------------------------- |
53+
| `EXTRACT_TARBALL` | ubuntu:22.04 | `(meta, tarball)` | `(meta, dir)` of JSON files |
54+
| `NORMALIZE_BENCHMARK_JSONL` | wave python/duckdb/jinja2/typer | `data_dir`, `cur_parquet` | `jsonl_bundle/` |
55+
| `AGGREGATE_BENCHMARK_REPORT_DATA` | wave python/duckdb/jinja2/typer | `jsonl_bundle/` | `report_data.json` |
56+
| `RENDER_BENCHMARK_REPORT` | wave python/duckdb/jinja2/typer | `report_data.json`, `brand.yml`, `logo.svg` | `benchmark_report.html` |
57+
58+
## Key Parameters
59+
60+
| Parameter | Default | Purpose |
61+
| --------------------------- | ----------------------------- | ------------------------------------------- |
62+
| `input` | required | CSV samplesheet of run IDs / external paths |
63+
| `outdir` | `results` | Output directory |
64+
| `generate_benchmark_report` | `false` | Enable the benchmark pipeline |
65+
| `benchmark_aws_cur_report` | `null` | AWS CUR parquet for cost analysis |
66+
| `seqera_api_endpoint` | `https://api.cloud.seqera.io` | Platform API base URL |
67+
| `java_truststore_path` | `null` | Custom Java truststore for private certs |
68+
| `java_truststore_password` | `null` | Truststore password |
69+
70+
## Input Schema
71+
72+
CSV with columns: `id` (required), `workspace` (required, `org/name` or `external`), `group` (optional), `logs` (path to tarball/dir for external), `platform` (per-row API URL override), `token_env` (per-row env var name for bearer token).
73+
74+
## Publish Structure
75+
76+
```
77+
results/
78+
├── benchmark_report/
79+
│ ├── benchmark_report.html
80+
│ ├── report_data.json
81+
│ └── jsonl_bundle/
82+
│ ├── runs.jsonl
83+
│ ├── tasks.jsonl
84+
│ └── metrics.jsonl
85+
└── pipeline_info/
86+
└── collated_software_versions.yml
87+
```
88+
89+
## Data Flow Details
90+
91+
1. **API path:** `SeqeraApi.fetchRunData()` runs in Groovy process memory (not a Nextflow task). It resolves workspace name → ID, then fetches `/workflow/{id}`, `/workflow/{id}/metrics`, `/workflow/{id}/tasks` (paginated), and `/workflow/{id}/progress`. Results are written to temp JSON files.
92+
2. **External path:** EXTRACT_TARBALL unpacks `.tar.gz` into a directory of JSON files. Directories are used directly.
93+
3. **All JSON files** are collected into a single temp directory and passed to the 3-stage Python pipeline: normalize → aggregate → render.
94+
4. The Python stages are separate Nextflow processes sharing one Wave container image (`python_duckdb_jinja2_typer_pruned`).
95+
5. JSONL is the handoff format — streaming-friendly for large run datasets and Fusion FS compatible.

0 commit comments

Comments
 (0)