Skip to content

Commit 4aceb4a

Browse files
fix(lint): satisfy pre-commit formatting
Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>
1 parent c26929b commit 4aceb4a

20 files changed

Lines changed: 5039 additions & 30 deletions

File tree

.seqera/context/ERRORS.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,4 +61,3 @@ Both runs `1cF65l2PDvxNd5` (maniac_celsius) and `5VGC0gjEmghOaz` (clever_kalman)
6161
- **No failed runs analysed:** Both Platform runs are currently successful. To populate this section with real failure analysis, re-run context generation after a failure occurs or provide historical failed run IDs.
6262
- **No STAR/alignment task data yet:** Both runs are still in early stages (QC/trim). Resource pressure and potential OOM failures during STAR genome generation and alignment are not yet observable.
6363
- **No cost data:** Neither run has completed, so total run cost is unavailable. Individual task costs are in the $0.001–$0.007 range for early QC tasks.
64-

.seqera/context/HISTORY.md

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,6 @@
4848

4949
Both runs are concurrent nf-core/rnaseq executions on the same workspace with identical test data (GM12878_REP1, GM12878_REP2). Key differences:
5050

51-
5251
| Metric | 1cF65l2PDvxNd5 | 5VGC0gjEmghOaz |
5352
| ----------------------- | ------------------------------------ | -------------------------- |
5453
| Submit time | 17:51:26Z | 17:50:14Z |
@@ -58,7 +57,6 @@ Both runs are concurrent nf-core/rnaseq executions on the same workspace with id
5857
| FQ_LINT avg realtime | ~605s | ~613s |
5958
| Cost pattern | Slightly higher per-task (m5d.large) | Lower per-task (c5d.large) |
6059

61-
6260
The timing variance in short tasks (SAMTOOLS_FAIDX: 28s vs 62s) likely reflects spot instance variability and S3 staging latency rather than algorithmic differences. Both runs are still in progress — STAR genome generation and alignment phases are pending.
6361

6462
## Local Development History (git log)
@@ -69,4 +67,3 @@ Recent development has focused on the **benchmark report rendering pipeline**:
6967
2. **Combined runtime charts** (18a64f4 → a49a9a4): Added combined task runtime visualisation with scheduling overhead, per-pipeline breakdowns, and ECharts integration
7068
3. **CUR cost streaming** (b96ecfd): Optimised AWS CUR parquet processing with batch streaming
7169
4. **Test colocation** (96b8b81): Moved benchmark tests to live alongside their modules
72-

.seqera/context/PATTERNS.md

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,6 @@ Early QC tasks cost $0.001–$0.007 each. The dominant cost will come from STAR
9393

9494
## File Naming Conventions
9595

96-
9796
| Pattern | Convention |
9897
| ---------------------- | ------------------------------------------- |
9998
| Modules | `modules/local/<name>/main.nf` |
@@ -104,11 +103,9 @@ Early QC tasks cost $0.001–$0.007 each. The dominant cost will come from STAR
104103
| Bin scripts | `bin/benchmark_report_<stage>.py` |
105104
| Agent docs | `AGENTS.md` at each directory level |
106105

107-
108106
## Git Workflow
109107

110108
- Feature branches named `<user>/<description>`
111109
- Conventional commits: `feat()`, `fix()`, `test()`, `refactor()`, `docs()`, `perf()`
112110
- Commit signing required (GPG via SSH/1Password)
113111
- Pre-commit hooks for linting
114-

.seqera/context/PIPELINE.md

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -39,29 +39,24 @@ Input CSV (id, workspace, group, logs, platform, token_env)
3939

4040
## Entry Points
4141

42-
4342
| File | Role |
4443
| -------------------------------- | -------------------------------------------------------------------------------- |
4544
| `main.nf` | Pipeline entry — validates params, parses samplesheet, delegates to NF_AGGREGATE |
4645
| `workflows/nf_aggregate/main.nf` | Core workflow — branching, API fetch, process chaining |
4746
| `lib/SeqeraApi.groovy` | Groovy API client — paginated task fetch, workspace resolution |
4847
| `bin/benchmark_report.py` | Typer CLI — normalize-jsonl, aggregate-report-data, render-html, fetch |
4948

50-
5149
## Processes (4 local modules)
5250

53-
5451
| Process | Container | Input | Output |
5552
| --------------------------------- | ------------------------------- | ------------------------------------------- | --------------------------- |
5653
| `EXTRACT_TARBALL` | ubuntu:22.04 | `(meta, tarball)` | `(meta, dir)` of JSON files |
5754
| `NORMALIZE_BENCHMARK_JSONL` | wave python/duckdb/jinja2/typer | `data_dir`, `cur_parquet` | `jsonl_bundle/` |
5855
| `AGGREGATE_BENCHMARK_REPORT_DATA` | wave python/duckdb/jinja2/typer | `jsonl_bundle/` | `report_data.json` |
5956
| `RENDER_BENCHMARK_REPORT` | wave python/duckdb/jinja2/typer | `report_data.json`, `brand.yml`, `logo.svg` | `benchmark_report.html` |
6057

61-
6258
## Key Parameters
6359

64-
6560
| Parameter | Default | Purpose |
6661
| --------------------------- | ----------------------------- | ------------------------------------------- |
6762
| `input` | required | CSV samplesheet of run IDs / external paths |
@@ -72,7 +67,6 @@ Input CSV (id, workspace, group, logs, platform, token_env)
7267
| `java_truststore_path` | `null` | Custom Java truststore for private certs |
7368
| `java_truststore_password` | `null` | Truststore password |
7469

75-
7670
## Input Schema
7771

7872
CSV with columns: `id` (required), `workspace` (required, `org/name` or `external`), `group` (optional), `logs` (path to tarball/dir for external), `platform` (per-row API URL override), `token_env` (per-row env var name for bearer token).
@@ -99,4 +93,3 @@ results/
9993
3. **All JSON files** are collected into a single temp directory and passed to the 3-stage Python pipeline: normalize → aggregate → render.
10094
4. The Python stages are separate Nextflow processes sharing one Wave container image (`python_duckdb_jinja2_typer_pruned`).
10195
5. JSONL is the handoff format — streaming-friendly for large run datasets and Fusion FS compatible.
102-

.seqera/skills/fetch-and-compare-platform-runs/SKILL.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ description: >
66
benchmark candidates, refreshing `.seqera/context`, or validating run-level
77
differences before building a report.
88
---
9+
910
# Fetch and compare Platform runs
1011

1112
Use this skill when the user wants to inspect one or more Seqera Platform runs
@@ -78,4 +79,3 @@ Build a full benchmark report when:
7879
- the user wants charts or HTML output
7980
- cost and aggregated process summaries matter more than raw run inspection
8081
- you need JSONL handoff artifacts for downstream debugging
81-

AGENTS.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -66,10 +66,10 @@ uv run --with typer --with pyyaml \
6666

6767
### Services overview
6868

69-
| Service | Purpose | Run command |
70-
|---------|---------|-------------|
71-
| Nextflow pipeline | Core product — aggregates metrics from Seqera Platform runs | `nextflow run . --input <csv> --outdir results -profile docker` |
72-
| Python benchmark_report.py | Normalizes JSON -> JSONL, aggregates report data, then renders HTML | See "Rebuild Command" section above |
69+
| Service | Purpose | Run command |
70+
| -------------------------- | ------------------------------------------------------------------- | --------------------------------------------------------------- |
71+
| Nextflow pipeline | Core product — aggregates metrics from Seqera Platform runs | `nextflow run . --input <csv> --outdir results -profile docker` |
72+
| Python benchmark_report.py | Normalizes JSON -> JSONL, aggregates report data, then renders HTML | See "Rebuild Command" section above |
7373

7474
### Running tests
7575

docs/DESIGN.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ python bin/benchmark_report.py normalize-jsonl --data-dir <run_json_dir> --outpu
3636
```
3737

3838
Responsibilities:
39+
3940
- read run JSON payloads
4041
- normalize runs/tasks/metrics rows
4142
- optionally normalize CUR parquet into `costs.jsonl`
@@ -54,6 +55,7 @@ python bin/benchmark_report.py aggregate-report-data --jsonl-dir <jsonl_bundle>
5455
```
5556

5657
Responsibilities:
58+
5759
- stream JSONL rows
5860
- compute report sections:
5961
- `benchmark_overview`
@@ -75,13 +77,15 @@ python bin/benchmark_report.py render-html --data <report_data.json> --output <b
7577
```
7678

7779
Responsibilities:
80+
7881
- load report JSON
7982
- apply brand/logo overrides when provided
8083
- render self-contained HTML from the Jinja template
8184

8285
## Benchmark CLI surface
8386

8487
`bin/benchmark_report.py` provides:
88+
8589
- `normalize-jsonl`
8690
- `aggregate-report-data`
8791
- `render-html`

modules/local/AGENTS.md

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,14 @@
22

33
Each module should keep one clear responsibility and its own terse `AGENTS.md`.
44

5-
| Module | Status | Purpose |
6-
| --------------------------------------- | ---------- | ---------------------------------------------- |
7-
| `normalize_benchmark_jsonl/` | **active** | Raw run JSON (+ optional CUR parquet) → JSONL |
8-
| `aggregate_benchmark_report_data/` | **active** | JSONL bundle → `report_data.json` |
9-
| `render_benchmark_report/` | **active** | `report_data.json` + branding → HTML |
10-
| `extract_tarball/` | active | Extract run-data tarballs for external runs |
5+
| Module | Status | Purpose |
6+
| ---------------------------------- | ---------- | --------------------------------------------- |
7+
| `normalize_benchmark_jsonl/` | **active** | Raw run JSON (+ optional CUR parquet) → JSONL |
8+
| `aggregate_benchmark_report_data/` | **active** | JSONL bundle → `report_data.json` |
9+
| `render_benchmark_report/` | **active** | `report_data.json` + branding → HTML |
10+
| `extract_tarball/` | active | Extract run-data tarballs for external runs |
1111

1212
Testing
13+
1314
- Keep stage-specific pytest tests beside the module under `tests/`.
1415
- Keep pipeline routing/integration scenarios under top-level `tests/`.

modules/local/aggregate_benchmark_report_data/AGENTS.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,32 @@
11
# aggregate_benchmark_report_data
22

33
Purpose
4+
45
- Convert the JSONL bundle into a single `report_data.json` document for rendering.
56

67
Owns
8+
79
- `AGGREGATE_BENCHMARK_REPORT_DATA` in `main.nf`
810
- Module-local CLI entrypoint in `bin/benchmark_report.py`
911
- Shared aggregation logic in repo-root `bin/benchmark_report_aggregate.py`
1012
- Stage-scoped tests under `tests/`
1113

1214
Run directly
15+
1316
- `nextflow run modules/local/aggregate_benchmark_report_data/main.nf -profile docker,arm --jsonl_bundle <jsonl_bundle_dir>`
1417
- The direct `nextflow run modules/local/...` path depends on the module-local `bin/benchmark_report.py` shim/CLI; keep that invocation shape working if you refactor the stage.
1518

1619
Inputs
20+
1721
- `jsonl_bundle/` from `normalize_benchmark_jsonl`
1822

1923
Outputs
24+
2025
- `report_data.json`
2126
- `versions.yml`
2227

2328
Invariants
29+
2430
- This is the boundary between streaming records and presentation data.
2531
- Prefer streaming iteration over JSONL inputs; avoid full-file eager loads unless clearly bounded.
2632
- Cost joins must key by `(run_id, process, hash)` to avoid cross-process collisions.
@@ -36,8 +42,10 @@ Invariants
3642
- `cost_overview`
3743

3844
Edit guidance
45+
3946
- Keep rendering/template logic out of this stage.
4047
- If output schema changes, update render tests and any HTML assertions.
4148

4249
Tests
50+
4351
- `pytest modules/local/aggregate_benchmark_report_data/tests/test_aggregate.py -q`
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,27 @@
11
# extract_tarball
22

33
Purpose
4+
45
- Extract external run-data tarballs into directories of JSON artifacts.
56

67
Owns
8+
79
- `EXTRACT_TARBALL` in `main.nf`
810

911
Inputs
12+
1013
- external tarball paths from the workflow routing layer
1114

1215
Outputs
16+
1317
- extracted directory per tarball
1418
- `versions.yml`
1519

1620
Invariants
21+
1722
- Extraction only. Do not mix benchmark parsing or aggregation into this module.
1823
- Downstream workflow code is responsible for collecting JSON files from extracted directories.
1924

2025
Edit guidance
26+
2127
- If tarball layout assumptions change, update the pipeline tarball scenario tests.

0 commit comments

Comments
 (0)