Merge pull request #125 from seqeralabs/edmundmiller/split-up-table-generation-and-report-generation

refactor: split benchmark report into JSONL stages

Author: edmundmiller

.gitignore

@@ -19,3 +19,4 @@ modules/local/benchmark_report_v2/tests/data/
 .beads/
 .lineage/
 .claude/
+.hermes/

AGENTS.md

@@ -8,8 +8,9 @@ Nextflow pipeline to aggregate metrics across Seqera Platform pipeline runs.
 input CSV (id, workspace, group, logs, fusion)
   → branch: api (SeqeraApi.fetchRunData) | external (EXTRACT_TARBALL)
   → collect JSON files
-  → BENCHMARK_REPORT process (benchmark_report.py build-db + report)
-  → benchmark.duckdb + benchmark_report.html
+  → NORMALIZE_BENCHMARK_JSONL (raw JSON -> jsonl_bundle/)
+  → AGGREGATE_BENCHMARK_REPORT_DATA (jsonl_bundle -> report_data.json)
+  → RENDER_BENCHMARK_REPORT (report_data.json -> benchmark_report.html)
 ```
 
 ## Key Params
@@ -32,26 +33,32 @@ input CSV (id, workspace, group, logs, fusion)
 ## Rebuild Command (local testing)
 
 ```bash
-# Build DuckDB from JSON data:
-uv run --with duckdb --with typer --with pyyaml --with pyarrow \
-  python bin/benchmark_report.py build-db \
-  --data-dir /path/to/json_data --output /tmp/benchmark.duckdb
+# Normalize raw run JSON (+ optional CUR parquet) to JSONL bundle:
+uv run --with typer --with pyyaml --with pyarrow \
+  python bin/benchmark_report.py normalize-jsonl \
+  --data-dir /path/to/json_data --output-dir /tmp/jsonl_bundle
 
-# Render HTML report from DuckDB:
-uv run --with duckdb --with jinja2 --with typer --with pyyaml \
-  python bin/benchmark_report.py report \
-  --db /tmp/benchmark.duckdb --brand assets/brand.yml --output /tmp/report.html
+# Aggregate JSONL bundle to report data:
+uv run --with typer --with pyyaml \
+  python bin/benchmark_report.py aggregate-report-data \
+  --jsonl-dir /tmp/jsonl_bundle --output /tmp/report_data.json
+
+# Render HTML report from report_data.json:
+uv run --with jinja2 --with typer --with pyyaml \
+  python bin/benchmark_report.py render-html \
+  --data /tmp/report_data.json --brand assets/brand.yml --output /tmp/report.html
 
 # Fetch run data from Seqera Platform API (standalone):
-uv run --with duckdb --with typer --with pyyaml --with httpx \
+uv run --with typer --with pyyaml \
   python bin/benchmark_report.py fetch \
   --run-ids <id> --workspace org/name --output-dir /tmp/json_data
 ```
 
 ## Gotchas
 
 - Wave freeze strategy: `['conda', 'container', 'dockerfile']` — no `spack` (breaks builds)
-- DuckDB `read_json_auto` needs file paths, not JSON strings — use temp files
+- JSONL is the primary handoff (`jsonl_bundle/`) for Fusion-friendly streaming
+- `report_data.json` is the explicit boundary between aggregation and rendering
 - `commit.gpgsign` must be true (SSH signing via 1Password)
 - RTK `buildOutputFiltering` / `testOutputAggregation` can swallow nf-test output — disable to debug
 
@@ -62,14 +69,22 @@ uv run --with duckdb --with typer --with pyyaml --with httpx \
 | Service | Purpose | Run command |
 |---------|---------|-------------|
 | Nextflow pipeline | Core product — aggregates metrics from Seqera Platform runs | `nextflow run . --input <csv> --outdir results -profile docker` |
-| Python benchmark_report.py | Builds DuckDB + renders HTML report | See "Rebuild Command" section above |
+| Python benchmark_report.py | Normalizes JSON -> JSONL, aggregates report data, then renders HTML | See "Rebuild Command" section above |
 
 ### Running tests
 
-- **pytest (Python unit tests):** `uv run --with duckdb --with typer --with pyyaml --with jinja2 --with pyarrow --with pytest --with httpx pytest bin/test_benchmark_report.py -v`
+- **pytest (Python unit tests):** `uv run --with typer --with pyyaml --with jinja2 --with pyarrow --with pytest --with httpx pytest -v modules/local/aggregate_benchmark_report_data/tests/test_aggregate.py modules/local/normalize_benchmark_jsonl/tests/test_normalize.py modules/local/render_benchmark_report/tests/test_render.py bin/test_benchmark_report_fetch.py`
 - **nf-test (pipeline integration tests):** `nf-test test --profile=+docker --verbose`
 - **Lint:** `pre-commit run --all-files`
 
+### Agent quick verify (offline)
+
+Run this exact sequence when validating split benchmark-report changes in Cursor Cloud:
+
+1. `uv run --with typer --with pyyaml --with jinja2 --with pyarrow --with pytest --with httpx pytest -v modules/local/aggregate_benchmark_report_data/tests/test_aggregate.py modules/local/normalize_benchmark_jsonl/tests/test_normalize.py modules/local/render_benchmark_report/tests/test_render.py bin/test_benchmark_report_fetch.py`
+2. `nf-test test --profile=+docker --verbose`
+3. `nextflow run . --input workflows/nf_aggregate/assets/test_benchmark.csv --generate_benchmark_report --outdir /tmp/nf-aggregate-e2e-results -profile docker`
+
 ### Docker in Cloud VM (cgroupv2 workaround)
 
 The Cloud VM runs inside a Firecracker VM where the root cgroupv2 hierarchy cannot delegate `memory`/`io` controllers. Docker containers that request resource limits (`--memory`, `--cpu-shares` — used by Nextflow's `process { memory; cpus }`) will fail with `"cannot enter cgroupv2 ... with domain controllers"`.

README.md

@@ -91,8 +91,9 @@ The results from the pipeline will be published in the path specified by the `--
 ```
 ./results
 ├── benchmark_report/
-│   ├── benchmark.duckdb
-│   └── benchmark_report.html                ## Benchmark report
+│   ├── benchmark_report.html                ## Benchmark report
+│   ├── report_data.json                     ## Aggregated report data boundary
+│   └── jsonl_bundle/                        ## Streaming stage handoff (runs/tasks/metrics[/costs].jsonl)
 └── pipeline_info/
 ```
 

bin/AGENTS.md

@@ -4,21 +4,34 @@ Python scripts executed inside Nextflow process containers. Placed in `bin/` so
 
 ## benchmark_report.py
 
-Unified benchmark report CLI. Typer app with 3 subcommands:
+Thin Typer CLI wrapper around focused modules.
 
 | Subcommand | Purpose |
 |---|---|
-| `build-db` | JSON files (+ optional CUR parquet) → `benchmark.duckdb` with normalized tables (runs, tasks, metrics, costs) |
-| `report` | Opens `.duckdb` file → runs 9 SQL queries → renders self-contained HTML with eCharts |
-| `fetch` | Calls Seqera Platform API → writes run JSON files (standalone use, not used by Nextflow pipeline) |
+| `normalize-jsonl` | Raw run JSON (+ optional CUR parquet) → `jsonl_bundle/` (`runs.jsonl`, `tasks.jsonl`, `metrics.jsonl`, optional `costs.jsonl`) |
+| `aggregate-report-data` | `jsonl_bundle/` → `report_data.json` |
+| `render-html` | `report_data.json` + brand/logo assets → self-contained HTML |
+| `report` | Convenience wrapper (`aggregate-report-data` + `render-html`) |
+| `fetch` | Calls Seqera Platform API → writes run JSON files (standalone use) |
 
-### DuckDB Tables
+## Focused modules
 
-- `runs` — one row per workflow run (id, group, pipeline, status, duration, efficiency, etc.)
-- `tasks` — one row per task (hash, process, cost, cpus, memory, realtime, etc.)
-- `metrics` — per-process resource stats (cpu/mem/time mean/min/q1-q3/max)
-- `costs` — optional, from AWS CUR parquet (run_id, process, hash, cost, used_cost, unused_cost)
+- `benchmark_report_normalize.py` — normalization stage
+- `benchmark_report_aggregate.py` — report aggregation stage
+- `benchmark_report_render.py` — HTML rendering stage
+- `benchmark_report_fetch.py` — Seqera API fetch helpers
 
-### Dependencies
+### Test ownership
 
-`duckdb`, `jinja2`, `typer`, `pyyaml`, `pyarrow`, `httpx` (fetch only)
+Keep tests close to the stage they exercise:
+
+- module-stage tests live under `modules/local/*/tests/`
+- CLI- and fetch-specific tests stay in `bin/`
+- shared pytest fixtures live in repo-root `conftest.py`
+
+### Dependencies by stage
+
+- normalize: `typer`, `pyyaml`, `pyarrow` (optional CUR parquet)
+- aggregate: stdlib (+ Typer in wrapper)
+- render: `jinja2`, `pyyaml`
+- fetch: stdlib networking