fix(report): dynamic height + force all labels on cost boxplot (#119)

* fix(report): dynamic height + force all labels on cost boxplot

With many processes (44+), the fixed 500px height caused ECharts to
hide Y-axis labels. RSEQC_TIN's $3-9 box appeared to belong to RUSTQC.

- Scale chart height to 25px per process (min 500px)
- Set interval: 0 to force all Y-axis labels visible

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: compute run metrics from tasks when workflowProgress is missing

When runs are imported from Nextflow log tarballs (rather than fetched
from the Platform API), the progress.workflowProgress object is empty.
This caused CPU time, CPU/memory efficiency, and I/O bytes to show as 0
in benchmark charts.

Added _compute_progress_from_tasks() fallback that aggregates metrics
from task-level data:
- cpuTime = sum(cpus * realtime)
- cpuLoad = sum(pcpu/100 * realtime)
- cpuEfficiency = cpuLoad / cpuTime * 100
- memoryEfficiency = peakRss / memoryReq * 100
- readBytes/writeBytes = sum from tasks

Also refactored common accessors into _run_group(), _run_workflow(),
and _task_payload() helpers.

* fix: avoid falsy-zero bug in _compute_progress_from_tasks

The `or` operator treats 0 as falsy, so fields like peakRss=0
would incorrectly fall through to rss. Use explicit None checks
via a _val() helper instead.

* refactor: restore two-query pattern in query_run_costs

Revert from string-concatenation SQL to two explicit queries
(CUR vs no-CUR). Each query is now self-contained and readable
without mental interpolation of fragment variables.

* ci: ignore pre-existing nf-core lint failures in .nf-core.yml

Add ignores for files_unchanged mismatches (CONTRIBUTING.md,
PULL_REQUEST_TEMPLATE.md, linting_comment.yml, .prettierignore),
schema_params (multiqc_title, modules_testdata_base_path,
max_multiqc_email_size), and nextflow_config
(params.max_multiqc_email_size). These are all pre-existing
template drift, not introduced by this PR.

* ci: retrigger pull_request workflows

* 🐛 fix: handle edge cases in _compute_progress_from_tasks and clean up query_run_costs

- Use `run.get("tasks") or []` to handle explicit null tasks from JSON
- Use `is not None` check for peakRss to avoid treating 0 as missing
- Fix pcpu comment to correctly describe aggregate CPU% semantics
- Revert query_run_costs to two separate clean SQL queries matching file convention
- Fix docstring to reference Seqera Platform tw run dumps instead of Nextflow log tarballs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* 🔥 remove(ci): delete .nf-core.yml superseded by dev branch cleanup

nf-core linting workflows were dropped on dev (ca70336), making
this config file obsolete and causing a merge conflict in the PR.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: FloWuenne <flowuenne@gmail.com>

Author: 3 people

bin/benchmark_report.py

@@ -41,6 +41,73 @@ def _api_get(url: str, headers: dict[str, str], params: dict[str, str] | None =
 # ── JSON normalization (from clean_json.py) ─────────────────────────────────
 
 
+def _run_group(run: dict) -> str:
+    """Return the benchmark group for a run."""
+    return run["meta"]["group"]
+
+
+def _run_workflow(run: dict) -> dict:
+    """Return the workflow payload for a run."""
+    return run["workflow"]
+
+
+def _task_payload(task_raw: dict) -> dict:
+    """Unwrap nested task payloads returned by the API."""
+    if isinstance(task_raw, dict) and "task" in task_raw:
+        return task_raw["task"]
+    return task_raw
+
+
+def _compute_progress_from_tasks(run: dict) -> dict:
+    """Compute workflowProgress metrics from task-level data.
+
+    This is a fallback for runs where aggregate progress is missing
+    (e.g. imported from Seqera Platform tw run dumps).
+    """
+    tasks = [_task_payload(t) for t in run.get("tasks") or []]
+    completed = [t for t in tasks if t.get("status") == "COMPLETED"]
+    if not completed:
+        return {}
+
+    def _val(d: dict, key: str, default: int = 0) -> int | float:
+        """Return d[key] if not None, else default. Avoids falsy-zero bug with `or`."""
+        v = d.get(key)
+        return v if v is not None else default
+
+    cpu_time = sum(
+        _val(t, "cpus") * _val(t, "realtime") for t in completed
+    )
+    # cpuLoad = actual CPU usage: pcpu is aggregate CPU% across all cores
+    # (e.g. 400 for a 4-core task at full load), so pcpu/100 * realtime
+    # gives core-milliseconds used.
+    cpu_load = sum(
+        _val(t, "pcpu") / 100.0 * _val(t, "realtime")
+        for t in completed
+    )
+    mem_rss = sum(
+        _val(t, "peakRss") if t.get("peakRss") is not None else _val(t, "rss")
+        for t in completed
+    )
+    mem_req = sum(_val(t, "memory") for t in completed)
+    read_bytes = sum(_val(t, "readBytes") for t in completed)
+    write_bytes = sum(_val(t, "writeBytes") for t in completed)
+
+    return {
+        "cpuTime": int(cpu_time),
+        "cpuLoad": int(cpu_load),
+        "cpuEfficiency": (
+            round(cpu_load / cpu_time * 100, 2) if cpu_time else None
+        ),
+        "memoryRss": mem_rss,
+        "memoryReq": mem_req,
+        "memoryEfficiency": (
+            round(mem_rss / mem_req * 100, 2) if mem_req else None
+        ),
+        "readBytes": read_bytes,
+        "writeBytes": write_bytes,
+    }
+
+
 def _write_tmp_json(rows: list[dict], name: str) -> str:
     """Write rows to a temporary JSON file for DuckDB to read."""
     path = os.path.join(tempfile.gettempdir(), f"nfagg_{name}.json")
@@ -62,8 +129,10 @@ def extract_runs(runs: list[dict]) -> list[dict]:
     """Extract run-level metadata from raw API data."""
     run_rows = []
     for r in runs:
-        wf = r["workflow"]
+        wf = _run_workflow(r)
         prog = r.get("progress", {}).get("workflowProgress", {})
+        if not prog:
+            prog = _compute_progress_from_tasks(r)
         stats = wf.get("stats", {})
         launch = r.get("launch", {}) or {}
         ce = r.get("computeEnv", {}) or {}
@@ -74,7 +143,7 @@ def extract_runs(runs: list[dict]) -> list[dict]:
 
         run_rows.append({
             "run_id": wf["id"],
-            "group": r["meta"]["group"],
+            "group": _run_group(r),
             "pipeline": (
                 wf.get("projectName")
                 or wf.get("repository", "").split("/")[-1]
@@ -120,14 +189,10 @@ def extract_tasks(runs: list[dict]) -> list[dict]:
     """Extract task-level data from raw API data."""
     task_rows = []
     for r in runs:
-        run_id = r["workflow"]["id"]
-        group = r["meta"]["group"]
+        run_id = _run_workflow(r)["id"]
+        group = _run_group(r)
         for t_raw in r.get("tasks", []):
-            t = (
-                t_raw.get("task", t_raw)
-                if isinstance(t_raw, dict) and "task" in t_raw
-                else t_raw
-            )
+            t = _task_payload(t_raw)
             task_rows.append({
                 "run_id": run_id,
                 "group": group,
@@ -164,8 +229,8 @@ def extract_metrics(runs: list[dict]) -> list[dict]:
     """Extract per-process metrics from raw API data."""
     metrics_rows = []
     for r in runs:
-        run_id = r["workflow"]["id"]
-        group = r["meta"]["group"]
+        run_id = _run_workflow(r)["id"]
+        group = _run_group(r)
         for m in r.get("metrics", []):
             row = {
                 "run_id": run_id,
@@ -186,15 +251,23 @@ def extract_metrics(runs: list[dict]) -> list[dict]:
 # ── CUR cost processing (from clean_cur.py) ────────────────────────────────
 
 
-def detect_cur_format(db: duckdb.DuckDBPyConnection, cur_path: str) -> str:
-    """Detect CUR format: 'map' (CUR 2.0) or 'flat' (CUR 1.0) or 'unknown'."""
-    cur_cols = {
-        r[0]
-        for r in db.execute(
+def _parquet_columns(
+    db: duckdb.DuckDBPyConnection,
+    cur_path: str,
+) -> set[str]:
+    """Return the available columns in a parquet file."""
+    return {
+        row[0]
+        for row in db.execute(
             f"SELECT column_name FROM (DESCRIBE SELECT * FROM read_parquet('{cur_path}'))"
         ).fetchall()
     }
 
+
+def detect_cur_format(db: duckdb.DuckDBPyConnection, cur_path: str) -> str:
+    """Detect CUR format: 'map' (CUR 2.0) or 'flat' (CUR 1.0) or 'unknown'."""
+    cur_cols = _parquet_columns(db, cur_path)
+
     is_map = (
         "resource_tags" in cur_cols
         and "resource_tags_user_unique_run_id" not in cur_cols
@@ -242,12 +315,7 @@ def build_costs_flat_format(
     db: duckdb.DuckDBPyConnection, cur_path: str
 ) -> None:
     """Build costs table from CUR 1.0 flattened format."""
-    cur_cols = {
-        r[0]
-        for r in db.execute(
-            f"SELECT column_name FROM (DESCRIBE SELECT * FROM read_parquet('{cur_path}'))"
-        ).fetchall()
-    }
+    cur_cols = _parquet_columns(db, cur_path)
 
     has_nf_run_id = "resource_tags_user_nf_unique_run_id" in cur_cols
     has_run_id = "resource_tags_user_unique_run_id" in cur_cols
@@ -443,6 +511,7 @@ def query_run_metrics(db: duckdb.DuckDBPyConnection) -> list[dict]:
 def query_run_costs(db: duckdb.DuckDBPyConnection) -> list[dict]:
     """Per-run cost from task-level sums + optional CUR costs."""
     has_cur = table_exists(db, "costs")
+
     if has_cur:
         return fetch_dicts(db, """
             SELECT
@@ -458,6 +527,7 @@ def query_run_costs(db: duckdb.DuckDBPyConnection) -> list[dict]:
             GROUP BY r.run_id, r."group"
             ORDER BY r."group"
         """)
+
     return fetch_dicts(db, """
         SELECT
             r.run_id,

bin/benchmark_report_template.html

@@ -697,7 +697,6 @@ <h2 id="task-metrics"><svg class="h-icon sm"><use href="#ic-scatter"/></svg> Tas
 
   const costBoxEl = document.createElement('div');
   costBoxEl.className = 'chart';
-  costBoxEl.style.height = '500px';
   container.appendChild(costBoxEl);
 
   setTimeout(() => {
@@ -718,12 +717,13 @@ <h2 id="task-metrics"><svg class="h-icon sm"><use href="#ic-scatter"/></svg> Tas
       }
     });
 
+    costBoxEl.style.height = Math.max(500, processes.length * 25 + 60) + 'px';
     echarts.init(costBoxEl, 'seqera').setOption({
       title: { text: 'Task cost ($) per process' },
       tooltip: { trigger: 'item' },
       grid: { top: 40, bottom: 20, containLabel: true },
       yAxis: { type: 'category', data: processes,
-               axisLabel: { width: 220, overflow: 'truncate' },
+               axisLabel: { width: 220, overflow: 'truncate', interval: 0 },
                inverse: true },
       xAxis: { type: 'value', name: 'Cost ($)', nameLocation: 'center', nameGap: 25 },
       series: [{ type: 'boxplot', data: boxData }],