🐛 fix(ic): stop summing EC2 instance and ECS split cost bases

AWS emits ECS split cost allocation rows *in addition to* the parent EC2
instance rows they were derived from, so the two describe the same compute.
Intelligent Compute tags the instances themselves, so an IC run on the ECS
architecture matched both row classes and their costs were added together.

Measured on a real July 2026 CUR export: all 204 ECS-based IC runs were
affected, median overstatement 1.545x (worst 1.93x). AWS Batch was never
affected because it tags only the ECS task, so its instance rows carry no
run tag.

normalize now emits unblended_cost / split_cost / unused_cost as separate
fields, with single-basis cost / used_cost conveniences that prefer the
billed instance charge. This matters because IC split rows carry no
pipeline_process or task_hash labels (0 of 49,106 in the real export), so
they collapse into the same (run_id, '', '') group as the instance rows —
the bases are separable only as columns, never by row class after grouping.

The IC report now carries both figures per run: `cost` (cost of record,
with cost_basis instance|split) and `comparable_cost` (the ECS split basis,
null when absent). Cross-engine comparison views use comparable_cost and
omit runs lacking it, since split cost is amortized and unblended is not;
the instance-type breakdown keeps using the billed cost.

Verified end to end against the real export: instance basis $2,193.68 and
comparable basis $46.58 reproduce independent DuckDB measurements exactly,
where the old code reported $2,240.27.

Tests cover the shapes that had none: an IC run with both bases in one
group, an IC run with task-labelled split rows, a VM-architecture run with
the instance basis only, and a Batch run with the split basis only.

Author: stefanoboriero

AGENTS.md

@@ -90,7 +90,7 @@ uv run --with typer --with pyyaml \
 - JSONL is the primary handoff (`jsonl_bundle/`) for Fusion-friendly streaming
 - `report_data.json` is the explicit boundary between aggregation and rendering
 - When a CUR file is supplied but a run has no matching cost rows, aggregation sets `run_costs[].cost_status` to `propagating` (run finished < 24h ago, per `_COST_PROPAGATION_WINDOW_HOURS`, so CUR data likely hasn't landed) or `not_found` (older run, genuinely absent). The report renders `pending` / `no data` instead of a bare dash. Runs with matched cost are `available`; no CUR at all leaves `cost_status` null.
-- **Intelligent Compute report cost basis.** The IC report (`benchmark_report_ic_aggregate.py`) wants ECS split cost allocation (used vs idle capacity) for *every* run — both Intelligent Compute and Batch. In practice Batch runs reliably carry split cost, but IC runs sometimes don't yet (we're still investigating why). When a run has genuine split cost (`split_cost_present`), `run_summary[].cost_basis` is `split` with `used_cost`/`unused_cost` populated; when it's absent we fall back to the unblended line-item cost — in practice this fallback only hits Intelligent Compute runs — so `cost_basis` is `blended` and used/unused are null. `cost` is always the billed total. It mirrors the benchmark report's `cost_status` (`available`/`propagating`/`not_found`/null). `normalize` emits `split_cost_present` (shared with the benchmark path). The report shows per-run cost-basis badges plus a Cost-section coverage note (how many runs had costs, and whether cost was split vs blended).
+- **Two CUR cost bases, never summed.** AWS emits ECS split cost allocation rows *in addition to* the parent EC2 instance rows they were derived from, so the two describe the same compute. Intelligent Compute tags the instances themselves, so an IC run on the ECS architecture matches both classes — summing them overstated affected runs by ~1.5x (median, measured on a real export). AWS Batch tags only the task, so its instance rows carry no run tag and it was never affected. `normalize` therefore emits `unblended_cost`, `split_cost` and `unused_cost` as separate fields (plus single-basis `cost`/`used_cost` conveniences that prefer the billed instance charge). Note IC split rows carry no `pipeline_process`/`task_hash` labels, so they collapse into the same `(run_id, '', '')` group as the instance rows — the bases are separable only as columns, never by row class after grouping. The IC report exposes `cost` (cost of record, with `cost_basis` = `instance`|`split`) alongside `comparable_cost` (the split basis, null when absent); cross-engine comparison views use `comparable_cost` and omit runs lacking it, since split cost is amortized and unblended is not. `cost_status` (`available`/`propagating`/`not_found`/null) mirrors the benchmark report.
 - `commit.gpgsign` must be true (SSH signing via 1Password)
 - RTK `buildOutputFiltering` / `testOutputAggregation` can swallow nf-test output — disable to debug
 - **Nextflow `include` statements in `main.nf` must be single-line.** `adamrtalbot/detect-nf-test-changes@v0.0.3` (used by CI) parses include lines and crashes on multi-line blocks. Write `include { A ; B ; C } from '...'` not multi-line blocks.

README.md

@@ -109,6 +109,22 @@ an AWS CUR export when `--benchmark_aws_cur_report` is supplied (same parameter
 benchmark report — see [Incorporate AWS split cost allocation data](#incorporate-aws-split-cost-allocation-data)); the
 cost column is otherwise blank. The Seqera cost estimate is never used.
 
+Two cost figures are reported per run, and they are deliberately never added together:
+
+- **Cost** — what AWS actually billed. For Intelligent Compute that is the EC2 charge for the
+  machines the scheduler used, including start-up and idle time.
+- **Comparable (split)** — the ECS split cost allocation figure, i.e. the instance cost divided
+  across the tasks that ran on it. AWS Batch only ever reports this basis, so it is what the
+  engine-to-engine comparison views use. It is blank for runs with no ECS tasks to split
+  (Intelligent Compute on the VM architecture), and those runs are omitted from the comparison
+  charts rather than compared on a different basis.
+
+AWS emits split cost allocation rows _in addition to_ the EC2 instance rows they were derived
+from, so the two describe the same compute — keeping them apart is what stops a run's cost being
+counted twice. Note that split costs are amortized (Savings Plan effective cost where one
+applies) while the billed instance charge is not, so the two are not directly comparable in
+absolute terms.
+
 By default, failed workflows are dropped from the IC report entirely and their CUR costs
 are not attributed. Pass `--include_failed_runs` to include failed runs (and their costs).
 Cancelled/aborted runs are always excluded.

bin/benchmark_report_ic_aggregate.py

@@ -188,28 +188,34 @@ def _run_cost_details(jsonl_dir: Path) -> dict[str, dict[str, Any]]:
     ``costs.jsonl`` is only written when a CUR parquet is supplied to the normalize
     step, so an absent file leaves run costs unset — we never fall back to Seqera's
     unreliable cost estimate. CUR rows are task-grained (run_id, process, task_hash);
-    they are summed here into one entry per run carrying:
-      - ``cost``               blended total (used + unused) — the amount actually billed
-      - ``used_cost``          cost of compute that was consumed (ECS split cost allocation)
-      - ``unused_cost``        cost of provisioned-but-idle capacity
+    they are summed here into one entry per run carrying the two cost bases SEPARATELY:
+      - ``unblended_cost``     instance-basis charge — what AWS actually billed for the
+                               instances, including boot/idle/drain time
+      - ``split_cost``         consumed capacity, from ECS split cost allocation
+      - ``unused_cost``        provisioned-but-idle capacity, from ECS split cost allocation
       - ``split_cost_present`` whether any row carried genuine split cost allocation
 
-    We want split cost allocation for every run (Intelligent Compute and Batch). In
-    practice Batch runs reliably carry it; IC runs sometimes don't yet — we're still
-    investigating why. A run with no split cost keeps ``split_cost_present`` False, and
-    its ``used_cost`` falls back to the unblended line-item cost (see ``normalize``); in
-    practice that fallback only affects IC runs.
+    The two bases must never be added: for an Intelligent Compute run on the ECS
+    architecture, the scheduler tags the EC2 instances, so both the instance rows and the
+    ECS split rows describing tasks on those instances carry the run-id tag — and the split
+    rows are a re-expression of the very cost the instance rows already state. Summing them
+    overstated affected runs by ~1.5x. See ``_normalize_cost_rows``.
     """
     details: dict[str, dict[str, Any]] = defaultdict(
-        lambda: {"cost": 0.0, "used_cost": 0.0, "unused_cost": 0.0, "split_cost_present": False}
+        lambda: {
+            "unblended_cost": 0.0,
+            "split_cost": 0.0,
+            "unused_cost": 0.0,
+            "split_cost_present": False,
+        }
     )
     for row in _iter_jsonl(jsonl_dir / "costs.jsonl"):
         run_id = str(row.get("run_id") or "")
         if not run_id:
             continue
         entry = details[run_id]
-        entry["cost"] += float(row.get("cost") or 0.0)
-        entry["used_cost"] += float(row.get("used_cost") or 0.0)
+        entry["unblended_cost"] += float(row.get("unblended_cost") or 0.0)
+        entry["split_cost"] += float(row.get("split_cost") or 0.0)
         entry["unused_cost"] += float(row.get("unused_cost") or 0.0)
         if row.get("split_cost_present"):
             entry["split_cost_present"] = True
@@ -293,23 +299,37 @@ def build_ic_report_data(
         req_mem_gib = round(res.get("mem_req", 0.0), 1)
         eff_mem_gib = round(res.get("mem_used", 0.0), 1)
 
-        # Cost, preferring ECS split cost allocation (used vs idle) when the CUR export
-        # carries it for this run, else the unblended total. We want split cost for every
-        # run (Intelligent Compute and Batch); in practice Batch reliably has it while IC
-        # sometimes doesn't yet (under investigation), so the blended fallback here in
-        # practice only affects IC runs. When a CUR file was supplied but no row matched
-        # this run, cost_status explains why (propagating vs not_found); with no CUR at all
-        # the whole cost story is off and every field stays None.
+        # Two cost figures, never summed (see _run_cost_details):
+        #   cost            the cost of record — the instance charge actually billed when we
+        #                   have it, else the split basis. Drives the Cost column and totals.
+        #   comparable_cost the ECS split basis (consumed + idle), or None when the run has no
+        #                   split rows. This is the only basis AWS Batch runs have, so it is
+        #                   what an IC-vs-Batch comparison must be built on.
+        # An IC run on the ECS architecture has both; one on the VM architecture has only the
+        # instance basis (AWS has no ECS tasks to split), and a Batch run only the split basis.
+        # When a CUR file was supplied but no row matched this run, cost_status explains why
+        # (propagating vs not_found); with no CUR at all every field stays None.
         detail = cost_details.get(run_id)
         if detail is not None:
             split_present = bool(detail["split_cost_present"])
-            cost = round(detail["cost"], 4)
-            used_cost = round(detail["used_cost"], 4) if split_present else None
+            instance_cost = round(detail["unblended_cost"], 4)
+            comparable_cost = (
+                round(detail["split_cost"] + detail["unused_cost"], 4) if split_present else None
+            )
+            used_cost = round(detail["split_cost"], 4) if split_present else None
             unused_cost = round(detail["unused_cost"], 4) if split_present else None
-            cost_basis = "split" if split_present else "blended"
+            # Prefer the real billed charge; fall back to the split basis only when there are
+            # no instance rows for this run at all (a Batch run, or IC instances left untagged).
+            if instance_cost:
+                cost, cost_basis = instance_cost, "instance"
+            elif split_present:
+                cost, cost_basis = comparable_cost, "split"
+            else:
+                cost, cost_basis = instance_cost, "instance"
             cost_status = "available"
         else:
             cost = used_cost = unused_cost = None
+            instance_cost = comparable_cost = None
             cost_basis = None
             cost_status = None if not cur_supplied else _classify_missing_cost(
                 run.get("complete") or started_at, now
@@ -344,10 +364,17 @@ def build_ic_report_data(
             # Real cost from the AWS CUR export; None when no CUR row matched this run
             # (renders as an em-dash). The Seqera estimate above is never used here.
             "cost": cost,
+            # Instance-basis charge on its own (None when no cost row matched). Same value as
+            # ``cost`` whenever cost_basis is "instance"; kept explicit so the report can show
+            # the billed amount alongside the comparable figure without re-deriving it.
+            "instance_cost": instance_cost,
+            # ECS split basis (consumed + idle) for like-for-like IC vs Batch comparison.
+            # None when the run has no split rows -> excluded from comparison charts.
+            "comparable_cost": comparable_cost,
             # Split cost allocation (used vs idle) when present, else None -> em-dash.
             "used_cost": used_cost,
             "unused_cost": unused_cost,
-            # "split" | "blended" | None (no cost row): how this run's cost was derived.
+            # "instance" | "split" | None (no cost row): which basis ``cost`` came from.
             "cost_basis": cost_basis,
             # "available" | "propagating" | "not_found" | None (no CUR at all).
             "cost_status": cost_status,
@@ -390,8 +417,14 @@ def build_ic_report_data(
             "cur_supplied": cur_supplied,
             # Cost-coverage tallies driving the report's cost-availability note.
             "n_runs_with_cost": len(runs_with_cost),
-            "n_runs_split_cost": sum(1 for r in runs_with_cost if r["cost_basis"] == "split"),
-            "n_runs_blended_cost": sum(1 for r in runs_with_cost if r["cost_basis"] == "blended"),
+            # How many runs carry a comparable (ECS split) figure, and how many are
+            # instance-basis only — the latter cannot appear in IC-vs-Batch comparisons.
+            "n_runs_comparable_cost": sum(
+                1 for r in runs_with_cost if r["comparable_cost"] is not None
+            ),
+            "n_runs_instance_only": sum(
+                1 for r in runs_with_cost if r["comparable_cost"] is None
+            ),
             "n_runs_missing_cost": sum(
                 1 for r in run_summary if r["cost_status"] in ("propagating", "not_found")
             ),