IC report double counts run cost: EC2 instance rows and ECS split rows are summed together

[stefanoboriero]

IC report double counts run cost: EC2 instance rows and ECS split rows are summed together

Summary

For Intelligent Compute runs on the ECS architecture, benchmark_report_normalize.py sums
both the EC2 instance usage rows and the ECS split cost allocation rows that describe
those same instances. AWS adds split line items as additional records rather than replacing
the parent instance charge, so counting both attributes the same compute twice.

Verified against a real AWS Data Exports (CUR 2.0) export covering billing period 2026-07:
every one of the 204 ECS-based IC runs in the export is affected, with a median
overstatement of 1.545× (worst 1.93×).

There is a design question attached to the fix — which figure should be reported, and for what
purpose. See Suggested direction; it is a proposal, not a
settled decision.

Impact

The inflation lands on ECS-based IC runs specifically, and it inflates them, so the report's
headline "X% lower cost with Seqera Intelligent Compute vs AWS Batch" figure
(benchmark_report_ic_template.html:956) understates real savings. The per-run Cost
column, the overview engine split, and the by-instance-type allocation are all affected, since
they all derive from run_summary[].cost.

Secondary damage: used_cost / unused_cost are also wrong (the parent's unblended cost is
added on top of the per-task split costs), so the used-vs-idle breakdown and the Split /
Blended basis chip are both misleading.

Evidence

Export: AWS Data Exports (CUR 2.0), billing period 2026-07, two shards, 1,789,898 rows,
378 distinct user_seqera_io_platform_workflow_id values.

Both row classes are present and carry the run-id tag

== tagged rows by product ==
  AmazonEC2            277876      <- IC tags the instances directly
  AmazonECS             49276      <- ECS split rows for those same instances
  AmazonVPC                 62
  AmazonCloudWatch          54

Only the ECS architecture is affected

Population	Cluster tag	Has split rows	Runs
ECS-based IC	clu-*:aws-ecs	yes	204
VM-based IC	clu-*:aws-vm	no	154
ECS clusters with no tagged instance cost	clu-*:aws-ecs	no	19
Other	—	no	1

VM-based IC runs have no split rows by architecture — they don't run on ECS, so AWS has
nothing to split. They are reported correctly today and must stay that way. All 204 runs on the
ECS architecture are affected; this is not a subset.

Magnitude, restricted to the 204 ECS-based IC runs

	Amount
Unblended total (actual bill)	$81.54
ECS split basis (split_cost + unused_cost)	$46.52
What the tool reports today (the two summed)	$128.06
Median overstatement per run	1.545× (worst 1.93×)

Example runs

workflow_id	sched_run_id	unblended rows / split rows	unblended	split basis	reported	overstated
92hf4CAcOpe51	run-0r0fkmes01j0a:aws-ecs	327 / 86	$2.9354	$2.0881	$5.0235	1.711×
4g91Lb1LKm0Ijq	run-0r0s7869g6hwc:aws-ecs	786 / 428	$1.2006	$0.8602	$2.0608	1.716×
4vpHpjQFPYkwdN	run-0qyw4c8j46q9v:aws-ecs	843 / 428	$1.2454	$0.7242	$1.9696	1.581×
5oEFZnGrCiuev5	run-0r0spq81c718t:aws-ecs	1018 / 428	$1.5429	$0.8828	$2.4257	1.572×
4JngcMANSkwUxp	run-0r0gexqc06jr2:aws-ecs	839 / 430	$1.1523	$0.5634	$1.7157	1.489×

Root cause

modules/local/normalize_benchmark_jsonl/bin/benchmark_report_normalize.py
(and the duplicated copy at bin/benchmark_report_normalize.py):

1. The only row filter is "does this row carry a run-id tag" (:407):

   WHERE run_id_expr IS NOT NULL AND run_id_expr <> ''

   No filter on line_item_product_code, line_item_line_item_type, or
   split_line_item_parent_resource_id, so a tagged EC2 usage row and the ECS split rows
   describing that same instance both pass.

2. used_cost collapses the two bases into one number per row (:394):

   CASE WHEN split_line_item_split_cost <> 0
        THEN split_line_item_split_cost
        ELSE line_item_unblended_cost END

   Being a per-row choice, it never notices that two different rows describe the same compute.
   The GROUP BY run_id, process, hash does separate them — instance rows carry no
   pipeline_process / task_hash tag, so they land in a process='', hash='' group — but
   _run_cost_details (benchmark_report_ic_aggregate.py:206) then sums every group for a
   run_id, recombining them.

3. split_cost_present is set by any row carrying split cost (:405), so affected runs are
   labelled cost_basis: "split" (benchmark_report_ic_aggregate.py:309) even though their
   total includes a full unblended parent charge.

Why AWS behaves this way

Per Understanding split cost allocation data:

"Split cost allocation data introduces new usage records and new cost metric columns for each
containerized resource ID … two new usage records are added for each ECS task and
Kubernetes pod per hour."

The parent instance charge is not removed or zeroed — it is the quantity being split. The
worked example
shows a $1/hr instance producing task rows whose split_cost + unused_cost sum to exactly $1.

Why AWS Batch is unaffected

Batch tags the ECS task, not the instance, so EC2 instance rows carry no run-id tag and are
dropped by the WHERE. Confirmed on the checked-in fixture
workflows/nf_aggregate/assets/test_benchmark_realworld_costs.parquet: 2,748 rows, all
AmazonECS, all with line_item_unblended_cost = 0, and zero AmazonEC2 rows. IC tagging
instances directly is what breaks that assumption.

Suggested direction (for discussion)

Whatever we do, the two figures must stop being added together — that summation is the bug
itself, and it is not in question.

What to report instead is worth deciding as a team. One option, put forward as a starting point
rather than a conclusion: carry both figures separately and let each serve its own purpose.

• Unblended total — a candidate cost of record. The amount actually billed for the instances
  the scheduler provisioned for the run, including boot, idle and drain time.
• ECS split basis (split_cost + unused_cost) — a candidate comparison figure. It is the
  only basis AWS Batch runs have, so it is the one that supports a like-for-like IC vs Batch
  comparison.

Two properties that bear on the choice, measured on the 204 ECS-based IC runs:

• The split basis is not a complete measure of IC spend. It accounts for 57.1% of the
  actual bill in aggregate (per-run median 54.5%, range 20.8%–93.0%), because only
  5,854 of 16,934 tagged instances (34.6%) carry split rows at all — hours in which an
  instance ran no ECS tasks produce no split line items.
• The two bases are not directly comparable in absolute terms. Split costs are amortized
  (SavingsPlanEffectiveCost where a Savings Plan applies); line_item_unblended_cost is not.
  The gap between them should not be read as pure idle time.

Supporting detail for attributing whole instances to a run: within this population instances are
never shared across workflows (16,934 tagged instances, 0 shared, max 1 workflow per instance),
so an unblended per-run total needs no proportional allocation.

Open questions

• Which figure should the per-run Cost column and the overview totals show?
• Should the IC vs Batch comparison chart use the split basis exclusively? If so, how should
  runs with no split basis (VM architecture) be presented — excluded, or shown on the
  unblended basis with a caveat?
• How should the amortized-vs-unblended distinction be surfaced in the UI, if at all?
• Does the per-task cost granularity from the split rows need to survive, or is a per-run
  figure sufficient for both reports?

Possible implementation

Sketch only, contingent on the direction above:

1. normalize — stop collapsing the bases. Emit unblended_cost, split_cost and
   unused_cost as separate fields per (run_id, process, hash) group instead of one
   used_cost chosen by a per-row CASE.
2. aggregate — expose the two per-run figures rather than one cost, and replace the
   cost_basis "split vs blended" choice with a description of which bases are available
   for that run.
3. report — decide per the open questions above which figure drives which view.
4. Apply to both copies of the script (bin/ and modules/local/normalize_benchmark_jsonl/bin/).

This also resolves the open question recorded in
modules/local/aggregate_ic_report_data/AGENTS.md:5 ("IC runs sometimes don't carry split cost
— under investigation"). ECS-based IC runs do carry split rows; VM-based IC runs cannot, by
architecture. The blended label was a symptom of which rows matched, not of what AWS exported.

Test gap

No existing test covers a run with both row classes:

• The realworld fixture has no AmazonEC2 rows at all.
• Every cost test in modules/local/normalize_benchmark_jsonl/tests/test_normalize.py supplies
  either split columns or unblended, never both under one run id.
  test_normalize_cost_rows_reads_directory_of_parquets comes closest — it sums $1.00 split +
  $2.00 unblended to $3.00 and asserts that as correct, which is exactly the double-count
  behaviour, just with no parent/child relationship between the rows.

Needs a fixture with one tagged AmazonEC2 usage row plus split children referencing it via
split_line_item_parent_resource_id, same run id, asserting the two bases are never summed. A
VM-architecture case (instance rows only) belongs alongside it as a regression guard.

Reproduction

A DuckDB script that reports per-run cost by row class and flags any run carrying both, with the
inflation multiple. Requires an export that has not been pre-filtered to ECS rows, with split
cost allocation enabled and the Seqera tags activated as cost allocation tags.

Also of note while investigating: a raw legacy CUR 1.0 delivery (camelCase lineItem/... column
names) silently yields zero cost rows rather than erroring, which is indistinguishable from
"no CUR supplied" in the report. CUR 2.0 / Data Exports and CUR 1.0 with the Athena integration
both work. The README's "currently only supporting CUR 1.0" is misleading on both counts.

[pditommaso]

WOW