Skip to content

45-multi-build-analytics.md

Latest commit

 

History

History
595 lines (410 loc) · 16 KB

45-multi-build-analytics.md

File metadata and controls

595 lines (410 loc) · 16 KB
Raw

Multi-Build Analytics And Comparison v0 Proposal

1. Status

This document is a v0 proposal for multi-build analytical composition.

It is downstream of the single-build analytical package described in 43-build-analytics.md.

2. Purpose

Single-build analytics explains one build as a knowledge artifact.

Multi-build analytics should explain change across builds.

Its purpose is to support:

  • degradation detection
  • metric drift analysis
  • failure-shape comparison
  • stability analysis by test identity
  • lineage-aware exploration across parent, child, shard, and subbuild relationships

3. Hard Boundary

Multi-build composition is also purely derived.

It MUST NOT alter:

  • the canonical report
  • the single-build cube contract
  • case identity
  • action identity
  • pass or fail semantics

The single-build cube remains the source analytical artifact.

4. Stage-2 Input Model

The first multi-build stage SHOULD consume only previously derived cube artifacts plus optional explicit relationship metadata.

Representative inputs:

  • build-1234.cube.json
  • build-1235.cube.json
  • build-compare.meta.json

The Stage-2 design SHOULD NOT require re-reading raw report JSONL when cube artifacts are already available.

5. Representative Command Shape

Representative command shape:

observer compare --cube build-1234.cube.json --cube build-1235.cube.json --out compare-1234-1235.json

Representative directory shape:

observer compare --cube-dir cubes/ --out compare-index.json

These command forms are future-facing Stage-2 sketches, not current implementation requirements.

An exact non-normative sample comparison artifact is provided in 92-build-compare-sample.json.

Exact non-normative sample source cubes for that pairwise artifact are provided in 102-build-compare-baseline-cube-sample.json and 103-build-compare-candidate-cube-sample.json.

An exact non-normative comparison-index sample artifact is provided in 99-build-compare-index-sample.json.

Exact non-normative sample source cubes for that comparison-index artifact are provided in 104-build-compare-index-build-1234-cube-sample.json, 105-build-compare-index-build-1235-cube-sample.json, and 106-build-compare-index-build-1236-cube-sample.json.

An invalid cube sample suitable for negative-path comparison testing is provided in 94-invalid-cube-sample.json.

Invalid comparison-metadata samples suitable for negative-path testing are provided in 96-invalid-compare-meta.json and 97-invalid-compare-relation-kind.json.

A valid comparison-metadata sample is provided in 98-build-compare-meta-sample.json.

5.1 Comparison Metadata Contract

Stage-2 comparison metadata is optional and purely relational.

It exists to supply explicit edges between builds when those edges are not already present in the comparison invocation context.

Representative metadata file shape:

{
	"relations": [
		{
			"from_build_id": "build-1234",
			"relation_kind": "previous",
			"to_build_id": "build-1235"
		}
	]
}

Required top-level field:

  • relations: array of relation rows

Required relation-row fields:

  • from_build_id: string
  • relation_kind: previous, parent, child, shard, or subbuild
  • to_build_id: string

Unknown top-level fields SHOULD cause Stage-2 derivation failure so the metadata contract remains explicit.

5.1.1 Directory Comparison Mode

The --cube-dir form is a distinct Stage-2 mode.

It SHOULD scan a directory of cube artifacts, derive a deterministic build set, apply explicit relation metadata when provided, and emit one comparison-index artifact.

Representative comparison-index artifact naming:

  • compare-index.json

The index artifact is not a pairwise delta artifact. It is an index over a build set plus any explicit pair links and relation edges.

The comparison-index explorer UX is described separately in 47-build-compare-index-explorer-ui.md.

5.2 Recommended Invalid Fixture Cases For observer compare

The comparison stage should also carry explicit invalid fixtures.

Recommended invalid cases include:

  • missing build.build_id in one or more source cubes
  • duplicate build_id values across input cubes when a comparison requires distinct builds
  • source cubes with unsupported k or v values
  • source cubes whose comparison anchors disagree structurally, such as missing item_id or test_name
  • metric comparisons attempted across differing telemetry units
  • relation metadata that references unknown builds
  • relation metadata that uses unsupported relation_kind
  • pairwise comparison invoked with fewer than two cubes
  • pairwise comparison invoked with more than two cubes unless an explicit multi-build mode is selected
  • directory mode invoked on an empty cube directory
  • directory mode invoked on a directory containing duplicate build_id values

5.3 Command Result And Error Payloads

The comparison command SHOULD emit one bounded machine-readable result payload on stdout.

Representative success payload:

{
	"k": "observer_compare_result",
	"v": "0",
	"out": "compare-1234-1235.json",
	"status": "ok",
	"build_count": 2
}

Required success fields:

  • k: fixed string observer_compare_result
  • v: fixed string 0
  • out: output comparison artifact path as supplied by the user
  • status: fixed string ok
  • build_count: non-negative integer

On failure, the command SHOULD emit one bounded machine-readable error payload on stderr.

Representative error payload:

{
	"k": "observer_compare_error",
	"v": "0",
	"code": "unknown_relation_build",
	"msg": "relation metadata references build id not present in input cubes",
	"status": "error"
}

Required error fields:

  • k: fixed string observer_compare_error
  • v: fixed string 0
  • code: stable error code string
  • msg: human-readable error message
  • status: fixed string error

Recommended Stage-2 error codes include:

  • invalid_cube_shape
  • duplicate_build_id
  • missing_build_id
  • invalid_compare_meta
  • unknown_relation_build
  • unsupported_relation_kind
  • invalid_compare_cardinality
  • empty_cube_dir

5.4 Directory-Mode Result Payload

Directory mode SHOULD emit one bounded machine-readable result payload on stdout.

Representative success payload:

{
	"k": "observer_compare_index_result",
	"v": "0",
	"out": "compare-index.json",
	"status": "ok",
	"build_count": 3,
	"pair_count": 2
}

Required fields:

  • k: fixed string observer_compare_index_result
  • v: fixed string 0
  • out: output comparison-index path as supplied by the user
  • status: fixed string ok
  • build_count: non-negative integer
  • pair_count: non-negative integer

5.5 Static Viewer Command Shapes

Pairwise and directory-mode outputs SHOULD also support static viewer emission.

Representative pairwise viewer command:

observer view --compare compare-1234-1235.json --out compare-1234-1235.view.html

Representative comparison-index viewer command:

observer view --compare-index compare-index.json --out compare-index.view.html

These viewer commands MUST read only the already-derived comparison artifact they are given.

They MUST NOT re-read raw reports or infer missing pair links.

Representative success payload:

{
	"k": "observer_view_result",
	"v": "0",
	"out": "compare-1234-1235.view.html",
	"view_kind": "compare",
	"status": "ok"
}

Representative comparison-index success payload:

{
	"k": "observer_view_result",
	"v": "0",
	"out": "compare-index.view.html",
	"view_kind": "compare_index",
	"status": "ok"
}

Exact non-normative HTML samples for the published Stage-2 artifacts are provided in 110-build-compare-view-sample.html and 111-build-compare-index-view-sample.html.

6. Output Model

The Stage-2 output SHOULD be a structured comparison artifact rather than an overloaded single-build cube.

Representative top-level shape:

{
	"k": "observer_build_compare",
	"v": "0",
	"builds": [ ... ],
	"relations": [ ... ],
	"comparisons": { ... },
	"measures": { ... }
}

Directory mode SHOULD emit a separate comparison-index artifact rather than forcing all builds into one pairwise delta shape.

Representative top-level index shape:

{
	"k": "observer_build_compare_index",
	"v": "0",
	"builds": [ ... ],
	"relations": [ ... ],
	"pairs": [ ... ],
	"measures": { ... }
}

6.1 Exact Top-Level Schema

The v0 comparison artifact SHOULD use this top-level schema:

{
	"k": "observer_build_compare",
	"v": "0",
	"builds": [ ... ],
	"relations": [ ... ],
	"comparisons": { ... },
	"measures": { ... }
}

Required top-level fields:

  • k: fixed string observer_build_compare
  • v: fixed string 0 for v0
  • builds: participating build identities
  • relations: explicit build-edge rows, possibly empty
  • comparisons: comparison tables
  • measures: aggregate comparison summaries

6.2 Build Entry Schema

Each builds row SHOULD contain:

  • build_id: string
  • inventory_sha256: string
  • suite_sha256: string
  • report_mode: default or golden
  • meta: object carrying explicit source-cube metadata fields

6.3 Relation Row Schema

Each relations row SHOULD contain:

  • from_build_id: string
  • relation_kind: string
  • to_build_id: string

The row MUST reference build ids present in builds.

6.4 Comparisons Object Schema

The comparisons object SHOULD contain these arrays:

  • case_status_change
  • action_fail_change
  • metric_delta
  • missing_or_added_tests

The arrays MAY be empty, but they SHOULD always be present in Stage-2 pairwise artifacts.

6.5 Case Status Change Row

Each case_status_change row SHOULD contain:

  • item_id: string
  • test_name: string
  • baseline_status: pass or fail or missing
  • candidate_status: pass or fail or missing
  • category: pass_to_pass, pass_to_fail, fail_to_pass, fail_to_fail, added, or removed

6.6 Action Fail Change Row

Each action_fail_change row SHOULD contain:

  • item_id: string
  • test_name: string
  • action: string
  • baseline_fail: non-negative integer
  • candidate_fail: non-negative integer
  • delta: signed integer

6.7 Metric Delta Row

Each metric_delta row SHOULD contain:

  • item_id: string
  • test_name: string
  • name: telemetry metric name
  • unit: telemetry metric unit
  • baseline_value: numeric
  • candidate_value: numeric
  • delta: numeric

Optional field:

  • delta_ratio: numeric when baseline is non-zero

6.8 Missing Or Added Test Row

Each missing_or_added_tests row SHOULD contain:

  • item_id: string
  • test_name: string
  • category: added or removed

6.9 Measures Schema

The measures object SHOULD contain:

  • build_count: non-negative integer
  • pair_count: non-negative integer
  • pass_to_fail: non-negative integer
  • fail_to_pass: non-negative integer
  • added_tests: non-negative integer
  • removed_tests: non-negative integer

6.10 Comparison-Index Schema

The comparison-index artifact SHOULD contain:

  • k: fixed string observer_build_compare_index
  • v: fixed string 0
  • builds: build entry rows
  • relations: explicit relation rows
  • pairs: explicit pair rows
  • measures: aggregate index summaries

Each pairs row SHOULD contain:

  • baseline_build_id: string
  • candidate_build_id: string
  • compare_artifact: string path or artifact identifier

Recommended measures fields for the index artifact:

  • build_count
  • pair_count
  • relation_count

7. Build Set

The builds array SHOULD list the participating build identities in deterministic order.

Recommended ordering:

  • by explicit input order for direct pair comparison
  • by build_id lexicographic order for directory or index derivation when no stronger order is supplied

Each build entry SHOULD include:

  • build_id
  • inventory_sha256
  • suite_sha256
  • report_mode
  • explicit external metadata fields already present in the source cube

8. Relationship Model

Multi-build analytics should support explicit edges between builds.

Representative relation row fields:

  • from_build_id
  • to_build_id
  • relation_kind

Permitted Stage-2 relation kinds may include:

  • previous
  • parent
  • child
  • shard
  • subbuild

These relations MUST be explicit. They MUST NOT be inferred heuristically from names.

9. Comparison Anchors

All comparison logic should anchor on stable identities already preserved in the single-build cube:

  • build_id
  • item_id
  • test_name
  • case_id only within one build
  • telemetry name
  • telemetry unit

For cross-build comparison, item_id plus test_name is the stable analytical key. case_id remains build-local.

10. Core Comparison Views

Stage-2 SHOULD support these comparisons:

  • case status change by item_id and test_name
  • assertion-failure change counts
  • action failure-kind shifts
  • telemetry metric deltas for scalar metrics with matching units
  • telemetry distribution drift for vectors

11. Delta Categories

Stage-2 SHOULD classify case movement explicitly.

Representative categories:

  • pass_to_pass
  • pass_to_fail
  • fail_to_pass
  • fail_to_fail
  • added
  • removed

These categories are mechanical derivations, not judgments.

12. Metric Delta Rules

Metric comparison MUST be explicit about units and comparability.

Rules:

  • metrics compare only when names match exactly
  • units compare only when units match exactly
  • missing metric values remain missing; they MUST NOT be coerced to zero
  • vector telemetry should not be flattened into scalar deltas in the normative artifact without an explicit summary rule

Representative scalar delta fields:

  • baseline_value
  • candidate_value
  • delta
  • delta_ratio when baseline is non-zero

13. Output Sections

The comparisons object SHOULD contain:

  • case_status_change
  • action_fail_change
  • metric_delta
  • missing_or_added_tests

The measures object SHOULD contain:

  • build_count
  • pair_count
  • pass_to_fail
  • fail_to_pass
  • added_tests
  • removed_tests

14. Determinism Constraints

Multi-build derivation remains subject to the determinism gate.

That means:

  • input cube order handling must be explicit
  • relation edges must be explicit
  • comparison keys must be explicit and stable
  • output arrays must use deterministic ordering
  • no inferred baseline or candidate choice is allowed without explicit rule

15. Recommended Ordering

Recommended array ordering:

  • builds: as defined in section 7
  • relations: by from_build_id, then relation_kind, then to_build_id
  • comparison rows: by item_id, then test_name, then comparison-specific secondary key

16. Explorer Follow-On

The static explorer in 44-build-explorer-ui.md may grow a Stage-2 comparison mode later.

That mode should add:

  • baseline and candidate build selectors
  • delta-focused scatter and heat-map views
  • lineage navigation across explicit relation edges

But the Stage-1 single-build explorer should remain separate and simpler.

The current reference workflow now includes explicit HTML emission for:

17. Stage-2 Recommendation

The first multi-build analytical stage should compare a small explicit set of cube artifacts and emit one deterministic comparison artifact.

It should prioritize:

  • exact identity alignment
  • explicit relation edges
  • deterministic delta rules
  • drill-through to the underlying single-build facts