Skip to content

Track expected pipelines (pytorch/jax enable-flags) for finalize rollup #57

Description

@HereThereBeDragons

Sibling to #56. Same root cause, one level up: #56 tracks expected archs
within a pipeline; this tracks the expected set of pipelines.

Right now rebuild_summary decides pipeline status from which leaves have
reported. pytorch and jax are fire-and-forget workflow-dispatch triggers,
gated on inputs.build_pytorch / inputs.build_jax. As such the orchestrator
will not wait for the results of those pipelines. It can finalize
(completed_at) while a pytorch/jax leaf has never reported.

That makes a no-leaf pytorch/jax pipeline at finalize ambiguous between two
states the detail tree can't distinguish:

  • disabled (build_pytorch: false) → should render skipped
  • dispatched but not reported yet → should stay in_progress (correctly
    holds the release)

Today both render in_progress, so a finalized release with pytorch/jax
disabled shows those children in_progress forever while overall_status is
success.

Side note: native packages and rocm build do not have this problem as
they do not have an if-guard and are always triggered via workflow call by
the orchestrator, so the orchestrator waits for their results.

The signal exists

Either via the job list ("conclusion": "skipped") of the platform
orchestrator (multi_arch_release_linux.yml) or via the workflow inputs of the
same workflow.

overall_status must consider this

Today completed_at is stamped when the top-level orchestrator run
finishes - but that run does not wait on the dispatched pytorch/jax runs, so
completed_at does not mean those pipelines are done. Any consumer (and the
rollup itself) treating orchestrator completed_at as "release done" will
finalize prematurely for dispatched pipelines.

Once we track the expected pipeline set, this resolves naturally:

  • An enabled pytorch/jax that hasn't reported stays in_progress, feeds the
    platform worst-of, and holds overall_status at in_progress until it
    reports a terminal leaf - regardless of the orchestrator's completed_at.
  • A disabled pytorch/jax renders skipped and does not hold the release.

So overall_status should only roll up to a finalized state (success /
failure / cancelled) once every expected pipeline has reached a terminal
state — not merely when the top-level orchestrator's completed_at is set. The
completed_at cap in rebuild_summary (the branch that currently drops the
unstarted-pipeline injection once finalized) needs to key off the expected set
instead of orchestrator completion alone.

Note

In practice the enabled-pipeline case is usually masked: the dispatched
pytorch/jax run reports a started leaf (holding overall_status at
in_progress) before the top-level orchestrator finalizes, because other
in-flight workflow_call work keeps the orchestrator alive long enough. But
that ordering is a timing coincidence, not a needs: guarantee - a fast
release, slow dispatch pickup, or a disabled pipeline (no leaf at all,
timing-independent) still exposes the inconsistency. Without the expected-set
tracking, overall_status can also flap: in_progress --> success
(orchestrator finalizes, pytorch silent) --> in_progress (pytorch dispatch
lands) --> success. This issue hardens the logic so correctness does not depend
on that slack.

Relates to #56, #41.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions