quickstart.yml

name: Quickstart

# Coverage note (#1755):
#
# Two guarantees have no equivalent in-repo test and are the reason this
# workflow still runs:
#   * Horizon ingest pipeline — a real horizon process consumes
#     LedgerCloseMeta from a henyey validator end-to-end (pubnet shard).
#   * Multi-service orchestration — docker startup order, port binding, and
#     inter-service pipes when core/rpc/horizon run together.
#
# Three other guarantees that previously only ran here now have fast
# deterministic in-repo tests:
#   * Two-process catchup → run handoff:
#     crates/henyey/tests/catchup_to_run_handoff.rs
#   * LedgerCloseMeta record-mark framing on fd:3: covered by the same
#     subprocess test reading via `XdrInputStream` (wire-compat-tested
#     against stellar-core at crates/common/src/xdr_stream.rs).
#   * JSON-RPC HTTP surface: crates/rpc/tests/http_dispatch.rs.
#
# Test orchestration (#2916):
#
# Image building is delegated to stellar/quickstart's reusable build
# workflow with test: false. Test orchestration lives in this repo so we
# can apply a timeout-only retry to the flaky testnet/core,horizon shard
# without forking the entire upstream test matrix.
# Each probe is run through scripts/ci/run-quickstart-test.sh which adds:
#   * GNU timeout with diagnostics capture on failure
#   * Exactly one retry for ANY probe on the testnet/core,horizon shard when the
#     exit is a transient-infra signature: exit 124 (timeout / slow start, #2916)
#     or exit 143 (SIGTERM — "runner has received a shutdown signal" /
#     spot-runner reclamation, #3131). Scope was widened from the single
#     horizon-core-up probe to the whole shard in #3185: testnet stellar-core's
#     slow catchup propagates to whichever probe runs next (horizon-ingesting
#     timed out right after horizon-core-up came up — run 27019344504), and that
#     probe had no retry. The retry stays scoped to this one shard.
#   * No retry for any other shard, or for any non-transient failure (e.g. a
#     genuine probe failure exit 1) — those still fail loudly.
#   * Soft-skip on testnet TIMEOUT (#3272): the testnet/core,horizon shard alone
#     passes --soft-on-timeout, so after the retry is exhausted a probe TIMEOUT
#     (exit 124 ONLY) is converted to a neutral exit 0 with a grep-able
#     SOFT-SKIP marker — the testnet shard depends on external network liveness
#     (slow catchup / checkpoint cadence / archive availability), not henyey
#     correctness, and a stuck sync probe used to red-roll the whole workflow.
#     A genuine assertion failure (any non-124 exit) on testnet STILL stays red,
#     so a real henyey-on-testnet break is never masked. Paired with a tighter
#     240s per-probe timeout on that shard so the soft-skip fires in minutes.
#
# Whole-runner reclamation (#3185 / #3193): when GitHub SIGTERMs/SIGKILLs the
# entire runner, the bash wrapper is killed too and never reaches its in-script
# retry. An in-RUN job cannot recover this — a job inside a run cannot
# re-dispatch (rerun --failed) its own still-running run (it fails with
# "This workflow is already running", #3193). Recovery therefore lives in a
# SEPARATE workflow_run-triggered workflow, .github/workflows/quickstart-retry.yml,
# which fires AFTER this run completes (so the rerun is accepted) and
# re-dispatches the failed jobs exactly once (conclusion == failure AND
# run_attempt == 1). A genuine failure reproduces on attempt 2 and stays red,
# so this absorbs exactly one transient reclamation without masking regressions.
#
# Upstream contract validation:
#   The validate-contract job fetches both build.yml and internal-build.yml
#   from the resolved SHA and validates them against the pinned contract at
#   scripts/ci/upstream-quickstart-contract.yml. This catches drift in the
#   input interface (build.yml) and the artifact/image interface
#   (internal-build.yml). Note: the `uses:` field calls build.yml@main
#   because GitHub Actions does not support dynamic refs for reusable
#   workflows. See the contract file header for the full limitation.
#
# Testnet shard (#1848): Temporarily downgraded from core,rpc,horizon to
# core,horizon. The test_stellar_rpc_healthy step requires continuous
# ledger close, but after initial catchup the node waits up to ~5 min
# for the next archive checkpoint (64-ledger cycle on testnet). The
# upstream timeout_multiplier is hardcoded to 4 (= 4 min on first
# attempt), which is insufficient. Re-enable RPC on testnet when the
# upstream timeout is configurable or the RPC healthy test is skipped
# for testnet (matching the pubnet precedent).

permissions:
  contents: read

# Per-probe timeout budget, mirroring upstream
# stellar/quickstart/.github/workflows/internal-test.yml, which sets
# `env.timeout_multiplier: 4` and computes each probe step's timeout-minutes as
# `github.run_attempt * timeout_multiplier`. We reproduce the same multiplier
# here so the in-repo orchestration preserves upstream timeout semantics (4 min
# on attempt 1, escalating on manual re-runs) instead of diverging silently.
# Pinned in scripts/ci/upstream-quickstart-contract.yml (timeout_multiplier: 4).
# Note: the validate-contract job only fetches/validates upstream build.yml and
# internal-build.yml, not internal-test.yml (where the multiplier lives), so it
# does NOT catch multiplier drift. Drift is caught by the harness/text
# assertions in scripts/test-quickstart-harness.sh instead (#2920).
env:
  timeout_multiplier: 4

on:
  pull_request:
  push:
    branches: [main]
    paths-ignore:
      - '*.md'
      - 'docs/**'
      - 'prompts/**'
      - 'reports/**'
      - 'ai-summary/**'
      - '.claude/**'
      - '.github/skills/**'

concurrency:
  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
  # Quickstart runs take 15–30 min and are expensive to restart. Let the
  # current run finish; GitHub collapses intermediate queued runs so only
  # the most recent queued run executes after the current one completes.
  cancel-in-progress: false

jobs:
  # Resolve the quickstart SHA once so build and test use the same version.
  setup:
    if: github.event_name == 'push' || github.event.pull_request.head.repo.private == false
    runs-on: ubuntu-latest
    outputs:
      quickstart-sha: ${{ steps.resolve.outputs.sha }}
    steps:
      - name: Resolve stellar/quickstart SHA
        id: resolve
        run: |
          SHA=$(git ls-remote https://github.com/stellar/quickstart.git HEAD | cut -f1)
          echo "sha=$SHA" >> "$GITHUB_OUTPUT"
          echo "Resolved stellar/quickstart SHA: $SHA"

  # Validate that our assumptions about the upstream contract still hold.
  # Fetches both the top-level build.yml (input contract) and the delegated
  # internal-build.yml (artifact/image contract) from the resolved SHA, then
  # validates both against scripts/ci/upstream-quickstart-contract.yml.
  #
  # The drift guard step verifies the resolved SHA == current main HEAD,
  # since GitHub Actions requires the literal `@main` ref in `uses:`. If they
  # diverge (e.g., setup resolved a stale SHA), the job fails immediately
  # rather than validating one revision while executing another.
  validate-contract:
    needs: setup
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Verify resolved SHA matches @main (drift guard)
        run: |
          SHA="${{ needs.setup.outputs.quickstart-sha }}"
          # We call build.yml@main but validate at $SHA. If main has advanced
          # past the SHA we resolved in setup, the executed workflow may differ
          # from what we validated — fail fast to avoid silent false-greens.
          MAIN_SHA=$(git ls-remote https://github.com/stellar/quickstart.git refs/heads/main | cut -f1)
          if [[ "$SHA" != "$MAIN_SHA" ]]; then
            echo "✗ Resolved SHA ($SHA) differs from current main ($MAIN_SHA)."
            echo "  The build job calls build.yml@main but we validated at $SHA."
            echo "  This means the executed workflow may have drifted from what"
            echo "  we validated. Re-run the workflow to pick up the latest SHA."
            exit 1
          fi
          echo "✓ Resolved SHA matches current main — no @main drift."

      - name: Fetch upstream workflows
        run: |
          SHA="${{ needs.setup.outputs.quickstart-sha }}"
          curl -sf "https://raw.githubusercontent.com/stellar/quickstart/$SHA/.github/workflows/build.yml" \
            -o /tmp/upstream-build.yml
          # internal-build.yml is the delegated workflow that produces artifacts.
          # It may not exist (older upstream versions inline it) — handle gracefully.
          curl -sf "https://raw.githubusercontent.com/stellar/quickstart/$SHA/.github/workflows/internal-build.yml" \
            -o /tmp/upstream-internal-build.yml || true

      - name: Validate upstream contract
        run: |
          CONTRACT="scripts/ci/upstream-quickstart-contract.yml"
          UPSTREAM="/tmp/upstream-build.yml"
          INTERNAL="/tmp/upstream-internal-build.yml"

          echo "=== Validating upstream quickstart contract ==="
          echo "SHA: ${{ needs.setup.outputs.quickstart-sha }}"
          echo "Contract file: $CONTRACT"
          echo ""

          ERRORS=0

          # --- 1. Validate build.yml input contract ---
          echo "--- build.yml input contract ---"

          # Read expected inputs from contract file
          if ! grep -q '^build_inputs:' "$CONTRACT"; then
            echo "✗ contract file missing build_inputs section"
            ERRORS=$((ERRORS + 1))
          else
            for input in ref test images archs; do
              if grep -q "  - $input" "$CONTRACT"; then
                if grep -q "$input:" "$UPSTREAM"; then
                  echo "✓ upstream accepts '$input' input (matches contract)"
                else
                  echo "✗ upstream missing '$input' input (contract expects it)"
                  ERRORS=$((ERRORS + 1))
                fi
              fi
            done
          fi

          # Verify 'test' is boolean-typed (we depend on test: false)
          if grep -q 'test:' "$UPSTREAM" && grep -A5 'test:' "$UPSTREAM" | grep -q 'type: boolean\|type:.*bool'; then
            echo "✓ 'test' input is boolean-typed"
          else
            echo "✗ 'test' input not found or not boolean-typed"
            ERRORS=$((ERRORS + 1))
          fi

          echo ""

          # --- 2. Validate artifact/image contract (internal-build.yml) ---
          echo "--- internal-build.yml artifact contract ---"

          # Read expected artifact name from contract
          EXPECTED_ARTIFACT=$(grep '^expected_artifact_name:' "$CONTRACT" | sed 's/.*: *"\(.*\)"/\1/')
          EXPECTED_TAG=$(grep '^expected_image_tag:' "$CONTRACT" | sed 's/.*: *"\(.*\)"/\1/')
          ARTIFACT_PATTERN=$(grep '^artifact_name_pattern:' "$CONTRACT" | sed 's/.*: *"\(.*\)"/\1/')

          if [[ -f "$INTERNAL" ]]; then
            echo "✓ internal-build.yml found at resolved SHA"

            # Check that it uploads artifacts (upload-artifact action present)
            if grep -q 'upload-artifact' "$INTERNAL"; then
              echo "✓ internal-build.yml uploads artifacts"
            else
              echo "✗ internal-build.yml does not use upload-artifact"
              ERRORS=$((ERRORS + 1))
            fi

            # --- Exact artifact name validation ---
            # The contract specifies artifact_name_pattern: "image-quickstart-{tag}-{arch}.tar"
            # Upstream internal-build.yml constructs the artifact name using expressions like:
            #   image-quickstart-<inputs.tag expr>-<matrix.arch expr>
            # (the angle-bracket placeholders stand in for GHA expression syntax;
            #  literal expression delimiters are avoided here so the workflow file
            #  itself stays parseable — see #2933.)
            # We validate: (a) the exact "image-quickstart-" prefix with tag/arch interpolation,
            # and (b) our concrete expected_artifact_name decomposes correctly from the pattern.
            #
            # Convert pattern "image-quickstart-{tag}-{arch}.tar" into a regex that matches
            # the upstream template expression (tag and arch are GHA expressions).
            ARTIFACT_PREFIX="image-quickstart-"
            if grep -q "${ARTIFACT_PREFIX}" "$INTERNAL"; then
              echo "✓ internal-build.yml artifact name uses prefix '${ARTIFACT_PREFIX}'"
            else
              echo "✗ internal-build.yml missing artifact name prefix '${ARTIFACT_PREFIX}'"
              echo "  Expected pattern: $ARTIFACT_PATTERN"
              echo "  Expected concrete: $EXPECTED_ARTIFACT"
              ERRORS=$((ERRORS + 1))
            fi

            # Validate the artifact name template follows the exact {tag}-{arch}.tar structure.
            # Upstream must produce: image-quickstart-<tag_expr>-<arch_expr> (with .tar optional
            # in the upload name). We check that the line contains the prefix, then a tag
            # expression, a literal hyphen separator, then an arch expression — in that order.
            # This rejects incompatible shapes like {arch}-{tag} or {tag}.zip.
            ARTIFACT_REGEX="${ARTIFACT_PREFIX}"'\$\{\{[^}]*(tag|inputs\.tag|matrix\.tag)[^}]*\}\}-\$\{\{[^}]*(arch|matrix\.arch|inputs\.arch)[^}]*\}\}'
            if grep -qE "$ARTIFACT_REGEX" "$INTERNAL"; then
              echo "✓ internal-build.yml artifact name matches {tag}-{arch} structure"
            else
              echo "✗ internal-build.yml artifact name does not match expected {tag}-{arch} structure"
              echo "  Expected regex: $ARTIFACT_REGEX"
              echo "  Expected pattern: $ARTIFACT_PATTERN (tag=testing-with-pr, arch=amd64)"
              ERRORS=$((ERRORS + 1))
            fi

            # --- Exact image tag validation ---
            # The contract specifies image_tag_pattern: "quickstart:{tag}-{arch}"
            # Upstream tags the image as quickstart:<inputs.tag expr>-<matrix.arch expr>
            # or similar. Validate the "quickstart:" prefix with tag-arch interpolation.
            IMAGE_TAG_PREFIX="quickstart:"
            if grep -q "${IMAGE_TAG_PREFIX}" "$INTERNAL"; then
              echo "✓ internal-build.yml image tag uses prefix '${IMAGE_TAG_PREFIX}'"
            else
              echo "✗ internal-build.yml missing image tag prefix '${IMAGE_TAG_PREFIX}'"
              echo "  Expected tag pattern: quickstart:{tag}-{arch}"
              echo "  Expected concrete: $EXPECTED_TAG"
              ERRORS=$((ERRORS + 1))
            fi

            # Validate the image tag template follows the exact {tag}-{arch} structure.
            # Upstream must tag as: quickstart:<tag_expr>-<arch_expr>.
            # Reject shapes like quickstart:<inputs.tag expr> (missing arch).
            TAG_REGEX="${IMAGE_TAG_PREFIX}"'\$\{\{[^}]*(tag|inputs\.tag|matrix\.tag)[^}]*\}\}-\$\{\{[^}]*(arch|matrix\.arch|inputs\.arch)[^}]*\}\}'
            if grep -qE "$TAG_REGEX" "$INTERNAL"; then
              echo "✓ internal-build.yml image tag matches {tag}-{arch} structure"
            else
              echo "✗ internal-build.yml image tag does not match expected {tag}-{arch} structure"
              echo "  Expected regex: $TAG_REGEX"
              echo "  Expected: dynamic tag with {tag}-{arch} pattern"
              ERRORS=$((ERRORS + 1))
            fi

            # Check docker save / image export pattern (/tmp/image is the conventional path)
            if grep -q '/tmp/image\|docker save' "$INTERNAL"; then
              echo "✓ internal-build.yml uses docker image export (matches artifact_layout)"
            else
              echo "✗ could not confirm /tmp/image or docker save in internal-build.yml"
              echo "  The 'Load Docker image' step depends on this layout — drift will break test runs."
              ERRORS=$((ERRORS + 1))
            fi
          else
            # internal-build.yml not found — check if build.yml itself handles artifacts
            echo "⚠ internal-build.yml not found at resolved SHA (may be inlined in build.yml)"
            if grep -q "${ARTIFACT_PREFIX:-image-quickstart-}" "$UPSTREAM" && grep -q 'upload-artifact' "$UPSTREAM"; then
              echo "✓ build.yml directly handles artifact upload with expected naming"
            else
              echo "✗ neither build.yml nor internal-build.yml confirms artifact contract"
              echo "  Expected artifact: $EXPECTED_ARTIFACT"
              ERRORS=$((ERRORS + 1))
            fi
          fi

          echo ""

          # --- 2b. Validate OUR consumer-side artifact/tag shape (anchored) ---
          # The checks above validate the UPSTREAM template (internal-build.yml).
          # This block validates the EXACT, concrete artifact filename and image
          # tag that *this* workflow consumes — the download-artifact `name:` and
          # the `docker run ... <tag>` line — against anchored ERE regexes stored
          # in the contract file (single source of truth, also used by
          # scripts/test-quickstart-harness.sh). Anchoring rejects incompatible
          # suffixes (.zip, .tar.gz, -debug) that a bare substring `grep -q`
          # would silently accept (#2932). The contract stores the regex body
          # verbatim (raw grep|sed extraction, no YAML unescaping).
          echo "--- consumer-side artifact/tag shape (this workflow) ---"
          ARTIFACT_NAME_REGEX=$(grep '^artifact_name_regex:' "$CONTRACT" | sed 's/^artifact_name_regex: *"\(.*\)"$/\1/')
          IMAGE_TAG_REGEX=$(grep '^image_tag_regex:' "$CONTRACT" | sed 's/^image_tag_regex: *"\(.*\)"$/\1/')
          WORKFLOW_SELF=".github/workflows/quickstart.yml"

          if [[ -z "$ARTIFACT_NAME_REGEX" || -z "$IMAGE_TAG_REGEX" ]]; then
            echo "✗ contract missing artifact_name_regex/image_tag_regex (anchored ERE)"
            ERRORS=$((ERRORS + 1))
          else
            if grep -Eq "$ARTIFACT_NAME_REGEX" "$WORKFLOW_SELF"; then
              echo "✓ workflow downloads the exact artifact shape (anchored)"
            else
              echo "✗ workflow has no line matching the anchored artifact regex"
              echo "  Regex: $ARTIFACT_NAME_REGEX"
              ERRORS=$((ERRORS + 1))
            fi
            if grep -Eq "$IMAGE_TAG_REGEX" "$WORKFLOW_SELF"; then
              echo "✓ workflow runs the exact image tag shape (anchored)"
            else
              echo "✗ workflow has no line matching the anchored image-tag regex"
              echo "  Regex: $IMAGE_TAG_REGEX"
              ERRORS=$((ERRORS + 1))
            fi
          fi

          echo ""

          # --- 3. Summary ---
          if [[ $ERRORS -gt 0 ]]; then
            echo "=== FAILED: $ERRORS contract violation(s) detected ==="
            echo "The upstream stellar/quickstart workflows at SHA ${{ needs.setup.outputs.quickstart-sha }}"
            echo "no longer match our pinned contract at $CONTRACT."
            echo "Update the contract file and workflow if the upstream change is intentional."
            exit 1
          fi

          echo "=== Upstream contract validation passed (${{ needs.setup.outputs.quickstart-sha }}) ==="

  # Build the quickstart image with test: false — testing is handled locally.
  #
  # NOTE: GitHub Actions does not support dynamic refs in `uses:` for reusable
  # workflows. We call build.yml@main; the `validate-contract` job's drift
  # guard ensures the resolved SHA == main HEAD before we reach this point,
  # so the validated and executed definitions are the same revision.
  build:
    needs: [setup, validate-contract]
    uses: stellar/quickstart/.github/workflows/build.yml@main
    with:
      ref: ${{ needs.setup.outputs.quickstart-sha }}
      test: false
      images: |
        [
          {
            "tag": "testing-with-pr",
            "inherit": "testing",
            "config": {
              "horizon_skip_protocol_version_check": true
            },
            "deps": [
              { "name": "core", "repo": "${{ github.event.pull_request.head.repo.full_name || github.repository }}", "ref": "${{ github.event.pull_request.head.sha || github.sha }}" }
            ]
          }
        ]
      archs: '["amd64"]'

  # Run upstream Go probes locally through the timeout/retry wrapper.
  test:
    needs: [setup, build]
    runs-on: ubuntu-latest
    timeout-minutes: 45
    strategy:
      fail-fast: false
      matrix:
        include:
          # Local shards (network=local)
          - network: local
            enable: core
            probes: "test_core.go"
          - network: local
            enable: rpc
            probes: "test_stellar_rpc_up.go test_stellar_rpc_healthy.go test_friendbot.go"
          - network: local
            enable: "core,rpc,horizon"
            probes: "test_core.go test_horizon_up.go test_horizon_core_up.go test_horizon_ingesting.go test_stellar_rpc_up.go test_stellar_rpc_healthy.go test_friendbot.go"
          # local/galexie soft-degate (#3563). The pinned galexie image
          # (galexie-v26.1.0) never exports a ledger-meta partition while
          # henyey core now runs Protocol 27, so test_galexie.go's
          # waitForURL(partitionURL) polls .../ledger-meta/FFFFFFFF--0-63999/
          # and gets HTTP 404 forever (unbounded for{} poll, no internal
          # deadline) until the wrapper timeout fires (exit 124). This is an
          # upstream IMAGE-VERSION incompatibility, NOT a henyey-correctness
          # break — tracked in #3565 — so this shard mirrors the testnet
          # shard's de-gate: soft_on_timeout converts a TIMEOUT (exit
          # 124/137) into a neutral SOFT-SKIP (exit 0, grep-able marker)
          # while any genuine assertion failure (non-timeout exit 1) STILL
          # stays RED. probe_timeout: 240 + step_timeout_minutes: 25 make the
          # soft-skip fire in minutes so the PR's own CI never reproduces the
          # ~55-min hang. The soft-skip is TIMEOUT-ONLY and TEMPORARY: remove
          # it once the galexie image is bumped to a Protocol-27-capable tag
          # (deliberately OUT OF SCOPE here — bumping a still-incompatible tag
          # would re-hang CI; see #3565).
          - network: local
            enable: galexie
            probes: "test_galexie.go"
            soft_on_timeout: true
            probe_timeout: 240
            step_timeout_minutes: 25
          # Additional testnet shard.
          #
          # De-gate `main` on testnet external-liveness (#3272): the testnet
          # shard flakes on external network health (slow catchup, checkpoint
          # cadence, archive availability) — NOT henyey correctness — and a
          # stuck sync probe TIMEOUT used to red-roll the whole workflow and
          # block every PR's merge gate. So this shard:
          #   * sets soft_on_timeout: true — a probe TIMEOUT (exit 124) is
          #     converted to a neutral soft-skip by run-quickstart-test.sh
          #     (a genuine assertion failure, non-124, still stays red), and
          #   * tightens probe_timeout to 240s (= timeout_multiplier(4) * 60,
          #     the upstream 4-min precedent already documented above) so a
          #     stuck probe fast-fails into the soft-skip in minutes with
          #     diagnostics uploaded, instead of hanging until the 45-min
          #     job wall-clock cancels the run.
          # This extends the existing testnet-RPC-disabled precedent (see the
          # "Testnet shard (#1848)" note above) to the sync-probe TIMEOUT
          # outcome. Scoped to THIS shard only — local/pubnet keep the default
          # 600s hard-fail-on-timeout and carry no soft flag.
          # Budget: 4 probes × 240s + one 240s retry = 1200s (~20 min), well
          # under the job's 45-min timeout-minutes.
          #
          # DIAGNOSTIC instrumentation (#3286): this shard also sets a TIGHT,
          # testnet-only step-level fail-fast bound via step_timeout_minutes: 25
          # (applied as the "Run probes through wrapper" step's timeout-minutes
          # below). Two wrapper-level fixes (#3273, #3287) both FAILED to bound
          # the ~55-min step-8 hang, and a 45-min JOB-level wall-clock is a
          # *cancel* — so `if: failure()` never fires and the diagnostics step
          # never uploads. A step-level timeout-minutes kill instead marks the
          # STEP failed (distinct from a job cancel), so the upload step runs.
          # 25 min > the ~20-min healthy budget above, so a healthy run never
          # trips it. Other shards inherit the generous 360-min default (a
          # no-op under the job's own 45-min cap) and stay byte-identical.
          - network: testnet
            enable: "core,horizon"
            probes: "test_core.go test_horizon_up.go test_horizon_core_up.go test_horizon_ingesting.go"
            soft_on_timeout: true
            probe_timeout: 240
            step_timeout_minutes: 25
          # Additional pubnet shard
          # Note: upstream excludes horizon_core_up, horizon_ingesting, and
          # stellar_rpc_healthy on pubnet (see internal-test.yml conditionals).
          - network: pubnet
            enable: "core,rpc,horizon"
            probes: "test_core.go test_horizon_up.go test_stellar_rpc_up.go"
    steps:
      - uses: actions/checkout@v4

      - name: Download quickstart image
        uses: actions/download-artifact@v4
        with:
          name: image-quickstart-testing-with-pr-amd64.tar
          path: /tmp/quickstart-image

      - name: Load Docker image
        run: |
          tarball="$(ls /tmp/quickstart-image/image 2>/dev/null || ls /tmp/quickstart-image/*.tar 2>/dev/null | head -1)"
          docker load -i "$tarball"

      - name: Checkout stellar/quickstart (tests)
        uses: actions/checkout@v4
        with:
          repository: stellar/quickstart
          ref: ${{ needs.setup.outputs.quickstart-sha }}
          path: quickstart

      - uses: actions/setup-go@v5
        with:
          go-version: 'stable'
          cache: false

      - name: Run quickstart container
        run: |
          docker run -d --name quickstart \
            -p 8000:8000 -p 11626:11626 -p 8001:8001 \
            quickstart:testing-with-pr-amd64 \
            --${{ matrix.network }} \
            --enable ${{ matrix.enable }}

      - name: Run probes through wrapper
        # DIAGNOSTIC fail-fast bound (#3286). Testnet sets
        # matrix.step_timeout_minutes: 25 (tighter than its ~20-min healthy
        # budget) so a hang here FAILS the STEP at ~25 min instead of running
        # to the 45-min JOB cancel — a step-timeout marks the step *failed*, so
        # the "Upload diagnostics" step (if: failure()) actually runs and the
        # watchdog's process-tree dump is uploaded. All other shards have no
        # step_timeout_minutes set and inherit the generous 360-min default
        # (a no-op under the job's 45-min cap), so they stay byte-identical.
        timeout-minutes: ${{ matrix.step_timeout_minutes || 360 }}
        env:
          NETWORK: ${{ matrix.network }}
          ENABLE: ${{ matrix.enable }}
          # Per-shard overrides (#3272). Unset matrix keys render as empty
          # strings; the loop below falls back to the default budget and omits
          # the soft flag when these are empty. Only the testnet shard sets them.
          SHARD_PROBE_TIMEOUT: ${{ matrix.probe_timeout }}
          SHARD_SOFT_ON_TIMEOUT: ${{ matrix.soft_on_timeout }}
        run: |
          # Per-probe budget = github.run_attempt * timeout_multiplier minutes,
          # converted to seconds for GNU `timeout` (* 60). Mirrors upstream
          # internal-test.yml's `minutes = github.run_attempt * timeout_multiplier`
          # applied as each probe step's timeout-minutes (attempt 1 -> 4 min;
          # manual re-runs escalate to 8, 12, ...). github.run_attempt is constant
          # across a job's steps, so computing it once here is equivalent to
          # upstream's per-step timeout-minutes. The budget is applied per-probe;
          # a shard running N probes can use up to N * budget wall-clock, bounded
          # overall by this job's timeout-minutes: 45. The targeted single retry
          # in run-quickstart-test.sh is layered on top of this budget (#2920).
          PROBE_TIMEOUT=$(( ${{ github.run_attempt }} * timeout_multiplier * 60 ))

          # Per-shard probe-timeout override (#3272). The testnet shard fast-
          # fails at a tighter budget (SHARD_PROBE_TIMEOUT=240) so a stuck sync
          # probe is soft-skipped in minutes; all other shards leave it empty and
          # keep the upstream run_attempt * multiplier * 60 budget unchanged.
          if [[ -n "$SHARD_PROBE_TIMEOUT" ]]; then
            PROBE_TIMEOUT="$SHARD_PROBE_TIMEOUT"
          fi

          # Per-shard soft-on-timeout flag (#3272). Only the testnet shard sets
          # matrix.soft_on_timeout: true; for every other shard SHARD_SOFT_ON_TIMEOUT
          # is empty and SOFT_FLAG stays empty so the wrapper invocation is
          # byte-identical to before (no --soft-on-timeout passed).
          SOFT_FLAG=""
          if [[ "$SHARD_SOFT_ON_TIMEOUT" == "true" ]]; then
            SOFT_FLAG="--soft-on-timeout"
          fi
          DIAG_DIR="/tmp/quickstart-diagnostics"

          # ---- DIAGNOSTIC hang watchdog (#3286), testnet shard only ----
          # Capture the step shell's PID HERE, in the step body — NOT inside a
          # subshell and NOT via $BASHPID — so the backgrounded watchdog can
          # dump the fds THIS shell holds. The watchdog sleeps WATCHDOG_DELAY
          # (default 1200s = 20 min, < the 25-min step_timeout_minutes bound)
          # then writes a process-tree + open-fd snapshot to the diagnostics
          # dir, so on the next hang the dump is on disk BEFORE the step kill
          # and gets swept into the uploaded artifact. It is reaped on the
          # healthy path (trap EXIT + explicit kill after the loop) so it never
          # lingers or emits a spurious dump on a passing run. This does NOT fix
          # the hang — its deliverable is DATA from the next hang.
          STEP_PID=$$
          WATCHDOG_PID=""
          if [[ "$NETWORK" == "testnet" ]]; then
            ( sleep "${WATCHDOG_DELAY:-1200}"; \
              bash scripts/ci/quickstart-hang-watchdog.sh \
                "$DIAG_DIR/testnet-hang-watchdog" "$STEP_PID" ) &
            WATCHDOG_PID=$!
            # Reap the watchdog on any step exit so a passing run never leaks
            # the backgrounded sleep.
            trap '[[ -n "$WATCHDOG_PID" ]] && kill "$WATCHDOG_PID" 2>/dev/null || true' EXIT
          fi

          for probe_file in ${{ matrix.probes }}; do
            probe_name="${probe_file%.go}"
            probe_name="${probe_name#test_}"
            # Normalize underscores to hyphens so probe names match the
            # retry policy (e.g. horizon_core_up → horizon-core-up).
            probe_name="${probe_name//_/-}"

            scripts/ci/run-quickstart-test.sh \
              $SOFT_FLAG \
              --network "$NETWORK" \
              --enable "$ENABLE" \
              --probe "$probe_name" \
              --timeout "$PROBE_TIMEOUT" \
              --diagnostics-dir "$DIAG_DIR/$NETWORK-$ENABLE-$probe_name" \
              -- go run "quickstart/tests/$probe_file"
          done

          # Healthy path: probes finished, so reap the watchdog now (explicit
          # kill, belt-and-suspenders with the EXIT trap) — no lingering sleep,
          # no spurious dump on a passing run.
          if [[ -n "$WATCHDOG_PID" ]]; then
            kill "$WATCHDOG_PID" 2>/dev/null || true
          fi

      - name: Upload diagnostics on failure
        # DIAGNOSTIC instrumentation (#3286): upload on BOTH a step-timeout
        # failure (the primary path — a testnet step_timeout_minutes kill marks
        # the step failed, so failure() fires) AND a residual job-level cancel
        # (belt-and-suspenders). The watchdog writes its dump before the step
        # kill, so the artifact exists for either disposition.
        if: ${{ failure() || cancelled() }}
        uses: actions/upload-artifact@v4
        with:
          name: quickstart-diagnostics-${{ matrix.network }}-${{ matrix.enable }}
          path: /tmp/quickstart-diagnostics/
          retention-days: 7

      - name: Cleanup
        if: always()
        run: |
          docker stop quickstart 2>/dev/null || true
          docker rm quickstart 2>/dev/null || true

  # NOTE (#3193): the previous in-run `rerun-on-transient` job was REMOVED. A job
  # inside a run cannot re-dispatch its own still-running run — the rerun-failed
  # API call returns "This workflow is already running" (exit 1), so the
  # auto-retry never fired and only added a spurious FAILURE check. Recovery from
  # a whole-runner reclamation now lives in .github/workflows/quickstart-retry.yml,
  # a separate workflow_run-triggered workflow that fires AFTER this run
  # completes (so the re-dispatch is accepted).