Make counter-streak dwell time-denominated to fix 2x streak acceleration #3246

Workflow file for this run

.github/workflows/quickstart.yml at df23ce4

	name: Quickstart

	# Coverage note (#1755):
	#
	# Two guarantees have no equivalent in-repo test and are the reason this
	# workflow still runs:
	# * Horizon ingest pipeline — a real horizon process consumes
	# LedgerCloseMeta from a henyey validator end-to-end (pubnet shard).
	# * Multi-service orchestration — docker startup order, port binding, and
	# inter-service pipes when core/rpc/horizon run together.
	#
	# Three other guarantees that previously only ran here now have fast
	# deterministic in-repo tests:
	# * Two-process catchup → run handoff:
	# crates/henyey/tests/catchup_to_run_handoff.rs
	# * LedgerCloseMeta record-mark framing on fd:3: covered by the same
	# subprocess test reading via `XdrInputStream` (wire-compat-tested
	# against stellar-core at crates/common/src/xdr_stream.rs).
	# * JSON-RPC HTTP surface: crates/rpc/tests/http_dispatch.rs.
	#
	# Test orchestration (#2916):
	#
	# Image building is delegated to stellar/quickstart's reusable build
	# workflow with test: false. Test orchestration lives in this repo so we
	# can apply a timeout-only retry to the flaky testnet/core,horizon shard
	# without forking the entire upstream test matrix.
	# Each probe is run through scripts/ci/run-quickstart-test.sh which adds:
	# * GNU timeout with diagnostics capture on failure
	# * Exactly one retry for ANY probe on the testnet/core,horizon shard when the
	# exit is a transient-infra signature: exit 124 (timeout / slow start, #2916)
	# or exit 143 (SIGTERM — "runner has received a shutdown signal" /
	# spot-runner reclamation, #3131). Scope was widened from the single
	# horizon-core-up probe to the whole shard in #3185: testnet stellar-core's
	# slow catchup propagates to whichever probe runs next (horizon-ingesting
	# timed out right after horizon-core-up came up — run 27019344504), and that
	# probe had no retry. The retry stays scoped to this one shard.
	# * No retry for any other shard, or for any non-transient failure (e.g. a
	# genuine probe failure exit 1) — those still fail loudly.
	# * Soft-skip on testnet TIMEOUT (#3272): the testnet/core,horizon shard alone
	# passes --soft-on-timeout, so after the retry is exhausted a probe TIMEOUT
	# (exit 124 ONLY) is converted to a neutral exit 0 with a grep-able
	# SOFT-SKIP marker — the testnet shard depends on external network liveness
	# (slow catchup / checkpoint cadence / archive availability), not henyey
	# correctness, and a stuck sync probe used to red-roll the whole workflow.
	# A genuine assertion failure (any non-124 exit) on testnet STILL stays red,
	# so a real henyey-on-testnet break is never masked. Paired with a tighter
	# 240s per-probe timeout on that shard so the soft-skip fires in minutes.
	#
	# Whole-runner reclamation (#3185 / #3193): when GitHub SIGTERMs/SIGKILLs the
	# entire runner, the bash wrapper is killed too and never reaches its in-script
	# retry. An in-RUN job cannot recover this — a job inside a run cannot
	# re-dispatch (rerun --failed) its own still-running run (it fails with
	# "This workflow is already running", #3193). Recovery therefore lives in a
	# SEPARATE workflow_run-triggered workflow, .github/workflows/quickstart-retry.yml,
	# which fires AFTER this run completes (so the rerun is accepted) and
	# re-dispatches the failed jobs exactly once (conclusion == failure AND
	# run_attempt == 1). A genuine failure reproduces on attempt 2 and stays red,
	# so this absorbs exactly one transient reclamation without masking regressions.
	#
	# Upstream contract validation:
	# The validate-contract job fetches both build.yml and internal-build.yml
	# from the resolved SHA and validates them against the pinned contract at
	# scripts/ci/upstream-quickstart-contract.yml. This catches drift in the
	# input interface (build.yml) and the artifact/image interface
	# (internal-build.yml). Note: the `uses:` field calls build.yml@main
	# because GitHub Actions does not support dynamic refs for reusable
	# workflows. See the contract file header for the full limitation.
	#
	# Testnet shard (#1848): Temporarily downgraded from core,rpc,horizon to
	# core,horizon. The test_stellar_rpc_healthy step requires continuous
	# ledger close, but after initial catchup the node waits up to ~5 min
	# for the next archive checkpoint (64-ledger cycle on testnet). The
	# upstream timeout_multiplier is hardcoded to 4 (= 4 min on first
	# attempt), which is insufficient. Re-enable RPC on testnet when the
	# upstream timeout is configurable or the RPC healthy test is skipped
	# for testnet (matching the pubnet precedent).

	permissions:
	contents: read

	# Per-probe timeout budget, mirroring upstream
	# stellar/quickstart/.github/workflows/internal-test.yml, which sets
	# `env.timeout_multiplier: 4` and computes each probe step's timeout-minutes as
	# `github.run_attempt * timeout_multiplier`. We reproduce the same multiplier
	# here so the in-repo orchestration preserves upstream timeout semantics (4 min
	# on attempt 1, escalating on manual re-runs) instead of diverging silently.
	# Pinned in scripts/ci/upstream-quickstart-contract.yml (timeout_multiplier: 4).
	# Note: the validate-contract job only fetches/validates upstream build.yml and
	# internal-build.yml, not internal-test.yml (where the multiplier lives), so it
	# does NOT catch multiplier drift. Drift is caught by the harness/text
	# assertions in scripts/test-quickstart-harness.sh instead (#2920).
	env:
	timeout_multiplier: 4

	on:
	pull_request:
	push:
	branches: [main]
	paths-ignore:
	- '*.md'
	- 'docs/**'
	- 'prompts/**'
	- 'reports/**'
	- 'ai-summary/**'
	- '.claude/**'
	- '.github/skills/**'

	concurrency:
	group: ${{ github.workflow }}-${{ github.event.pull_request.number \|\| github.ref }}
	# Quickstart runs take 15–30 min and are expensive to restart. Let the
	# current run finish; GitHub collapses intermediate queued runs so only
	# the most recent queued run executes after the current one completes.
	cancel-in-progress: false

	jobs:
	# Resolve the quickstart SHA once so build and test use the same version.
	setup:
	if: github.event_name == 'push' \|\| github.event.pull_request.head.repo.private == false
	runs-on: ubuntu-latest
	outputs:
	quickstart-sha: ${{ steps.resolve.outputs.sha }}
	steps:
	- name: Resolve stellar/quickstart SHA
	id: resolve
	run: \|
	SHA=$(git ls-remote https://github.com/stellar/quickstart.git HEAD \| cut -f1)
	echo "sha=$SHA" >> "$GITHUB_OUTPUT"
	echo "Resolved stellar/quickstart SHA: $SHA"

	# Validate that our assumptions about the upstream contract still hold.
	# Fetches both the top-level build.yml (input contract) and the delegated
	# internal-build.yml (artifact/image contract) from the resolved SHA, then
	# validates both against scripts/ci/upstream-quickstart-contract.yml.
	#
	# The drift guard step verifies the resolved SHA == current main HEAD,
	# since GitHub Actions requires the literal `@main` ref in `uses:`. If they
	# diverge (e.g., setup resolved a stale SHA), the job fails immediately
	# rather than validating one revision while executing another.
	validate-contract:
	needs: setup
	runs-on: ubuntu-latest
	steps:
	- uses: actions/checkout@v4

	- name: Verify resolved SHA matches @main (drift guard)
	run: \|
	SHA="${{ needs.setup.outputs.quickstart-sha }}"
	# We call build.yml@main but validate at $SHA. If main has advanced
	# past the SHA we resolved in setup, the executed workflow may differ
	# from what we validated — fail fast to avoid silent false-greens.
	MAIN_SHA=$(git ls-remote https://github.com/stellar/quickstart.git refs/heads/main \| cut -f1)
	if [[ "$SHA" != "$MAIN_SHA" ]]; then
	echo "✗ Resolved SHA ($SHA) differs from current main ($MAIN_SHA)."
	echo " The build job calls build.yml@main but we validated at $SHA."
	echo " This means the executed workflow may have drifted from what"
	echo " we validated. Re-run the workflow to pick up the latest SHA."
	exit 1
	fi
	echo "✓ Resolved SHA matches current main — no @main drift."

	- name: Fetch upstream workflows
	run: \|
	SHA="${{ needs.setup.outputs.quickstart-sha }}"
	curl -sf "https://raw.githubusercontent.com/stellar/quickstart/$SHA/.github/workflows/build.yml" \
	-o /tmp/upstream-build.yml
	# internal-build.yml is the delegated workflow that produces artifacts.
	# It may not exist (older upstream versions inline it) — handle gracefully.
	curl -sf "https://raw.githubusercontent.com/stellar/quickstart/$SHA/.github/workflows/internal-build.yml" \
	-o /tmp/upstream-internal-build.yml \|\| true

	- name: Validate upstream contract
	run: \|
	CONTRACT="scripts/ci/upstream-quickstart-contract.yml"
	UPSTREAM="/tmp/upstream-build.yml"
	INTERNAL="/tmp/upstream-internal-build.yml"

	echo "=== Validating upstream quickstart contract ==="
	echo "SHA: ${{ needs.setup.outputs.quickstart-sha }}"
	echo "Contract file: $CONTRACT"
	echo ""

	ERRORS=0

	# --- 1. Validate build.yml input contract ---
	echo "--- build.yml input contract ---"

	# Read expected inputs from contract file
	if ! grep -q '^build_inputs:' "$CONTRACT"; then
	echo "✗ contract file missing build_inputs section"
	ERRORS=$((ERRORS + 1))
	else
	for input in ref test images archs; do
	if grep -q " - $input" "$CONTRACT"; then
	if grep -q "$input:" "$UPSTREAM"; then
	echo "✓ upstream accepts '$input' input (matches contract)"
	else
	echo "✗ upstream missing '$input' input (contract expects it)"
	ERRORS=$((ERRORS + 1))
	fi
	fi
	done
	fi

	# Verify 'test' is boolean-typed (we depend on test: false)
	if grep -q 'test:' "$UPSTREAM" && grep -A5 'test:' "$UPSTREAM" \| grep -q 'type: boolean\\|type:.*bool'; then
	echo "✓ 'test' input is boolean-typed"
	else
	echo "✗ 'test' input not found or not boolean-typed"
	ERRORS=$((ERRORS + 1))
	fi

	echo ""

	# --- 2. Validate artifact/image contract (internal-build.yml) ---
	echo "--- internal-build.yml artifact contract ---"

	# Read expected artifact name from contract
	EXPECTED_ARTIFACT=$(grep '^expected_artifact_name:' "$CONTRACT" \| sed 's/.: "$.*$"/\1/')
	EXPECTED_TAG=$(grep '^expected_image_tag:' "$CONTRACT" \| sed 's/.: "$.*$"/\1/')
	ARTIFACT_PATTERN=$(grep '^artifact_name_pattern:' "$CONTRACT" \| sed 's/.: "$.*$"/\1/')

	if [[ -f "$INTERNAL" ]]; then
	echo "✓ internal-build.yml found at resolved SHA"

	# Check that it uploads artifacts (upload-artifact action present)
	if grep -q 'upload-artifact' "$INTERNAL"; then
	echo "✓ internal-build.yml uploads artifacts"
	else
	echo "✗ internal-build.yml does not use upload-artifact"
	ERRORS=$((ERRORS + 1))
	fi

	# --- Exact artifact name validation ---
	# The contract specifies artifact_name_pattern: "image-quickstart-{tag}-{arch}.tar"
	# Upstream internal-build.yml constructs the artifact name using expressions like:
	# image-quickstart-<inputs.tag expr>-<matrix.arch expr>
	# (the angle-bracket placeholders stand in for GHA expression syntax;
	# literal expression delimiters are avoided here so the workflow file
	# itself stays parseable — see #2933.)
	# We validate: (a) the exact "image-quickstart-" prefix with tag/arch interpolation,
	# and (b) our concrete expected_artifact_name decomposes correctly from the pattern.
	#
	# Convert pattern "image-quickstart-{tag}-{arch}.tar" into a regex that matches
	# the upstream template expression (tag and arch are GHA expressions).
	ARTIFACT_PREFIX="image-quickstart-"
	if grep -q "${ARTIFACT_PREFIX}" "$INTERNAL"; then
	echo "✓ internal-build.yml artifact name uses prefix '${ARTIFACT_PREFIX}'"
	else
	echo "✗ internal-build.yml missing artifact name prefix '${ARTIFACT_PREFIX}'"
	echo " Expected pattern: $ARTIFACT_PATTERN"
	echo " Expected concrete: $EXPECTED_ARTIFACT"
	ERRORS=$((ERRORS + 1))
	fi

	# Validate the artifact name template follows the exact {tag}-{arch}.tar structure.
	# Upstream must produce: image-quickstart-<tag_expr>-<arch_expr> (with .tar optional
	# in the upload name). We check that the line contains the prefix, then a tag
	# expression, a literal hyphen separator, then an arch expression — in that order.
	# This rejects incompatible shapes like {arch}-{tag} or {tag}.zip.
	ARTIFACT_REGEX="${ARTIFACT_PREFIX}"'\$\{\{[^}](tag\|inputs\.tag\|matrix\.tag)[^}]\}\}-\$\{\{[^}](arch\|matrix\.arch\|inputs\.arch)[^}]\}\}'
	if grep -qE "$ARTIFACT_REGEX" "$INTERNAL"; then
	echo "✓ internal-build.yml artifact name matches {tag}-{arch} structure"
	else
	echo "✗ internal-build.yml artifact name does not match expected {tag}-{arch} structure"
	echo " Expected regex: $ARTIFACT_REGEX"
	echo " Expected pattern: $ARTIFACT_PATTERN (tag=testing-with-pr, arch=amd64)"
	ERRORS=$((ERRORS + 1))
	fi

	# --- Exact image tag validation ---
	# The contract specifies image_tag_pattern: "quickstart:{tag}-{arch}"
	# Upstream tags the image as quickstart:<inputs.tag expr>-<matrix.arch expr>
	# or similar. Validate the "quickstart:" prefix with tag-arch interpolation.
	IMAGE_TAG_PREFIX="quickstart:"
	if grep -q "${IMAGE_TAG_PREFIX}" "$INTERNAL"; then
	echo "✓ internal-build.yml image tag uses prefix '${IMAGE_TAG_PREFIX}'"
	else
	echo "✗ internal-build.yml missing image tag prefix '${IMAGE_TAG_PREFIX}'"
	echo " Expected tag pattern: quickstart:{tag}-{arch}"
	echo " Expected concrete: $EXPECTED_TAG"
	ERRORS=$((ERRORS + 1))
	fi

	# Validate the image tag template follows the exact {tag}-{arch} structure.
	# Upstream must tag as: quickstart:<tag_expr>-<arch_expr>.
	# Reject shapes like quickstart:<inputs.tag expr> (missing arch).
	TAG_REGEX="${IMAGE_TAG_PREFIX}"'\$\{\{[^}](tag\|inputs\.tag\|matrix\.tag)[^}]\}\}-\$\{\{[^}](arch\|matrix\.arch\|inputs\.arch)[^}]\}\}'
	if grep -qE "$TAG_REGEX" "$INTERNAL"; then
	echo "✓ internal-build.yml image tag matches {tag}-{arch} structure"
	else
	echo "✗ internal-build.yml image tag does not match expected {tag}-{arch} structure"
	echo " Expected regex: $TAG_REGEX"
	echo " Expected: dynamic tag with {tag}-{arch} pattern"
	ERRORS=$((ERRORS + 1))
	fi

	# Check docker save / image export pattern (/tmp/image is the conventional path)
	if grep -q '/tmp/image\\|docker save' "$INTERNAL"; then
	echo "✓ internal-build.yml uses docker image export (matches artifact_layout)"
	else
	echo "✗ could not confirm /tmp/image or docker save in internal-build.yml"
	echo " The 'Load Docker image' step depends on this layout — drift will break test runs."
	ERRORS=$((ERRORS + 1))
	fi
	else
	# internal-build.yml not found — check if build.yml itself handles artifacts
	echo "⚠ internal-build.yml not found at resolved SHA (may be inlined in build.yml)"
	if grep -q "${ARTIFACT_PREFIX:-image-quickstart-}" "$UPSTREAM" && grep -q 'upload-artifact' "$UPSTREAM"; then
	echo "✓ build.yml directly handles artifact upload with expected naming"
	else
	echo "✗ neither build.yml nor internal-build.yml confirms artifact contract"
	echo " Expected artifact: $EXPECTED_ARTIFACT"
	ERRORS=$((ERRORS + 1))
	fi
	fi

	echo ""

	# --- 2b. Validate OUR consumer-side artifact/tag shape (anchored) ---
	# The checks above validate the UPSTREAM template (internal-build.yml).
	# This block validates the EXACT, concrete artifact filename and image
	# tag that this workflow consumes — the download-artifact `name:` and
	# the `docker run ... <tag>` line — against anchored ERE regexes stored
	# in the contract file (single source of truth, also used by
	# scripts/test-quickstart-harness.sh). Anchoring rejects incompatible
	# suffixes (.zip, .tar.gz, -debug) that a bare substring `grep -q`
	# would silently accept (#2932). The contract stores the regex body
	# verbatim (raw grep\|sed extraction, no YAML unescaping).
	echo "--- consumer-side artifact/tag shape (this workflow) ---"
	ARTIFACT_NAME_REGEX=$(grep '^artifact_name_regex:' "$CONTRACT" \| sed 's/^artifact_name_regex: "$.$"$/\1/')
	IMAGE_TAG_REGEX=$(grep '^image_tag_regex:' "$CONTRACT" \| sed 's/^image_tag_regex: "$.$"$/\1/')
	WORKFLOW_SELF=".github/workflows/quickstart.yml"

	if [[ -z "$ARTIFACT_NAME_REGEX" \|\| -z "$IMAGE_TAG_REGEX" ]]; then
	echo "✗ contract missing artifact_name_regex/image_tag_regex (anchored ERE)"
	ERRORS=$((ERRORS + 1))
	else
	if grep -Eq "$ARTIFACT_NAME_REGEX" "$WORKFLOW_SELF"; then
	echo "✓ workflow downloads the exact artifact shape (anchored)"
	else
	echo "✗ workflow has no line matching the anchored artifact regex"
	echo " Regex: $ARTIFACT_NAME_REGEX"
	ERRORS=$((ERRORS + 1))
	fi
	if grep -Eq "$IMAGE_TAG_REGEX" "$WORKFLOW_SELF"; then
	echo "✓ workflow runs the exact image tag shape (anchored)"
	else
	echo "✗ workflow has no line matching the anchored image-tag regex"
	echo " Regex: $IMAGE_TAG_REGEX"
	ERRORS=$((ERRORS + 1))
	fi
	fi

	echo ""

	# --- 3. Summary ---
	if [[ $ERRORS -gt 0 ]]; then
	echo "=== FAILED: $ERRORS contract violation(s) detected ==="
	echo "The upstream stellar/quickstart workflows at SHA ${{ needs.setup.outputs.quickstart-sha }}"
	echo "no longer match our pinned contract at $CONTRACT."
	echo "Update the contract file and workflow if the upstream change is intentional."
	exit 1
	fi

	echo "=== Upstream contract validation passed (${{ needs.setup.outputs.quickstart-sha }}) ==="

	# Build the quickstart image with test: false — testing is handled locally.
	#
	# NOTE: GitHub Actions does not support dynamic refs in `uses:` for reusable
	# workflows. We call build.yml@main; the `validate-contract` job's drift
	# guard ensures the resolved SHA == main HEAD before we reach this point,
	# so the validated and executed definitions are the same revision.
	build:
	needs: [setup, validate-contract]
	uses: stellar/quickstart/.github/workflows/build.yml@main
	with:
	ref: ${{ needs.setup.outputs.quickstart-sha }}
	test: false
	images: \|
	[
	{
	"tag": "testing-with-pr",
	"inherit": "testing",
	"config": {
	"horizon_skip_protocol_version_check": true
	},
	"deps": [
	{ "name": "core", "repo": "${{ github.event.pull_request.head.repo.full_name \|\| github.repository }}", "ref": "${{ github.event.pull_request.head.sha \|\| github.sha }}" }
	]
	}
	]
	archs: '["amd64"]'

	# Run upstream Go probes locally through the timeout/retry wrapper.
	test:
	needs: [setup, build]
	runs-on: ubuntu-latest
	timeout-minutes: 45
	strategy:
	fail-fast: false
	matrix:
	include:
	# Local shards (network=local)
	- network: local
	enable: core
	probes: "test_core.go"
	- network: local
	enable: rpc
	probes: "test_stellar_rpc_up.go test_stellar_rpc_healthy.go test_friendbot.go"
	- network: local
	enable: "core,rpc,horizon"
	probes: "test_core.go test_horizon_up.go test_horizon_core_up.go test_horizon_ingesting.go test_stellar_rpc_up.go test_stellar_rpc_healthy.go test_friendbot.go"
	# local/galexie soft-degate (#3563). The pinned galexie image
	# (galexie-v26.1.0) never exports a ledger-meta partition while
	# henyey core now runs Protocol 27, so test_galexie.go's
	# waitForURL(partitionURL) polls .../ledger-meta/FFFFFFFF--0-63999/
	# and gets HTTP 404 forever (unbounded for{} poll, no internal
	# deadline) until the wrapper timeout fires (exit 124). This is an
	# upstream IMAGE-VERSION incompatibility, NOT a henyey-correctness
	# break — tracked in #3565 — so this shard mirrors the testnet
	# shard's de-gate: soft_on_timeout converts a TIMEOUT (exit
	# 124/137) into a neutral SOFT-SKIP (exit 0, grep-able marker)
	# while any genuine assertion failure (non-timeout exit 1) STILL
	# stays RED. probe_timeout: 240 + step_timeout_minutes: 25 make the
	# soft-skip fire in minutes so the PR's own CI never reproduces the
	# ~55-min hang. The soft-skip is TIMEOUT-ONLY and TEMPORARY: remove
	# it once the galexie image is bumped to a Protocol-27-capable tag
	# (deliberately OUT OF SCOPE here — bumping a still-incompatible tag
	# would re-hang CI; see #3565).
	- network: local
	enable: galexie
	probes: "test_galexie.go"
	soft_on_timeout: true
	probe_timeout: 240
	step_timeout_minutes: 25
	# Additional testnet shard.
	#
	# De-gate `main` on testnet external-liveness (#3272): the testnet
	# shard flakes on external network health (slow catchup, checkpoint
	# cadence, archive availability) — NOT henyey correctness — and a
	# stuck sync probe TIMEOUT used to red-roll the whole workflow and
	# block every PR's merge gate. So this shard:
	# * sets soft_on_timeout: true — a probe TIMEOUT (exit 124) is
	# converted to a neutral soft-skip by run-quickstart-test.sh
	# (a genuine assertion failure, non-124, still stays red), and
	# * tightens probe_timeout to 240s (= timeout_multiplier(4) * 60,
	# the upstream 4-min precedent already documented above) so a
	# stuck probe fast-fails into the soft-skip in minutes with
	# diagnostics uploaded, instead of hanging until the 45-min
	# job wall-clock cancels the run.
	# This extends the existing testnet-RPC-disabled precedent (see the
	# "Testnet shard (#1848)" note above) to the sync-probe TIMEOUT
	# outcome. Scoped to THIS shard only — local/pubnet keep the default
	# 600s hard-fail-on-timeout and carry no soft flag.
	# Budget: 4 probes × 240s + one 240s retry = 1200s (~20 min), well
	# under the job's 45-min timeout-minutes.
	#
	# DIAGNOSTIC instrumentation (#3286): this shard also sets a TIGHT,
	# testnet-only step-level fail-fast bound via step_timeout_minutes: 25
	# (applied as the "Run probes through wrapper" step's timeout-minutes
	# below). Two wrapper-level fixes (#3273, #3287) both FAILED to bound
	# the ~55-min step-8 hang, and a 45-min JOB-level wall-clock is a
	# cancel — so `if: failure()` never fires and the diagnostics step
	# never uploads. A step-level timeout-minutes kill instead marks the
	# STEP failed (distinct from a job cancel), so the upload step runs.
	# 25 min > the ~20-min healthy budget above, so a healthy run never
	# trips it. Other shards inherit the generous 360-min default (a
	# no-op under the job's own 45-min cap) and stay byte-identical.
	- network: testnet
	enable: "core,horizon"
	probes: "test_core.go test_horizon_up.go test_horizon_core_up.go test_horizon_ingesting.go"
	soft_on_timeout: true
	probe_timeout: 240
	step_timeout_minutes: 25
	# Additional pubnet shard
	# Note: upstream excludes horizon_core_up, horizon_ingesting, and
	# stellar_rpc_healthy on pubnet (see internal-test.yml conditionals).
	- network: pubnet
	enable: "core,rpc,horizon"
	probes: "test_core.go test_horizon_up.go test_stellar_rpc_up.go"
	steps:
	- uses: actions/checkout@v4

	- name: Download quickstart image
	uses: actions/download-artifact@v4
	with:
	name: image-quickstart-testing-with-pr-amd64.tar
	path: /tmp/quickstart-image

	- name: Load Docker image
	run: \|
	tarball="$(ls /tmp/quickstart-image/image 2>/dev/null \|\| ls /tmp/quickstart-image/*.tar 2>/dev/null \| head -1)"
	docker load -i "$tarball"

	- name: Checkout stellar/quickstart (tests)
	uses: actions/checkout@v4
	with:
	repository: stellar/quickstart
	ref: ${{ needs.setup.outputs.quickstart-sha }}
	path: quickstart

	- uses: actions/setup-go@v5
	with:
	go-version: 'stable'
	cache: false

	- name: Run quickstart container
	run: \|
	docker run -d --name quickstart \
	-p 8000:8000 -p 11626:11626 -p 8001:8001 \
	quickstart:testing-with-pr-amd64 \
	--${{ matrix.network }} \
	--enable ${{ matrix.enable }}

	- name: Run probes through wrapper
	# DIAGNOSTIC fail-fast bound (#3286). Testnet sets
	# matrix.step_timeout_minutes: 25 (tighter than its ~20-min healthy
	# budget) so a hang here FAILS the STEP at ~25 min instead of running
	# to the 45-min JOB cancel — a step-timeout marks the step failed, so
	# the "Upload diagnostics" step (if: failure()) actually runs and the
	# watchdog's process-tree dump is uploaded. All other shards have no
	# step_timeout_minutes set and inherit the generous 360-min default
	# (a no-op under the job's 45-min cap), so they stay byte-identical.
	timeout-minutes: ${{ matrix.step_timeout_minutes \|\| 360 }}
	env:
	NETWORK: ${{ matrix.network }}
	ENABLE: ${{ matrix.enable }}
	# Per-shard overrides (#3272). Unset matrix keys render as empty
	# strings; the loop below falls back to the default budget and omits
	# the soft flag when these are empty. Only the testnet shard sets them.
	SHARD_PROBE_TIMEOUT: ${{ matrix.probe_timeout }}
	SHARD_SOFT_ON_TIMEOUT: ${{ matrix.soft_on_timeout }}
	run: \|
	# Per-probe budget = github.run_attempt * timeout_multiplier minutes,
	# converted to seconds for GNU `timeout` (* 60). Mirrors upstream
	# internal-test.yml's `minutes = github.run_attempt * timeout_multiplier`
	# applied as each probe step's timeout-minutes (attempt 1 -> 4 min;
	# manual re-runs escalate to 8, 12, ...). github.run_attempt is constant
	# across a job's steps, so computing it once here is equivalent to
	# upstream's per-step timeout-minutes. The budget is applied per-probe;
	# a shard running N probes can use up to N * budget wall-clock, bounded
	# overall by this job's timeout-minutes: 45. The targeted single retry
	# in run-quickstart-test.sh is layered on top of this budget (#2920).
	PROBE_TIMEOUT=$(( ${{ github.run_attempt }} * timeout_multiplier * 60 ))

	# Per-shard probe-timeout override (#3272). The testnet shard fast-
	# fails at a tighter budget (SHARD_PROBE_TIMEOUT=240) so a stuck sync
	# probe is soft-skipped in minutes; all other shards leave it empty and
	# keep the upstream run_attempt * multiplier * 60 budget unchanged.
	if [[ -n "$SHARD_PROBE_TIMEOUT" ]]; then
	PROBE_TIMEOUT="$SHARD_PROBE_TIMEOUT"
	fi

	# Per-shard soft-on-timeout flag (#3272). Only the testnet shard sets
	# matrix.soft_on_timeout: true; for every other shard SHARD_SOFT_ON_TIMEOUT
	# is empty and SOFT_FLAG stays empty so the wrapper invocation is
	# byte-identical to before (no --soft-on-timeout passed).
	SOFT_FLAG=""
	if [[ "$SHARD_SOFT_ON_TIMEOUT" == "true" ]]; then
	SOFT_FLAG="--soft-on-timeout"
	fi
	DIAG_DIR="/tmp/quickstart-diagnostics"

	# ---- DIAGNOSTIC hang watchdog (#3286), testnet shard only ----
	# Capture the step shell's PID HERE, in the step body — NOT inside a
	# subshell and NOT via $BASHPID — so the backgrounded watchdog can
	# dump the fds THIS shell holds. The watchdog sleeps WATCHDOG_DELAY
	# (default 1200s = 20 min, < the 25-min step_timeout_minutes bound)
	# then writes a process-tree + open-fd snapshot to the diagnostics
	# dir, so on the next hang the dump is on disk BEFORE the step kill
	# and gets swept into the uploaded artifact. It is reaped on the
	# healthy path (trap EXIT + explicit kill after the loop) so it never
	# lingers or emits a spurious dump on a passing run. This does NOT fix
	# the hang — its deliverable is DATA from the next hang.
	STEP_PID=$$
	WATCHDOG_PID=""
	if [[ "$NETWORK" == "testnet" ]]; then
	( sleep "${WATCHDOG_DELAY:-1200}"; \
	bash scripts/ci/quickstart-hang-watchdog.sh \
	"$DIAG_DIR/testnet-hang-watchdog" "$STEP_PID" ) &
	WATCHDOG_PID=$!
	# Reap the watchdog on any step exit so a passing run never leaks
	# the backgrounded sleep.
	trap '[[ -n "$WATCHDOG_PID" ]] && kill "$WATCHDOG_PID" 2>/dev/null \|\| true' EXIT
	fi

	for probe_file in ${{ matrix.probes }}; do
	probe_name="${probe_file%.go}"
	probe_name="${probe_name#test_}"
	# Normalize underscores to hyphens so probe names match the
	# retry policy (e.g. horizon_core_up → horizon-core-up).
	probe_name="${probe_name//_/-}"

	scripts/ci/run-quickstart-test.sh \
	$SOFT_FLAG \
	--network "$NETWORK" \
	--enable "$ENABLE" \
	--probe "$probe_name" \
	--timeout "$PROBE_TIMEOUT" \
	--diagnostics-dir "$DIAG_DIR/$NETWORK-$ENABLE-$probe_name" \
	-- go run "quickstart/tests/$probe_file"
	done

	# Healthy path: probes finished, so reap the watchdog now (explicit
	# kill, belt-and-suspenders with the EXIT trap) — no lingering sleep,
	# no spurious dump on a passing run.
	if [[ -n "$WATCHDOG_PID" ]]; then
	kill "$WATCHDOG_PID" 2>/dev/null \|\| true
	fi

	- name: Upload diagnostics on failure
	# DIAGNOSTIC instrumentation (#3286): upload on BOTH a step-timeout
	# failure (the primary path — a testnet step_timeout_minutes kill marks
	# the step failed, so failure() fires) AND a residual job-level cancel
	# (belt-and-suspenders). The watchdog writes its dump before the step
	# kill, so the artifact exists for either disposition.
	if: ${{ failure() \|\| cancelled() }}
	uses: actions/upload-artifact@v4
	with:
	name: quickstart-diagnostics-${{ matrix.network }}-${{ matrix.enable }}
	path: /tmp/quickstart-diagnostics/
	retention-days: 7

	- name: Cleanup
	if: always()
	run: \|
	docker stop quickstart 2>/dev/null \|\| true
	docker rm quickstart 2>/dev/null \|\| true

	# NOTE (#3193): the previous in-run `rerun-on-transient` job was REMOVED. A job
	# inside a run cannot re-dispatch its own still-running run — the rerun-failed
	# API call returns "This workflow is already running" (exit 1), so the
	# auto-retry never fired and only added a spurious FAILURE check. Recovery from
	# a whole-runner reclamation now lives in .github/workflows/quickstart-retry.yml,
	# a separate workflow_run-triggered workflow that fires AFTER this run
	# completes (so the re-dispatch is accepted).

Navigation Menu

Uh oh!

Make counter-streak dwell time-denominated to fix 2x streak acceleration #3246

Workflow file

Make counter-streak dwell time-denominated to fix 2x streak acceleration #3246

Uh oh!

Workflow file for this run