Testing guide + coverage matrix: exercise every connector lifecycle, with a two-tier CI gate before main

[zozo123]

Summary

Crabbox has 71 registered providers but no single place that says how to test a connector's full lifecycle, no complete picture of which providers are covered, and no CI gate that exercises any lifecycle before code lands on main. This issue documents the testing method, publishes a verified coverage matrix, and proposes a two-tier CI design that stays inside the constraints from the closing guidance on #176 and #175.

Verified against current main:

• 71 providers registered in internal/providers/all/all.go, matching docs/providers/provider-metadata.json and the 71 pages under docs/providers/.
• 30 providers have a dedicated smoke runner in scripts/ (29 live-*-smoke.sh shell scripts; codesandbox uses Node runners scripts/live-codesandbox-smoke.test.js + live-codesandbox-smoke-classify.test.js).
• 43 providers are drivable through the scripts/live-smoke.sh matrix (CRABBOX_LIVE_PROVIDERS), via its generic provider_smoke() path or an inline custom function.
• Only 4 providers have hermetic lifecycle unit tests that run in go test -race ./... today: scaleway (internal/providers/scaleway/backend_lifecycle_test.go), nebius, nomad, firecracker.
• .github/workflows/ contains zero smoke or lifecycle invocations — no workflow references CRABBOX_LIVE or any live-*-smoke.sh.
• 20 providers have no lifecycle coverage of any kind (list below).

The lifecycle a connector test must exercise

From the backend contract in internal/cli/provider_backend.go:

Stage	Contract	CLI stage used by smoke scripts
Acquire / Warmup	SSHLeaseBackend.Acquire, DelegatedRunBackend.Warmup	crabbox warmup
Resolve / readiness	SSHLoginBackend.Resolve, delegated Status	crabbox status --wait
Exec + sync	DelegatedRunBackend.Run (RunRequest incl. downloads/artifact globs), SSH exec	crabbox run --shell, crabbox ssh
Touch (keepalive)	LeaseTouchBackend.Touch	exercised by status --wait
List / inventory	SSHLeaseBackend.List, JSONListBackend.ListJSON	crabbox list --json, crabbox inspect --json
Copy / ports	CopyBackend.Copy, PortsBackend.Ports	crabbox cp, crabbox ports
Snapshot / checkpoint (optional)	NativeCheckpointLifecycleProvider.*	provider-specific
Pause / Resume (optional)	PausableBackend.Pause/Resume	live-exercised only for tenki and codesandbox today
Release / Stop	SSHLeaseBackend.ReleaseLease (with ProviderIdentityExpectation validation), delegated Stop	crabbox stop
Orphan cleanup	CleanupBackend.Cleanup (DryRun supported)	crabbox cleanup --dry-run, scripts/aws-crabbox-orphan-audit.sh

How to test a connector today (the recipe this issue wants documented in docs/)

1. Hermetic lifecycle unit test — a table-driven backend_lifecycle_test.go against a fake API (pattern: internal/providers/scaleway/backend_lifecycle_test.go, e.g. TestScalewayAcquireListResolveTouchReleaseLifecycle). Runs in CI via go test -race ./.... This is the only tier that can gate merges for every provider.
2. Guarded live smoke — CRABBOX_LIVE=1 CRABBOX_LIVE_PROVIDERS=<p> CRABBOX_BIN=bin/crabbox scripts/live-smoke.sh, or the provider's dedicated scripts/live-<p>-smoke.sh. All scripts arm EXIT/RETURN traps that crabbox stop their leases, and several classify provider-capacity failures (quota_blocked / environment_blocked) separately from regressions.
3. Opt-in Go live smokes — //go:build smoke tests (islo, morph, wandb) run only with the smoke tag plus provider env gates (e.g. ISLO_API_KEY + CRABBOX_LIVE_ISLO_PAUSE_RESUME=1). These are excluded from CI by build tag and undocumented in docs/features/provider-live-smoke.md.

Coverage matrix — the gaps

20 providers with zero lifecycle coverage (no dedicated smoke runner, not in the live-smoke.sh matrix, no lifecycle unit test):

apple-machine, ascii-box, azure, azure-dynamic-sessions, cloudflare-sandbox, exe-dev, fastapi-cloud, freestyle, gcp, hostinger, hyperv, islo, mxc, opencomputer, parallels, railway, ssh, tensorlake, upstash-box, windows-sandbox

Notable: azure and gcp are majors with open cleanup-safety issues (#930, #931) yet no lifecycle test exercises their release path. ssh should be testable hermetically against 127.0.0.1 on any runner. islo has live Go tests but no shell smoke and is absent from live-smoke.sh (it needs a custom function like e2b_smoke because generic crabbox ssh requires the external islo CLI).

Only 4/71 providers have hermetic lifecycle unit tests. That is the cheapest, highest-leverage gap: those tests need no secrets and gate every PR.

Proposal: .github/workflows/connector-lifecycles.yml, two tiers

Designed to address the reasons #175 was closed (secret-bearing runners, automation risk, provider-list drift):

Tier 1 — hermetic, required check on every PR / push to main. No secrets at all.

• docker-sandbox full lifecycle (scripts/live-docker-sandbox-smoke.sh — needs no env), docker-sandbox trust boundary, local-container via live-smoke.sh, firecracker readiness (read-only doctor contract), ssh-to-localhost lifecycle (new small script), apple-vm on macos-15.
• Lives as a matrix job; can sit inside ci.yml so dependabot-auto-merge.yml (which keys off the CI workflow conclusion) inherits it without changes.

Tier 2 — live cloud matrix: schedule (nightly) + workflow_dispatch only. Never runs on pull_request, so fork PRs can never reach secrets.

• Secrets scoped to a dedicated live-smoke GitHub environment (required-reviewer gating for dispatches comes free — same pattern coordinator-deploy.yml uses for coordinator).
• Green-skip rows whose secret is unset (the coordinator-deploy.yml probe pattern), so partially provisioned secrets never red-X the nightly.
• fail-fast: false, max-parallel: 4 (spend/rate cap), timeout-minutes: 30, per-provider concurrency group with cancel-in-progress: false — cancelling mid-provision is how orphans happen.
• Treat quota_blocked / environment_blocked classifications as warnings, not failures.
• Trailing orphan-audit job (if: always()) running scripts/aws-crabbox-orphan-audit.sh read-only (its --terminate fails closed by design) that fails when crabbox-tagged instances outlive their leases.
• No second provider list: generate the live matrix from docs/providers/provider-metadata.json (extend scripts/generate-provider-matrix.mjs) or reuse CRABBOX_LIVE_PROVIDERS defaults, so the workflow cannot drift from the registry — this was an explicit concern on Proposal: add live provider E2E GitHub Actions matrix #176.
• All uses: SHA-pinned (enforced by scripts/workflow_action_pins.go via the CI Scripts job) and a scripts/connector-lifecycles-workflow.test.js shape test asserting: no secrets in tier 1, no pull_request trigger on tier 2, continue-on-error on live rows.

Secret names match what the scripts already read from env: AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, HCLOUD_TOKEN, DIGITALOCEAN_TOKEN, LINODE_TOKEN, VULTR_API_KEY, SCW_*, TENCENTCLOUD_*, LAMBDA_API_KEY, RUNPOD_API_KEY, VAST_API_KEY (+ CRABBOX_LIVE_VAST_MAX_DPH_TOTAL price cap), BL_API_KEY/BL_WORKSPACE, VERCEL_TOKEN, SMOLMACHINES_API_KEY, SUPERSERVE_API_KEY, OPEN_SANDBOX_API_KEY, CROWNEST_API_KEY, WANDB_API_KEY/WANDB_ENTITY_NAME. Self-hosted-infra providers (nomad, proxmox, xcp-ng, kubevirt, agent-sandbox, phala) stay out of hosted runners; the hydrate.yml self-hosted-label pattern covers them later.

Task checklist

• Add a docs/ testing guide capturing the lifecycle table and the three-tier recipe above (candidate: extend docs/features/provider-live-smoke.md).
• Document the //go:build smoke live-test env vars (ISLO_API_KEY, CRABBOX_LIVE_ISLO_PAUSE_RESUME, CRABBOX_LIVE_ISLO_IMAGE, morph/wandb equivalents) in that guide.
• Tier 1 workflow (hermetic matrix) + workflow shape test; wire as required check.
• scripts/live-ssh-localhost-smoke.sh (hermetic, mirrors live-docker-sandbox-smoke.sh).
• Hermetic backend_lifecycle_test.go for high-traffic providers missing one (start: aws, azure, gcp, e2b, daytona, islo, railway) — pattern from internal/providers/scaleway/backend_lifecycle_test.go; long-term goal: all 71.
• islo smoke coverage: islo_smoke() function in scripts/live-smoke.sh (warmup → status --wait → run → pause → resume → stop; skip generic crabbox ssh).
• Dedicated or matrix smoke coverage for the remaining zero-coverage providers listed above.
• Tier 2 live workflow behind the live-smoke environment — only after maintainers decide the repository secret/provider policy, per the closing guidance on Proposal: add live provider E2E GitHub Actions matrix #176; dispatch it once and attach the run link as live proof.

I can send tier 1 (hermetic gate + docs guide + shape test) as a PR first since it needs no secret policy decision, and hold tier 2 until the policy call is made.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed July 5, 2026, 11:39 PM ET / 03:39 UTC.

Summary
Keep open: current main has provider-live-smoke guidance and opt-in smoke scripts, but it does not have the requested lifecycle coverage matrix or two-tier lifecycle CI gate, and the related secret-bearing workflow proposal was closed pending maintainer policy.

Reproducibility: not applicable. this is a CI/docs coverage proposal, not a runtime bug. Source inspection confirms current main lacks the requested lifecycle workflow gate and complete coverage matrix.

Ways to help us reproduce this

• Add a screenshot or short recording showing the behavior.
• Add expected vs actual behavior.

Next step
Maintainers need to scope the no-secret tier separately from the policy-sensitive live cloud tier before this becomes safe repair-lane work.

Maintainer decision needed

• Question: Should Crabbox split this into a no-secret tier-1 lifecycle guide/gate now and defer the secret-backed live cloud tier until repository secret/provider policy is decided?
• Rationale: Current main lacks the requested coverage/gate, but the live workflow portion repeats a policy-sensitive direction that maintainers already closed pending secret and runner decisions.
• Likely owner: vincentkoc — vincentkoc closed the related live workflow PR and has recent history on provider live-smoke coverage, making them the best visible owner for the policy split.
• Options:
  • Split tier 1 first (recommended): Accept the no-secret guide, generated matrix, hermetic smoke, and shape-test work as the next scoped direction while keeping live cloud CI out until policy is settled.
  • Keep one umbrella item: Use this issue as the long-running tracker for docs, hermetic tests, and live workflow policy, but require human scoping before any PR is queued.
  • Defer all CI workflow work: Pause both tiers until maintainers decide the broader provider CI and secret policy, leaving only ad hoc live-smoke documentation in place.

Review details

Best possible solution:

Split the work: land a maintainer-approved no-secret lifecycle guide, generated coverage matrix, and hermetic tier-1 gate first, then handle the secret-backed live cloud workflow in a separate policy-backed maintainer PR.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is a CI/docs coverage proposal, not a runtime bug. Source inspection confirms current main lacks the requested lifecycle workflow gate and complete coverage matrix.

Is this the best way to solve the issue?

Unclear as one bundle: the no-secret guide and hermetic tier are aligned and maintainable, but the live cloud tier should wait for maintainer secret/provider policy.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• The secret-backed live tier still needs explicit maintainer policy for repository secrets, trusted runners, spend limits, and live dispatch proof.
• The requested all-provider lifecycle coverage is broad enough that a single automated repair PR would likely mix docs, CI policy, scripts, and many provider tests.
• The coverage matrix should be generated from provider metadata or another source of truth so it does not drift as providers are added.

Codex review notes: model internal, reasoning high; reviewed against 7e1da0a52260.

Label changes

Label changes:

• add P3: This is a maintainer CI/docs coverage improvement rather than an urgent shipped-user regression.
• add impact:other: The issue affects maintainer release confidence and provider lifecycle validation coverage, which is meaningful but outside the specific runtime impact labels.
• add issue-rating: 🌊 off-meta tidepool: Current issue advisory state selects this label.
• add clawsweeper:no-new-fix-pr: Current issue advisory state selects this label.
• add clawsweeper:needs-maintainer-review: Current issue advisory state selects this label.
• add clawsweeper:needs-product-decision: Current issue advisory state selects this label.

Label justifications:

• P3: This is a maintainer CI/docs coverage improvement rather than an urgent shipped-user regression.
• impact:other: The issue affects maintainer release confidence and provider lifecycle validation coverage, which is meaningful but outside the specific runtime impact labels.

Evidence reviewed

What I checked:

• Repository policy read: AGENTS.md was read fully; its provider-neutral architecture, test guidance, and secret-handling rules are relevant to this CI/provider lifecycle proposal. (AGENTS.md:9, 7e1da0a52260)
• Vision fit: VISION.md says core remains provider-neutral, provider adapters own lifecycle rules, and funded or remote providers need real create/use/destroy proof with zero residue before merge. (VISION.md:3, 7e1da0a52260)
• Current provider surface: Current main has 71 provider imports, 71 provider docs pages, and 71 provider metadata entries, matching the issue's broad provider-coverage scope. (internal/providers/all/all.go:4, 7e1da0a52260)
• Existing guide is narrower: The current Provider Live Smoke guide explains opt-in live proof and smoke contracts, but it does not publish the requested complete lifecycle coverage matrix or CI gate design. (docs/features/provider-live-smoke.md:10, 7e1da0a52260)
• Existing smoke script is opt-in: scripts/live-smoke.sh is gated by CRABBOX_LIVE and CRABBOX_LIVE_PROVIDERS, with provider_smoke covering a lease lifecycle, but it is not a required CI workflow. (scripts/live-smoke.sh:4, 7e1da0a52260)
• No lifecycle workflow on main: Searching .github/workflows found no CRABBOX_LIVE, live-smoke, connector-lifecycles, or provider-live-smoke workflow references. (7e1da0a52260)

Likely related people:

• vincentkoc: Vincent Koc closed the related workflow PR with policy guidance and appears in history defining provider live-smoke contracts and adding live-smoke dispatch coverage. (role: prior workflow reviewer and live-smoke area contributor; confidence: high; commits: 47736cf94dfa, c6dd7cb474e7, 0ec69d642764; files: docs/features/provider-live-smoke.md, scripts/live-smoke.sh, .github/workflows)
• steipete: Current workflow files, provider registry, and provider-live-smoke docs blame to Peter Steinberger in the current release baseline, and steipete gave the closing guidance on the older workflow proposal. (role: recent workflow and provider baseline contributor; confidence: high; commits: 4ea85a4f1b7e, 7e1da0a52260, 0301236b2752; files: .github/workflows/ci.yml, .github/workflows/hydrate.yml, docs/features/provider-live-smoke.md)
• zozo123: zozo123 opened this issue and the earlier workflow PR, and git history shows recent provider work that overlaps the requested coverage targets. (role: proposal contributor and recent provider contributor; confidence: medium; commits: e079b3dde742, 1609f8da54ab, 973efedd2c87; files: internal/providers/fastapicloud, internal/providers/islo, internal/providers/windowssandbox)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[steipete]

The safe tier-one slice landed in #948 as 5b39acb1d8d0530824f1acc25d73b94bc9a0c773.

It adds a source-discovered, no-secret lifecycle CI gate plus a generated 71-provider proof matrix. The issue remains open for tier two: authenticated create/use/destroy workflows, credential governance, spend ceilings, cancellation, and orphan cleanup.