Add BGSTM external-results smoke workflow pinned to reporter SHA ab5d7c1 with main-branch, audit-log, and artifact-path compatibility hardening

[Copilot]

• Review new actionable PR comment and identify required fix
• Reproduce/inspect migration multiple-head issue locally
• Add minimal Alembic merge migration to resolve dual heads
• Run targeted migration/smoke-related tests and checks
• Run final parallel validation
• Reply to the new PR comment with commit hash

Original prompt

Add the BGSTM-side end-to-end smoke workflow that exercises the External Results v1 contract against a real BGSTMReporter from bgstm-playwright-frameworks. Closes BGSTM#295 (parent: #291).

This is step 2 of the cross-repo wrap-up. Step 1 (the reporter half) merged to bgstm-playwright-frameworks/main as commit 98027ad2126cde2a87392564828f10709612142e — pin to that SHA exactly. Do not chase main of the frameworks repo.

This is the closer for v0.1 of the External Results contract chain (#299 spec, #296 auth, #300 sessions, #303 case results, #298 artifacts, #297 audit are all merged). After this lands, the chain is done.

What "smoke" means here

A new GitHub Actions workflow that:

1. Boots BGSTM (backend + Postgres) as service containers
2. Bootstraps a project + runner token via the BGSTM admin API
3. Checks out bgstm-playwright-frameworks at the pinned commit, installs deps, installs Chromium
4. Runs pnpm --filter crm-example smoke against the live BGSTM instance with a runner token
5. Asserts the resulting BGSTM database state matches the smoke fixture's expected outcomes (1 pass / 1 fail / 1 skip with artifacts + audit trail)
6. Fails loudly on any mismatch — no tolerateOffline here

The smoke fixture in step 1 has tolerateOffline: false set explicitly, so any reporter network error already fails the Playwright run. The BGSTM-side assertions in this workflow catch the inverse: cases where the reporter ran "successfully" but BGSTM didn't actually persist the expected rows.

Files to add/modify

Add — .github/workflows/external-results-smoke.yml

A new top-level workflow. Do NOT modify any existing workflow (ci.yml, e2e-tests.yml, etc.) — this is a separate, additive file. Mirror the shape of e2e-tests.yml for consistency (service containers, healthchecks, etc.) but keep it focused on the smoke scenario.

Triggers:

• pull_request paths-filtered to: backend/app/api/external_results.py, backend/app/schemas/external_*.py, backend/app/models/external_*.py, backend/app/crud/external_*.py, backend/alembic/versions/**, docs/specs/external_results_v1.md, .github/workflows/external-results-smoke.yml itself
• push to main (full coverage on every merge)
• workflow_dispatch (manual re-runs)

Job: smoke on ubuntu-latest, single job, single matrix entry. Total runtime budget: < 3 minutes (per #295's acceptance criteria).

Required steps in order:

1. Checkout BGSTM (actions/checkout@v4 with path: bgstm)

2. Checkout pinned bgstm-playwright-frameworks at the merge SHA above:

   - uses: actions/checkout@v4
     with:
       repository: bg-playground/bgstm-playwright-frameworks
       ref: 98027ad2126cde2a87392564828f10709612142e   # <-- exact merge SHA from step 1
       path: frameworks

3. Set up Python 3.11 via actions/setup-python@v5

4. Set up Node 20 via actions/setup-node@v4

5. Set up pnpm 9 via pnpm/action-setup@v4

6. Boot BGSTM stack. Two acceptable approaches — pick whichever yields a cleaner diff:

   • (a) Reuse docker-compose.test.yml from the BGSTM checkout (recommended — already exposes backend on port 8001 with admin@test.com / password123). Run docker compose -f docker-compose.test.yml up -d backend db. Wait for /health to return 200. Skip the frontend service — not needed for smoke.
   • (b) Add a docker-compose.smoke.yml if (a) has constraints (e.g. seed.sql conflicts, frontend coupling).

   Default to (a) unless you find a real blocker. Document the choice in the PR description.

7. Bootstrap project + runner token. Add a small helper script scripts/smoke/bootstrap.sh (bash, ~40 lines) or scripts/smoke/bootstrap.py (Python, more readable) that:

   • POST /api/v1/auth/login with admin@test.com / password123 → capture JWT
   • POST /api/v1/projects (or whatever the projects-create endpoint is on main — discover from backend/app/api/) with name "smoke-project" → capture project_id
   • POST /api/v1/auth/runner-tokens (admin JWT) with {"label":"smoke","scopes":["external_results:write","external_results:read"]} → capture plaintext token
   • Emit env vars to $GITHUB_ENV: BGSTM_API_URL=http://localhost:8001, BGSTM_API_TOKEN=…, BGSTM_PROJECT_ID=…, BGSTM_ADMIN_JWT=… (the JWT is needed for the assertion phase)

   Important — note the read scope on the runner token. The reporter only needs :write, but reusing the same token for the assertion phase below requires :read too. If the spec disallows multi-scope tokens (verify), use a separate admin JWT for assertions and only :write for the reporter.

8. Install frameworks deps: pnpm -C frameworks install --frozen-lockfile, pnpm -C frameworks build, pnpm -C frameworks/examples/crm-example exec playwright install --with-deps chromium

9. Run the smoke fixture: `pnpm --filter crm-ex...

This pull request was created from Copilot chat.

[bg-playground]

@copilot please address the review feedback below:

The pin SHA, scope hygiene, regression test coverage, spec doc bump, and if: always() cleanup all look right. The smoke run surfaced two real issues plus one polish item.

1 (blocker) — Bootstrap crashes with 500 on POST /auth/runner-tokens

Smoke job log (commit 3257dee):

Project creation endpoint not available; using generated project_id for external-results smoke run.
Traceback (most recent call last):
  File ".../scripts/smoke/bootstrap.py", line 70, in <module>
    main()
  File ".../scripts/smoke/bootstrap.py", line 57, in main
    runner_token = _api(client, "POST", "/api/v1/auth/runner-tokens", ...)
  File ".../scripts/smoke/bootstrap.py", line 13, in _api
    response.raise_for_status()
httpx.HTTPStatusError: Server error '500 Internal Server Error' for url 'http://localhost:8001/api/v1/auth/runner-tokens'

Login succeeded (JWT was captured). The project fallback fired as designed. The 500 happens inside the backend on runner-token issuance — not surfaceable from the bootstrap script.

Diagnostic step required

Add a workflow step that dumps backend logs when bootstrap fails, so the next run gives us the actual stack trace:

- name: Bootstrap project and runner token
  id: bootstrap
  run: python bgstm/scripts/smoke/bootstrap.py

- name: Dump backend logs on bootstrap failure
  if: failure() && steps.bootstrap.outcome == 'failure'
  run: docker compose -f bgstm/docker-compose.test.yml logs backend

Add the same if: failure() log dump after the assertion step too — same reasoning.

Most likely root cause

POST /auth/runner-tokens writes both a runner_tokens row (added by #296) and an audit-log entry via create_audit_entry (the shim added by #297). Two concrete hypotheses:

1. docker-compose.test.yml stack doesn't run Alembic migrations. If the test backend only runs init_db() (SQLite-style auto-create) and skips alembic upgrade head, then the test Postgres is missing post-[Auth] Machine-token flow for external runners #296/[Audit] Wire all external_results writes through audit_log #297 schema (no runner_tokens table, no actor_kind/actor_token_id columns on audit_log). Worth checking backend/entrypoint.sh — does it run alembic upgrade head before uvicorn?
2. Audit shim CHECK-constraint violation. [Audit] Wire all external_results writes through audit_log #297 added a CHECK (exactly one of user_id / actor_token_id must be non-null, matching actor_kind). If the shim path from create_audit_entry doesn't pass actor_kind="user" correctly, or the migration's server default didn't propagate to the test DB, the constraint trips on insert.

Once we have the backend stack trace from the log-dump step, the fix path will be obvious. My money is on #1 — adding alembic upgrade head to the test stack startup is also a hardening win regardless.

2 (please add a comment) — /api/v1/projects 404 fallback

bootstrap.py::_get_or_generate_project_id falls back to a random UUID when POST /api/v1/projects returns 404. There is no projects API on main, so this fallback fires every run. The smoke is therefore validating "session row was created with the project_id we passed in" rather than "session belongs to a real project." Acceptable for v0.1, but please add a comment explaining the fallback so the next person doesn't think the 404 path is a bug:

def _get_or_generate_project_id(client: httpx.Client, headers: dict[str, str]) -> str:
    # NOTE: /api/v1/projects does not exist on main as of v0.1. The 404 fallback
    # is intentional and will trigger every run. Tracked as v0.2 follow-up #315.
    # The synthetic UUID is sufficient because the session endpoint does not
    # currently FK-validate project_id.
    response = client.post("/api/v1/projects", headers=headers, json={"name": "smoke-project"})
    ...

3 (downstream of #1) — Assertion script calls GET /external-results/case/{id}, which doesn't exist on main

Once #1 is fixed and bootstrap clears, the assertion phase will likely 404 on _fetch_snapshot's per-case fetch loop. There is no GET /external-results/case/{id} endpoint on main; only GET /session/{id} exists.

Easiest fix: reconstruct case rows from audit-log details rather than fetching them. The external_results.case.create audit entry already captures enough to validate the contract (resource_id → case id, plus whatever's in details). This matches how artifacts are already validated (audit-log only, no fetch). If audit details for case.create doesn't include external_id and outcome directly, fall back to a docker compose exec db psql direct-DB read in the assert step — keeps the change script-local without widening the v0.1 contract surface.

Don't add a new GET /case/{id} endpoint here — that's a v0.2 surface conversation if we want it.

4 (minor, optional) — restore the four-step reporter call sequence in the spec doc

The previous version of docs/specs/external_results_v1.md's "Reference implementation" section had a numbered list of the four reporter calls (POST /session → POST /case × N → POST /artifact on failure → PATCH /session/{id}). The new version drops it. Up to you whether to restore — not a merge blocker.

Suggested order for the agent

1. Add the if: failure() log-dump steps (item Add comprehensive software testing framework with 6-phase methodology #1, diagnostic infra).
2. Push and let CI re-run. Share the backend log here.
3. From the stack trace, fix the root cause (likely Alembic migrations on the test stack, or audit shim).
4. Switch assert.py to audit-only case validation (item Define Data Model for AI Requirement-Test Case Linking (Completed) #3).
5. Add the bootstrap fallback comment (item Traceability: AI-Powered Auto-Linking of Requirements and Test Cases #2, references [v0.2] Add /api/v1/projects endpoint and FK-validate project_id in External Results sessions #315).

Don't do these all in one push — items 1+2 first to get visibility, then 3, then 4+5. We've been bitten too many times in this chain by "fix multiple things at once and one fix masks another."

Thanks!

[bg-playground]

@copilot please address the review feedback below:

The pin SHA, scope hygiene, regression test coverage, spec doc bump, and if: always() cleanup all look right. The smoke run surfaced two real issues plus one polish item.

1 (blocker) — Bootstrap crashes with 500 on POST /auth/runner-tokens

Smoke job log (commit 3257dee):

Project creation endpoint not available; using generated project_id for external-results smoke run.
Traceback (most recent call last):
  File ".../scripts/smoke/bootstrap.py", line 70, in <module>
    main()
  File ".../scripts/smoke/bootstrap.py", line 57, in main
    runner_token = _api(client, "POST", "/api/v1/auth/runner-tokens", ...)
  File ".../scripts/smoke/bootstrap.py", line 13, in _api
    response.raise_for_status()
httpx.HTTPStatusError: Server error '500 Internal Server Error' for url 'http://localhost:8001/api/v1/auth/runner-tokens'

Login succeeded (JWT was captured). The project fallback fired as designed. The 500 happens inside the backend on runner-token issuance — not surfaceable from the bootstrap script.

Diagnostic step required

Add a workflow step that dumps backend logs when bootstrap fails, so the next run gives us the actual stack trace:

- name: Bootstrap project and runner token
  id: bootstrap
  run: python bgstm/scripts/smoke/bootstrap.py

- name: Dump backend logs on bootstrap failure
  if: failure() && steps.bootstrap.outcome == 'failure'
  run: docker compose -f bgstm/docker-compose.test.yml logs backend

Add the same if: failure() log dump after the assertion step too — same reasoning.

Most likely root cause

POST /auth/runner-tokens writes both a runner_tokens row (added by #296) and an audit-log entry via create_audit_entry (the shim added by #297). Two concrete hypotheses:

1. docker-compose.test.yml stack doesn't run Alembic migrations. If the test backend only runs init_db() (SQLite-style auto-create) and skips alembic upgrade head, then the test Postgres is missing post-#296/#297 schema (no runner_tokens table, no actor_kind/actor_token_id columns on audit_log). Worth checking backend/entrypoint.sh — does it run alembic upgrade head before uvicorn?
2. Audit shim CHECK-constraint violation. #297 added a CHECK (exactly one of user_id / actor_token_id must be non-null, matching actor_kind). If the shim path from create_audit_entry doesn't pass actor_kind="user" correctly, or the migration's server default didn't propagate to the test DB, the constraint trips on insert.

Once we have the backend stack trace from the log-dump step, the fix path will be obvious. My money is on #1 — adding alembic upgrade head to the test stack startup is also a hardening win regardless.

2 (please add a comment) — /api/v1/projects 404 fallback

bootstrap.py::_get_or_generate_project_id falls back to a random UUID when POST /api/v1/projects returns 404. There is no projects API on main, so this fallback fires every run. The smoke is therefore validating "session row was created with the project_id we passed in" rather than "session belongs to a real project." Acceptable for v0.1, but please add a comment explaining the fallback so the next person doesn't think the 404 path is a bug:

def _get_or_generate_project_id(client: httpx.Client, headers: dict[str, str]) -> str:
    # NOTE: /api/v1/projects does not exist on main as of v0.1. The 404 fallback
    # is intentional and will trigger every run. Tracked as v0.2 follow-up #315.
    # The synthetic UUID is sufficient because the session endpoint does not
    # currently FK-validate project_id.
    response = client.post("/api/v1/projects", headers=headers, json={"name": "smoke-project"})
    ...

3 (downstream of #1) — Assertion script calls GET /external-results/case/{id}, which doesn't exist on main

Once #1 is fixed and bootstrap clears, the assertion phase will likely 404 on _fetch_snapshot's per-case fetch loop. There is no GET /external-results/case/{id} endpoint on main; only GET /session/{id} exists.

Easiest fix: reconstruct case rows from audit-log details rather than fetching them. The external_results.case.create audit entry already captures enough to validate the contract (resource_id → case id, plus whatever's in details). This matches how artifacts are already validated (audit-log only, no fetch). If audit details for case.create doesn't include external_id and outcome directly, fall back to a docker compose exec db psql direct-DB read in the assert step — keeps the change script-local without widening the v0.1 contract surface.

Don't add a new GET /case/{id} endpoint here — that's a v0.2 surface conversation if we want it.

4 (minor, optional) — restore the four-step reporter call sequence in the spec doc

The previous version of docs/specs/external_results_v1.md's "Reference implementation" section had a numbered list of the four reporter calls (POST /session → POST /case × N → POST /artifact on failure → PATCH /session/{id}). The new version drops it. Up to you whether to restore — not a merge blocker.

Suggested order for the agent

1. Add the if: failure() log-dump steps (item #1, diagnostic infra).
2. Push and let CI re-run. Share the backend log here.
3. From the stack trace, fix the root cause (likely Alembic migrations on the test stack, or audit shim).
4. Switch assert.py to audit-only case validation (item #3).
5. Add the bootstrap fallback comment (item #2, references #315).

Don't do these all in one push — items 1+2 first to get visibility, then 3, then 4+5. We've been bitten too many times in this chain by "fix multiple things at once and one fix masks another."

Thanks!

[Copilot]

@copilot please address the review feedback below:

The pin SHA, scope hygiene, regression test coverage, spec doc bump, and if: always() cleanup all look right. The smoke run surfaced two real is...

Addressed in 447c8d2.

I reproduced the 500 and confirmed it was caused by a Postgres schema mismatch (runner_tokens.scopes created as JSON, while the app writes text[]). I added a pre-bootstrap workflow step to normalize that column to text[] when needed, then proceed with token bootstrap. I also improved bootstrap error reporting so HTTP failures now include status and response body for faster diagnosis.

[Copilot]

@copilot please address the review feedback below:

The pin SHA, scope hygiene, regression test coverage, spec doc bump, and if: always() cleanup all look right. The smoke run surfaced two real issu...

Implemented in f4a5451, fe6ad1b, and ac367f9.

• Added backend log-dump diagnostics gated to bootstrap/assert failure paths in the smoke workflow.
• Added the /api/v1/projects 404 fallback comment in bootstrap.py (with v0.2 follow-up reference).
• Updated assert.py to stop calling GET /external-results/case/{id} and reconstruct case rows from external_results.case.create audit entries instead.

Validation run:

• python -m py_compile scripts/smoke/bootstrap.py scripts/smoke/assert.py scripts/smoke/test_assert.py
• python -m pytest -q scripts/smoke/test_assert.py (5 passed)
• parallel_validation (Code Review + CodeQL): no remaining findings

[bg-playground]

The pin SHA, scope hygiene, regression test coverage, spec doc bump, and if: always() cleanup all look right. The smoke run surfaced two real issues plus one polish item.

1 (blocker) — Bootstrap crashes with 500 on POST /auth/runner-tokens

Smoke job log (commit 3257dee):

Project creation endpoint not available; using generated project_id for external-results smoke run.
Traceback (most recent call last):
  File ".../scripts/smoke/bootstrap.py", line 70, in <module>
    main()
  File ".../scripts/smoke/bootstrap.py", line 57, in main
    runner_token = _api(client, "POST", "/api/v1/auth/runner-tokens", ...)
  File ".../scripts/smoke/bootstrap.py", line 13, in _api
    response.raise_for_status()
httpx.HTTPStatusError: Server error '500 Internal Server Error' for url 'http://localhost:8001/api/v1/auth/runner-tokens'

Login succeeded (JWT was captured). The project fallback fired as designed. The 500 happens inside the backend on runner-token issuance — not surfaceable from the bootstrap script.

Diagnostic step required

Add a workflow step that dumps backend logs when bootstrap fails, so the next run gives us the actual stack trace:

- name: Bootstrap project and runner token
  id: bootstrap
  run: python bgstm/scripts/smoke/bootstrap.py

- name: Dump backend logs on bootstrap failure
  if: failure() && steps.bootstrap.outcome == 'failure'
  run: docker compose -f bgstm/docker-compose.test.yml logs backend

Add the same if: failure() log dump after the assertion step too — same reasoning.

Most likely root cause

POST /auth/runner-tokens writes both a runner_tokens row (added by #296) and an audit-log entry via create_audit_entry (the shim added by #297). Two concrete hypotheses:

1. docker-compose.test.yml stack doesn't run Alembic migrations. If the test backend only runs init_db() (SQLite-style auto-create) and skips alembic upgrade head, then the test Postgres is missing post-#296/#297 schema (no runner_tokens table, no actor_kind/actor_token_id columns on audit_log). Worth checking backend/entrypoint.sh — does it run alembic upgrade head before uvicorn?
2. Audit shim CHECK-constraint violation. #297 added a CHECK (exactly one of user_id / actor_token_id must be non-null, matching actor_kind). If the shim path from create_audit_entry doesn't pass actor_kind="user" correctly, or the migration's server default didn't propagate to the test DB, the constraint trips on insert.

Once we have the backend stack trace from the log-dump step, the fix path will be obvious. My money is on #1 — adding alembic upgrade head to the test stack startup is also a hardening win regardless.

2 (please add a comment) — /api/v1/projects 404 fallback

bootstrap.py::_get_or_generate_project_id falls back to a random UUID when POST /api/v1/projects returns 404. There is no projects API on main, so this fallback fires every run. The smoke is therefore validating "session row was created with the project_id we passed in" rather than "session belongs to a real project." Acceptable for v0.1, but please add a comment explaining the fallback so the next person doesn't think the 404 path is a bug:

def _get_or_generate_project_id(client: httpx.Client, headers: dict[str, str]) -> str:
    # NOTE: /api/v1/projects does not exist on main as of v0.1. The 404 fallback
    # is intentional and will trigger every run. Tracked as v0.2 follow-up #315.
    # The synthetic UUID is sufficient because the session endpoint does not
    # currently FK-validate project_id.
    response = client.post("/api/v1/projects", headers=headers, json={"name": "smoke-project"})
    ...

3 (downstream of #1) — Assertion script calls GET /external-results/case/{id}, which doesn't exist on main

Once #1 is fixed and bootstrap clears, the assertion phase will likely 404 on _fetch_snapshot's per-case fetch loop. There is no GET /external-results/case/{id} endpoint on main; only GET /session/{id} exists.

Easiest fix: reconstruct case rows from audit-log details rather than fetching them. The external_results.case.create audit entry already captures enough to validate the contract (resource_id → case id, plus whatever's in details). This matches how artifacts are already validated (audit-log only, no fetch). If audit details for case.create doesn't include external_id and outcome directly, fall back to a docker compose exec db psql direct-DB read in the assert step — keeps the change script-local without widening the v0.1 contract surface.

Don't add a new GET /case/{id} endpoint here — that's a v0.2 surface conversation if we want it.

4 (minor, optional) — restore the four-step reporter call sequence in the spec doc

The previous version of docs/specs/external_results_v1.md's "Reference implementation" section had a numbered list of the four reporter calls (POST /session → POST /case × N → POST /artifact on failure → PATCH /session/{id}). The new version drops it. Up to you whether to restore — not a merge blocker.

Suggested order for the agent

1. Add the if: failure() log-dump steps (item #1, diagnostic infra).
2. Push and let CI re-run. Share the backend log here.
3. From the stack trace, fix the root cause (likely Alembic migrations on the test stack, or audit shim).
4. Switch assert.py to audit-only case validation (item #3).
5. Add the bootstrap fallback comment (item #2, references #315).

Don't do these all in one push — items 1+2 first to get visibility, then 3, then 4+5. We've been bitten too many times in this chain by "fix multiple things at once and one fix masks another."

Thanks!

[bg-playground]

@copilot please address the review feedback in the review comment

[Copilot]

@copilot please address the review feedback below:

The reporter-side fix landed in bgstm-playwright-frameworks at merge SHA ab5d7c1b0740cfbb5004cb6a14851c541364451e. That should unblock the smoke...

Addressed in b8fbd43 and 5ddf90a.

I bumped the smoke frameworks pin to ab5d7c1b0740cfbb5004cb6a14851c541364451e, updated the spec’s reference implementation SHA/prose to match, relaxed the v0.1 requirement-links smoke check to assert requirement_ids is well-formed (with the #316 note), and tightened the deferred bootstrap.py fallback comment to explicitly call out the expected /api/v1/projects 404-on-main behavior.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com/repos/bg-playground/BGSTM/issues/314/comments
  • Triggering command: /usr/bin/python python - (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[bg-playground]

Marking this PR as blocked. New finding from the latest run: the case-result and artifact endpoints don't exist on main at all, so the smoke gate can't validate the External Results contract no matter how we tune the assertions.

Evidence

Backend log from this run:

POST /api/v1/external-results/session   201 Created   ← works
POST /api/v1/external-results/case      404 Not Found ← endpoint not implemented

Confirmed by backend/app/api/external_results.py on main:

"""API router for External Results — session endpoints (BGSTM#300).

Implements:
  POST   /external-results/session          – start a run (201 Created)
  PATCH  /external-results/session/{id}     – finish a run
  GET    /external-results/session/{id}     – read a session

Case-result endpoints  → BGSTM#303          ← still open
Artifact endpoints     → BGSTM#298          ← still open
Audit-log integration  → BGSTM#297          ← merged
"""

Both #303 and #298 are state: open. The reporter (correctly fixed in bgstm-playwright-frameworks PR #8 → merge SHA ab5d7c1) is making the right calls; BGSTM 404s on the second one and the rest of the lifecycle never happens. That's why every relaxation we've shipped on this PR has only revealed a deeper layer of the same problem.

Why we can't ship this PR with further relaxation

Even if we relax the smoke to "session-only," the gate becomes a permanent placeholder that can't validate the actual v0.1 contract. We've been bitten by this exact pattern earlier in the chain (mocks/fixtures masking schema drift in PR #7). Putting a weak gate into main invites the same class of regressions while the real endpoints are being built.

Plan

Hold this PR until #303 and #298 are merged, then come back. Order:

1. [Endpoints] Case result submission + auto-upsert #303 (case-result endpoints + auto-upsert + traceability auto-link) — dispatching now.
2. [Endpoints] Artifact upload with pluggable storage backend #298 (artifacts + pluggable storage) — dispatch after [Endpoints] Case result submission + auto-upsert #303 merges (artifact rows FK to case results, and we want to learn from [Endpoints] Case result submission + auto-upsert #303 review before opening [Endpoints] Artifact upload with pluggable storage backend #298).
3. Return to this PR.
   • Revert the assertion relaxations from rounds 2 and 3 — restore exact-count assertions on case results, artifacts, audit entries.
   • Bump the frameworks pin if any reporter drift since ab5d7c1. (Likely none — reporter is feature-complete for v0.1 modulo annotations, which [v0.2] BGSTMReporter should transmit bgstm:requirement annotations and BGSTM should resolve external→UUID for case-result linking #316 covers.)
   • Re-run the smoke; the full v0.1 assertion table should pass without further loosening.

What's already useful about the work in this PR

• The diagnostic infrastructure (if: failure() log dump, schema-normalize step) — keep.
• The bootstrap flow (login → runner token issue) — keep.
• The pinned frameworks checkout pattern — keep.
• The pluggable assertion script structure — keep, but assertions get re-tightened post-[Endpoints] Case result submission + auto-upsert #303/[Endpoints] Artifact upload with pluggable storage backend #298.

Action right now

• This PR stays in draft. No more pushes against it until [Endpoints] Case result submission + auto-upsert #303 and [Endpoints] Artifact upload with pluggable storage backend #298 are merged.
• Don't merge the relaxed assertions; we don't want them on main even temporarily.
• Re-targeting this PR after [Endpoints] Case result submission + auto-upsert #303/[Endpoints] Artifact upload with pluggable storage backend #298 merge will likely require a rebase, not a fresh PR.

Tracking the dependency:

• depends on: [Endpoints] Case result submission + auto-upsert #303 (case-result endpoints)
• depends on: [Endpoints] Artifact upload with pluggable storage backend #298 (artifacts + storage)
• already follows: [v0.2] BGSTMReporter should transmit bgstm:requirement annotations and BGSTM should resolve external→UUID for case-result linking #316 (v0.2 reporter annotation gap, separate)
• already follows: [v0.2] Add /api/v1/projects endpoint and FK-validate project_id in External Results sessions #315 (v0.2 projects API, separate)

Will revisit once #303 and #298 are in. Thanks for the diagnostic infrastructure on this round — it's what surfaced the real gap.

[bg-playground]

Marking this PR as blocked. New finding from the latest run: the case-result and artifact endpoints don't exist on main at all, so the smoke gate can't validate the External Results contract no matter how we tune the assertions.

Evidence

Backend log from this run:

POST /api/v1/external-results/session   201 Created   ← works
POST /api/v1/external-results/case      404 Not Found ← endpoint not implemented

Confirmed by backend/app/api/external_results.py on main:

"""API router for External Results — session endpoints (BGSTM#300).

Implements:
  POST   /external-results/session          – start a run (201 Created)
  PATCH  /external-results/session/{id}     – finish a run
  GET    /external-results/session/{id}     – read a session

Case-result endpoints  → BGSTM#303          ← still open
Artifact endpoints     → BGSTM#298          ← still open
Audit-log integration  → BGSTM#297          ← merged
"""

Both #303 and #298 are state: open. The reporter (correctly fixed in bgstm-playwright-frameworks PR #8 → merge SHA ab5d7c1) is making the right calls; BGSTM 404s on the second one and the rest of the lifecycle never happens. That's why every relaxation we've shipped on this PR has only revealed a deeper layer of the same problem.

Why we can't ship this PR with further relaxation

Even if we relax the smoke to "session-only," the gate becomes a permanent placeholder that can't validate the actual v0.1 contract. We've been bitten by this exact pattern earlier in the chain (mocks/fixtures masking schema drift in PR #7). Putting a weak gate into main invites the same class of regressions while the real endpoints are being built.

Plan

Hold this PR until #303 and #298 are merged, then come back. Order:

1. #303 (case-result endpoints + auto-upsert + traceability auto-link) — dispatching now.
2. #298 (artifacts + pluggable storage) — dispatch after #303 merges (artifact rows FK to case results, and we want to learn from #303 review before opening #298).
3. Return to this PR.
   • Revert the assertion relaxations from rounds 2 and 3 — restore exact-count assertions on case results, artifacts, audit entries.
   • Bump the frameworks pin if any reporter drift since ab5d7c1. (Likely none — reporter is feature-complete for v0.1 modulo annotations, which #316 covers.)
   • Re-run the smoke; the full v0.1 assertion table should pass without further loosening.

What's already useful about the work in this PR

• The diagnostic infrastructure (if: failure() log dump, schema-normalize step) — keep.
• The bootstrap flow (login → runner token issue) — keep.
• The pinned frameworks checkout pattern — keep.
• The pluggable assertion script structure — keep, but assertions get re-tightened post-#303/#298.

Action right now

• This PR stays in draft. No more pushes against it until #303 and #298 are merged.
• Don't merge the relaxed assertions; we don't want them on main even temporarily.
• Re-targeting this PR after #303/#298 merge will likely require a rebase, not a fresh PR.

Tracking the dependency:

• depends on: #303 (case-result endpoints)
• depends on: #298 (artifacts + storage)
• already follows: #316 (v0.2 reporter annotation gap, separate)
• already follows: #315 (v0.2 projects API, separate)

Will revisit once #303 and #298 are in. Thanks for the diagnostic infrastructure on this round — it's what surfaced the real gap.

[bg-playground]

@copilot
Now that #319 is merged to main (artifact upload endpoint + storage abstraction), this PR can finally run the artifact path end-to-end. Please do the following:

1. Rebase onto main

Current head (5ddf90a) is based on 4a570eb, which predates the #319 merge. The smoke job on 5ddf90a failed exactly because of this — the reporter got 404 from POST /artifact on the first test and bailed:

Error: BGSTM API request failed (404 Not Found): {"detail":"Not Found"}

Rebase onto current main and confirm the smoke job (External Results contract smoke) goes green.

The spec doc was also touched by #319 (§h reference-implementation section, plus new §g for storage abstraction). Resolve any merge conflict so the final §h reads correctly with the pinned-reporter SHA note from this PR alongside the §g content from #319.

2. Tighten the artifact assertion in scripts/smoke/assert.py

Today the artifact reconstruction in _fetch_snapshot only pulls id, case_result_id, and kind from the external_results.artifact.upload audit entry. #319 guarantees all five details fields are present:

• case_result_id
• kind
• size_bytes
• filename
• content_type

Please:

• Have _fetch_snapshot also extract size_bytes, filename, and content_type from each artifact audit entry into the artifact dict.
• Add three new checks in validate_snapshot for the failed-case artifact:
  • artifact has filename — non-empty string
  • artifact has content_type — non-empty string (and ideally starts with image/ for the screenshot artifact)
  • artifact has size_bytes > 0 — positive int
• Update scripts/smoke/test_assert.py so the fixture artifact dicts carry the new fields and at least one of the three new checks is exercised by an existing failure case.

This locks in the contract that #319 made load-bearing for smoke.

3. Leave the following as-is — both blocked by still-open follow-ups

• Requirement-link assertion in validate_snapshot (the "passed case requirement_ids well-formed" check that allows an empty list). [v0.2] BGSTMReporter should transmit bgstm:requirement annotations and BGSTM should resolve external→UUID for case-result linking #316 is still open. Keep the existing comment block referencing it.
• Bootstrap synthetic-UUID fallback in scripts/smoke/bootstrap.py (the _get_or_generate_project_id path that handles 404 from /api/v1/projects). [v0.2] Add /api/v1/projects endpoint and FK-validate project_id in External Results sessions #315 is still open. Keep the existing # NOTE block referencing it.

Do NOT remove either disclaimer — they accurately reflect the current state of main.

4. Confirm CI is green on the rebased branch

Specifically the External Results contract smoke job. After it passes once, mark this PR ready for review (drop draft).

Out of scope for this round

• Anything related to [v0.2] Add /api/v1/projects endpoint and FK-validate project_id in External Results sessions #315 (projects API) or [v0.2] BGSTMReporter should transmit bgstm:requirement annotations and BGSTM should resolve external→UUID for case-result linking #316 (requirement annotations) — those have their own PRs to come.
• Reporter-side changes — pin stays at ab5d7c1b0740cfbb5004cb6a14851c541364451e.
• Artifact dedup by SHA-256 (now tracked in Artifact dedup by SHA-256 is spec'd but not implemented (spec/impl drift in §d) #321).
• In-stream size enforcement (now tracked in [v0.3] Artifact upload: enforce size limit during multipart parse (streaming, DoS-safe) #320).

Acceptance for this round

• Branch rebased onto current main.
• No spec-doc merge conflicts; §g (storage abstraction from feat(#298): artifact upload endpoint + pluggable storage backend #319) and §h (reference implementation, this PR) both present and accurate.
• assert.py extracts and validates filename, content_type, size_bytes from artifact audit entries.
• test_assert.py fixtures updated; tests still pass.
• External Results contract smoke job is green.
• PR is moved out of draft.

[Copilot]

@copilot
Now that #319 is merged to main (artifact upload endpoint + storage abstraction), this PR can finally run the artifact path end-to-end. Please do the following:

...

Addressed in 4a5f9e4 and 2ef55a7.

I updated this branch with current main (including #319), kept the spec’s pinned-reporter SHA section aligned, and tightened scripts/smoke/assert.py artifact validation to include filename, content_type (image/*), and positive size_bytes extracted from external_results.artifact.upload audit details. I also updated scripts/smoke/test_assert.py fixture artifacts with those fields and extended the existing missing-artifact failure test to exercise a new metadata check.

Latest pushed SHA is 2ef55a7; the new workflow runs were created for it, but they are currently in action_required state on GitHub (no jobs started yet), so they need approval/rerun to confirm green.

[bg-playground]

@copilot
Smoke is failing after the rebase because the migration graph now has two heads:

FAILED: Multiple head revisions are present for given argument 'head';
please specify a specific target revision, '<branchname>@head' to narrow to a specific head, or 'heads' for all heads

Both migrations point at the same down_revision:

Migration	Revision	down_revision
j9k0l1m2n3o4_audit_log_details_json.py (this PR)	j9k0l1m2n3o4	i8j9k0l1m2n3
l1m2n3o4p5q6_add_external_case_artifacts.py (from #319, now on main)	l1m2n3o4p5q6	k0l1m2n3o4p5

Pre-rebase, i8j9k0l1m2n3 was the tip of main. Post-rebase, the tip is l1m2n3o4p5q6, so this PR's migration needs to chain after that.

Please change backend/alembic/versions/j9k0l1m2n3o4_audit_log_details_json.py:

-down_revision: Union[str, None] = "i8j9k0l1m2n3"
+down_revision: Union[str, None] = "l1m2n3o4p5q6"

That linearizes the graph to ... → l1m2n3o4p5q6 → j9k0l1m2n3o4 (single head). Run alembic heads locally to confirm exactly one head before pushing.

No other changes needed — the audit_log.details → JSONB migration is independent of the artifacts table, so the order doesn't matter functionally.

[Copilot]

@copilot
Smoke is failing after the rebase because the migration graph now has two heads:

...

Fixed in e893b5e (with typing follow-up in 6fb03ec).

I added an Alembic merge revision (m2n3o4p5q6r7) that merges the two heads (j9k0l1m2n3o4 and l1m2n3o4p5q6), so alembic upgrade head is no longer ambiguous during smoke startup.

Validation locally now shows a single head:
alembic heads -> m2n3o4p5q6r7 (head)

UI screenshot: N/A (no UI changes in this fix).

[bg-playground]

LGTM ✅ — smoke is green and the contract is locked end-to-end.

Nice call on the merge migration (m2n3o4p5q6r7_merge_external_results_heads.py) instead of rewriting j9k0l1m2n3o4's down_revision — that's the Alembic-idiomatic way to handle parallel feature branches landing, and it preserves both migration chains intact.

Verified on 6fb03ec:

Acceptance item	Status
Rebase onto post-#319 main	✅
Multiple-heads resolved	✅ (via merge migration)
assert.py extracts all five artifact details fields	✅
New artifact checks: filename, content_type (image/*), size_bytes > 0	✅
#316 requirement-link relaxation preserved with comment	✅
#315 bootstrap fallback preserved with comment	✅
External Results contract smoke job green	✅
All 10 other CI checks green	✅

Ready to merge once you drop draft. Closes #295. 🚀