Add Projects API and enforce project_id existence for External Results sessions

[Copilot]

This PR closes the v0.2 gap where /api/v1/projects was missing and external session creation accepted non-existent project_id values. It adds a real Projects API, enforces project existence at session creation time, and updates smoke bootstrap/specs to match enforced behavior.

• Projects API surface (/api/v1/projects)

  • Added POST, paginated GET list, GET {id}, and PATCH {id} endpoints.
  • Wired router into main.py under settings.API_V1_PREFIX with tags=["projects"].
  • Kept DELETE out of scope.

• Projects domain implementation

  • Added Project SQLAlchemy model and Alembic migration for projects table.
  • Added Pydantic schemas:
    • ProjectCreate
    • ProjectResponse (from_attributes=True)
    • ProjectUpdate
  • Added async CRUD helpers:
    • create_project
    • get_project
    • list_projects
    • update_project

• Auth + audit behavior

  • Write endpoints (POST, PATCH) require reviewer/admin.
  • Read endpoints require authenticated user.
  • Added audit log writes for:
    • project.create (full payload snapshot)
    • project.update (changed-fields diff)
  • Audit shape uses actor_kind="user", user_id=current_user.id, actor_token_id=None, resource_type="project".

• External Results FK validation

  • In create_session, added pre-insert lookup against projects.
  • Unknown project now raises the structured error:

    raise ValueError(
        {
            "code": "session.project_not_found",
            "message": f"Project {payload.project_id} does not exist.",
            "details": None,
        }
    )

  • In create_external_session, extended existing ValueError handling to map this code to HTTP 400.

• Smoke bootstrap alignment

  • Removed synthetic UUID fallback in scripts/smoke/bootstrap.py.
  • Bootstrap now always creates a real project through POST /api/v1/projects and fails loudly on error.

• Tests

  • Added backend/tests/integration/test_projects.py covering auth matrix, CRUD behavior, 404s, pagination shape, and audit entries.
  • Added backend/tests/integration/test_external_results_session_project_fk.py for:
    • unknown project_id → 400 session.project_not_found
    • valid project_id → 201
  • Updated existing external-results tests to seed real projects so session creation remains valid under enforced FK checks.

• Spec updates

  • Updated docs/specs/external_results_v1.md to state session.project_not_found is enforced.
  • Added a concise “Project lifecycle” subsection documenting creation flow, session reference requirement, and read/write auth rules.

Original prompt

Goal

Implement issue #315 — add a real /api/v1/projects API and FK-validate project_id in External Results sessions. This closes one of two v0.2 gaps the smoke workflow surfaced (the other is #316).

Closes: #315
Related: #314 (smoke workflow — bootstrap synthetic-UUID fallback gets removed by this PR), #316 (separate v0.2 follow-up, do not touch in this PR)

Scope

Backend — new projects API

• backend/app/api/projects.py — new router with:
  • POST /api/v1/projects → 201 Created with ProjectResponse
  • GET /api/v1/projects → list, paginated mirroring /api/v1/requirements shape
  • GET /api/v1/projects/{project_id} → 200 or 404
  • PATCH /api/v1/projects/{project_id} → partial update
  • No DELETE endpoint in this PR (separate concern; cascades not designed yet)
• backend/app/schemas/project.py — ProjectCreate, ProjectResponse, ProjectUpdate. Match the existing Project model's columns; mirror the patterns in schemas/requirement.py (Pydantic v2 model_config, from_attributes=True).
• backend/app/crud/project.py — async helpers: create_project, get_project, list_projects, update_project. Mirror crud/requirement.py style.
• backend/app/main.py — register the new router under settings.API_V1_PREFIX with tags=["projects"].

Auth and audit

• Write endpoints (POST, PATCH) require reviewer or admin role — mirror the requirements API guard exactly.
• Read endpoints accept any authenticated user (admin / reviewer / viewer).
• Audit log entries on every state-changing endpoint, matching the established pattern (e.g. external_results.session.start):
  • actor_kind = "user", user_id = current_user.id, actor_token_id = None
  • action = "project.create" or "project.update"
  • resource_type = "project", resource_id = project.id
  • details includes the relevant changed fields (for create: full payload snapshot; for update: changed-fields diff)

External Results FK validation

• In backend/app/crud/external_results.create_session: before insert, verify payload.project_id resolves to an existing row in the projects table. If not, raise ValueError with the spec-aligned shape:

  raise ValueError({"code": "session.project_not_found", "message": f"Project {payload.project_id} does not exist.", "details": None})

• In backend/app/api/external_results.py create_external_session: catch the new ValueError shape and map to 400 Bad Request (per existing spec § c.1 — the table already lists this code). Add to the existing ValueError exception handler — don't introduce a parallel one.

Smoke workflow updates (this same PR)

• In scripts/smoke/bootstrap.py:
  • Remove the _get_or_generate_project_id synthetic-UUID fallback. Always create a real project via the new POST /api/v1/projects endpoint.
  • Delete the # NOTE: /api/v1/projects currently 404s on main comment block.
  • If creation fails for any reason, fail the bootstrap loudly (no fallback).
• The smoke workflow itself (.github/workflows/external-results-smoke.yml) should not need changes — bootstrap.py is invoked the same way.

Tests

• backend/tests/integration/test_projects.py — mirror test_requirements.py:
  • 201 create with admin / reviewer; 403 with viewer; 401 unauthenticated
  • 200 list / get with any role; 404 on unknown id
  • 200 patch with admin / reviewer; 403 with viewer; 404 on unknown id
  • Audit log entries written on create / patch with correct shape
• backend/tests/integration/test_external_results_session_project_fk.py — new file:
  • Session create with unknown project_id → 400 with code: "session.project_not_found"
  • Session create with valid project_id → 201 (regression for the happy path)
• scripts/smoke/test_assert.py — no changes expected (it doesn't exercise bootstrap).

Spec doc

• docs/specs/external_results_v1.md § c.1 ("Create session" → "Error codes"):
  • Update the prose around the 400 session.project_not_found row to indicate it is now enforced (was previously aspirational).
• Add a short "Project lifecycle" subsection — placement at your discretion, but probably before § c.1 or in a new § c.0. One paragraph documenting:
  • Projects are created via POST /api/v1/projects.
  • Sessions reference a project by project_id and require it to exist.
  • Reads accept any authenticated user; writes require reviewer or admin.

Out of scope

• Project membership / per-project authorization scoping (separate v0.2+ concern).
• Soft-delete on projects.
• Adding project_id columns to other tables (requirements, test_cases, etc.) — projects table only.
• Reporter bgstm:requirement annotation handling — tracked in [v0.2] BGSTMReporter should transmit bgstm:requirement annotations and BGSTM should resolve external→UUID for case-result linking #316.
• Any changes to External Results schemas beyond catching the new ValueError.
• Any DELETE endpoint for projects.

Acceptance criteria

• POST /api/v1/projects returns 201 with ProjectResponse; persists row.
• `GET ...

This pull request was created from Copilot chat.

[bg-playground]

LGTM ✅ — comprehensive implementation, CI green including the smoke workflow.

Strong work on the second-order details:

• Update audit shape uses {field: {from, to}} diff — clean and machine-parseable. Better than I'd have prescribed.
• Cross-file fixture updates — proactively seeding Project rows in the existing external-results test fixtures (test_external_results_session.py, test_external_results_artifact.py, test_external_results_case.py, test_external_results_audit.py) so the new FK validation doesn't break their happy paths. That kind of second-order thinking saves a review round.
• Migration down_revision correctly chains off m2n3o4p5q6r7 (the merge migration from Add BGSTM external-results smoke workflow pinned to reporter SHA ab5d7c1 with main-branch, audit-log, and artifact-path compatibility hardening #314) — single-head graph preserved.
• Spec § c.0 placement before § c.1 reads naturally — context first, enforcement details second.

Acceptance check:

Item	Status
Projects API surface (POST/GET list/GET id/PATCH id; no DELETE)	✅
Auth guards mirror requirements API (writes: reviewer/admin; reads: any authenticated)	✅
Audit on create (full payload) and update (from/to diff)	✅
create_session FK validation → ValueError({"code": "session.project_not_found", ...})	✅
Handler maps to 400 via the existing ValueError exception path	✅
Bootstrap synthetic-UUID fallback fully removed; bootstrap fails loudly on non-2xx	✅
test_projects.py covers auth matrix + audit shape + 404	✅
test_external_results_session_project_fk.py covers 400/201	✅
Spec § c.0 (Project lifecycle) + § c.1 enforcement prose	✅
All 11 CI checks green incl. External Results contract smoke	✅

Closes #315. v0.2 step 1 of 2 done; #316 is the remaining piece. 🚀

Two non-blocking nits for awareness, no action needed:

1. Project.name has no UniqueConstraint; smoke runs will accumulate "smoke-project" rows over time. Trivial cleanup if it ever matters.
2. There's no DB-level FK from external_run_sessions.project_id to projects.id — validation is at the CRUD layer only. Fine for v0.2; worth eventually adding the FK constraint when project deletion is introduced.