fix(smoke): add per-call timing, bump RUN_TIMEOUT 180→300s, document flake (GLA-634) (#109)

* feat(api): Wave B BE bundle — alerts aggregator, feedback distribution, TraceRead fields, trace time-range (GLA-598)

Four additive read-only pieces required by Wave B observability FE (T2/T8):

a) GET /projects/{id}/alerts — project-scope alert aggregator joining
   alerts→runs, ordered newest-first, optional `since`/`limit` params.
b) GET /projects/{id}/feedback/distribution — server-side width_bucket
   histogram (bins/avg/p10/n) over feedback.key_values->>scorer.
c) TraceRead gains nullable score/tokens + error_count; list_traces
   computes them via correlated subqueries over spans+feedback.
d) list_traces accepts created_after/created_before query params.

All endpoints enforce org-membership auth. 535 tests pass, no regressions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(smoke): add per-call timing, bump RUN_TIMEOUT to 300s, document flake

Addresses GLA-634: cold-start hang on first log() call caused rc=124 timeout
on GitHub-hosted runners pulling cold images.

- scripts/smoke_run.py: emit [smoke_run] timing lines for init, each log step,
  and finish so run.log identifies exactly which call hung
- .github/workflows/smoke.yml: bump SMOKE_RUN_TIMEOUT 180→300s to give the
  cold-start path (MinIO bucket create, first-request auth) enough headroom
- docs/dev/smoke.md: document the 8-step smoke harness, per-call timing output
  format, and the rc=124 failure mode with diagnosis steps

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: jqueguiner

.github/workflows/smoke.yml

@@ -49,7 +49,7 @@ jobs:
           # slower image-pull + first-request latency than a dev box; a 90s
           # SDK budget timed out on the first PR run (rc=124).
           SMOKE_HEALTH_TIMEOUT: "300"
-          SMOKE_RUN_TIMEOUT: "180"
+          SMOKE_RUN_TIMEOUT: "300"
         run: make smoke
 
       - name: Upload smoke logs on failure

docs/dev/smoke.md

@@ -0,0 +1,95 @@
+# Smoke (E2E) test — developer guide
+
+The smoke test is the wandb-compat invariant gate. It boots the full Docker stack
+from a clean checkout, registers a user, runs the SDK round-trip, and asserts the
+run is visible via the API. A single honest assertion — no retries on the SDK step.
+
+## Running locally
+
+```bash
+make smoke
+```
+
+Env-var overrides (all optional):
+
+| Variable | Default | Purpose |
+|---|---|---|
+| `SMOKE_HEALTH_TIMEOUT` | 180 | Seconds to wait for `/health/full` |
+| `SMOKE_RUN_TIMEOUT` | 60 | Seconds budget for `smoke_run.py` |
+| `SMOKE_KEEP` | 0 | Set to `1` to leave stack + venv up for triage |
+| `SMOKE_API_BASE` | auto | Override the discovered API base URL |
+| `SMOKE_SECRET_KEY` | auto-gen | Pin a specific `SECRET_KEY` for triage |
+
+CI uses `SMOKE_HEALTH_TIMEOUT=300` and `SMOKE_RUN_TIMEOUT=300` because
+GitHub-hosted runners have slower image-pull and first-request latency.
+
+## Log files
+
+On failure, CI uploads `.smoke-logs/` as a workflow artifact (7-day retention).
+Locally the same directory is written; pass `SMOKE_KEEP=1` to preserve it.
+
+| File | Contents |
+|---|---|
+| `.smoke-logs/compose.log` | All `docker compose` output |
+| `.smoke-logs/run.log` | SDK script stdout/stderr + per-call timings |
+| `.smoke-logs/run_id` | Run UUID written by `smoke_run.py` |
+
+## Per-call timing lines
+
+`smoke_run.py` emits `[smoke_run]` lines to stdout (captured in `run.log`):
+
+```
+[smoke_run] start
+[smoke_run] init done (1.23s) run_id=<uuid>
+[smoke_run] first_log done (0.45s)
+[smoke_run] log_step1 done (0.12s)
+[smoke_run] log_step2 done (0.11s)
+[smoke_run] log_step3 done (0.11s)
+[smoke_run] last_log done (0.10s)
+[smoke_run] finish done (0.22s)
+[smoke_run] total (2.34s)
+SMOKE_RUN_ID=<uuid>
+```
+
+When a hang occurs, the last `[smoke_run]` line before the timeout identifies
+which call blocked. The most common cold-start culprit is `init` (first call
+triggers MinIO bucket creation and any pending migrations).
+
+## Known failure modes
+
+### rc=124 — `SMOKE_RUN_TIMEOUT` exceeded
+
+**Symptom:** `smoke_run.py exited rc=124`. Step 6/8 logs `init` success then goes
+silent until the timeout fires.
+
+**Root cause (GLA-634):** Cold-start latency on the API container. Postgres, Redis,
+and MinIO report healthy via `/health/full`, but the *first* SDK `log()` call
+triggers lazy work inside the API: MinIO bucket creation, first-request auth
+overhead, and any deferred migration steps. On a GitHub-hosted runner pulling cold
+images this added ~2 min over the previous 90s budget.
+
+**Diagnosis from `run.log`:**
+1. Find the last `[smoke_run] … done` line — that identifies the completed call.
+2. The *next* call (or absence of any line) is where the hang occurred.
+3. Check `compose.log` for API-side errors around the same timestamp.
+
+**Fix applied:** `SMOKE_RUN_TIMEOUT` bumped to 300s in CI. The local default
+(60s) is intentionally kept lower; pass `SMOKE_RUN_TIMEOUT=300 make smoke` on a
+slow dev box if needed.
+
+**If the flake recurs:**
+- Check `compose.log` for MinIO or Postgres startup warnings after `/health/full` turned green.
+- Run with `SMOKE_KEEP=1`, then `docker compose exec api journalctl` (or equivalent) to inspect API startup logs.
+- Consider adding an explicit MinIO bucket pre-warm step to `smoke.sh` before step 6.
+
+### rc=1 — stack failed to start
+
+See `compose.log`. Common causes: port collision (another stack on the same host),
+image build failure, or insufficient disk space for `.data/` bind-mounts.
+
+### Metric assertion failure
+
+`loss` / `accuracy` not found in the metrics response. Usually means the
+`openrunner.log()` flush did not complete before `finish()` returned, or the
+metrics endpoint returned an unexpected shape. Check `run.log` for `finish done`
+timing — if it was near-instant, the flush may have been skipped.

scripts/smoke_run.py

@@ -9,43 +9,66 @@
 Reads OPENRUNNER_API_KEY / OPENRUNNER_BASE_URL / OPENRUNNER_PROJECT from env
 (set by scripts/smoke.sh). Writes the run id to SMOKE_RUN_ID_FILE for the
 caller to verify via the API.
+
+Per-call timings are printed so cold-start hangs are diagnosable from run.log.
 """
 
 from __future__ import annotations
 
 import os
 import sys
+import time
+
+
+def _ts() -> float:
+    return time.monotonic()
+
 
-import openrunner
+def _elapsed(t0: float) -> str:
+    return f"{time.monotonic() - t0:.2f}s"
 
 
 def main() -> int:
+    import openrunner
+
     project = os.environ.get("OPENRUNNER_PROJECT", "smoke")
     run_id_file = os.environ.get("SMOKE_RUN_ID_FILE")
 
+    smoke_start = _ts()
+    print(f"[smoke_run] start", flush=True)
+
     # save_code=False keeps the SDK init/log/finish round-trip inside
     # SMOKE_RUN_TIMEOUT on slow GitHub-hosted runners. The SDK default
     # uploads every non-binary file in the CWD as a code-snapshot artifact
     # via sequential presigned PUTs (~80 files for this repo) which is out
     # of scope for the smoke test (artifact ingestion is covered elsewhere).
+    t = _ts()
     run = openrunner.init(
         project=project,
         name="smoke-run",
         config={"learning_rate": 0.001, "batch_size": 32, "epochs": 1},
         tags=["smoke", "ci"],
         save_code=False,
     )
+    print(f"[smoke_run] init done ({_elapsed(t)}) run_id={run.id}", flush=True)
 
     for step in range(5):
+        t = _ts()
         openrunner.log(
             {
                 "loss": 1.0 / (step + 1),
                 "accuracy": 0.1 * (step + 1),
             },
             step=step,
         )
+        label = "first_log" if step == 0 else ("last_log" if step == 4 else f"log_step{step}")
+        print(f"[smoke_run] {label} done ({_elapsed(t)})", flush=True)
 
+    t = _ts()
     openrunner.finish()
+    print(f"[smoke_run] finish done ({_elapsed(t)})", flush=True)
+
+    print(f"[smoke_run] total ({_elapsed(smoke_start)})", flush=True)
 
     if run_id_file:
         with open(run_id_file, "w", encoding="utf-8") as fh:

src/api/app/api/v1/alerts.py

@@ -2,19 +2,22 @@
 
 from uuid import UUID
 
-from fastapi import APIRouter, Depends, HTTPException, status
+from fastapi import APIRouter, Depends, HTTPException, Query, status
 from pydantic import BaseModel, ConfigDict
 from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 from datetime import datetime
 
 from app.api.deps import get_current_user, get_db
+from app.models.organization import OrgMember
+from app.models.project import Project
 from app.models.run import Run
 from app.models.user import User
 from app.schemas.alert import AlertCreate, AlertRead
 from app.services import alert as alert_service
 
 router = APIRouter(prefix="/runs")
+project_router = APIRouter(prefix="/projects")
 
 
 class WebhookDeliveryRead(BaseModel):
@@ -125,3 +128,62 @@ async def list_alert_deliveries(
     """
     await _verify_run_access(db, run_id, current_user.id)
     return await alert_service.get_delivery_history(db, alert_id)
+
+
+# ---------------------------------------------------------------------------
+# Project-scope aggregator
+# ---------------------------------------------------------------------------
+
+
+async def _verify_project_membership(
+    db: AsyncSession, project_id: UUID, user_id: UUID
+) -> Project:
+    """Verify project exists and the caller is a member of its org."""
+    result = await db.execute(
+        select(Project).where(Project.id == project_id)
+    )
+    project = result.scalar_one_or_none()
+    if project is None:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail="Project not found",
+        )
+
+    member_result = await db.execute(
+        select(OrgMember).where(
+            OrgMember.org_id == project.org_id,
+            OrgMember.user_id == user_id,
+        )
+    )
+    if member_result.scalar_one_or_none() is None:
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail="Not a member of the project's organization",
+        )
+    return project
+
+
+@project_router.get(
+    "/{project_id}/alerts",
+    response_model=list[AlertRead],
+)
+async def list_project_alerts(
+    project_id: UUID,
+    since: datetime | None = Query(
+        default=None,
+        description="ISO8601 lower bound on Alert.created_at",
+    ),
+    limit: int = Query(default=20, ge=1, le=100),
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    """List recent alerts across all runs in a project, newest first.
+
+    Aggregator for the FE "Recent alerts (24h)" card. ``Alert`` has no
+    resolved-state column, so this returns a flat recent-activity feed
+    rather than a status-filtered list.
+    """
+    await _verify_project_membership(db, project_id, current_user.id)
+    return await alert_service.list_for_project(
+        db, project_id, since=since, limit=limit
+    )

src/api/app/api/v1/feedback.py

@@ -1,8 +1,9 @@
 """Feedback endpoints -- user reactions and notes on traced spans."""
 
+from datetime import datetime
 from uuid import UUID
 
-from fastapi import APIRouter, Depends, HTTPException, status
+from fastapi import APIRouter, Depends, HTTPException, Query, status
 from sqlalchemy import delete, select
 from sqlalchemy.ext.asyncio import AsyncSession
 
@@ -12,7 +13,13 @@
 from app.models.trace import Span, Trace
 from app.models.project import Project
 from app.models.user import User
-from app.schemas.feedback import FeedbackCreate, FeedbackRead, VALID_REACTIONS
+from app.schemas.feedback import (
+    DistributionRead,
+    FeedbackCreate,
+    FeedbackRead,
+    VALID_REACTIONS,
+)
+from app.services import feedback as feedback_service
 
 router = APIRouter()
 
@@ -162,3 +169,70 @@ async def delete_feedback(
     await db.execute(
         delete(Feedback).where(Feedback.id == feedback_id)
     )
+
+
+# ---------------------------------------------------------------------------
+# Project-scope aggregator
+# ---------------------------------------------------------------------------
+
+
+async def _verify_project_membership(
+    db: AsyncSession, project_id: UUID, user_id: UUID
+) -> Project:
+    """Verify project exists and the caller is a member of its org."""
+    result = await db.execute(
+        select(Project).where(Project.id == project_id)
+    )
+    project = result.scalar_one_or_none()
+    if project is None:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail="Project not found",
+        )
+
+    member_result = await db.execute(
+        select(OrgMember).where(
+            OrgMember.org_id == project.org_id,
+            OrgMember.user_id == user_id,
+        )
+    )
+    if member_result.scalar_one_or_none() is None:
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail="Not a member of the project's organization",
+        )
+    return project
+
+
+@router.get(
+    "/projects/{project_id}/feedback/distribution",
+    response_model=DistributionRead,
+)
+async def feedback_distribution(
+    project_id: UUID,
+    scorer: str | None = Query(
+        default=None,
+        description="JSON key inside feedback.key_values to bucketise (default 'score')",
+    ),
+    since: datetime | None = Query(
+        default=None,
+        description="ISO8601 lower bound on Feedback.created_at",
+    ),
+    bins: int = Query(default=20, ge=1, le=100),
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    """Server-side score-distribution histogram for a project.
+
+    Joins ``feedback → spans → traces → projects`` and bucketises the
+    numeric value at ``key_values->>scorer`` (default ``score``) into
+    ``bins`` equal-width buckets between min and max.
+    """
+    await _verify_project_membership(db, project_id, current_user.id)
+    return await feedback_service.get_distribution(
+        db,
+        project_id,
+        scorer=scorer,
+        since=since,
+        bins=bins,
+    )

src/api/app/api/v1/router.py

@@ -2,6 +2,7 @@
 
 from fastapi import APIRouter
 
+from app.api.v1.alerts import project_router as alerts_project_router
 from app.api.v1.alerts import router as alerts_router
 from app.api.v1.automations import project_router as automation_project_router
 from app.api.v1.automations import router as automations_router
@@ -87,6 +88,7 @@ async def list_tenants():
 api_router.include_router(storage_router, tags=["storage"])
 api_router.include_router(export_router, tags=["export"])
 api_router.include_router(alerts_router, tags=["alerts"])
+api_router.include_router(alerts_project_router, tags=["alerts"])
 api_router.include_router(reports_router, tags=["reports"])
 api_router.include_router(reports_public_router, tags=["reports"])
 api_router.include_router(sweep_project_router, tags=["sweeps"])