Harden QA reporting, thresholds, and multi-target flows

Add PR quality comment publishing, threshold-driven regression gates, multi-target onboarding/intake handling, and CI coverage updates while finalizing remaining issue-era references to the QA-first runtime model.

Author: kenahrens

.github/workflows/server-mode-smoke.yml

@@ -35,8 +35,8 @@ jobs:
 
       - name: Start intake API
         run: |
-          INTAKE_REPO_APP_MAP_JSON='{"speedscale/demo":"examples/apps/demo-node/agentapp.yaml"}' \
-          INTAKE_ALLOWED_REPOS='speedscale/demo' \
+          INTAKE_REPO_APP_MAP_JSON='{"speedscale/demo":"examples/apps/demo-node/agentapp.yaml","speedscale/demo-multi":"examples/apps/demo-node-multi-target/agentapp.yaml"}' \
+          INTAKE_ALLOWED_REPOS='speedscale/demo,speedscale/demo-multi' \
           npm run intake-api > intake-api.log 2>&1 &
           echo $! > intake-api.pid
 
@@ -94,6 +94,13 @@ jobs:
             --data-binary @webhook-pr.json > webhook-response.json
           node -e 'const fs=require("fs"); const payload=JSON.parse(fs.readFileSync("webhook-response.json","utf8")); if(!payload.runs?.[0]?.metadata?.name){console.error(payload); process.exit(1)};'
 
+      - name: Submit multi-target intake request
+        run: |
+          curl -sS -X POST http://127.0.0.1:8080/qa/runs \
+            -H "content-type: application/json" \
+            --data-binary @examples/runs/demo-node-multi-target-pr-quality-intake.json > multi-target-response.json
+          node -e 'const fs=require("fs"); const payload=JSON.parse(fs.readFileSync("multi-target-response.json","utf8")); const runs=payload.runs||[]; if(runs.length!==2){console.error(payload); process.exit(1)}; const names=runs.map((r)=>r.metadata?.name||""); if(!names.every((name)=>name.includes("node-api")||name.includes("node-worker"))){console.error(names); process.exit(1)};'
+
       - name: Stop intake API
         if: always()
         run: |
@@ -113,3 +120,4 @@ jobs:
             intake-response.json
             comparison-response.json
             webhook-response.json
+            multi-target-response.json

AGENTS.md

@@ -2,9 +2,9 @@
 
 ## Purpose
 
-This repository is a public reference architecture for an autonomous issue-to-fix workflow centered on the inner loop:
+This repository is a public reference architecture for an autonomous quality-validation workflow centered on the inner loop:
 
-`issue -> plan -> build -> validate`
+`request -> baseline -> compare -> report`
 
 Keep changes aligned with that goal. Prefer small, explicit contracts over broad platform abstractions.
 
@@ -20,7 +20,7 @@ Keep changes aligned with that goal. Prefer small, explicit contracts over broad
 
 - The first implementation should target one simple demo application.
 - The agent should operate against an app manifest rather than hardcoded repo logic.
-- The initial system should emit artifacts for every step: triage, plan, patch, build logs, validation result.
+- The initial system should emit artifacts for every step: request, baseline target, build logs, validation result, quality report.
 
 ## Repository Conventions
 

docs/phase-b-first-run.md

@@ -2,14 +2,14 @@
 
 Audience: Agent Factory developers and operators tracking Phase B execution outcomes.
 
-This is the concrete execution plan for the first real-ticket autonomous run.
+This document is historical context from the issue-first phase. Current operator flows are PR/manual QA request-first.
 
 ## Selected target
 
 - repo: `speedscale/microsvc`
 - issue: `#58`
 - issue URL: `https://github.com/speedscale/microsvc/issues/58`
-- intake payload: `examples/runs/microsvc-user-service-intake.json`
+- intake payload (legacy): `examples/runs/microsvc-user-service-intake.json`
 
 ## Why this issue
 
@@ -20,7 +20,7 @@ This is the concrete execution plan for the first real-ticket autonomous run.
 ## Execution steps
 
 1. start intake and worker in server mode
-2. submit `microsvc-user-service-intake.json` to intake API
+2. submit a QA intake payload to `/qa/runs` (legacy flow used `microsvc-user-service-intake.json`)
 3. wait for run phase to reach `succeeded` or `failed`
 4. collect evidence bundle from `artifacts/<run-name>/`
 5. evaluate against `docs/autonomy-mvp.md` rubric

docs/server.md

@@ -62,6 +62,7 @@ GitHub PR webhook intake (optional):
 - auth options for GitHub API calls:
   - preferred: `GITHUB_APP_ID` + `GITHUB_APP_PRIVATE_KEY`
   - fallback: `GITHUB_BOT_TOKEN` or `GH_TOKEN`
+  - worker uses same credentials to publish/update PR quality comments
 
 GitHub poller mode (optional):
 
@@ -73,8 +74,10 @@ GitHub poller mode (optional):
 - polls open issues/PRs in `INTAKE_ALLOWED_REPOS`
 - loads repo manifests from `INTAKE_REPO_APP_MAP_FILE` or `INTAKE_REPO_APP_MAP_JSON`
 - queues runs for events that satisfy required labels
+- queues one run per onboarded quality target when target is not explicitly specified
 - posts one bot comment for missing-label or missing-manifest cases
 - GitHub auth uses same precedence as webhook intake (App first, token fallback)
+- worker posts/updates one PR quality comment per target run
 
 Worker trigger mode (optional):
 
@@ -92,6 +95,7 @@ Run operations:
 - list runs: `npm run runs -- list [--phase <phase>]`
 - retry a failed run: `npm run runs -- retry <run-name>`
 - queue onboarding baseline run: `npm run runs -- baseline examples/apps/demo-node/agentapp.yaml --target demo-node`
+- queue all targets from multi-target manifest: `npm run runs -- baseline examples/apps/demo-node-multi-target/agentapp.yaml`
 
 When Redis backend is enabled, intake and retry operations enqueue run names to Redis and workers consume from Redis.
 

docs/users.md

@@ -53,6 +53,14 @@ curl -sS -X POST http://127.0.0.1:8080/qa/runs \
   --data-binary @examples/runs/demo-node-pr-quality-intake.json
 ```
 
+Multi-target example (queues one run per target):
+
+```bash
+curl -sS -X POST http://127.0.0.1:8080/qa/runs \
+  -H "content-type: application/json" \
+  --data-binary @examples/runs/demo-node-multi-target-pr-quality-intake.json
+```
+
 Queue onboarding baseline from manifest:
 
 ```bash
@@ -76,6 +84,7 @@ For real PR requests, treat successful command execution as necessary but not su
 - provide endpoint-level replay outcomes when performance is in scope
 - keep `build.test` and `validate.proxymock.command` meaningful (no no-op placeholders)
 - ensure baseline artifacts are current for each onboarded quality target
+- ensure GitHub bot auth is configured so PR quality comments can be posted/updated
 
 ## Operational Baselines
 

examples/apps/demo-node-multi-target/agentapp.yaml

@@ -0,0 +1,57 @@
+apiVersion: agents.speedscale.io/v1alpha1
+kind: AgentApp
+metadata:
+  name: demo-node-multi-target
+spec:
+  repo:
+    provider: github
+    url: https://github.com/speedscale/demo
+    defaultBranch: main
+    workdir: node
+  issue:
+    labels:
+      include:
+        - agent
+        - bug
+  quality:
+    trigger:
+      pullRequest: true
+      manualRequest: true
+      prePrRequest: true
+    baseline:
+      strategy: multi-project
+      targets:
+        - name: node-api
+          workdir: node
+          baselineRef: baseline/demo/node-api
+          command: npm test
+        - name: node-worker
+          workdir: node
+          baselineRef: baseline/demo/node-worker
+          command: npm test
+    reporting:
+      formats:
+        - json
+        - markdown
+      failOnRegression: true
+      thresholds:
+        maxBuildStderrLineDelta: 20
+        maxValidationStderrLineDelta: 20
+  build:
+    install: npm ci
+    test: npm test
+    start: npm start
+  validate:
+    proxymock:
+      dataset: demo-node-404
+      mode: replay-with-mocks
+      command: proxymock replay
+      service:
+        command: npm start
+        host: 127.0.0.1
+        port: 3000
+        startupTimeoutSeconds: 30
+  policy:
+    autoBranch: true
+    autoMr: true
+    autoMerge: false

examples/apps/demo-node/agentapp.yaml

@@ -30,6 +30,9 @@ spec:
         - json
         - markdown
       failOnRegression: true
+      thresholds:
+        maxBuildStderrLineDelta: 20
+        maxValidationStderrLineDelta: 20
   build:
     install: npm ci
     test: npm test

examples/apps/microsvc-user-service/agentapp.yaml

@@ -30,6 +30,9 @@ spec:
         - json
         - markdown
       failOnRegression: true
+      thresholds:
+        maxBuildStderrLineDelta: 50
+        maxValidationStderrLineDelta: 50
   build:
     install: make build
     test: make test

examples/runs/demo-node-multi-target-pr-quality-intake.json

@@ -0,0 +1,27 @@
+{
+  "source": "developer",
+  "repository": {
+    "provider": "github",
+    "owner": "speedscale",
+    "name": "demo-multi"
+  },
+  "appRef": {
+    "name": "demo-node-multi-target"
+  },
+  "request": {
+    "mode": "comparison",
+    "pullRequest": {
+      "number": 999,
+      "url": "https://github.com/speedscale/demo/pull/999",
+      "headSha": "abc123",
+      "baseSha": "def456"
+    }
+  },
+  "requestedBy": {
+    "type": "user",
+    "login": "developer"
+  },
+  "metadata": {
+    "reason": "multi-target-intake-smoke"
+  }
+}

schemas/agentapp.schema.yaml

@@ -115,6 +115,14 @@ properties:
                     - markdown
               failOnRegression:
                 type: boolean
+              thresholds:
+                type: object
+                properties:
+                  maxBuildStderrLineDelta:
+                    type: number
+                  maxValidationStderrLineDelta:
+                    type: number
+                additionalProperties: false
             additionalProperties: false
         additionalProperties: false
       build: