From 91aa37b8d9ef82fb60fc6525b50150be324c0e0c Mon Sep 17 00:00:00 2001 From: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Mon, 24 Aug 2026 12:57:21 +0000 Subject: [PATCH 1/3] Add always-on drift report and bug triage automation artifacts --- docs/automations/README.md | 29 +++ docs/automations/bug-triage.md | 90 ++++++++ docs/automations/prompts/bug-triage.md | 91 ++++++++ .../prompts/weekly-drift-report.md | 92 ++++++++ .../automations/remediation-queue.schema.json | 116 ++++++++++ docs/automations/weekly-drift-report.md | 116 ++++++++++ scripts/drift_scan.py | 204 ++++++++++++++++++ 7 files changed, 738 insertions(+) create mode 100644 docs/automations/README.md create mode 100644 docs/automations/bug-triage.md create mode 100644 docs/automations/prompts/bug-triage.md create mode 100644 docs/automations/prompts/weekly-drift-report.md create mode 100644 docs/automations/remediation-queue.schema.json create mode 100644 docs/automations/weekly-drift-report.md create mode 100755 scripts/drift_scan.py diff --git a/docs/automations/README.md b/docs/automations/README.md new file mode 100644 index 00000000..c65db67c --- /dev/null +++ b/docs/automations/README.md @@ -0,0 +1,29 @@ +# Always-on automations + +Two Devin Automations run against this repository. Both are defined in the Devin +Automations UI; this directory holds their reviewable source of truth: the design, +the guardrails, and the exact prompt each one runs. + +| Automation | Trigger | Human checkpoint | +| --- | --- | --- | +| [Weekly drift report](weekly-drift-report.md) | Schedule — Mondays 06:00 UTC | Nobody may act on the queue until a maintainer picks items; the automation never opens remediation PRs | +| [Bug triage](bug-triage.md) | GitHub issue labeled `bug` in this repo | Anything above a trivial fix waits for an explicit "go ahead" comment on the issue | + +Both automations run as their creator (creator permissions, visible to the creator +and org admins) and deliver everything to GitHub — no Slack, no email fan-out. + +Prompts live in [`prompts/`](prompts/) and are the text pasted into each automation. +Change the prompt file and the automation together, in the same PR, so the repo never +disagrees with what actually runs. + +## Scanners + +Each repository in scope carries its own read-only scanner that emits the same +normalized JSON, so the weekly report is reproducible rather than hand-assembled: + +- `timesheet-app` — [`scripts/drift_scan.py`](../../scripts/drift_scan.py) (npm workspaces, GitHub advisories via `npm audit`) +- `petclinic-microservices` — `scripts/drift_scan.py` (Maven coordinates, Maven Central, OSV.dev) +- `timesheet-infra` — `scripts/drift_scan.py` (Terraform providers, registry modules, `required_version`) + +The shared item schema, ranking formula, and sizing rubric are documented in +[weekly-drift-report.md](weekly-drift-report.md#normalized-scanner-output). diff --git a/docs/automations/bug-triage.md b/docs/automations/bug-triage.md new file mode 100644 index 00000000..06d4f109 --- /dev/null +++ b/docs/automations/bug-triage.md @@ -0,0 +1,90 @@ +# Automation 2 — Ticket-triggered bug triage + +Every new bug report in `timesheet-app` gets a reproduction attempt, a named code path +and a severity before a human reads it — and nothing above a trivial fix happens +without a human saying so. + +- **Scope:** `timesheet-app` only (`backend/src/`, `frontend/src/`) +- **Prompt:** [`prompts/bug-triage.md`](prompts/bug-triage.md) +- **Delivery:** a comment on the triggering issue + +## Trigger + +GitHub issue event on `Cognition-Partner-Workshops/timesheet-app` where the issue +carries the label `bug`: + +- `action = opened` with the `bug` label, or +- `action = labeled` with `label.name = bug` (so triage still fires when a maintainer + triages an existing report into the bug queue). + +Issues without the `bug` label are ignored; the label is the intake gate, which keeps +questions and feature requests out of the reproduction loop. + +## Procedure + +1. **Read the report.** Extract expected vs. actual behaviour, steps, environment and + any error text. If the report has no observable symptom at all, say so and ask the + reporter for the missing detail instead of guessing. +2. **Reproduce first.** Install, run `cd backend && npm test` and the relevant flows; + start `backend` (port 3001) and `frontend` (port 5173) when the symptom is + user-visible. Reproduction is attempted *before* any severity claim. +3. **Locate the responsible code path.** Follow the failing behaviour to specific files + and line ranges under `backend/src/` or `frontend/src/`, then state the call chain + (route → handler → query, or component → hook → API call) and the root cause. +4. **Assess severity** with the rubric below. +5. **Post findings** as one comment on the issue: reproduction result, code path, + root cause, severity with justification, and a proposed fix with its size. +6. **Stop and ask** unless the fix is trivial (see below). + +## Severity rubric + +| Severity | Criteria | +| --- | --- | +| `S1 critical` | data loss or corruption, auth/authorization bypass, secret exposure, or the app unusable for all users | +| `S2 high` | a core timesheet flow (create/edit/submit/approve entries, login) broken with no workaround | +| `S3 medium` | a non-core flow broken, or a core flow degraded but with a workaround | +| `S4 low` | cosmetic, copy, or a rare edge case with no functional impact | +| `unreproducible` | the symptom did not occur — reported with exactly what was tried, never silently downgraded | + +Severity is reported with the evidence that justifies it (failing test, HTTP status, +console error, screenshot), so the label is auditable rather than a feeling. + +## Trivial vs. everything else + +**Trivial** — may be fixed in the same run, in a PR, without waiting: + +- a typo, a wrong string/label, a wrong constant or off-by-one in one expression +- a missing null/undefined guard in one function +- confined to a single file, under roughly 10 changed lines, no schema/API/auth change, + and covered by existing tests that pass + +**Everything else waits**: schema or migration changes, auth or permission logic, +API contract changes, shared component or state refactors, anything touching more than +one file or needing a new test to prove the fix, and anything the automation is not +certain about. Uncertainty counts as non-trivial. + +## Guardrails + +- **Reproduce before diagnosing.** No severity, no root cause and no fix without either + a reproduction or an explicit statement that reproduction failed and why. +- **No fix above trivial without confirmation** — see the human checkpoint. +- **Never weaken tests.** No editing, skipping or loosening a test to make a failure go + away; if a test looks wrong, say so and stop. +- **No production behaviour changes as a side effect** of triage: no dependency bumps, + no refactors, no formatting sweeps, no unrelated file edits. +- **No secrets in comments.** No tokens, connection strings, `.env` contents or real + user data; redact anything sensitive pulled from logs. +- **No destructive actions.** Never delete or force-push branches, never close or + relabel the reporter's issue, never touch production or live infrastructure. +- **One comment per triage run,** so the issue stays readable. +- **Stay in `timesheet-app`.** A cause outside `backend/src/` or `frontend/src/` + (infrastructure, upstream dependency) is reported, not fixed here. +- **Time-boxed.** If reproduction does not succeed within the run, post what was tried + and what is missing rather than continuing to guess. + +## Human checkpoint + +**The confirmation gate on the issue.** For anything above trivial, the automation posts +its findings plus a proposed fix and size, then explicitly asks for confirmation and +stops. It resumes only when a maintainer replies on the issue with an explicit go-ahead. +Trivial fixes still land as a PR for review — nothing is merged by the automation. diff --git a/docs/automations/prompts/bug-triage.md b/docs/automations/prompts/bug-triage.md new file mode 100644 index 00000000..337be452 --- /dev/null +++ b/docs/automations/prompts/bug-triage.md @@ -0,0 +1,91 @@ +# Prompt — Bug triage + +This is the exact prompt run by the "Bug Triage — timesheet-app" automation (trigger: +GitHub issue labeled `bug` in `Cognition-Partner-Workshops/timesheet-app`). Keep this file +and the automation in sync in one PR. Design and rubric: +[../bug-triage.md](../bug-triage.md). + +--- + +A bug was reported in `Cognition-Partner-Workshops/timesheet-app`. The triggering issue +(number, title, body, labels) is in the trigger event payload. Triage it, post your +findings on that issue, and do not fix anything above trivial without confirmation. + +## 1. Understand the report + +Read the issue with the built-in git tools, including existing comments. Extract expected +behaviour, actual behaviour, steps, environment and error text. If there is no observable +symptom to chase, post what is missing and ask the reporter — do not invent a repro. + +## 2. Reproduce before you diagnose + +Work on a clean checkout of `main`. + +- Backend: `cd backend && npm ci && npm test`, then `npm run dev` (port 3001) and exercise + the reported request path directly. +- Frontend: `cd frontend && npm ci && npm run lint && npm run build`, then `npm run dev` + (port 5173) and drive the reported UI flow in the browser. +- Capture the evidence: failing test output, HTTP status and response body, console error, + or a screenshot of the broken UI. + +If you cannot reproduce it, say so explicitly, list exactly what you tried and what you +would need from the reporter. Never mark an unreproducible report as low severity by +default. + +## 3. Locate the responsible code path + +Trace the symptom to concrete files and line ranges under `backend/src/` or +`frontend/src/` and state the chain (route → handler → query, or component → hook → API +call) plus the root cause: which expression or missing condition produces the observed +behaviour. Cite `path:line` for every claim. If the cause is outside those two trees +(infrastructure, an upstream dependency), report that and stop there. + +## 4. Assess severity + +- `S1 critical` — data loss/corruption, auth or authorization bypass, secret exposure, or + the app unusable for everyone +- `S2 high` — a core flow (create/edit/submit/approve timesheet entries, login) broken + with no workaround +- `S3 medium` — a non-core flow broken, or a core flow degraded with a workaround +- `S4 low` — cosmetic/copy or a rare edge case with no functional impact +- `unreproducible` — symptom did not occur; report what was tried + +Justify the severity with the evidence from step 2. + +## 5. Post findings — one comment on the triggering issue + +Include, in this order: reproduction result (with the command/flow used and its output), +the responsible code path with `path:line` references, the root cause, the severity with +its justification, and a proposed fix with its size and risk. + +## 6. Fix only what is trivial + +**Trivial** (fix it in this run, in a PR — do not merge it): a typo, wrong string, wrong +constant, off-by-one in a single expression, or a missing null/undefined guard — confined +to one file, roughly under 10 changed lines, no schema/API/auth change, with existing +tests passing afterwards. Run `cd frontend && npm run lint`, `cd backend && npm test` and +`cd frontend && npm run build` before opening it, and link the PR in your comment. + +**Everything else**: do not change any code. Post the proposed fix, its size, its risk and +the files it would touch, then ask on the issue for explicit confirmation before +proceeding, and stop. This includes schema/migration changes, auth or permission logic, +API contract changes, shared component or state refactors, anything touching more than one +file, anything needing a new test to prove the fix, and anything you are unsure about — +uncertainty counts as non-trivial. + +## Guardrails + +- Never edit, skip or loosen a test to make a failure disappear. If a test looks wrong, + say so and stop. +- No unrelated changes: no dependency bumps, no refactors, no formatting sweeps. +- No secrets or real user data in the comment; redact anything sensitive from logs. +- Do not close, relabel or edit the reporter's issue, do not force-push or delete + branches, and never touch live infrastructure. +- Exactly one findings comment per run. +- Time-box the reproduction: post what you have with the gaps named rather than guessing + indefinitely. + +## Human checkpoint + +For anything above trivial, the run ends at "here is the fix I propose — confirm and I'll +do it". Resume only on an explicit go-ahead from a maintainer on the issue. diff --git a/docs/automations/prompts/weekly-drift-report.md b/docs/automations/prompts/weekly-drift-report.md new file mode 100644 index 00000000..0a13ec2e --- /dev/null +++ b/docs/automations/prompts/weekly-drift-report.md @@ -0,0 +1,92 @@ +# Prompt — Weekly drift report + +This is the exact prompt run by the "Weekly Drift Report — 3 repos" automation +(schedule: Mondays 06:00 UTC). Keep this file and the automation in sync in one PR. +Design, ranking and guardrail rationale: [../weekly-drift-report.md](../weekly-drift-report.md). + +--- + +Produce this week's dependency drift report across three repositories and publish it, +with a ranked and sized remediation queue, as a single GitHub issue. + +Repositories (clone each with the built-in git tools if it is not already on the box): + +- `Cognition-Partner-Workshops/timesheet-app` +- `Cognition-Partner-Workshops/petclinic-microservices` +- `Cognition-Partner-Workshops/timesheet-infra` + +## 1. Scan + +Run each repository's own read-only scanner on a clean checkout of `main` and keep the +JSON output: + +- `timesheet-app`: `python3 scripts/drift_scan.py --out /tmp/drift-timesheet-app.json` + (run `npm ci` in `backend/` and `frontend/` first so `npm outdated`/`npm audit` see an + installed tree; if `npm ci` fails, record it as a scan error and continue) +- `petclinic-microservices`: `python3 scripts/drift_scan.py --out /tmp/drift-petclinic.json` + (add `--skip-maven` if `./mvnw dependency:list` cannot resolve; that fallback parses the + poms directly and is recorded in `errors`) +- `timesheet-infra`: `python3 scripts/drift_scan.py --out /tmp/drift-infra.json` + +Do not modify anything: no installs that write to `package.json`, no lockfile updates, no +`pom.xml` edits, no `terraform init`/`plan`/`apply`, no `.tf` or `.terraform.lock.hcl` +edits, no source changes. Do not create branches or PRs in any of the three repositories. + +If a scanner exits non-zero or its `errors` array is non-empty, the report is **partial**. +Say so explicitly and list every error. Never present an unreachable registry or advisory +API as "no findings". + +## 2. Report + +Open one issue in `Cognition-Partner-Workshops/timesheet-app`: + +- Title: `Drift report — ` +- Label: `drift-report` (create the label if it does not exist) + +Body sections, in this order: + +1. **Scan health** — per repository: scanner status, fallbacks used, and every `errors` + entry verbatim. State plainly whether the report is complete or partial. +2. **Summary** — totals by severity, by repository, and by size (`XS`/`S`/`M`/`L`). +3. **Advisories** — every item with `advisory_count > 0`, worst severity first, with + advisory IDs, links, affected component, location, current and target version. +4. **Version drift** — dependency/plugin items grouped by repository, `current → latest` + with the major/minor/patch gap. +5. **Terraform** — for each root under `timesheet-infra/terraform/`: provider and module + constraint, lock-file pin, newest release. Call out where the declared constraint + itself blocks the newest release (`constraint_allows_latest: false`) and where no + `.terraform.lock.hcl` pins the version. +6. **Remediation queue** — see below. +7. **Raw data** — the three JSON documents in fenced blocks (truncate the `advisories` + arrays if the body would exceed GitHub's size limit, and say that you truncated). + +## 3. Remediation queue (input to the Track 2 fan-out) + +A single table, ordered by `rank_score` descending across all three repositories, with +these columns: `rank`, `repo`, `component`, `location`, `current → target`, +`advisories`, `max severity`, `size`, `blast radius`, `evidence`. + +- One row = one unit of work in one repository. Never bundle ("upgrade all Spring"). +- `target` is a concrete released version. Never `latest`, never an unbounded range, + never a prerelease. +- `blast radius`: which modules/workspaces/Terraform roots the change touches. +- `evidence`: the scanner location plus advisory IDs — enough for a worker to verify the + row without re-deriving it. +- Keep component/location identity stable week over week so recurring rows are traceable. +- Cap the table at the top 25 rows and state the total count of remaining rows. + +## Guardrails + +- Read-only across all three repositories. No PRs, no commits, no upgrades, and do not + start the fan-out yourself. +- No secrets in the issue: version metadata only — no tokens, no `.tfvars` values, no + state contents. +- Do not open per-finding issues; exactly one issue per run. +- If a scan cannot be completed, publish the partial report rather than retrying + indefinitely or skipping the report. + +## Human checkpoint + +Stop after the issue is published. The queue is a proposal: a maintainer decides which +rows get picked up and whether a Track 2 fan-out starts. End your final message with the +issue link and the top three rows. diff --git a/docs/automations/remediation-queue.schema.json b/docs/automations/remediation-queue.schema.json new file mode 100644 index 00000000..777c62ac --- /dev/null +++ b/docs/automations/remediation-queue.schema.json @@ -0,0 +1,116 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/Cognition-Partner-Workshops/timesheet-app/blob/main/docs/automations/remediation-queue.schema.json", + "title": "Drift scan document", + "description": "Output of every per-repository drift scanner (scripts/drift_scan.py in timesheet-app, petclinic-microservices and timesheet-infra). The weekly drift report concatenates these documents, sorts by rank_score, and publishes the result as the remediation queue that a Track 2 fan-out consumes.", + "type": "object", + "required": ["repo", "generated_at", "scanner", "items", "errors"], + "additionalProperties": false, + "properties": { + "repo": { + "type": "string", + "description": "owner/name of the scanned repository" + }, + "generated_at": { + "type": "string", + "format": "date-time", + "description": "UTC timestamp of the scan" + }, + "scanner": { + "type": "string", + "description": "path of the scanner inside the scanned repository" + }, + "advisory_coverage": { + "type": "string", + "description": "set when the ecosystem has no advisory feed, so an empty advisories array is not read as 'no known vulnerabilities'" + }, + "terraform_roots": { + "type": "array", + "description": "timesheet-infra only: the Terraform roots that were scanned", + "items": {"type": "string"} + }, + "items": { + "type": "array", + "items": {"$ref": "#/$defs/item"} + }, + "errors": { + "type": "array", + "description": "every failed lookup, unparseable manifest and degraded fallback. A non-empty array means the scan is incomplete and MUST be reported as such rather than treated as clean.", + "items": {"type": "string"} + } + }, + "$defs": { + "item": { + "type": "object", + "required": [ + "component", "ecosystem", "kind", "location", "current", "latest", + "exposure", "gap", "advisories", "advisory_count", "rank_score", "size" + ], + "additionalProperties": false, + "properties": { + "component": { + "type": "string", + "description": "package name (npm), group:artifact (maven), provider/module source (terraform), or 'java'/'terraform' for a runtime" + }, + "ecosystem": {"enum": ["npm", "maven", "terraform"]}, + "kind": {"enum": ["dependency", "plugin", "provider", "module", "runtime"]}, + "location": { + "type": "string", + "description": "manifest or Terraform root the finding was read from, relative to the repository root" + }, + "current": {"type": "string"}, + "latest": {"type": "string"}, + "constraint": { + "type": "string", + "description": "terraform only: the declared version constraint" + }, + "constraint_allows_latest": { + "type": "boolean", + "description": "terraform only: false means the constraint itself must be widened before the upgrade is possible" + }, + "pinned_by_lockfile": { + "type": "boolean", + "description": "terraform only: whether current came from .terraform.lock.hcl rather than from the constraint" + }, + "exposure": { + "enum": ["runtime", "dev", "test", "transitive", "plugin", "build_parent", "provider", "module", "runtime_platform"], + "description": "how the component is reachable; feeds rank_score" + }, + "gap": { + "type": "object", + "required": ["major", "minor", "patch"], + "additionalProperties": false, + "properties": { + "major": {"type": "integer", "minimum": 0}, + "minor": {"type": "integer", "minimum": 0}, + "patch": {"type": "integer", "minimum": 0} + } + }, + "advisories": { + "type": "array", + "description": "highest severity first, truncated to --max-advisories; advisory_count keeps the untruncated total", + "items": { + "type": "object", + "required": ["id", "severity", "summary", "url"], + "additionalProperties": false, + "properties": { + "id": {"type": "string"}, + "severity": {"enum": ["CRITICAL", "HIGH", "MODERATE", "MEDIUM", "LOW", "INFO"]}, + "summary": {"type": "string"}, + "url": {"type": "string"} + } + } + }, + "advisory_count": {"type": "integer", "minimum": 0}, + "rank_score": { + "type": "integer", + "description": "worst advisory severity + advisory volume + version staleness + exposure. Higher ranks first." + }, + "size": { + "enum": ["XS", "S", "M", "L"], + "description": "remediation effort: XS lockfile-only, S single manifest bump, M breaking major in one component, L high blast radius / cross-module" + } + } + } + } +} diff --git a/docs/automations/weekly-drift-report.md b/docs/automations/weekly-drift-report.md new file mode 100644 index 00000000..7239d211 --- /dev/null +++ b/docs/automations/weekly-drift-report.md @@ -0,0 +1,116 @@ +# Automation 1 — Weekly drift report + +One report per week covering three repositories, plus a ranked and sized +remediation queue that a Track 2 fan-out consumes item by item. + +- **Scope:** `petclinic-microservices`, `timesheet-app`, `timesheet-infra` +- **Prompt:** [`prompts/weekly-drift-report.md`](prompts/weekly-drift-report.md) +- **Delivery:** a single GitHub issue in `timesheet-app`, labeled `drift-report` + +## Trigger + +Schedule — **every Monday at 06:00 UTC** (`FREQ=WEEKLY;BYDAY=MO;BYHOUR=6;BYMINUTE=0`), +one run at a time. Monday morning means the queue exists before the week is planned; +serializing runs means a slow scan can never overlap the next one and produce two +competing reports. + +There is no push or PR trigger: drift is a function of *upstream* releases, not of +this week's commits, so scanning on commit would add noise without adding findings. + +## What is collected + +| Repository | Scanner | Sources | +| --- | --- | --- | +| `timesheet-app` | `scripts/drift_scan.py` | `npm outdated` for `backend/` and `frontend/`, `npm audit` for advisories | +| `petclinic-microservices` | `scripts/drift_scan.py` | `mvn dependency:list` (falls back to parsing every `pom.xml`), Maven Central `maven-metadata.xml`, OSV.dev for advisories, `` vs. newest Java LTS | +| `timesheet-infra` | `scripts/drift_scan.py` | `required_providers` and `.terraform.lock.hcl` in every root under `terraform/`, registry modules, `required_version`, Terraform Registry + HashiCorp releases API | + +Each scanner is read-only and self-contained: no installs, no `terraform init`, no +plugin that rewrites versions, no state access. + +## Normalized scanner output + +All three scanners emit the same document, validated against +[`remediation-queue.schema.json`](remediation-queue.schema.json). That is what makes +one report out of three ecosystems possible, and it is the contract the fan-out reads. + +Per item: `component`, `ecosystem`, `kind`, `location`, `current`, `latest`, +`exposure`, `gap` (major/minor/patch), `advisories` (id, severity, summary, url), +`advisory_count`, `rank_score`, `size`. Terraform items add `constraint`, +`constraint_allows_latest` and `pinned_by_lockfile`. + +### Ranking + +`rank_score` = worst advisory severity (CRITICAL 50 / HIGH 30 / MODERATE 12 / LOW 4) +\+ advisory volume (5 per extra advisory, capped at 15) +\+ staleness (8 per major behind capped at 24, minor behind capped at 6-8, +2 for patch) +\+ exposure (runtime and provider weigh more than dev/test/plugin). + +The report sorts across all three repositories by this score, so a CRITICAL advisory +in a Terraform provider outranks a cosmetic major bump in a dev dependency without +anyone arbitrating by hand. + +### Sizing + +| Size | Meaning | Rough effort | +| --- | --- | --- | +| `XS` | lockfile-only or patch bump | minutes, no review risk | +| `S` | one manifest bump inside the same major | one short session | +| `M` | breaking major in a single component | one session, needs tests read | +| `L` | high blast radius: Spring Boot parent, Spring Cloud BOM, React/Vite/TypeScript, `hashicorp/aws`, Java or Terraform itself | multiple sessions, needs a plan first | + +## Report layout + +The issue body contains, in order: + +1. **Scan health** — per repository: scanner exit state and every entry from `errors`. + A run with errors is published as *partial*, never as clean. +2. **Summary** — counts by severity, by repository, and by size. +3. **Advisories** — every item with `advisory_count > 0`, highest severity first. +4. **Version drift** — dependency and plugin items, grouped by repository. +5. **Terraform** — providers and modules per root under `timesheet-infra/terraform/`, + flagging where the constraint (`~> 5.0`) itself blocks the newest release and where + no `.terraform.lock.hcl` pins the version. +6. **Remediation queue** — the fan-out input (below). + +## Remediation queue — Track 2 fan-out contract + +The queue is a table of the top items, each row carrying everything a fan-out worker +needs to start without re-deriving anything: + +`rank` · `repo` · `component` · `location` · `current` → `target` · `advisories` +· `severity` · `size` · `blast radius` · `evidence` + +Rules that keep the queue machine-consumable: + +- One row is one unit of work in one repository — never "upgrade all Spring". +- `target` is always a concrete released version. Never `latest`, never an unbounded + range, never a prerelease. +- Rows are ordered by `rank_score` and are stable: a row that reappears next week keeps + the same component/location identity so progress is visible across reports. +- The raw JSON for all three repositories is attached to the issue as a fenced block or + gist link, so a worker can consume the queue programmatically instead of parsing prose. + +## Guardrails + +- **Read-only.** No dependency, lockfile, `pom.xml`, `package.json`, `.tf` or source + edits during a drift run. +- **No autonomous remediation.** The automation never opens upgrade PRs and never + starts the fan-out. It publishes one issue and stops. +- **No silent green.** Every failed registry/advisory lookup is reported in *Scan + health*; an unreachable API is a partial scan, not an absence of findings. +- **Evidence required.** Every queue row cites location, current version, target + version, advisory IDs and severity — no unsourced claims. +- **No `latest`, no unbounded ranges** as a recommendation. +- **One issue per run.** Reuse the label `drift-report`; do not spam per-finding issues. +- **No secrets.** Scanner output is version metadata only; tokens, state files and + `.tfvars` values never appear in the issue. +- **Bounded runtime.** If a scanner cannot finish, report the partial result rather + than retrying indefinitely. + +## Human checkpoint + +**The queue is a proposal.** The automation stops at the published issue; a maintainer +decides which rows are picked up, in what order, and whether any `L` row needs a design +discussion first. Nothing is upgraded — and no Track 2 fan-out is started — until a +maintainer says so on the issue. diff --git a/scripts/drift_scan.py b/scripts/drift_scan.py new file mode 100755 index 00000000..2571826b --- /dev/null +++ b/scripts/drift_scan.py @@ -0,0 +1,204 @@ +#!/usr/bin/env python3 +"""Dependency drift scanner for the timesheet-app npm workspaces. + +Collects, for backend/ and frontend/: + - installed vs. latest versions (npm outdated) + - known advisories (npm audit) + +Emits one normalized JSON document on stdout (or to --out) that matches +docs/automations/remediation-queue.schema.json, so the weekly drift report and +its remediation queue are reproducible instead of hand-assembled. + +Read-only: never installs, never writes to package.json or a lockfile. +""" + +import argparse +import json +import os +import re +import subprocess +import sys +from datetime import datetime, timezone + +REPO = "Cognition-Partner-Workshops/timesheet-app" +WORKSPACES = ("backend", "frontend") + +SEVERITY_POINTS = {"CRITICAL": 50, "HIGH": 30, "MODERATE": 12, "MEDIUM": 12, "LOW": 4, "INFO": 1} +EXPOSURE_POINTS = {"runtime": 10, "dev": 3} +# Components whose major bumps ripple through the whole app. +HIGH_BLAST_RADIUS = { + "react", "react-dom", "react-router-dom", "@mui/material", "vite", "typescript", + "express", "sqlite3", "jsonwebtoken", "jest", "eslint", +} + + +def run(cmd, cwd): + proc = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True) + return proc.returncode, proc.stdout, proc.stderr + + +def run_json(cmd, cwd, errors): + """npm outdated/audit exit non-zero when they have findings; only parse failures matter.""" + code, out, err = run(cmd, cwd) + if not out.strip(): + errors.append(f"{' '.join(cmd)} in {cwd}: no output (exit {code}): {err.strip()[:200]}") + return None + try: + return json.loads(out) + except json.JSONDecodeError as exc: + errors.append(f"{' '.join(cmd)} in {cwd}: unparseable output: {exc}") + return None + + +def parse_version(value): + parts = [int(p) for p in re.findall(r"\d+", value or "")[:3]] + return parts + [0] * (3 - len(parts)) + + +def version_gap(current, latest): + cur, new = parse_version(current), parse_version(latest) + return { + "major": max(0, new[0] - cur[0]), + "minor": max(0, new[1] - cur[1]) if new[0] == cur[0] else 0, + "patch": max(0, new[2] - cur[2]) if new[:2] == cur[:2] else 0, + } + + +def rank_score(item): + severities = [SEVERITY_POINTS.get(a["severity"].upper(), 4) for a in item["advisories"]] + severity = max(severities) if severities else 0 + volume = min(15, 5 * max(0, len(item["advisories"]) - 1)) + gap = item["gap"] + staleness = min(24, 8 * gap["major"]) + min(6, gap["minor"]) + (2 if gap["patch"] else 0) + return severity + volume + staleness + EXPOSURE_POINTS.get(item["exposure"], 5) + + +def size_estimate(item): + gap = item["gap"] + if gap["major"] >= 1: + return "L" if item["component"] in HIGH_BLAST_RADIUS else "M" + if gap["minor"] >= 1: + return "S" + # No version gap but an advisory still needs a lockfile bump or an override. + return "S" if item["advisories"] else "XS" + + +def collect_advisories(cwd, errors): + """Map package name -> advisories, from `npm audit --json`.""" + report = run_json(["npm", "audit", "--json"], cwd, errors) + advisories = {} + if not report: + return advisories + for name, vuln in (report.get("vulnerabilities") or {}).items(): + entries = [] + for via in vuln.get("via") or []: + if isinstance(via, dict): + entries.append({ + "id": str(via.get("source") or via.get("url") or "unknown"), + "severity": (via.get("severity") or vuln.get("severity") or "low").upper(), + "summary": via.get("title") or "", + "url": via.get("url") or "", + }) + if not entries: + entries.append({ + "id": f"npm-audit:{name}", + "severity": (vuln.get("severity") or "low").upper(), + "summary": f"transitive advisory via {', '.join(str(v) for v in vuln.get('via') or [])}", + "url": "", + }) + advisories[name] = entries + return advisories + + +def declared_exposure(manifest, name): + if name in (manifest.get("dependencies") or {}): + return "runtime" + if name in (manifest.get("devDependencies") or {}): + return "dev" + return "transitive" + + +def scan_workspace(workspace, root, errors): + cwd = os.path.join(root, workspace) + manifest_path = os.path.join(cwd, "package.json") + if not os.path.isfile(manifest_path): + errors.append(f"{workspace}: no package.json") + return [] + with open(manifest_path) as handle: + manifest = json.load(handle) + + outdated = run_json(["npm", "outdated", "--json", "--long"], cwd, errors) or {} + advisories = collect_advisories(cwd, errors) + + items = [] + for name, info in outdated.items(): + current = info.get("current") or info.get("wanted") or "" + latest = info.get("latest") or "" + items.append({ + "component": name, + "ecosystem": "npm", + "kind": "dependency", + "location": f"{workspace}/package.json", + "current": current, + "latest": latest, + "exposure": declared_exposure(manifest, name), + "gap": version_gap(current, latest), + "advisories": advisories.pop(name, []), + }) + + # Advisories on packages npm outdated did not report (already latest, or transitive). + for name, entries in advisories.items(): + items.append({ + "component": name, + "ecosystem": "npm", + "kind": "dependency", + "location": f"{workspace}/package-lock.json", + "current": "installed", + "latest": "see advisory", + "exposure": declared_exposure(manifest, name), + "gap": {"major": 0, "minor": 0, "patch": 0}, + "advisories": entries, + }) + return items + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--out", help="write JSON here instead of stdout") + parser.add_argument("--root", default=os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")) + parser.add_argument("--max-advisories", type=int, default=5, + help="advisories kept per component (highest severity first); the rest are counted only") + args = parser.parse_args() + + root = os.path.abspath(args.root) + errors = [] + items = [] + for workspace in WORKSPACES: + items.extend(scan_workspace(workspace, root, errors)) + + for item in items: + item["rank_score"] = rank_score(item) + item["size"] = size_estimate(item) + item["advisory_count"] = len(item["advisories"]) + item["advisories"].sort(key=lambda a: -SEVERITY_POINTS.get(a["severity"].upper(), 4)) + del item["advisories"][args.max_advisories:] + items.sort(key=lambda i: (-i["rank_score"], i["size"], i["component"])) + + document = { + "repo": REPO, + "generated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "scanner": "scripts/drift_scan.py", + "items": items, + "errors": errors, + } + payload = json.dumps(document, indent=2) + if args.out: + with open(args.out, "w") as handle: + handle.write(payload + "\n") + else: + print(payload) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From ba95966a5e75199363719e1ea444d184753127fe Mon Sep 17 00:00:00 2001 From: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Mon, 24 Aug 2026 13:01:08 +0000 Subject: [PATCH 2/3] Validate scanner paths against an allowed base and simplify advisory parsing --- scripts/drift_scan.py | 79 +++++++++++++++++++++++++++++-------------- 1 file changed, 54 insertions(+), 25 deletions(-) diff --git a/scripts/drift_scan.py b/scripts/drift_scan.py index 2571826b..4a2f073c 100755 --- a/scripts/drift_scan.py +++ b/scripts/drift_scan.py @@ -18,6 +18,7 @@ import re import subprocess import sys +import tempfile from datetime import datetime, timezone REPO = "Cognition-Partner-Workshops/timesheet-app" @@ -32,6 +33,28 @@ } +def resolve_within(base, *parts): + """Path under base, or None when the joined path escapes it (--root is user input).""" + base = os.path.realpath(base) + candidate = os.path.realpath(os.path.join(base, *parts)) + return candidate if candidate == base or candidate.startswith(base + os.sep) else None + + +def output_bases(): + """Directories --out may write into: the working tree and the temp directories.""" + candidates = [os.getcwd(), tempfile.gettempdir(), "/tmp"] + return tuple(sorted({os.path.realpath(c) for c in candidates if os.path.isdir(c)})) + + +def resolve_output(path): + """--out is user input, so it is validated against output_bases() before opening.""" + candidate = os.path.realpath(path) + bases = output_bases() + if any(candidate.startswith(base + os.sep) for base in bases): + return candidate + raise SystemExit(f"--out must be inside one of {', '.join(bases)}: {path}") + + def run(cmd, cwd): proc = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True) return proc.returncode, proc.stdout, proc.stderr @@ -83,31 +106,37 @@ def size_estimate(item): return "S" if item["advisories"] else "XS" +def advisory_entries(name, vuln): + """npm audit lists direct advisories as objects in `via`; transitive ones as names.""" + entries = [ + { + "id": str(via.get("source") or via.get("url") or "unknown"), + "severity": (via.get("severity") or vuln.get("severity") or "low").upper(), + "summary": via.get("title") or "", + "url": via.get("url") or "", + } + for via in vuln.get("via") or [] if isinstance(via, dict) + ] + if entries: + return entries + chain = ", ".join(str(via) for via in vuln.get("via") or []) + return [{ + "id": f"npm-audit:{name}", + "severity": (vuln.get("severity") or "low").upper(), + "summary": f"transitive advisory via {chain}", + "url": "", + }] + + def collect_advisories(cwd, errors): """Map package name -> advisories, from `npm audit --json`.""" report = run_json(["npm", "audit", "--json"], cwd, errors) - advisories = {} if not report: - return advisories - for name, vuln in (report.get("vulnerabilities") or {}).items(): - entries = [] - for via in vuln.get("via") or []: - if isinstance(via, dict): - entries.append({ - "id": str(via.get("source") or via.get("url") or "unknown"), - "severity": (via.get("severity") or vuln.get("severity") or "low").upper(), - "summary": via.get("title") or "", - "url": via.get("url") or "", - }) - if not entries: - entries.append({ - "id": f"npm-audit:{name}", - "severity": (vuln.get("severity") or "low").upper(), - "summary": f"transitive advisory via {', '.join(str(v) for v in vuln.get('via') or [])}", - "url": "", - }) - advisories[name] = entries - return advisories + return {} + return { + name: advisory_entries(name, vuln) + for name, vuln in (report.get("vulnerabilities") or {}).items() + } def declared_exposure(manifest, name): @@ -119,9 +148,9 @@ def declared_exposure(manifest, name): def scan_workspace(workspace, root, errors): - cwd = os.path.join(root, workspace) - manifest_path = os.path.join(cwd, "package.json") - if not os.path.isfile(manifest_path): + cwd = resolve_within(root, workspace) + manifest_path = resolve_within(root, workspace, "package.json") if cwd else None + if not manifest_path or not os.path.isfile(manifest_path): errors.append(f"{workspace}: no package.json") return [] with open(manifest_path) as handle: @@ -193,7 +222,7 @@ def main(): } payload = json.dumps(document, indent=2) if args.out: - with open(args.out, "w") as handle: + with open(resolve_output(args.out), "w") as handle: handle.write(payload + "\n") else: print(payload) From 4377bc25b42e4feac52d21aa93e724fe6cb24ced Mon Sep 17 00:00:00 2001 From: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Mon, 24 Aug 2026 13:02:59 +0000 Subject: [PATCH 3/3] Restrict scanner output paths to the working tree or platform temp dir --- docs/automations/prompts/weekly-drift-report.md | 6 +++--- scripts/drift_scan.py | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/automations/prompts/weekly-drift-report.md b/docs/automations/prompts/weekly-drift-report.md index 0a13ec2e..fe4aab0b 100644 --- a/docs/automations/prompts/weekly-drift-report.md +++ b/docs/automations/prompts/weekly-drift-report.md @@ -20,13 +20,13 @@ Repositories (clone each with the built-in git tools if it is not already on the Run each repository's own read-only scanner on a clean checkout of `main` and keep the JSON output: -- `timesheet-app`: `python3 scripts/drift_scan.py --out /tmp/drift-timesheet-app.json` +- `timesheet-app`: `python3 scripts/drift_scan.py --out drift-timesheet-app.json` (run `npm ci` in `backend/` and `frontend/` first so `npm outdated`/`npm audit` see an installed tree; if `npm ci` fails, record it as a scan error and continue) -- `petclinic-microservices`: `python3 scripts/drift_scan.py --out /tmp/drift-petclinic.json` +- `petclinic-microservices`: `python3 scripts/drift_scan.py --out drift-petclinic.json` (add `--skip-maven` if `./mvnw dependency:list` cannot resolve; that fallback parses the poms directly and is recorded in `errors`) -- `timesheet-infra`: `python3 scripts/drift_scan.py --out /tmp/drift-infra.json` +- `timesheet-infra`: `python3 scripts/drift_scan.py --out drift-infra.json` Do not modify anything: no installs that write to `package.json`, no lockfile updates, no `pom.xml` edits, no `terraform init`/`plan`/`apply`, no `.tf` or `.terraform.lock.hcl` diff --git a/scripts/drift_scan.py b/scripts/drift_scan.py index 4a2f073c..3e388d76 100755 --- a/scripts/drift_scan.py +++ b/scripts/drift_scan.py @@ -41,8 +41,8 @@ def resolve_within(base, *parts): def output_bases(): - """Directories --out may write into: the working tree and the temp directories.""" - candidates = [os.getcwd(), tempfile.gettempdir(), "/tmp"] + """Directories --out may write into: the working tree and the platform temp directory.""" + candidates = [os.getcwd(), tempfile.gettempdir()] return tuple(sorted({os.path.realpath(c) for c in candidates if os.path.isdir(c)}))