[ci] Autonomous CI flake triage - #1737
Conversation
Deploy previewBundle sizeTotal Size Change: 0B(0.00%) - Total Gzip Change: 0B(0.00%) Show details for 69 more bundles@mui/internal-docs-infra/abstractCreateDemo parsed: 0B(0.00%) gzip: 0B(0.00%) PerformanceTotal duration: 17.03 ms ▼-4.44 ms(-20.7%) | Renders: 5 (+0) | Paint: 58.54 ms ▼-18.71 ms(-24.2%)
6 tests within noise — details Check out the code infra dashboard for more information about this PR. |
Fetch CircleCI data in a workflow step so the token never coexists with the agent, narrow the Bash allowlist, and split publishing so a git/gh failure can't swallow the week's report.
Skip the agent entirely on a green week, check for an open PR before paying for a clone, and pass the CircleCI token on argv instead of staging it on disk. Teach the skill a pre-fetched mode so the prompt stops overriding it.
A week whose workflows failed without producing a failed job was published as '0/N failed'; report the real counts and flag it as CircleCI-side. Bound the agent with a step timeout, since a job timeout cancels the publish steps.
The workflow was parsing summary.txt and re-rendering the skill's report header in bash, duplicating knowledge only fetch.mjs has. It now writes report.md itself for all four no-failed-jobs cases; the workflow just checks for it.
|
@claude review |
This comment was marked as outdated.
This comment was marked as outdated.
summary.txt and report.md become one result.json: a status of issues/clean/ no-job-failures/no-data, the totals, and either the job list or the finished report. Consumers read one field instead of parsing text.
The jobs array grew with the corpus (~300B per failure, ~60KB at the --max ceiling) and nothing read it: the workflow uses status/report and the skill greps the job files. Counts stay, the list goes.
Each job read is ~4KB per failed step and persists for the rest of the session, with the loop running until every job matched. Stop at 12 markers (the report can't show more) and read via a subagent past ~20 jobs.
The >20-file threshold left the orchestrator carrying every log it read on a small window, and it carries them through the fix phase too. Delegate unconditionally so its context is flat in the number of classes.
The description claimed repo inference that automation does not use. The two CIRCLECI_TOKEN tests were the same condition written twice.
The hook existed because a hard stop used to lose the run's report. The publish steps now run on !cancelled() and the report is written first, so hitting the cap costs the week's fix rather than the week.
The step was 45 lines of bash building a --token flag, mapping exit codes and parsing result.json. The fetcher now emits its own ::notice::/::error:: and classify output under GITHUB_ACTIONS, so the step is one command. Output is flat, so report.md has one path whoever wrote it.
Nothing parsed it: the report is read from report.md and the skip decision from the classify step output, so the field was a second copy of the report that no consumer touched.
The Actions check was at the call sites, which meant the verdict line and the auth error were suppressed entirely outside CI rather than just unprefixed. Only the formatting is environment-specific.
A report-only week caused by an open fix PR said nothing about which PR, so the comment gave no reason for the missing fix. The preflight knew the number and discarded it.
Restore the warning severity for no-data/no-job-failures, tell a workflow-filter miss apart from an empty window, verify a subagent's marker before trusting it, and stop the tracking issue skipping a week when the fetcher fails.
Count error/unauthorized workflows as failures, tolerate transient API errors instead of losing the week, name the no-work reason at each call site rather than inferring it, surface the --max cap, refuse a corpus with no log text, and keep a fix when only the report file is missing.
Replaces every gh CLI call with Octokit: no --json/--jq parsing, core.setOutput instead of appending to GITHUB_OUTPUT, and try/catch around label creation. The PR step splits so git stays in bash and the API calls do not.
Truncate step logs at fetch time rather than at use, retry rate limits instead of dropping the window, resolve three contradictory rules for marker overlap into one counting formula, and drop a dead step output.
The skill is only ever wanted at mui-public's tip; a pinned ref was an option nobody needs.
|
@claude review |
This comment was marked as outdated.
This comment was marked as outdated.
The Bash matcher strips a bare `xargs` and matches the inner command, and stops stripping at the first flag. So `Bash(xargs grep:*)` matched neither form the skill runs: `xargs grep -m1` resolves to `grep` and was already covered, while the counting step's `xargs -r grep` is matched as an `xargs` command and was not covered at all.
Janpot
added
the
scope: code-infra
`Edit` and `Write` are separate tools with separate rules: an `Edit` grant does not authorize `Write`, and an `Edit(<path>)` deny does not restrain it. The report, PR title and PR body are all new files, so every run would have failed to produce any of them and published "No report this week" instead. Found by running the skill end-to-end against these exact lists. Deny `Write` on the same two paths as `Edit`, since the grant would otherwise leave `.github/workflows/**` reachable, and warn in the skill that a leading `OUT=` assignment voids approval for the rest of a compound command.
A path-scoped Edit(<path>) does authorize the Write tool for that path, which is why claude-review.yml works with one writable file; a bare Edit does not, and Write(<path>) grants nothing on the allow list. The previous note flattened that into "Edit never authorizes Write", which reads as a bug in claude-review.yml where there isn't one.
A report-only run told the agent to write report.md and, two lines later, not to edit any file. Verified in CI: the agent completed 15 turns with no permission denials and wrote nothing, so the run published "No report this week" — the one output a report-only run exists to produce.
Both cost a working run and neither is visible from the failure. A repo created after 2026-07-15 presents an immutable OIDC subject with numeric owner and repo IDs, so the federation rule needs that form; and the token exchange happens inside the CLI, not the action, so a rejected subject produces a step with no error text at all.
Every fix-proposing run logs permission denials. Extracting them shows two kinds: compound commands and shell variables, which the matcher refuses on shape whatever they contain, and attempts to read the CircleCI token file, which must stay refused. Neither wants a new allowlist entry, so tell the agent about the first and correct the token comment, which claimed cat was not path-confined.
Replaces the allowlist-as-security model with isolation: the agent runs --dangerously-skip-permissions in a container holding no credentials. A sidecar (proxy.mjs) on the same docker network mints and exchanges the Anthropic WIF token and injects it; the CircleCI token is spent by a trusted host step before the container starts; the GitHub write token lives only in a separate publish job that trusts nothing from the sandbox except a patch it re-vets (inspect-patch.mjs). A compromised agent can waste model spend and propose a bad diff to a draft PR, nothing more. The payload (fetch.mjs, proxy.mjs, bootstrap.sh, prompt.txt, inspect-patch.mjs) lives tested under .github/sandbox/ and is inlined into the workflow as base64 by generate.mjs, so a caller pinning the workflow by SHA gets the exact code that runs. The interactive circleci-why-flaky skill is removed; its methodology folds into the agent prompt and on-demand runs use workflow_dispatch.
Janpot
changed the title
Drops the base64-inlined container sandbox for a plainer design. Three jobs each hold one credential class and pass work via artifacts: fetch (CircleCI token) → triage (id-token only, runs claude-code-action --dangerously-skip-permissions) → publish (GitHub write token, re-vets the patch). Isolation comes from the job split, not a tool allowlist, so the agent can run unrestricted with nothing to steal or push. Host-run, no container — acceptable for a public repo on an ephemeral runner. The two scripts (fetch, patch inspection) become SHA-pinnable composite actions under .github/actions/, keeping them in their own tested files with no base64. Removes the hand-rolled proxy sidecar, bootstrap, and generate.mjs.
Janpot
changed the title
Callers run the workflow via workflow_call and pin it by SHA; this file in turn pins its two internal composite actions by SHA (a reusable workflow can't reference its own SHA, so it's a separate pin bumped on release). Everything is SHA-pinned end to end.
Replaces the two flake-fetch/flake-publish composite actions with plain scripts under .github/flake-fix/, fetched by the jobs that need them via a pinned checkout of mui-public (FLAKE_SRC_REF). Only fetch and publish check out; triage uses claude-code-action. The publish orchestration (inspect/apply/PR/issue) is now visible in the workflow rather than hidden in an action — better for reviewing what runs with the write token.
Everything SHA-pinned end to end: callers pin the workflow; the workflow pins the script checkout to the commit where .github/flake-fix/ is final.
…rompt Adds a circleci-branch input so a caller can triage another project's CircleCI while editing its own tree, and injects a report-only instruction into the agent prompt so a report-only run skips the fix step instead of doing wasted work.
Rewrite the issue body each run (Renovate Dependency Dashboard style) instead of appending a weekly comment: one always-current view of what's flaky, quieter for subscribers. History lives in the Actions log and closed fix PRs.
The triage report becomes the dashboard issue body, so steer it terse and jargon-free: one or two plain lines per problem, one example link, a one-line takeaway — no error codes, logs, or exhaustive job lists.
checkout v4.2.2 -> v7.0.1 (the version already used elsewhere in the repo) and upload-artifact v4.6.2 -> v5.0.0, pairing with the download-artifact v5 already in use. Clears the Node 20 deprecation warning.
Fixes the dashboard formatting: single-line bullets (GitHub renders stray newlines as breaks), markdown [example](url) links, no agent-written title (the issue title serves that) or job-count subtitle. The workflow now owns a one-line purpose intro and a footer that says the issue is kept current by the action and must not be edited.
|
@claude review |
This comment was marked as resolved.
This comment was marked as resolved.
Workflow (Bugs 1-6, Docs-1): - Bug 2: drop FLAKE_SRC_REF; a resolve-ref job reads the run's referenced_workflows to get the exact mui-public commit the workflow was called at (fallback github.sha), so the scripts always match the workflow version with nothing to hand-maintain. - Bug 3: the fetch step's inputs go through env, not interpolated into the shell. - Bug 5: the publish precheck now also requires pr-title.txt. - Bug 6: the agent stages its fix and writes fix.patch itself; the git add -A capture is gone, so install/test byproducts can't leak into the patch. - Bug 1 / Docs-1: header restated plainly and accurately — the real containment is environment-gated secrets + branch-secretless CI, not the guard; caller precondition documented. Scripts: - Bug 1: the guard also blocks .github/actions/**, and its message states policy (Docs-2). Parses with core.quotePath=false. Refactored to export inspectPatch / destinationPath so they can be unit-tested. - Simplifications: PAD_WIDTH constant; --max/maxJobs renamed to --max-workflows/ maxWorkflows to match what it caps. Tests (Tests 1-3): converted to vitest and registered .github/flake-fix as a vitest project so pnpm test runs them; added the missing guard cases (composite-action edit, rename into .github/actions, .git write, MAX_FILES) and a direct destinationPath unit test; tightened the oversized-patch assertion. Dropped the node:test eslint override.
inspect-patch.mjs is a CLI that prints its result to stdout; disable no-console in the file rather than carving out .github/flake-fix in the shared eslint config. fetch.mjs needs nothing (it only uses console.error).
Janpot
changed the title
…regex util.stripVTControlCharacters handles OSC sequences (e.g. terminal hyperlinks) that the old CSI-only regex missed, and drops the no-control-regex eslint-disable. Built-in, no dependency.
Per review discussion, the safety is that a fix branch's CI holds no important secrets and a person reviews the draft PR — not path restrictions. So: - The patch may change any file. inspect-patch.mjs drops the path/rename rules and only checks the patch is valid and not a runaway diff (size caps). destinationPath and its tests are gone. - The publish job no longer applies the patch into the working copy it operates in with the write token. It builds the fix commit through a throwaway git index (read-tree, apply --cached, write-tree, commit-tree), so the changed files are never written to disk and nothing in the patch can run; the token is used only on the final push, and the target checkout uses persist-credentials: false.
Its validity check was redundant with the publish job's git apply --cached --check, and with path rules gone its only unique job was a size cap — now a one-line file-count check inline. The publish job no longer checks out any script, so it drops the scripts checkout and its resolve-ref dependency.
1 participant
Weekly CircleCI flake triage that runs Claude fully autonomously (
--dangerously-skip-permissions) with nothing to leak or push.Four jobs, each holding one credential and passing work through artifacts. fetch has the CircleCI token and downloads the failure logs. triage has only the Anthropic identity token and runs the agent unrestricted. publish has the GitHub write token: it builds the agent's patch into a commit through a throwaway git index — so nothing in the patch runs next to the token — then opens a draft PR and updates a rolling tracking issue. A small resolve-ref job finds the exact mui-public commit so
fetch/publishcan check out the fetcher script.The safety is the job split, not a tool allowlist: triage holds no write token and no CircleCI token, so an unrestricted or prompt-injected agent has nothing to steal and nothing to push with. The fix branch's CI reaches no important secrets (those sit behind protected environments), and a person reviews the draft PR — CI on that PR is the real verification.
Exercised end-to-end from a test repo against material-ui's CircleCI: the cross-repo call, the WIF exchange, the agent run, and the publish path all work.