Skip to content

Fix #646: make Stage 2 publish by command, and fail the job when it doesn't #649

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
johanzander merged 2 commits into from
Aug 18, 2026
Merged

Fix #646: make Stage 2 publish by command, and fail the job when it doesn't #649

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b087f5c
fix: make Stage 2 publish by command, and fail the job when it doesn't
johanzander Aug 18, 2026
b890c1d
Merge remote-tracking branch 'origin/main' into fix/issue-646-stage2-…
johanzander Aug 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 113 additions & 16 deletions .github/workflows/issue-analyze.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,13 @@ jobs:
app-id: ${{ secrets.CLAUDE_REVIEWER_APP_ID }}
private-key: ${{ secrets.CLAUDE_REVIEWER_PRIVATE_KEY }}

# Marks the point the agent started, so the verification step below can
# tell a comment IT posted from the `@claude-bot analyze` trigger comment
# that started this run (which is always already present).
- name: Mark analysis start
id: started
run: echo "at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$GITHUB_OUTPUT"

- uses: anthropics/claude-code-action@v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
Expand Down Expand Up @@ -109,40 +116,65 @@ jobs:
the reporter's), file:line references, and whether the
debug bundle reveals a different root cause than claimed."

3. After the sub-agent reports back, **independently verify** by
4. After the sub-agent reports back, **independently verify** by
reading the file:line locations it cited. Do not just trust
the summary — quote the actual code.

4. Post ONE comment on the issue with this structure:
5. **Publish the analysis by RUNNING these two commands.** Writing
the diagnosis as your reply does NOT publish it — you are running
headless, nothing is watching your output, and a final message
reaches nobody. The issue only changes if `gh` changes it.

Write the body to a file first, then post it. Use a file rather
than `-b` because the body contains backticks, quotes and
newlines that do not survive shell quoting:

cat > /tmp/analysis.md <<'ANALYSIS_EOF'
## Root cause
<one-paragraph plain-English explanation>

## Evidence
- `path/to/file.py:LINE` — <quoted code excerpt>
- <additional file:line refs as needed>

## Root cause
<one-paragraph plain-English explanation>
## Proposed fix
<high-level approach in 2-4 sentences — NO code yet>

## Evidence
- `path/to/file.py:LINE` — <quoted code excerpt>
- <additional file:line refs as needed>
## Risks / open questions
<anything you couldn't determine, edge cases, or test gaps>

## Proposed fix
<high-level approach in 2-4 sentences — NO code yet>
---
Reply `@claude-bot fix` to implement and open a draft PR.
ANALYSIS_EOF

## Risks / open questions
<anything you couldn't determine, edge cases, or test gaps>
gh issue comment ${{ github.event.issue.number }} --body-file /tmp/analysis.md

---
Reply `@claude-bot fix` to implement and open a draft PR.
Check that the command succeeded. If it failed, fix the problem
and run it again — do not proceed to step 6 with the comment
unposted, and do not end the run having only described it.

5. Update labels:
6. Update labels:
gh issue edit ${{ github.event.issue.number }} --add-label analyzed --remove-label ready-for-analysis

──────────────────────────────────────────────────────────────────
IF YOU CAN'T REACH A CONCLUSION
──────────────────────────────────────────────────────────────────
Post what you found, what you read, and what specific information
is still missing. Then:
Publish what you found, what you read, and what specific
information is still missing — again by RUNNING the commands, not
by describing the outcome:

cat > /tmp/analysis.md <<'ANALYSIS_EOF'
## Inconclusive
<what you read, what you ruled out, what is still missing>
ANALYSIS_EOF

gh issue comment ${{ github.event.issue.number }} --body-file /tmp/analysis.md
gh issue edit ${{ github.event.issue.number }} --add-label needs-human-review

Be honest — "I couldn't find the bug in the current code" is a
valid outcome. Do not invent a root cause to fill the template.
An inconclusive comment that is actually posted is worth far more
than a confident one that is not.

──────────────────────────────────────────────────────────────────
HARD CONSTRAINTS
Expand All @@ -151,4 +183,69 @@ jobs:
- DO NOT open a PR.
- DO NOT skip the bess-analyst delegation step — it's the whole
point of this stage.
- DO NOT end the run without having RUN `gh issue comment`. A
diagnosis you only wrote out is a diagnosis nobody receives,
and the job will fail the verification step below.
- Quote real code. Don't paraphrase.

# The acceptance criterion of #646: a Stage 2 run must not be able to
# exit `success` having written nothing to the issue.
#
# Five of nine non-skipped runs did exactly that over three days, at
# $0.33-1.58 each, and every signal said they were healthy -- green run,
# is_error false, permission_denials_count 0. The `ready-for-analysis`
# label staying put is indistinguishable from "nobody has run analyze
# yet", so the backlog pass could not see it either. On #593 the
# maintainer eventually wrote the diagnosis by hand: the stage was paid
# for and the work was done twice.
#
# `if: always()` is load-bearing -- the whole point is to catch the run
# that the agent step already reported as successful.
#
# This step ASSERTS; it deliberately does not repair. Posting the comment
# on the agent's behalf would be a second publishing path whose only job
# is to route around the first one failing, which masks the regression
# instead of surfacing it (docs/agents/rules.md, Debugging Protocol
# step 8). A loud red run is the product here.
- name: Verify the analysis reached the issue
if: always()
env:
GH_TOKEN: ${{ steps.app-token.outputs.token }}
ISSUE: ${{ github.event.issue.number }}
SINCE: ${{ steps.started.outputs.at }}
run: |
set -euo pipefail

posted=$(gh issue view "$ISSUE" --json comments \
--jq "[.comments[] | select(.createdAt > \"$SINCE\")] | length")
labels=$(gh issue view "$ISSUE" --json labels --jq '[.labels[].name] | join(",")')

echo "Comments posted since ${SINCE}: ${posted}"
echo "Labels now: ${labels:-<none>}"

failed=0

if [ "$posted" -eq 0 ]; then
echo "::error::Stage 2 posted NO comment on issue #${ISSUE}. The run"
echo "::error::billed for an analysis that reached nobody. Do not re-fire"
echo "::error::blindly -- check whether the agent described the comment"
echo "::error::instead of running 'gh issue comment' (see #646)."
failed=1
fi

case ",$labels," in
*,analyzed,*|*,needs-human-review,*) ;;
*)
echo "::error::Stage 2 applied neither 'analyzed' nor"
echo "::error::'needs-human-review' to issue #${ISSUE}. Without one of"
echo "::error::them the issue is indistinguishable from one that was"
echo "::error::never analysed, and the backlog pass will skip it."
failed=1
;;
esac

if [ "$failed" -eq 1 ]; then
exit 1
fi

echo "✅ Analysis published: ${posted} new comment(s), labels: ${labels}"
3 changes: 3 additions & 0 deletions docs/agents/workflow.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ Issue opened/edited → issue-triage.yml [auto, ~$0.05]
→ delegates to `bess-analyst` sub-agent
→ posts Root cause / Evidence / Proposed fix
→ label `analyzed` (or `needs-human-review`)
→ job FAILS if it posted no comment or applied
neither label — a run that analysed nothing
must not report success (#646)

@claude-bot fix → issue-fix.yml [manual, ~$1–4]
→ reads Stage 2 diagnosis comment
Expand Down
92 changes: 92 additions & 0 deletions scripts/quality-check.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -420,6 +420,98 @@ then
ERRORS=$((ERRORS + 1))
fi

echo ""
echo "📋 Checking bot workflow publish contract..."
echo "-------------------------------------------"

# Stage 2 (issue-analyze.yml) spent three days exiting `success` while posting
# nothing -- five of nine non-skipped runs produced no comment and no label,
# burning $0.33-1.58 each (#646). Every layer reported healthy: the run was
# green, is_error false, permission_denials_count 0.
#
# Two independent things have to hold, and this gate checks BOTH because either
# one alone leaves the failure silent:
#
# 1. The prompt must tell the agent to RUN a command to publish. Stage 2 said
# "Post ONE comment on the issue with this structure:" followed by a
# markdown template -- a description of a document, not an instruction to
# execute anything. These workflows run in AGENT mode (use_sticky_comment
# false, track_progress false), so the action posts nothing on the agent's
# behalf: an agent that renders the template as its final assistant message
# has, by its own lights, finished. It then never reaches the labelling
# step, which is why the label was untouched too.
#
# The contrast is what makes this more than a story. On the SAME action
# version (459ad358 / CLI 2.1.234), Stage 4's review -- whose prompt names
# `gh pr review` explicitly -- landed an APPROVED verdict, while Stage 2
# went silent three times in three seconds. Stage 2 was the only bot
# workflow whose publish step was prose, and the only one that failed to
# publish.
#
# 2. The workflow must VERIFY it afterwards. The prompt fix is a behavioural
# argument about what a model will infer, so it cannot be trusted on its
# own -- the next upstream version may infer differently, exactly as this
# one did. Only a post-condition on the job makes that loud instead of
# green.
#
# Deliberately NOT checked for, and deliberately not built: a step that posts
# the comment on the agent's behalf when it didn't. That is a second publishing
# path whose only job is to route around the first one failing, and it would
# mask the regression rather than surface it (docs/agents/rules.md, Debugging
# Protocol step 8). The guard asserts; it does not repair.
if ! python3 - <<'PY'
import pathlib, re, sys

WORKFLOWS = pathlib.Path(".github/workflows")

# Workflows whose whole product is a comment on the issue. Stage 3 is excluded
# on purpose: its product is a PR, which is observable without this check.
PUBLISHERS = {
"issue-triage.yml": "gh issue comment",
"issue-analyze.yml": "gh issue comment",
}

bad = []

for name, command in PUBLISHERS.items():
path = WORKFLOWS / name
if not path.is_file():
print(f"❌ {name} is missing -- the publish contract cannot be checked")
bad.append(name)
continue
if command not in path.read_text():
print(
f"❌ {name} never names `{command}`. Its publish step is prose, so "
"the agent can render it as a final message and exit success "
"having written nothing to GitHub (#646)."
)
bad.append(name)

# The post-agent guard: a run that published nothing must fail the job. Checked
# by the shape that actually does the work -- a step keyed on `if: always()`
# that re-reads the issue -- rather than by step name, which renames freely.
analyze = WORKFLOWS / "issue-analyze.yml"
if analyze.is_file():
text = analyze.read_text()
has_always = re.search(r"^\s*if:\s*always\(\)\s*$", text, re.MULTILINE)
has_readback = "--json comments" in text and "--json labels" in text
if not (has_always and has_readback):
print(
"❌ issue-analyze.yml has no post-agent verification step. A run "
"that posts no comment and applies no label must FAIL the job, "
"not exit success (#646 acceptance criterion)."
)
bad.append("issue-analyze.yml:guard")

if bad:
sys.exit(1)
print(f"✅ Bot workflow publish contract intact ({len(PUBLISHERS)} publishers "
"name their command, Stage 2 verifies it posted)")
PY
then
ERRORS=$((ERRORS + 1))
fi

echo ""
echo "📋 Checking scenario discovery coverage..."
echo "-------------------------------------------"
Expand Down
Loading