fix: make Stage 2 publish by command, and fail the job when it doesn't (#649)

Stage 2 (issue-analyze.yml) exited `success` while posting nothing on
five of nine non-skipped runs over three days, billing $0.33-1.58 each.
Every signal said the run was healthy: green job, is_error false,
permission_denials_count 0. The `ready-for-analysis` label staying put
is indistinguishable from "nobody has run analyze yet", so the backlog
pass could not see it either. On #593 the maintainer eventually wrote
the diagnosis by hand -- the stage was paid for and the work done twice.

The prompt never told the agent to RUN anything to publish. Step 4 read
"Post ONE comment on the issue with this structure:" followed by a
markdown template -- a description of a document, not an instruction to
execute a command. These workflows run in agent mode
(use_sticky_comment false, track_progress false), so the action posts
nothing on the agent's behalf and creates no tracking comment. An agent
that renders the template as its final assistant message has, by its own
lights, finished; it then never reaches the labelling step, which is why
the label was untouched too.

The contrast is what identifies it. On the same action version
(459ad358 / CLI 2.1.234), Stage 4 -- whose prompt names `gh pr review`
explicitly -- landed an APPROVED verdict, while Stage 2 went silent three
times within three seconds. Stage 2 was the only bot workflow whose
publish step was prose, and the only one that failed to publish.

So publishing is now a literal `gh issue comment --body-file` command on
both the conclusive and inconclusive paths, and PROCESS is renumbered
(it had two steps numbered 3, so "am I done with step 3" was satisfiable
by the wrong one).

That fix is a behavioural argument about what a model will infer, so it
is not trusted on its own: a post-agent step re-reads the issue and fails
the job when no comment was posted or neither label applied. It asserts
and deliberately does not repair -- posting on the agent's behalf would
be a second publishing path routing around the first one failing, which
masks the regression instead of surfacing it.

quality-check.sh gains a matching static gate so neither half can be
removed silently.

Closes #646

Author: johanzander

.github/workflows/issue-analyze.yml

@@ -35,6 +35,13 @@ jobs:
           app-id: ${{ secrets.CLAUDE_REVIEWER_APP_ID }}
           private-key: ${{ secrets.CLAUDE_REVIEWER_PRIVATE_KEY }}
 
+      # Marks the point the agent started, so the verification step below can
+      # tell a comment IT posted from the `@claude-bot analyze` trigger comment
+      # that started this run (which is always already present).
+      - name: Mark analysis start
+        id: started
+        run: echo "at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$GITHUB_OUTPUT"
+
       - uses: anthropics/claude-code-action@v1
         with:
           claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
@@ -109,40 +116,65 @@ jobs:
                    the reporter's), file:line references, and whether the
                    debug bundle reveals a different root cause than claimed."
 
-            3. After the sub-agent reports back, **independently verify** by
+            4. After the sub-agent reports back, **independently verify** by
                reading the file:line locations it cited. Do not just trust
                the summary — quote the actual code.
 
-            4. Post ONE comment on the issue with this structure:
+            5. **Publish the analysis by RUNNING these two commands.** Writing
+               the diagnosis as your reply does NOT publish it — you are running
+               headless, nothing is watching your output, and a final message
+               reaches nobody. The issue only changes if `gh` changes it.
+
+               Write the body to a file first, then post it. Use a file rather
+               than `-b` because the body contains backticks, quotes and
+               newlines that do not survive shell quoting:
+
+                 cat > /tmp/analysis.md <<'ANALYSIS_EOF'
+                 ## Root cause
+                 <one-paragraph plain-English explanation>
+
+                 ## Evidence
+                 - `path/to/file.py:LINE` — <quoted code excerpt>
+                 - <additional file:line refs as needed>
 
-                  ## Root cause
-                  <one-paragraph plain-English explanation>
+                 ## Proposed fix
+                 <high-level approach in 2-4 sentences — NO code yet>
 
-                  ## Evidence
-                  - `path/to/file.py:LINE` — <quoted code excerpt>
-                  - <additional file:line refs as needed>
+                 ## Risks / open questions
+                 <anything you couldn't determine, edge cases, or test gaps>
 
-                  ## Proposed fix
-                  <high-level approach in 2-4 sentences — NO code yet>
+                 ---
+                 Reply `@claude-bot fix` to implement and open a draft PR.
+                 ANALYSIS_EOF
 
-                  ## Risks / open questions
-                  <anything you couldn't determine, edge cases, or test gaps>
+                 gh issue comment ${{ github.event.issue.number }} --body-file /tmp/analysis.md
 
-                  ---
-                  Reply `@claude-bot fix` to implement and open a draft PR.
+               Check that the command succeeded. If it failed, fix the problem
+               and run it again — do not proceed to step 6 with the comment
+               unposted, and do not end the run having only described it.
 
-            5. Update labels:
+            6. Update labels:
                  gh issue edit ${{ github.event.issue.number }} --add-label analyzed --remove-label ready-for-analysis
 
             ──────────────────────────────────────────────────────────────────
             IF YOU CAN'T REACH A CONCLUSION
             ──────────────────────────────────────────────────────────────────
-            Post what you found, what you read, and what specific information
-            is still missing. Then:
+            Publish what you found, what you read, and what specific
+            information is still missing — again by RUNNING the commands, not
+            by describing the outcome:
+
+                 cat > /tmp/analysis.md <<'ANALYSIS_EOF'
+                 ## Inconclusive
+                 <what you read, what you ruled out, what is still missing>
+                 ANALYSIS_EOF
+
+                 gh issue comment ${{ github.event.issue.number }} --body-file /tmp/analysis.md
                  gh issue edit ${{ github.event.issue.number }} --add-label needs-human-review
 
             Be honest — "I couldn't find the bug in the current code" is a
             valid outcome. Do not invent a root cause to fill the template.
+            An inconclusive comment that is actually posted is worth far more
+            than a confident one that is not.
 
             ──────────────────────────────────────────────────────────────────
             HARD CONSTRAINTS
@@ -151,4 +183,69 @@ jobs:
             - DO NOT open a PR.
             - DO NOT skip the bess-analyst delegation step — it's the whole
               point of this stage.
+            - DO NOT end the run without having RUN `gh issue comment`. A
+              diagnosis you only wrote out is a diagnosis nobody receives,
+              and the job will fail the verification step below.
             - Quote real code. Don't paraphrase.
+
+      # The acceptance criterion of #646: a Stage 2 run must not be able to
+      # exit `success` having written nothing to the issue.
+      #
+      # Five of nine non-skipped runs did exactly that over three days, at
+      # $0.33-1.58 each, and every signal said they were healthy -- green run,
+      # is_error false, permission_denials_count 0. The `ready-for-analysis`
+      # label staying put is indistinguishable from "nobody has run analyze
+      # yet", so the backlog pass could not see it either. On #593 the
+      # maintainer eventually wrote the diagnosis by hand: the stage was paid
+      # for and the work was done twice.
+      #
+      # `if: always()` is load-bearing -- the whole point is to catch the run
+      # that the agent step already reported as successful.
+      #
+      # This step ASSERTS; it deliberately does not repair. Posting the comment
+      # on the agent's behalf would be a second publishing path whose only job
+      # is to route around the first one failing, which masks the regression
+      # instead of surfacing it (docs/agents/rules.md, Debugging Protocol
+      # step 8). A loud red run is the product here.
+      - name: Verify the analysis reached the issue
+        if: always()
+        env:
+          GH_TOKEN: ${{ steps.app-token.outputs.token }}
+          ISSUE: ${{ github.event.issue.number }}
+          SINCE: ${{ steps.started.outputs.at }}
+        run: |
+          set -euo pipefail
+
+          posted=$(gh issue view "$ISSUE" --json comments \
+            --jq "[.comments[] | select(.createdAt > \"$SINCE\")] | length")
+          labels=$(gh issue view "$ISSUE" --json labels --jq '[.labels[].name] | join(",")')
+
+          echo "Comments posted since ${SINCE}: ${posted}"
+          echo "Labels now: ${labels:-<none>}"
+
+          failed=0
+
+          if [ "$posted" -eq 0 ]; then
+            echo "::error::Stage 2 posted NO comment on issue #${ISSUE}. The run"
+            echo "::error::billed for an analysis that reached nobody. Do not re-fire"
+            echo "::error::blindly -- check whether the agent described the comment"
+            echo "::error::instead of running 'gh issue comment' (see #646)."
+            failed=1
+          fi
+
+          case ",$labels," in
+            *,analyzed,*|*,needs-human-review,*) ;;
+            *)
+              echo "::error::Stage 2 applied neither 'analyzed' nor"
+              echo "::error::'needs-human-review' to issue #${ISSUE}. Without one of"
+              echo "::error::them the issue is indistinguishable from one that was"
+              echo "::error::never analysed, and the backlog pass will skip it."
+              failed=1
+              ;;
+          esac
+
+          if [ "$failed" -eq 1 ]; then
+            exit 1
+          fi
+
+          echo "✅ Analysis published: ${posted} new comment(s), labels: ${labels}"

docs/agents/workflow.md

@@ -43,6 +43,9 @@ Issue opened/edited  → issue-triage.yml      [auto, ~$0.05]
                        → delegates to `bess-analyst` sub-agent
                        → posts Root cause / Evidence / Proposed fix
                        → label `analyzed` (or `needs-human-review`)
+                       → job FAILS if it posted no comment or applied
+                         neither label — a run that analysed nothing
+                         must not report success (#646)
 
 @claude-bot fix      → issue-fix.yml        [manual, ~$1–4]
                        → reads Stage 2 diagnosis comment

scripts/quality-check.sh

@@ -420,6 +420,98 @@ then
     ERRORS=$((ERRORS + 1))
 fi
 
+echo ""
+echo "📋 Checking bot workflow publish contract..."
+echo "-------------------------------------------"
+
+# Stage 2 (issue-analyze.yml) spent three days exiting `success` while posting
+# nothing -- five of nine non-skipped runs produced no comment and no label,
+# burning $0.33-1.58 each (#646). Every layer reported healthy: the run was
+# green, is_error false, permission_denials_count 0.
+#
+# Two independent things have to hold, and this gate checks BOTH because either
+# one alone leaves the failure silent:
+#
+#   1. The prompt must tell the agent to RUN a command to publish. Stage 2 said
+#      "Post ONE comment on the issue with this structure:" followed by a
+#      markdown template -- a description of a document, not an instruction to
+#      execute anything. These workflows run in AGENT mode (use_sticky_comment
+#      false, track_progress false), so the action posts nothing on the agent's
+#      behalf: an agent that renders the template as its final assistant message
+#      has, by its own lights, finished. It then never reaches the labelling
+#      step, which is why the label was untouched too.
+#
+#      The contrast is what makes this more than a story. On the SAME action
+#      version (459ad358 / CLI 2.1.234), Stage 4's review -- whose prompt names
+#      `gh pr review` explicitly -- landed an APPROVED verdict, while Stage 2
+#      went silent three times in three seconds. Stage 2 was the only bot
+#      workflow whose publish step was prose, and the only one that failed to
+#      publish.
+#
+#   2. The workflow must VERIFY it afterwards. The prompt fix is a behavioural
+#      argument about what a model will infer, so it cannot be trusted on its
+#      own -- the next upstream version may infer differently, exactly as this
+#      one did. Only a post-condition on the job makes that loud instead of
+#      green.
+#
+# Deliberately NOT checked for, and deliberately not built: a step that posts
+# the comment on the agent's behalf when it didn't. That is a second publishing
+# path whose only job is to route around the first one failing, and it would
+# mask the regression rather than surface it (docs/agents/rules.md, Debugging
+# Protocol step 8). The guard asserts; it does not repair.
+if ! python3 - <<'PY'
+import pathlib, re, sys
+
+WORKFLOWS = pathlib.Path(".github/workflows")
+
+# Workflows whose whole product is a comment on the issue. Stage 3 is excluded
+# on purpose: its product is a PR, which is observable without this check.
+PUBLISHERS = {
+    "issue-triage.yml": "gh issue comment",
+    "issue-analyze.yml": "gh issue comment",
+}
+
+bad = []
+
+for name, command in PUBLISHERS.items():
+    path = WORKFLOWS / name
+    if not path.is_file():
+        print(f"❌ {name} is missing -- the publish contract cannot be checked")
+        bad.append(name)
+        continue
+    if command not in path.read_text():
+        print(
+            f"❌ {name} never names `{command}`. Its publish step is prose, so "
+            "the agent can render it as a final message and exit success "
+            "having written nothing to GitHub (#646)."
+        )
+        bad.append(name)
+
+# The post-agent guard: a run that published nothing must fail the job. Checked
+# by the shape that actually does the work -- a step keyed on `if: always()`
+# that re-reads the issue -- rather than by step name, which renames freely.
+analyze = WORKFLOWS / "issue-analyze.yml"
+if analyze.is_file():
+    text = analyze.read_text()
+    has_always = re.search(r"^\s*if:\s*always\(\)\s*$", text, re.MULTILINE)
+    has_readback = "--json comments" in text and "--json labels" in text
+    if not (has_always and has_readback):
+        print(
+            "❌ issue-analyze.yml has no post-agent verification step. A run "
+            "that posts no comment and applies no label must FAIL the job, "
+            "not exit success (#646 acceptance criterion)."
+        )
+        bad.append("issue-analyze.yml:guard")
+
+if bad:
+    sys.exit(1)
+print(f"✅ Bot workflow publish contract intact ({len(PUBLISHERS)} publishers "
+      "name their command, Stage 2 verifies it posted)")
+PY
+then
+    ERRORS=$((ERRORS + 1))
+fi
+
 echo ""
 echo "📋 Checking scenario discovery coverage..."
 echo "-------------------------------------------"