validationCommand verifier treats non-zero exit as failure, so negative-assertion tasks (grep-finds-nothing) loop forever

[S0l0m0n8und9]

This bug report was generated by Claude Code (AI) at the maintainer's request, from a real RalphDex run.

Summary

RalphDex's validationCommand verifier treats any non-zero exit code as a verification failure. This breaks every task whose natural validation is a negative assertion — i.e. "this thing should not be present". For such tasks, success is grep finding nothing, which exits 1. RalphDex reads that exit 1 as "task failed", so a task that is genuinely complete can never be auto-verified or auto-closed, and the loop retries it forever until the watchdog escalates to a human.

A real task (T2-3) hit this and burned iterations 7, 8, and 9 (plus a watchdog pass) on work that was already done.

Environment

• RalphDex: s0l0m0n8und9.ralphdex v1.3.1 (VS Code extension)
• Host OS / shell: Windows 11 Enterprise, PowerShell
• Provider / model: claude / claude-sonnet-4-6, tier simple, reasoning medium
• Adapter: cliExec
• Workspace: a Python project (dataverse-mcp-python), git repo, workspace-root selection strategy

The task that triggered it

Backlog task T2-3 — "Resolve or remove untracked dataverse-workbench-cowork.zip":

{
  "id": "T2-3",
  "validation": "git status --short | grep cowork",
  "acceptance": [
    "dataverse-workbench-cowork.zip is either listed in .gitignore or its contents have been integrated and the file removed",
    "git status shows no untracked zip files at repo root"
  ]
}

The validation command git status --short | grep cowork is a negative assertion: the task is done precisely when that command produces no output (the zip is gone / gitignored). grep exits 1 when it finds no match.

What happened

The underlying work was completed in two prior commits on the branch:

• 409db57 — chore(gitignore): remove committed zip artifact and add *.zip rule
• 3e41e02 — chore: untrack dataverse-workbench-cowork.zip build artifact

Both acceptance criteria were satisfied. But every iteration after that classified T2-3 as no_progress and retried it:

Iteration	Prompt kind	Execution	Verification	Outcome
7	iteration	succeeded	failed	no_progress
8	fix-failure	succeeded	failed	no_progress
9 (watchdog)	fix-failure	succeeded	failed	no_progress → escalate_to_human

The recorded failure signature was identical each time:

validationFailureSignature: "git status --short | grep cowork::exit:1::no output"

The gitDiff verifier also failed ("Detected N change(s), but none outside Ralph-managed files") and taskState was skipped — both correct, because there was genuinely no further work to do. So all three verifiers reinforced a false "no progress" verdict on an already-complete task.

The watchdog's own assessment (iteration 9 last-message) nailed it:

The validation command git status --short | grep cowork is returning exit:1 with no output — but for this task, no output is the success condition. grep exits 1 when it finds no matches. The Ralph verifier is treating grep's "not found" exit code as a failure, when "not found" is exactly what passing looks like here.

Expected behaviour

A task whose validation is a "should-not-exist" check should be able to pass when the check finds nothing. RalphDex needs a way to express that success ≠ exit 0.

Reproduction

1. Create a backlog task whose validation is a negative-assertion command, e.g. git status --short | grep cowork (or any grep that should find nothing on success).
2. Make the workspace satisfy the task (so the grep matches nothing).
3. Run the loop on that task.
4. Observe: execution succeeds, but validationCommand records exit:1, the iteration is classified no_progress, and the same task is re-selected indefinitely until the watchdog escalates.

Root cause

The validationCommand verifier hard-codes exitCode == 0 ⇒ pass, exitCode != 0 ⇒ fail. There is no notion of:

• negative assertion ("command should find nothing / exit non-zero"), or
• expected-empty-output semantics, or
• a per-task way to invert / customise the success predicate.

Compounding issue: POSIX grep on a Windows PowerShell host

The preflight already warned that this validation command uses POSIX grep syntax while the host shell is Windows PowerShell. Even setting the exit-code semantics aside, grep is not a native PowerShell command, so these validation commands are non-portable across hosts. Worth handling alongside the primary bug.

Impact

• Infinite no-progress loop on completed tasks → wasted provider iterations/tokens and unnecessary watchdog escalations.
• Any task expressed as "verify X is absent / removed / cleaned up" is structurally unverifiable today.
• Undermines deterministic resume: a fresh run re-selects the same already-done task.

Suggested directions (non-prescriptive)

• Let a task declare validation success semantics, e.g. validationExpect: "empty" | "non-empty" | "exit-zero" | "exit-non-zero", or an explicit expectedExitCode.
• Support a "must produce no output" predicate distinct from exit code.
• For portability, either normalise common POSIX idioms (grep) to the host shell, or document/require shell-agnostic validation commands and surface the preflight warning more loudly.
• Consider: when execution succeeds, the diff shows no relevant changes, and the task's acceptance criteria are already met, treat repeated identical-signature failures as a candidate "already complete" state rather than pure no_progress.