Bug: `completion_indicators` exit check ignores Claude's explicit `EXIT_SIGNAL: false`

[amgohan]

Description

Ralph exits prematurely after 2 loops based on confidence-scoring heuristics, even when Claude explicitly reports EXIT_SIGNAL: false in the RALPH_STATUS block. This prevents projects requiring multiple iterations from running to completion.

Environment

• Ralph Version: v0.9.8
• OS: macOS / Linux
• Shell: Bash

Steps to Reproduce

1. Create a project with iterative stop conditions (e.g., quality score >= 95%)
2. Run Ralph: ralph --timeout 30 --allowed-tools "Write,Edit,Read,Bash(*)" --verbose
3. Observe Claude doing productive work and reporting:

   ---RALPH_STATUS---
   STATUS: IN_PROGRESS
   EXIT_SIGNAL: false
   RECOMMENDATION: Continue processing...
   ---END_RALPH_STATUS---
   

4. Ralph exits after 2 loops with: Exit condition: Strong completion indicators (2)

Expected Behavior

Ralph should respect Claude's explicit EXIT_SIGNAL: false and continue iterating until Claude reports EXIT_SIGNAL: true (indicating project requirements are met).

Actual Behavior

Ralph exits after 2 loops regardless of Claude's EXIT_SIGNAL value. The .response_analysis file shows:

{
  "analysis": {
    "exit_signal": false,
    "has_completion_signal": false,
    "confidence_score": 70
  }
}

Despite exit_signal: false, Ralph triggers exit because confidence_score >= 60 adds the loop to completion_indicators, and 2+ indicators triggers exit.

Root Cause

In ralph_loop.sh, the should_exit_gracefully() function (~line 313) checks:

if [[ $recent_completion_indicators -ge 2 ]]; then
    log_status "WARN" "Exit condition: Strong completion indicators ($recent_completion_indicators)"
    echo "project_complete"
    return 0
fi

This check does not consult Claude's explicit exit_signal from .response_analysis. The confidence scoring triggers on natural language patterns like "Complete", "Perfect", "successfully" - even during productive IN_PROGRESS iterations.

Proposed Fix

Modify the completion indicators check to respect Claude's explicit EXIT_SIGNAL:

# 3. Strong completion indicators (only if Claude's EXIT_SIGNAL is true)
local claude_exit_signal="false"
if [[ -f ".response_analysis" ]]; then
    claude_exit_signal=$(jq -r '.analysis.exit_signal // false' ".response_analysis" 2>/dev/null || echo "false")
fi

if [[ $recent_completion_indicators -ge 2 ]] && [[ "$claude_exit_signal" == "true" ]]; then
    log_status "WARN" "Exit condition: Strong completion indicators ($recent_completion_indicators) with EXIT_SIGNAL=true" >&2
    echo "project_complete"
    return 0
elif [[ $recent_completion_indicators -ge 2 ]]; then
    log_status "INFO" "DEBUG: Completion indicators ($recent_completion_indicators) but EXIT_SIGNAL=false, continuing..." >&2
fi

Important: The >&2 is critical - without it, log_status output goes to stdout and gets captured as the function's return value (non-empty = exit reason), causing unintended exits.

Behavior After Fix

Scenario	completion_indicators	EXIT_SIGNAL	Result
Work in progress	>= 2	false	Continue (fixed)
Project complete	>= 2	true	Exit
Early iterations	< 2	false	Continue

Verified Fix

I've tested this fix on a documentation refinement project:

• Before fix: Exited after 2-3 loops at 65% quality
• After fix: Ran 17+ loops, reached 94% quality and continuing

Related Files

• ralph_loop.sh - should_exit_gracefully() function
• lib/response_analyzer.sh - Parses RALPH_STATUS and populates .response_analysis
• .response_analysis - Contains exit_signal from Claude
• .exit_signals - Tracks completion_indicators array

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Related PRs

#6 - Fix circuit breaker false positives from JSON field names [merged]
#8 - docs: update CLAUDE.md and README.md with recent improvements [merged]
#47 - [P1] feat(cli): add modern CLI commands with JSON output support (Phase 1.1) [merged]
#58 - fix(loop): replace non-existent --prompt-file with -p flag [merged]
#66 - feat(session): add session lifecycle management with auto-reset triggers [merged]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[traycerai]

Plan

Observations

The bug occurs in file:ralph_loop.sh at the should_exit_gracefully() function (lines 313-318). The function exits when completion_indicators >= 2 based solely on confidence scoring heuristics (triggered by natural language patterns), completely ignoring Claude's explicit EXIT_SIGNAL: false in the RALPH_STATUS block. The file:lib/response_analyzer.sh already parses and stores the exit_signal in .response_analysis, but this value is never consulted during exit condition checks. This causes premature exits during productive iterations.

Approach

The fix requires modifying the completion indicators check in should_exit_gracefully() to respect Claude's explicit intent. Before exiting on completion indicators, the function must read exit_signal from .response_analysis and only exit when BOTH conditions are true: completion_indicators >= 2 AND exit_signal == true. This ensures Ralph continues iterating when Claude reports productive work in progress, while still respecting completion signals when the project is genuinely complete. The fix also requires proper stderr redirection to prevent stdout capture issues.

Implementation Steps

1. Modify Exit Condition Logic in file:ralph_loop.sh

Location: Lines 313-318 in the should_exit_gracefully() function

Current behavior: Exits immediately when recent_completion_indicators >= 2

Required changes:

• Before the completion indicators check, read exit_signal from .response_analysis using jq
• Default to "false" if the file doesn't exist or parsing fails
• Modify the condition to check BOTH recent_completion_indicators >= 2 AND claude_exit_signal == "true"
• Add an informational log when completion indicators are present but EXIT_SIGNAL is false
• Ensure all log_status calls use >&2 to prevent stdout capture

Specific modifications:

# 3. Strong completion indicators (only if Claude's EXIT_SIGNAL is true)
local claude_exit_signal="false"
if [[ -f ".response_analysis" ]]; then
    claude_exit_signal=$(jq -r '.analysis.exit_signal // false' ".response_analysis" 2>/dev/null || echo "false")
fi

if [[ $recent_completion_indicators -ge 2 ]] && [[ "$claude_exit_signal" == "true" ]]; then
    log_status "WARN" "Exit condition: Strong completion indicators ($recent_completion_indicators) with EXIT_SIGNAL=true" >&2
    echo "project_complete"
    return 0
elif [[ $recent_completion_indicators -ge 2 ]]; then
    log_status "INFO" "DEBUG: Completion indicators ($recent_completion_indicators) but EXIT_SIGNAL=false, continuing..." >&2
fi

2. Update Unit Tests in file:tests/unit/test_exit_detection.bats

Add new test cases to verify the fix:

• Test: Completion indicators with EXIT_SIGNAL=false should continue

  • Setup: Create .exit_signals with 2+ completion indicators
  • Setup: Create .response_analysis with exit_signal: false
  • Expected: should_exit_gracefully returns empty string (continues)

• Test: Completion indicators with EXIT_SIGNAL=true should exit

  • Setup: Create .exit_signals with 2+ completion indicators
  • Setup: Create .response_analysis with exit_signal: true
  • Expected: should_exit_gracefully returns "project_complete"

• Test: Completion indicators without .response_analysis file should continue

  • Setup: Create .exit_signals with 2+ completion indicators
  • Setup: Delete .response_analysis file
  • Expected: should_exit_gracefully returns empty string (continues, defaults to false)

• Test: Completion indicators with malformed .response_analysis should continue

  • Setup: Create .exit_signals with 2+ completion indicators
  • Setup: Create corrupted .response_analysis (invalid JSON)
  • Expected: should_exit_gracefully returns empty string (continues, defaults to false)

Update existing test at line 142:

• Modify test "should_exit_gracefully exits on 2 completion indicators"
• Add .response_analysis file with exit_signal: true to maintain current behavior

3. Update Test Helper Function

Location: file:tests/unit/test_exit_detection.bats lines 29-83

Required changes:

• Update the should_exit_gracefully() helper function to match the new implementation
• Add the claude_exit_signal check before the completion indicators condition
• Ensure consistency with the main implementation

4. Add Integration Tests in file:tests/integration/test_edge_cases.bats

Add new edge case tests:

• Test: Multiple loops with high confidence but EXIT_SIGNAL=false

  • Simulate 5 loops with confidence >= 60 (triggers completion_indicators)
  • Each loop has .response_analysis with exit_signal: false
  • Verify Ralph continues through all loops without premature exit

• Test: Transition from IN_PROGRESS to COMPLETE

  • Loop 1-2: confidence >= 60, exit_signal: false → continues
  • Loop 3: confidence >= 60, exit_signal: true → exits with "project_complete"

• Test: Missing .response_analysis with completion indicators

  • Simulate scenario where .response_analysis is deleted mid-loop
  • Verify graceful handling (defaults to continue)

5. Verify Behavior Table

After implementation, the exit logic should follow this decision matrix:

completion_indicators	EXIT_SIGNAL	.response_analysis exists	Result
>= 2	true	Yes	Exit (project_complete)
>= 2	false	Yes	Continue
>= 2	N/A	No	Continue (defaults to false)
< 2	true	Yes	Continue (threshold not met)
< 2	false	Yes	Continue

6. Documentation Updates

Update relevant documentation:

• Add comment in should_exit_gracefully() explaining the dual-condition check
• Document the relationship between completion_indicators (heuristic) and exit_signal (explicit)
• Note that exit_signal takes precedence as Claude's explicit intent

Sequence Diagram

sequenceDiagram
    participant Loop as ralph_loop.sh
    participant Analyzer as response_analyzer.sh
    participant Claude as Claude Response
    participant ExitCheck as should_exit_gracefully()

    Loop->>Claude: Execute Claude Code
    Claude-->>Loop: Response with RALPH_STATUS
    Note over Claude: EXIT_SIGNAL: false<br/>STATUS: IN_PROGRESS
    
    Loop->>Analyzer: analyze_response()
    Analyzer->>Analyzer: Parse RALPH_STATUS
    Analyzer->>Analyzer: Calculate confidence_score
    Note over Analyzer: confidence >= 60<br/>(natural language patterns)
    Analyzer-->>Loop: Write .response_analysis
    Note over Loop: exit_signal: false<br/>confidence_score: 70
    
    Loop->>Analyzer: update_exit_signals()
    Analyzer->>Analyzer: Add to completion_indicators
    Note over Analyzer: completion_indicators: [1, 2]
    Analyzer-->>Loop: Write .exit_signals
    
    Loop->>ExitCheck: Check exit conditions
    ExitCheck->>ExitCheck: Read .exit_signals
    Note over ExitCheck: completion_indicators >= 2 ✓
    ExitCheck->>ExitCheck: Read .response_analysis
    Note over ExitCheck: exit_signal: false
    ExitCheck->>ExitCheck: Check BOTH conditions
    Note over ExitCheck: indicators >= 2 ✓<br/>exit_signal == true ✗
    ExitCheck-->>Loop: Continue (empty string)
    
    Note over Loop: Loop continues...<br/>Claude does more work
    
    Loop->>Claude: Execute Claude Code (later)
    Claude-->>Loop: Response with RALPH_STATUS
    Note over Claude: EXIT_SIGNAL: true<br/>STATUS: COMPLETE
    
    Loop->>Analyzer: analyze_response()
    Analyzer-->>Loop: Write .response_analysis
    Note over Loop: exit_signal: true
    
    Loop->>ExitCheck: Check exit conditions
    ExitCheck->>ExitCheck: Read .response_analysis
    Note over ExitCheck: exit_signal: true
    ExitCheck->>ExitCheck: Check BOTH conditions
    Note over ExitCheck: indicators >= 2 ✓<br/>exit_signal == true ✓
    ExitCheck-->>Loop: Exit (project_complete)
    
    Loop->>Loop: Graceful shutdown

Loading

Import In IDE

🤖 Prompt for AI Agents

## Observations

The bug occurs in `file:ralph_loop.sh` at the `should_exit_gracefully()` function (lines 313-318). The function exits when `completion_indicators >= 2` based solely on confidence scoring heuristics (triggered by natural language patterns), completely ignoring Claude's explicit `EXIT_SIGNAL: false` in the RALPH_STATUS block. The `file:lib/response_analyzer.sh` already parses and stores the `exit_signal` in `.response_analysis`, but this value is never consulted during exit condition checks. This causes premature exits during productive iterations.

## Approach

The fix requires modifying the completion indicators check in `should_exit_gracefully()` to respect Claude's explicit intent. Before exiting on completion indicators, the function must read `exit_signal` from `.response_analysis` and only exit when BOTH conditions are true: `completion_indicators >= 2` AND `exit_signal == true`. This ensures Ralph continues iterating when Claude reports productive work in progress, while still respecting completion signals when the project is genuinely complete. The fix also requires proper stderr redirection to prevent stdout capture issues.

## Implementation Steps

### 1. Modify Exit Condition Logic in `file:ralph_loop.sh`

**Location**: Lines 313-318 in the `should_exit_gracefully()` function

**Current behavior**: Exits immediately when `recent_completion_indicators >= 2`

**Required changes**:
- Before the completion indicators check, read `exit_signal` from `.response_analysis` using `jq`
- Default to `"false"` if the file doesn't exist or parsing fails
- Modify the condition to check BOTH `recent_completion_indicators >= 2` AND `claude_exit_signal == "true"`
- Add an informational log when completion indicators are present but EXIT_SIGNAL is false
- Ensure all `log_status` calls use `>&2` to prevent stdout capture

**Specific modifications**:
```bash
# 3. Strong completion indicators (only if Claude's EXIT_SIGNAL is true)
local claude_exit_signal="false"
if [[ -f ".response_analysis" ]]; then
    claude_exit_signal=$(jq -r '.analysis.exit_signal // false' ".response_analysis" 2>/dev/null || echo "false")
fi

if [[ $recent_completion_indicators -ge 2 ]] && [[ "$claude_exit_signal" == "true" ]]; then
    log_status "WARN" "Exit condition: Strong completion indicators ($recent_completion_indicators) with EXIT_SIGNAL=true" >&2
    echo "project_complete"
    return 0
elif [[ $recent_completion_indicators -ge 2 ]]; then
    log_status "INFO" "DEBUG: Completion indicators ($recent_completion_indicators) but EXIT_SIGNAL=false, continuing..." >&2
fi
```

### 2. Update Unit Tests in `file:tests/unit/test_exit_detection.bats`

**Add new test cases** to verify the fix:

- **Test: Completion indicators with EXIT_SIGNAL=false should continue**
  - Setup: Create `.exit_signals` with 2+ completion indicators
  - Setup: Create `.response_analysis` with `exit_signal: false`
  - Expected: `should_exit_gracefully` returns empty string (continues)

- **Test: Completion indicators with EXIT_SIGNAL=true should exit**
  - Setup: Create `.exit_signals` with 2+ completion indicators
  - Setup: Create `.response_analysis` with `exit_signal: true`
  - Expected: `should_exit_gracefully` returns "project_complete"

- **Test: Completion indicators without .response_analysis file should continue**
  - Setup: Create `.exit_signals` with 2+ completion indicators
  - Setup: Delete `.response_analysis` file
  - Expected: `should_exit_gracefully` returns empty string (continues, defaults to false)

- **Test: Completion indicators with malformed .response_analysis should continue**
  - Setup: Create `.exit_signals` with 2+ completion indicators
  - Setup: Create corrupted `.response_analysis` (invalid JSON)
  - Expected: `should_exit_gracefully` returns empty string (continues, defaults to false)

**Update existing test** at line 142:
- Modify test "should_exit_gracefully exits on 2 completion indicators"
- Add `.response_analysis` file with `exit_signal: true` to maintain current behavior

### 3. Update Test Helper Function

**Location**: `file:tests/unit/test_exit_detection.bats` lines 29-83

**Required changes**:
- Update the `should_exit_gracefully()` helper function to match the new implementation
- Add the `claude_exit_signal` check before the completion indicators condition
- Ensure consistency with the main implementation

### 4. Add Integration Tests in `file:tests/integration/test_edge_cases.bats`

**Add new edge case tests**:

- **Test: Multiple loops with high confidence but EXIT_SIGNAL=false**
  - Simulate 5 loops with confidence >= 60 (triggers completion_indicators)
  - Each loop has `.response_analysis` with `exit_signal: false`
  - Verify Ralph continues through all loops without premature exit

- **Test: Transition from IN_PROGRESS to COMPLETE**
  - Loop 1-2: confidence >= 60, `exit_signal: false` → continues
  - Loop 3: confidence >= 60, `exit_signal: true` → exits with "project_complete"

- **Test: Missing .response_analysis with completion indicators**
  - Simulate scenario where `.response_analysis` is deleted mid-loop
  - Verify graceful handling (defaults to continue)

### 5. Verify Behavior Table

After implementation, the exit logic should follow this decision matrix:

| completion_indicators | EXIT_SIGNAL | .response_analysis exists | Result |
|----------------------|-------------|---------------------------|---------|
| >= 2 | true | Yes | **Exit** (project_complete) |
| >= 2 | false | Yes | **Continue** |
| >= 2 | N/A | No | **Continue** (defaults to false) |
| < 2 | true | Yes | **Continue** (threshold not met) |
| < 2 | false | Yes | **Continue** |

### 6. Documentation Updates

**Update relevant documentation**:
- Add comment in `should_exit_gracefully()` explaining the dual-condition check
- Document the relationship between `completion_indicators` (heuristic) and `exit_signal` (explicit)
- Note that `exit_signal` takes precedence as Claude's explicit intent

## Sequence Diagram

```mermaid
sequenceDiagram
    participant Loop as ralph_loop.sh
    participant Analyzer as response_analyzer.sh
    participant Claude as Claude Response
    participant ExitCheck as should_exit_gracefully()

    Loop->>Claude: Execute Claude Code
    Claude-->>Loop: Response with RALPH_STATUS
    Note over Claude: EXIT_SIGNAL: false<br/>STATUS: IN_PROGRESS
    
    Loop->>Analyzer: analyze_response()
    Analyzer->>Analyzer: Parse RALPH_STATUS
    Analyzer->>Analyzer: Calculate confidence_score
    Note over Analyzer: confidence >= 60<br/>(natural language patterns)
    Analyzer-->>Loop: Write .response_analysis
    Note over Loop: exit_signal: false<br/>confidence_score: 70
    
    Loop->>Analyzer: update_exit_signals()
    Analyzer->>Analyzer: Add to completion_indicators
    Note over Analyzer: completion_indicators: [1, 2]
    Analyzer-->>Loop: Write .exit_signals
    
    Loop->>ExitCheck: Check exit conditions
    ExitCheck->>ExitCheck: Read .exit_signals
    Note over ExitCheck: completion_indicators >= 2 ✓
    ExitCheck->>ExitCheck: Read .response_analysis
    Note over ExitCheck: exit_signal: false
    ExitCheck->>ExitCheck: Check BOTH conditions
    Note over ExitCheck: indicators >= 2 ✓<br/>exit_signal == true ✗
    ExitCheck-->>Loop: Continue (empty string)
    
    Note over Loop: Loop continues...<br/>Claude does more work
    
    Loop->>Claude: Execute Claude Code (later)
    Claude-->>Loop: Response with RALPH_STATUS
    Note over Claude: EXIT_SIGNAL: true<br/>STATUS: COMPLETE
    
    Loop->>Analyzer: analyze_response()
    Analyzer-->>Loop: Write .response_analysis
    Note over Loop: exit_signal: true
    
    Loop->>ExitCheck: Check exit conditions
    ExitCheck->>ExitCheck: Read .response_analysis
    Note over ExitCheck: exit_signal: true
    ExitCheck->>ExitCheck: Check BOTH conditions
    Note over ExitCheck: indicators >= 2 ✓<br/>exit_signal == true ✓
    ExitCheck-->>Loop: Exit (project_complete)
    
    Loop->>Loop: Graceful shutdown
```

---

Execution Information

Branch: main
Commit: 3dc3479

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.