Skip to content

Commit 5dcba17

Browse files
frankbriaTest User
andauthored
feat(queue): batch processing and issue queue management (#72) (#293)
* feat(queue): add queue_manager.sh state library (#72) Persistent priority/dependency-aware queue at .ralph/queue.json backing batch issue processing. Functions: init/add/remove/clear, status counts, priority extraction + sorting, dependency parsing + circular detection, dependency-aware get_next_issue, and status transitions. 36 unit tests in test_queue_manager.bats. * feat(queue): add ralph-queue command for batch processing (#72) ralph_queue.sh: add (from GitHub filters/explicit list/PRD), status, next, remove, clear, reorder, validate, process/resume. Reuses ralph_import.sh gh helpers + lib/queue_manager.sh. Sequential processor runs the loop per ready item (priority + dependency aware), commits Fix #N per issue, supports --halt-on-failure; RALPH_LOOP_CMD is a mockable seam. 20 integration tests. * feat(queue): install ralph-queue + wire ralph --queue-* flags (#72) install.sh installs the ralph-queue command and copies ralph_queue.sh; uninstall removes it. ralph_loop.sh delegates --queue-status, --process-queue, --resume-queue, --queue-next, --queue-clear, --queue-remove to ralph-queue (subcommand-and-exit pattern). Adds install + loop-delegation tests. * feat(monitor): show issue-queue progress in dashboard (#72) display_queue_status() renders a queue section (completed/total, pending/ active/failed, current item) when .ralph/queue.json has entries; hidden otherwise. 3 monitor tests. * docs(queue): document batch processing and ralph-queue (#72) README Batch Processing section + command reference; new docs/QUEUE_MANAGEMENT.md guide; CLAUDE.md library entries (ralph_queue.sh, lib/queue_manager.sh) + Key Commands + behaviors. * fix(queue): address codex review findings (#72) - init_queue: build JSON via jq so a repository with quotes/backslashes can't corrupt queue.json (Major) - cmd_add: guard options that take a value so a missing trailing value fails fast instead of hanging on shift 2 (Major) - processor: only commit residue when the loop did not already commit (HEAD unchanged); warn instead of silently swallowing a commit failure (Major) - harden generated PROMPT.md to treat spec content as untrusted requirements data; document the trust boundary (Critical/injection) +2 regression tests. * fix(queue): address CodeRabbit + claude-review findings (#72) - deps outside the queue are treated as satisfied external prereqs in is_dependency_satisfied/get_next_issue (were blocking forever) (Major) - monitor sanitizes issue titles (strip control chars + printf %s) to block terminal-escape injection (Major) - cmd_reorder reports failure instead of false success (Major) - _prepare_work fails when format_issue_as_prd fails (Major) - commit failure marks the item failed (not completed) and respects --halt-on-failure; _finalize_commit surfaces the git error (Major/Low) - cmd_process resets items orphaned in 'processing' by an interrupted run so resume can retry them (Medium) - prompt-injection fence is appended to a pre-existing PROMPT.md that lacks it, not only when PROMPT.md is absent (Low/Medium) +3 regression tests (external-dep readiness x2, orphan recovery). * fix(queue): namespace PRD specs by id; document fetch/PROMPT behavior (#72) - _prepare_work copies PRD specs as <id>-<basename> so two PRDs with the same filename from different dirs don't overwrite each other (LOW, claude-review) - docs: note that processing re-fetches issues at runtime (needs gh) and may append the security fence to a pre-existing PROMPT.md +1 regression test. --------- Co-authored-by: Test User <test@example.com>
1 parent 1fe9af1 commit 5dcba17

12 files changed

Lines changed: 2027 additions & 3 deletions

CLAUDE.md

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ Ralph for Claude Code — an autonomous AI development loop system enabling cont
2222
- One `gh issue list --limit 500` query (server-side where gh supports it; exclude-label and title matching client-side via jq); candidates sorted oldest-first. Cap-hit handling (gh returns newest-first, so a capped set may be missing the true oldest matches): WARN for server-side-only queries; hard ERROR when client-side filters are active (they could pick the wrong issue or report zero matches over a truncated set)
2323
- `--select first|interactive|priority` picks among multiple matches: priority understands bare `P0``P9` and `priority: PN` labels (ties → oldest, none → first); interactive falls back to first on non-TTY/EOF. `--dry-run` previews the match table + would-be selection without importing
2424
- Completeness assessment + plan generation (Issue #70): issues are scored 0–100 via `lib/issue_analyzer.sh`; below `--completeness-threshold` (default 60) an implementation plan is generated via Claude CLI (`--plan-model` passthrough) and appended to the PRD before conversion, plus saved to `.ralph/specs/implementation-plan.md`. Flags: `--generate-plan` (force), `--no-generate-plan` (fail if below threshold), `--plan-model <model>`, `--completeness-threshold <0-100>`, `--auto-approve` (skip the approval prompt; non-TTY sessions auto-accept)
25+
- **ralph_queue.sh**`ralph-queue` command: batch processing and issue queue management (Issue #72). Builds a persistent queue at `.ralph/queue.json` of GitHub issues (reuses `ralph_import.sh`'s `gh` machinery: `resolve_github_issue_candidates`, `fetch_github_issue`, `format_issue_as_prd`) or local PRD specs. Subcommands: `add` (`--github-issues N,N` | filter flags | `--prd <file>`), `status [--json]`, `next`, `remove`, `clear`, `reorder`, `validate`, `process [--halt-on-failure]`, `resume`. The sequential processor stages `.ralph/` from each ready item (priority + dependency order), runs the loop via `RALPH_LOOP_CMD` (default `ralph_loop.sh`; overridable for tests), commits `Fix #N: <title>` per issue, skips failures (or halts), and writes `.ralph/logs/queue_processing.log`. Single branch, no concurrency. See [docs/QUEUE_MANAGEMENT.md](docs/QUEUE_MANAGEMENT.md)
2526
- **ralph_enable.sh** — interactive wizard enabling Ralph in existing projects (environment detection, task source selection, generates `.ralphrc`)
2627
- **ralph_enable_ci.sh** — non-interactive version for CI/automation; `--json` output mode; exit codes: 0 (success), 1 (error), 2 (already enabled)
2728

@@ -37,6 +38,7 @@ Ralph for Claude Code — an autonomous AI development loop system enabling cont
3738
- **issue_analyzer.sh**`assess_issue_completeness()`: deterministic 0–100 heuristic scoring of issue PRDs (acceptance criteria +25, checklists/code blocks/sections/keywords/length +15 each); JSON output with `confidence_score`, `completeness_level`, `missing_elements`, `recommendation`; `log_issue_analysis()` for summaries
3839
- **file_protection.sh**`validate_ralph_integrity()` checks `RALPH_REQUIRED_PATHS` exist; runs every loop iteration; `get_integrity_report()` for recovery instructions
3940
- **log_utils.sh**`rotate_logs()` rotates `$LOG_DIR/ralph.log` at 10MB, keeping 4 archives (`.log.1``.log.4`); GNU `stat -c%s` with BSD `stat -f%z` fallback
41+
- **queue_manager.sh** — queue state primitives backing `ralph-queue` (Issue #72). State at `.ralph/queue.json` (`{version, created_at, updated_at, repository, queue:[…]}`); entries carry `id`/`source` (`github`|`prd`)/`issue_number`/`path`/`title`/`priority`/`labels`/`milestone`/`dependencies`/`status`/timestamps. Functions: `init_queue`, `add_to_queue` (dedupe by id, fills defaults; rc 0/1/2), `remove_from_queue`, `clear_queue`, `mark_issue_status` (validates status, stamps started/completed), `get_queue_status` (counts JSON), `sort_queue_by_priority` (rank then FIFO), `get_priority_from_labels` (reuses the P0–P9 / `priority: PN` parser), `parse_issue_dependencies` (`depends on/blocked by/requires #N`), `is_dependency_satisfied`, `get_next_issue` (ready+priority+FIFO), `validate_dependencies` (jq cycle detection). All mutations are atomic temp-file+`mv` via `_queue_apply`
4042

4143
## Key Commands
4244

@@ -81,6 +83,28 @@ ralph --rollback # List available backup branches
8183
ralph --rollback ralph-backup-loop-3-1775155286 # Roll back to a specific backup
8284
```
8385

86+
### Batch Processing / Issue Queue (Issue #72)
87+
```bash
88+
# Build a queue (GitHub issues or local PRDs)
89+
ralph-queue add --github-label "bug,P0" # filters reuse the ralph-import flags
90+
ralph-queue add --github-issues 69,70,71 # explicit list
91+
ralph-queue add --github-milestone "v1.0"
92+
ralph-queue add --prd ./docs/feature.md
93+
94+
# Manage
95+
ralph-queue status [--json] # also: ralph --queue-status
96+
ralph-queue next # also: ralph --queue-next
97+
ralph-queue reorder # sort by priority (P0 first)
98+
ralph-queue validate # detect circular dependencies
99+
ralph-queue remove 69 # also: ralph --queue-remove 69
100+
ralph-queue clear # also: ralph --queue-clear
101+
102+
# Process sequentially (priority + dependency order; one commit per issue)
103+
ralph --process-queue # also: ralph-queue process
104+
ralph --process-queue --halt-on-failure
105+
ralph --resume-queue # continue remaining pending items
106+
```
107+
84108
### Monitoring
85109
```bash
86110
ralph-monitor # Manual monitoring in separate terminal
@@ -269,7 +293,7 @@ project-name/
269293
## Global Installation
270294

271295
`./install.sh` installs:
272-
- **Commands**`~/.local/bin/`: `ralph`, `ralph-monitor`, `ralph-setup`, `ralph-import`, `ralph-migrate`, `ralph-enable`, `ralph-enable-ci`, `ralph-stats`
296+
- **Commands**`~/.local/bin/`: `ralph`, `ralph-monitor`, `ralph-setup`, `ralph-import`, `ralph-queue`, `ralph-migrate`, `ralph-enable`, `ralph-enable-ci`, `ralph-stats`
273297
- **Scripts + templates**`~/.ralph/` (main scripts, `templates/`, `lib/`)
274298

275299
**External dependencies**: Claude Code CLI (execution engine), tmux (integrated monitoring), git (projects must be repos), jq (JSON processing), standard Unix tools.

README.md

Lines changed: 61 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -179,7 +179,7 @@ cd ralph-claude-code
179179
./install.sh
180180
```
181181

182-
This adds `ralph`, `ralph-monitor`, `ralph-setup`, `ralph-import`, `ralph-migrate`, `ralph-enable`, and `ralph-enable-ci` commands to your PATH.
182+
This adds `ralph`, `ralph-monitor`, `ralph-setup`, `ralph-import`, `ralph-queue`, `ralph-migrate`, `ralph-enable`, and `ralph-enable-ci` commands to your PATH.
183183

184184
> **Note**: You only need to do this once per system. After installation, you can delete the cloned repository if desired.
185185
@@ -485,6 +485,57 @@ Generated plans are shown for approval before conversion. Non-interactive sessio
485485
### Troubleshooting
486486
- **"GitHub CLI (gh) is not installed"** — install it from https://cli.github.com
487487
- **"GitHub CLI is not authenticated"** — run `gh auth login`
488+
489+
## Batch Processing and Issue Queue
490+
491+
For larger efforts, the `ralph-queue` command builds a persistent queue of work items (GitHub issues or local PRD specs) and processes them sequentially, respecting priority and dependencies. The queue is stored at `.ralph/queue.json` and survives restarts, so an interrupted run can be resumed. See [docs/QUEUE_MANAGEMENT.md](docs/QUEUE_MANAGEMENT.md) for the full guide.
492+
493+
### Building a queue
494+
495+
```bash
496+
# Queue issues by metadata filter (reuses the import filter flags)
497+
ralph-queue add --github-label "bug,P0"
498+
ralph-queue add --github-milestone "v1.0"
499+
500+
# Queue specific issues by number
501+
ralph-queue add --github-issues 69,70,71
502+
503+
# Queue a local PRD/spec file
504+
ralph-queue add --prd ./docs/feature.md
505+
```
506+
507+
When a GitHub issue is queued, its priority is read from `P0``P9` / `priority: PN` labels and its dependencies are parsed from the body (`depends on #N`, `blocked by #N`, `requires #N`).
508+
509+
### Managing the queue
510+
511+
```bash
512+
ralph-queue status # Show the queue (counts + items); --json for machine output
513+
ralph-queue next # Print the id of the next ready item
514+
ralph-queue reorder # Sort the queue by priority (P0 first)
515+
ralph-queue validate # Check for circular dependencies
516+
ralph-queue remove 69 # Remove an item by issue number or id
517+
ralph-queue clear # Empty the queue
518+
```
519+
520+
These are also available through `ralph` itself: `ralph --queue-status`, `ralph --queue-next`, `ralph --queue-clear`, and `ralph --queue-remove <id|N>`.
521+
522+
### Processing the queue
523+
524+
```bash
525+
# Process all ready pending items in priority/dependency order
526+
ralph --process-queue
527+
ralph-queue process
528+
529+
# Stop at the first failure instead of skipping it
530+
ralph --process-queue --halt-on-failure
531+
532+
# Resume after an interruption (only pending items are picked up)
533+
ralph --resume-queue
534+
```
535+
536+
For each ready item the processor stages the project from the issue/spec, runs the Ralph loop, commits the work as `Fix #N: <title>` (one commit per issue when in a git repo), then advances. Failed items are marked `failed` and skipped by default (or halt the run with `--halt-on-failure`); items whose dependencies never complete remain `pending`. Progress is written to `.ralph/logs/queue_processing.log` and shown in the `ralph-monitor` dashboard.
537+
538+
Concurrent (parallel) processing is intentionally out of scope — items are processed one at a time on a single branch.
488539
- **"Could not fetch issue #N"** — check the issue number, your repo access, and the `--repo` value
489540
- **"No issues match the specified filters"** — relax or remove some filters; only open issues are matched unless `--github-state closed|all` is given. `--dry-run` shows what a filter set matches.
490541

@@ -952,6 +1003,12 @@ ralph [OPTIONS]
9521003
--circuit-status Show circuit breaker status
9531004
--auto-reset-circuit Auto-reset circuit breaker on startup (bypasses cooldown)
9541005
--reset-session Reset session state manually
1006+
--queue-status Show the issue queue and exit
1007+
--process-queue Process pending queued issues sequentially (--halt-on-failure to stop on first failure)
1008+
--resume-queue Resume processing the remaining pending issues
1009+
--queue-next Print the id of the next ready queued issue
1010+
--queue-clear Remove all items from the queue
1011+
--queue-remove <id|N> Remove one item from the queue
9551012
```
9561013
9571014
> **Full reference**: every flag is documented in depth, with examples and `.ralphrc` patterns, in [docs/CLI_OPTIONS.md](docs/CLI_OPTIONS.md).
@@ -962,6 +1019,9 @@ ralph-setup project-name # Create new Ralph project
9621019
ralph-enable # Enable Ralph in existing project (interactive)
9631020
ralph-enable-ci # Enable Ralph in existing project (non-interactive)
9641021
ralph-import prd.md project # Convert PRD/specs to Ralph project
1022+
ralph-queue add --github-label bug # Build a batch queue of issues
1023+
ralph-queue status # Show the issue queue
1024+
ralph --process-queue # Process the queue sequentially
9651025
ralph --monitor # Start with integrated monitoring
9661026
ralph --status # Check current loop status
9671027
ralph --verbose # Enable detailed progress updates

docs/QUEUE_MANAGEMENT.md

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# Batch Processing and Issue Queue Management
2+
3+
Ralph can work through multiple GitHub issues (or local PRD specs) in one
4+
autonomous session using a persistent, priority- and dependency-aware queue.
5+
This guide covers building, managing, and processing that queue.
6+
7+
> Introduced in Issue #72. Requires the GitHub CLI (`gh`) and `jq` for the
8+
> GitHub-issue paths; the PRD path needs neither.
9+
10+
## Concepts
11+
12+
- **Queue file**`.ralph/queue.json` in the current Ralph project. It is the
13+
single source of truth and persists across restarts, so an interrupted run
14+
can be resumed. It is created automatically the first time you add an item.
15+
- **Item** — one unit of work. A *github* item carries the issue number,
16+
title, priority, labels, milestone, and parsed dependencies. A *prd* item
17+
carries a path to a local spec file and a derived title.
18+
- **Status** — each item is `pending`, `processing`, `completed`, `failed`,
19+
or `skipped`.
20+
- **Priority** — read from `P0``P9` or `priority: PN` labels (lower number =
21+
higher priority). Items without a priority label sort last.
22+
- **Dependencies** — parsed from the issue body via `depends on #N`,
23+
`blocked by #N`, and `requires #N` (case-insensitive). An item only becomes
24+
*ready* once every dependency it references in the queue is `completed`.
25+
26+
## Building a queue
27+
28+
`ralph-queue add` accepts three kinds of sources.
29+
30+
```bash
31+
# 1) Metadata filters — reuses the same flags as `ralph-import`
32+
ralph-queue add --github-label "bug,P0" # ALL labels (comma = AND)
33+
ralph-queue add --github-milestone "v1.0"
34+
ralph-queue add --github-search "login timeout"
35+
ralph-queue add --github-title "[P0]*" # * is the only wildcard
36+
ralph-queue add --github-assignee @me # or a username, or none
37+
ralph-queue add --github-label bug --exclude-label wontfix
38+
ralph-queue add --github-state all # open (default), closed, all
39+
ralph-queue add --github-label bug --repo owner/repo
40+
41+
# 2) Explicit issue numbers
42+
ralph-queue add --github-issues 69,70,71
43+
44+
# 3) A local PRD/spec file
45+
ralph-queue add --prd ./docs/feature.md
46+
```
47+
48+
Adding is idempotent — an item already in the queue (matched by id) is skipped
49+
with a warning, so re-running a filter add only appends what's new.
50+
51+
## Managing the queue
52+
53+
```bash
54+
ralph-queue status # Human-readable table of items + counts
55+
ralph-queue status --json # { total, pending, processing, completed, failed, skipped }
56+
ralph-queue next # Print the id of the next ready item (or exit 1)
57+
ralph-queue reorder # Persist a priority sort (P0 first, FIFO within a tier)
58+
ralph-queue validate # Exit non-zero if a circular dependency exists
59+
ralph-queue remove 69 # Remove by issue number or id (e.g. github-69, prd-feature-md)
60+
ralph-queue clear # Remove every item
61+
```
62+
63+
The most common queries are mirrored on the `ralph` command itself:
64+
65+
```bash
66+
ralph --queue-status
67+
ralph --queue-next
68+
ralph --queue-clear
69+
ralph --queue-remove 69
70+
```
71+
72+
## Processing the queue
73+
74+
```bash
75+
ralph --process-queue # or: ralph-queue process
76+
ralph --process-queue --halt-on-failure
77+
ralph --resume-queue # continue with the remaining pending items
78+
```
79+
80+
For each ready item, in priority then FIFO order, the processor:
81+
82+
1. Marks the item `processing`.
83+
2. Stages the project from the source — for a GitHub issue it writes
84+
`.ralph/specs/issue-<N>.md` and points `.ralph/fix_plan.md` at it; for a PRD
85+
it copies the spec into `.ralph/specs/`.
86+
3. Runs the Ralph loop (`ralph_loop.sh`) until it exits.
87+
4. On success, commits the work as `Fix #N: <title>` (one commit per issue,
88+
when the project is a git repo) and marks the item `completed`.
89+
5. On a non-zero loop exit, marks the item `failed` and either skips it
90+
(default) or halts the whole run (`--halt-on-failure`).
91+
6. Re-evaluates dependencies and moves to the next ready item.
92+
93+
Items whose dependencies never complete stay `pending` and are reported at the
94+
end of the run. Resuming simply re-runs `process`: completed and failed items
95+
are left alone, and only ready `pending` items are picked up. Any item left in
96+
`processing` by an interrupted run (SIGKILL, power loss) is reset to `pending`
97+
at the start of the next `process`/`resume` so it is retried.
98+
99+
Two things to know about processing:
100+
101+
- **GitHub connectivity is needed at process time.** Each GitHub item is
102+
re-fetched when it is processed (to use the freshest issue body), so
103+
`process` requires `gh` access even when the items are already in
104+
`queue.json`.
105+
- **PROMPT.md may gain a security fence.** If the project's `.ralph/PROMPT.md`
106+
predates the "Handling Spec Content" untrusted-input fence, the processor
107+
appends it (once) so the trust boundary described above is always present.
108+
109+
### Progress and logging
110+
111+
- `.ralph/logs/queue_processing.log` captures the loop output per item.
112+
- `ralph-queue status` (and `ralph --queue-status`) show live counts.
113+
- The `ralph-monitor` dashboard renders an **Issue Queue** panel
114+
(completed/total, pending/active/failed, and the item currently processing)
115+
whenever a non-empty `.ralph/queue.json` is present.
116+
117+
## Security: queued content is untrusted input
118+
119+
Processing a queued item feeds the issue body (or PRD) into an autonomous agent
120+
as the work to implement. Two protections apply:
121+
122+
- **Comments are excluded.** Only the issue *body* is used (via
123+
`format_issue_as_prd … false`); on public repos anyone can comment, so comment
124+
text — a prime prompt-injection surface — never reaches the agent.
125+
- **Spec content is marked as data.** The generated `.ralph/PROMPT.md` instructs
126+
the agent to treat spec files as requirements describing *what* to build and to
127+
ignore any embedded instructions that try to change its task or tool
128+
permissions (the same posture as `ralph-import`).
129+
130+
Even so, the body is the work instruction by design. **Only queue issues you
131+
trust** (e.g. your own backlog or a maintained milestone), and review unfamiliar
132+
issues before processing them unattended.
133+
134+
## Design notes and limits
135+
136+
- **Single branch, one commit per issue.** Items are processed sequentially on
137+
the current branch; there is no per-issue branching.
138+
- **Failures are isolated.** A failed item does not abort the run unless you
139+
pass `--halt-on-failure`; this maximizes throughput across a backlog.
140+
- **Circular dependencies are refused.** `process` runs `validate` first and
141+
stops if the dependency graph contains a cycle. Fix the cycle (or
142+
`remove` an offending item) and retry.
143+
- **No concurrency.** Parallel processing is intentionally out of scope — the
144+
queue is processed one item at a time.
145+
146+
## Troubleshooting
147+
148+
| Symptom | Cause / fix |
149+
|---|---|
150+
| `GitHub CLI (gh) is not installed` | Install `gh` (https://cli.github.com) |
151+
| `GitHub CLI is not authenticated` | `gh auth login` |
152+
| `circular dependency detected` | Run `ralph-queue validate`; remove or fix the cycle |
153+
| Items stuck `pending` after a run | Their dependencies failed or aren't in the queue — add/complete the dependency, then resume |
154+
| Query results were capped | Narrow your filters; very large result sets are truncated by `gh` |

0 commit comments

Comments
 (0)