Configure Claude agent system prompt for docs-bi repository#214
Configure Claude agent system prompt for docs-bi repository#214Sachindu-Nethmin wants to merge 8 commits intowso2:mainfrom
Conversation
Add Claude workflow for automated PR creation on issue comments
Sachindu-Nethmin
requested review from
KavinduZoysa,
NipunaRanasinghe,
anupama-pathirage and
rosensilva
as code owners
WalkthroughAdds an AI-driven documentation pipeline: a new Claude system prompt and two GitHub Actions workflows that auto-classify issues, queue doc-fix requests, render and inject prompts, invoke Claude to create per-version documentation PRs, and manage issue labels and queue progression. (50 words) Changes
Sequence DiagramsequenceDiagram
participant User as User (issue)
participant GH as GitHub (Issues & Labels)
participant Auto as Auto-Label Workflow
participant Runner as Claude Runner Workflow
participant Claude as Claude (AI Agent)
participant Repo as Target Repository
Note over User,GH: Issue opened/edited triggers auto-label workflow
GH->>Auto: webhook (issue opened/edited)
Auto->>Claude: classification request
Claude-->>Auto: category or skip
alt category → queued
Auto->>GH: add label `AI-Agent/Queued`
else
Auto-->>GH: no label change
end
Note over GH,Runner: scheduled/comment trigger starts runner
Runner->>GH: query oldest `AI-Agent/Queued`
alt issue found
GH-->>Runner: return issue `#N`
Runner->>GH: remove `AI-Agent/Queued`, add `AI-Agent/In-Progress`
Runner->>GH: post trigger comment
Runner->>Repo: checkout repository (full history)
Runner->>Runner: render system prompt (envsubst) + set env
Runner->>Claude: invoke claude-code-action with prompt & tools
Claude->>Repo: analyze docs and create per-version PR(s)
Claude-->>Runner: return results/metadata
else none
GH-->>Runner: queue empty — exit
end
Note over Runner,GH: continue processing queue
Runner->>GH: fetch next `AI-Agent/Queued`
alt next exists
GH-->>Runner: issue `#N`+1
Runner->>GH: update labels & post trigger
else empty
GH-->>Runner: complete
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
Fix all issues with AI Agents 🤖
In @.github/claude/system_prompt.txt:
- Around line 49-52: The branch-name snippet mixes `${ISSUE_NUMBER}` with
`{VERSION}`, causing inconsistent variable syntax; update `{VERSION}` to
`${VERSION}` (or `${VERSION:-default}` if you need a default) so TIMESTAMP,
BRANCH_NAME, ISSUE_NUMBER and VERSION all use shell variable substitution
consistently, or alternatively add a comment above the block clarifying that
`{VERSION}` is an external placeholder to be replaced by the AI and not a shell
variable—pick one approach and apply it to the BRANCH_NAME and git checkout
lines.
- Line 45: Replace the inconsistent phrase "master branch" with "main branch" to
match the rest of the document; update the occurrence on line 45 and also search
for and replace any other occurrences (e.g., lines 106, 249 and any remaining
"master") so all references consistently use "main branch".
- Line 78: Fix the typo in the system prompt sentence by replacing "accroding"
with the correct spelling "according" in the line that reads "First need to
understand what the issue is related to by reading the issue very carefully.Then
accroding to the issue type,apply the solution as below."
- Around line 246-249: Update the inconsistent branch/version wording: change
the list that currently reads "docs-1.4.0, docs-1.3.0" to include "and main
branch" so it matches the earlier line that lists "docs-1.4.0, docs-1.3.0 and
main branch", and replace the phrase "master branch" with "main branch" so all
references consistently use "main branch".
In @.github/workflows/claude_runner.yml:
- Around line 72-73: Quote the secret interpolations in the git config commands
to avoid word splitting or shell parsing issues: update the two commands that
set user.name and user.email (the lines with git config user.name
${{secrets.DOC_FIXING_AGENT_GIT_USER_NAME}} and git config user.email
${{secrets.DOC_FIXING_AGENT_GIT_USER_EMAIL}}) so the secret values are enclosed
in quotes when passed to git config.
- Around line 92-94: The heredoc uses a fixed delimiter "EOF" which can collide
if $SYSTEM_PROMPT contains that literal; change the workflow to use a unique
delimiter token for the heredoc (e.g., include a timestamp/PID/hash like
EOF_$RANDOM or EOF_$(date +%s) or a generated UUID) when writing SYSTEM_PROMPT
into $GITHUB_ENV, and make sure the opening line echo "SYSTEM_PROMPT<<DELIM" >>
$GITHUB_ENV and the closing echo "DELIM" >> $GITHUB_ENV use the same unique
token so the heredoc cannot terminate early.
- Around line 3-8: The issue is that the issue_comment trigger allows any
commenter to mention the bot and run the run_claude job; update the job-level if
condition (the if on the run_claude job) to require both the mention and that
the commenter is an org member/owner or specific team member by adding an actor
check—e.g. combine contains(github.event.comment.body, '@wso2-engineering-bot')
with github.event.comment.author_association == 'OWNER' ||
github.event.comment.author_association == 'MEMBER' (use parentheses to ensure
correct boolean logic), or call the GitHub API to verify team membership before
executing the Claude steps; apply this change to the run_claude job condition
and any jobs that use the same comment-based guard.
🧹 Nitpick comments (5)
.github/workflows/claude_runner.yml (4)
21-24: Unnecessary installation of pre-installed tool.The
ghCLI is pre-installed onubuntu-latestrunners. Onlyjqneeds to be installed.🔎 Proposed fix
- name: Install GitHub CLI run: | sudo apt-get update - sudo apt-get install -y gh jq + sudo apt-get install -y jq
31-38: Edge case:jq -r .numberreturns literal string "null" when issue is null.When
ISSUE_JSONis"null", runningjq -r .numberon it will output the literal string"null", not empty. The check on line 33 handles"null"but the subsequentjq -r .numbercould still output"null".🔎 Proposed fix to add explicit null handling
ISSUE_JSON=$(gh issue list --label "AI-Agent/Queued" --state open --json number,createdAt --limit 100 \ | jq '[.[]] | sort_by(.createdAt) | .[0]') if [ "$ISSUE_JSON" = "null" ] || [ -z "$ISSUE_JSON" ]; then echo "oldest_issue_number=" >> $GITHUB_OUTPUT exit 0 fi - ISSUE_NUMBER=$(echo "$ISSUE_JSON" | jq -r .number) + ISSUE_NUMBER=$(echo "$ISSUE_JSON" | jq -r '.number // empty') + if [ -z "$ISSUE_NUMBER" ]; then + echo "oldest_issue_number=" >> $GITHUB_OUTPUT + exit 0 + fi echo "oldest_issue_number=$ISSUE_NUMBER" >> $GITHUB_OUTPUT
130-134: Unnecessaryexit 0in conditional step.This step is already guarded by
if: steps.get_next_queued.outputs.next_issue_number == ''. Theexit 0does not prevent subsequent steps from running because each step'sifcondition is evaluated independently. The next step on line 136 already has its own guard condition.The step is informational but doesn't affect flow. You could remove it or convert it to a simple echo without exit:
🔎 Proposed simplification
- name: Exit if no queued issues if: steps.get_next_queued.outputs.next_issue_number == '' run: | echo "No queued issues." - exit 0
136-148: Duplicate logic for label update and trigger.Lines 136-148 duplicate the logic from lines 40-52 in the
get_issue_and_assignjob. Consider extracting to a composite action or reusable workflow to reduce maintenance burden..github/claude/system_prompt.txt (1)
184-186: Clarify that${BRANCH_NAME}is a runtime placeholder, not envsubst variable.
${BRANCH_NAME}here refers to the branch name Claude should use at runtime (as defined in the workflow section lines 50-51), not an environment variable substituted byenvsubst. This is clear in context but could be made explicit.
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
.github/claude/system_prompt.txt.github/workflows/claude_runner.yml
🧰 Additional context used
🪛 LanguageTool
.github/claude/system_prompt.txt
[style] ~78-~78: As an alternative to the over-used intensifier ‘very’, consider replacing this phrase.
Context: ...ssue is related to by reading the issue very carefully.Then accroding to the issue type,apply ...
(EN_WEAK_ADJECTIVE)
[uncategorized] ~98-~98: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...n state or cookie requirements * Rate limiting or temporary access restrictions -...
(EN_COMPOUND_ADJECTIVE_INTERNAL)
🔇 Additional comments (5)
.github/workflows/claude_runner.yml (2)
10-12: LGTM!The concurrency configuration correctly prevents parallel executions while not canceling in-progress runs, which is appropriate for an agent that processes issues sequentially.
112-114:Bash(*)grants unrestricted shell access.The
Bash(*)tool permission allows Claude to execute arbitrary shell commands. While other Bash permissions are scoped (e.g.,Bash(git:*),Bash(mkdocs:*)), the wildcard overrides these restrictions. If the intent is to restrict to specific tools, removeBash(*).Is the
Bash(*)permission intentional, or should Claude be restricted to only the explicitly listed tool patterns (git,mkdocs,python3,pip3)?.github/claude/system_prompt.txt (3)
13-16: LGTM on multi-version PR requirements.The emphasis on creating PRs for all versions is clear and well-documented with appropriate priority indicators.
91-101: Well-designed external link verification guidance.The instructions appropriately account for false positives from automated HTTP checks and provide reasonable fallback strategies for handling transient access issues.
169-180: Clear scope differentiation for style compliance.The distinction between "full compliance for new documents" vs "only new content for existing documents" is well-defined and prevents unnecessary churn in existing documentation.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI Agents
In @.github/workflows/auto_label.yml:
- Around line 76-86: The curl call that sets RESPONSE lacks HTTP/status and
error handling so failures will be parsed as if successful; update the request
around the RESPONSE variable to capture both body and HTTP status (e.g., using
curl -w to append status or separate variables), check the status code before
piping to jq, log the full RESPONSE on non-2xx, and either exit the workflow or
set a safe default CATEGORY; ensure the CATEGORY extraction only runs when the
response structure is valid (check for .content[0].text) and handle jq failures
gracefully.
🧹 Nitpick comments (2)
.github/workflows/auto_label.yml (2)
16-16: Remove unused dependency.The
@anthropic-ai/claude-codepackage is installed but never used in this workflow. The Claude API is called directly viacurlat lines 77-81 instead.🔎 Proposed fix
- npm install -g @anthropic-ai/claude-code
88-94: Validate category against expected values.The current logic accepts any non-empty response from Claude that isn't "null" or "none". If Claude returns unexpected text, the label will still be added incorrectly. Consider validating against the expected categories.
🔎 Proposed fix
# Add claude:queue label if a valid category is returned - if [[ "$CATEGORY" != "" && "$CATEGORY" != "null" && "$CATEGORY" != "none" ]]; then + VALID_CATEGORIES="broken-links spelling-mistakes grammatical-errors formatting-issues suggestions" + if [[ " $VALID_CATEGORIES " =~ " $CATEGORY " ]]; then echo "Adding claude:queue label" gh issue edit $ISSUE_NUMBER --add-label "AI-Agent/Queued" --repo ${{ github.repository }} else - echo "No valid category returned. Not adding label." + echo "No valid category returned (got: '$CATEGORY'). Not adding label." fi
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
.github/workflows/auto_label.yml
🔇 Additional comments (2)
.github/workflows/auto_label.yml (2)
18-42: LGTM!The pre-check logic correctly filters out issues that already have assignees or are tagged for Hacktoberfest, preventing unnecessary processing.
80-80: Theanthropic-version: 2023-06-01header is the current Anthropic API version as of January 2026 and requires no changes.Likely an incorrect or invalid review comment.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In @.github/workflows/auto_label.yml:
- Around line 24-25: The workflow currently assigns ISSUE_TITLE and ISSUE_BODY
as shell variables then interpolates them into PROMPT, which risks shell
injection; instead stop direct shell interpolation and let jq construct the full
prompt string safely by passing ISSUE_TITLE and ISSUE_BODY as jq --arg inputs
(or by reading them inside jq) and building the JSON/prompt there, then use jq's
output as PROMPT — update where PROMPT is created (references to PROMPT,
ISSUE_TITLE, ISSUE_BODY) to use jq for safe string construction rather than
embedding untrusted content into shell variables.
🧹 Nitpick comments (2)
.github/workflows/auto_label.yml (2)
12-16: Remove unnecessary CLI installations.The
ghCLI is pre-installed on GitHub-hosted runners, andclaude-codeis installed but never used—the workflow calls the Claude API directly viacurl.♻️ Proposed fix
- name: Install GitHub & Claude CLI run: | - sudo apt-get update - sudo apt-get install -y gh - npm install -g @anthropic-ai/claude-code + echo "Using pre-installed gh CLI"
100-106: Consider validating the category against expected values.The condition allows any category value that isn't empty,
null, ornone. While the label applied is alwaysAI-Agent/Queued, validating against the expected categories would make the workflow more robust against unexpected Claude responses.♻️ Proposed fix
# Add AI-Agent/Queued label if a valid category is returned - if [[ "$CATEGORY" != "" && "$CATEGORY" != "null" && "$CATEGORY" != "none" ]]; then + VALID_CATEGORIES="broken-links spelling-mistakes grammatical-errors formatting-issues suggestions" + if [[ " $VALID_CATEGORIES " =~ " $CATEGORY " ]]; then echo "Adding AI-Agent/Queued label" gh issue edit $ISSUE_NUMBER --add-label "AI-Agent/Queued" --repo ${{ github.repository }} else - echo "No valid category returned or parsing failed. Not adding label." + echo "Category '$CATEGORY' not in valid list or parsing failed. Not adding label." fi
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
.github/workflows/auto_label.yml
🔇 Additional comments (3)
.github/workflows/auto_label.yml (3)
1-6: LGTM!The workflow trigger configuration is appropriate for auto-labeling issues on creation and edit.
76-98: Error handling looks good.The implementation properly captures HTTP status, fails on non-2xx responses, and handles JSON parsing errors gracefully with a fallback. This addresses the concerns from the previous review.
71-71: No action needed — model identifier is valid.The model
claude-sonnet-4-5-20250929is a valid Anthropic API model identifier released in 2025. The date-based naming format follows Anthropic's standard convention (also aliased asclaude-sonnet-4-5).
…ally for improved context
…for improved context
1 participant
Purpose
Enable automated documentation issue fixing for the WSO2 BI documentation repository. Addresses the need for faster resolution of common documentation issues (broken links, spelling, grammar, formatting) across multiple documentation versions (docs-1.4.0, docs-1.3.0, main).
Goals
Approach
Added three configuration files:
1. Claude Runner Workflow (
.github/workflows/claude_runner.yml):@wso2-engineering-botmentions in issue commentsAI-Agent/Queuedlabelmkdocs build --strict2. Auto Label Workflow (
.github/workflows/auto_label.yml):Hacktoberfestlabelbroken-links,spelling-mistakes,grammatical-errors,formatting-issues, orsuggestionsAI-Agent/Queuedlabel if a valid documentation issue is detected3. System Prompt (
.github/claude/system_prompt.txt):fixing-issue-{ISSUE_NUMBER}-{VERSION}-{TIMESTAMP}Summary by CodeRabbit
New Features
Chores
✏️ Tip: You can customize this high-level summary in your review settings.