Skip to content

Commit 2179059

Browse files
critesjoshcritesjosh
authored
Merge pull request #613 from AztecProtocol/jc/docs-source-reviewer
feat: Add Claude-powered documentation suggestions tool
2 parents b771771 + 7d59bff commit 2179059

File tree

14 files changed

+2731
-0
lines changed

14 files changed

+2731
-0
lines changed

‎.github/workflows/doc-suggestions.yml‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
name: Generate Documentation Suggestions
2+
3+
on:
4+
schedule:
5+
# Run nightly at 2 AM UTC
6+
- cron: '0 2 * * *'
7+
workflow_dispatch:
8+
inputs:
9+
lookback_days:
10+
description: 'Number of days to look back for changes'
11+
default: '7'
12+
type: string
13+
14+
jobs:
15+
generate:
16+
runs-on: ubuntu-latest
17+
permissions:
18+
contents: write
19+
pull-requests: write
20+
21+
steps:
22+
- name: Checkout dev-rel
23+
uses: actions/checkout@v4
24+
25+
- name: Setup Node.js
26+
uses: actions/setup-node@v4
27+
with:
28+
node-version: '20'
29+
cache: 'npm'
30+
cache-dependency-path: tooling/doc-suggestions/package-lock.json
31+
32+
- name: Install dependencies
33+
working-directory: tooling/doc-suggestions
34+
run: npm install
35+
36+
- name: Generate suggestions report
37+
working-directory: tooling/doc-suggestions
38+
env:
39+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
40+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
41+
LOOKBACK_DAYS: ${{ inputs.lookback_days || '7' }}
42+
run: npx tsx src/index.ts
43+
44+
- name: Upload report as artifact
45+
uses: actions/upload-artifact@v4
46+
with:
47+
name: doc-suggestions-${{ github.run_id }}
48+
path: tooling/doc-suggestions/reports/
49+
retention-days: 30
50+
51+
- name: Create PR with report
52+
env:
53+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
54+
run: |
55+
# Check if latest.md exists and has content
56+
if [ ! -f tooling/doc-suggestions/reports/latest.md ]; then
57+
echo "No report generated"
58+
exit 0
59+
fi
60+
61+
DATE=$(date +%Y-%m-%d)
62+
BRANCH_NAME="docs/suggestions-${DATE}"
63+
64+
git config user.name "github-actions[bot]"
65+
git config user.email "github-actions[bot]@users.noreply.github.com"
66+
67+
# Create new branch
68+
git checkout -b "$BRANCH_NAME"
69+
70+
# Copy to dated report in repo root reports directory
71+
mkdir -p reports
72+
cp tooling/doc-suggestions/reports/latest.md "reports/doc-suggestions-${DATE}.md"
73+
74+
# Stage changes
75+
git add reports/
76+
git add tooling/doc-suggestions/reports/
77+
78+
# Only proceed if there are changes
79+
if git diff --staged --quiet; then
80+
echo "No changes to commit"
81+
exit 0
82+
fi
83+
84+
git commit -m "docs: Add documentation suggestions for ${DATE}"
85+
git push -u origin "$BRANCH_NAME"
86+
87+
# Count suggestions by priority
88+
HIGH=$(grep -c "^### High Priority" "reports/doc-suggestions-${DATE}.md" || echo "0")
89+
MEDIUM=$(grep -c "^### Medium Priority" "reports/doc-suggestions-${DATE}.md" || echo "0")
90+
LOW=$(grep -c "^### Low Priority" "reports/doc-suggestions-${DATE}.md" || echo "0")
91+
92+
# Create PR and assign to DevRel team
93+
gh pr create \
94+
--title "docs: Documentation suggestions for ${DATE}" \
95+
--body "## Documentation Update Suggestions
96+
97+
This PR contains automatically generated suggestions for documentation updates based on recent source code changes in aztec-packages.
98+
99+
**Review the report**: \`reports/doc-suggestions-${DATE}.md\`
100+
101+
### How to use
102+
1. Review the suggestions in the report
103+
2. For items to address, copy the relevant section
104+
3. Open Claude Code in aztec-packages and paste the suggestion
105+
106+
---
107+
🤖 Generated by the doc-suggestions tool" \
108+
--assignee "@AztecProtocol/devrel"

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,4 @@ proofs
33
debug_*.json
44
crs
55
yarn-error.log
6+
.env

‎reports/.gitkeep‎

Whitespace-only changes.

‎tooling/doc-suggestions/.gitignore‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Dependencies
2+
node_modules/
3+
4+
# Build output
5+
dist/
6+
7+
# Reports (except .gitkeep)
8+
reports/*
9+
!reports/.gitkeep
10+
11+
# Environment files
12+
.env
13+
.env.local
14+
15+
# IDE
16+
.idea/
17+
.vscode/
18+
*.swp
19+
*.swo
20+
21+
# OS
22+
.DS_Store
23+
Thumbs.db

‎tooling/doc-suggestions/README.md‎

Lines changed: 239 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,239 @@
1+
# Documentation Suggestions Tool
2+
3+
A Claude-powered tool that scans the `aztec-packages` repository for source code changes and generates suggestions for documentation updates.
4+
5+
## Overview
6+
7+
This tool addresses a common DevRel challenge: keeping documentation in sync with rapidly evolving source code. It:
8+
9+
1. **Scans** recent commits in `aztec-packages` for file changes
10+
2. **Identifies** documentation files that reference those changed files
11+
3. **Analyzes** the changes using Claude to determine if docs need updating
12+
4. **Generates** a prioritized markdown report with specific suggestions
13+
14+
The output is designed to be fed directly to Claude Code for making the actual documentation updates.
15+
16+
## How It Works
17+
18+
```
19+
┌─────────────────────────────────────────────────────────────────┐
20+
│ Nightly Cron (2 AM UTC) │
21+
└─────────────────────────────────────────────────────────────────┘
22+
│
23+
▼
24+
┌─────────────────────────────────────────────────────────────────┐
25+
│ GitHub Scanner │
26+
│ - Fetches commits from configured branch (default: next) │
27+
│ - Gets file changes and associated PRs │
28+
│ - Scans docs/ directory for `references:` frontmatter │
29+
└─────────────────────────────────────────────────────────────────┘
30+
│
31+
▼
32+
┌─────────────────────────────────────────────────────────────────┐
33+
│ Reference Analyzer │
34+
│ - Matches changed files to doc references │
35+
│ - Identifies stale docs (source newer than doc) │
36+
│ - Calculates staleness metrics │
37+
└─────────────────────────────────────────────────────────────────┘
38+
│
39+
▼
40+
┌─────────────────────────────────────────────────────────────────┐
41+
│ Claude Analysis │
42+
│ - Reviews source diff + current doc content │
43+
│ - Determines if update is needed │
44+
│ - Generates specific update suggestions │
45+
│ - Assigns priority (high/medium/low) │
46+
└─────────────────────────────────────────────────────────────────┘
47+
│
48+
▼
49+
┌─────────────────────────────────────────────────────────────────┐
50+
│ Report Generator │
51+
│ - Creates markdown report grouped by priority │
52+
│ - Includes copy-paste sections for Claude Code │
53+
│ - Saves to reports/ directory │
54+
└─────────────────────────────────────────────────────────────────┘
55+
```
56+
57+
## Documentation References
58+
59+
The tool relies on documentation files having a `references` field in their YAML frontmatter:
60+
61+
```markdown
62+
---
63+
title: CLI Reference
64+
references: ["yarn-project/aztec/src/cli/cli.ts", "yarn-project/aztec/src/cli/cmds/*.ts"]
65+
---
66+
67+
# CLI Reference
68+
69+
...
70+
```
71+
72+
When files matching these references change, the tool flags the documentation for review.
73+
74+
## Installation
75+
76+
```bash
77+
cd tooling/doc-suggestions
78+
npm install
79+
```
80+
81+
## Usage
82+
83+
### Local Development
84+
85+
```bash
86+
# Set required environment variables
87+
export GITHUB_TOKEN="ghp_your_token_here"
88+
export ANTHROPIC_API_KEY="sk-ant-your_key_here"
89+
90+
# Optional: customize settings
91+
export LOOKBACK_DAYS=14 # Default: 7 days
92+
export BRANCH=next # Default: next (aztec-packages main branch)
93+
94+
# Run the tool
95+
npx tsx src/index.ts
96+
```
97+
98+
### GitHub Actions
99+
100+
The tool runs automatically via GitHub Actions:
101+
102+
- **Schedule**: Daily at 2 AM UTC
103+
- **Manual**: Trigger via Actions tab with optional lookback days parameter
104+
105+
Reports are:
106+
1. Uploaded as workflow artifacts (retained 30 days)
107+
2. Committed to the `reports/` directory in the repo
108+
109+
## Configuration
110+
111+
| Environment Variable | Required | Default | Description |
112+
|---------------------|----------|---------|-------------|
113+
| `GITHUB_TOKEN` | Yes | - | GitHub API token for reading aztec-packages |
114+
| `ANTHROPIC_API_KEY` | Yes | - | Claude API key for generating suggestions |
115+
| `LOOKBACK_DAYS` | No | `7` | Number of days to scan for changes |
116+
| `OUTPUT_DIR` | No | `./reports` | Directory for output reports |
117+
| `REPO` | No | `aztec-packages` | Repository to scan |
118+
| `BRANCH` | No | `next` | Branch to scan (aztec-packages uses `next` as main) |
119+
120+
## Output Format
121+
122+
Reports are generated as markdown files with the following structure:
123+
124+
```markdown
125+
# Documentation Update Suggestions
126+
127+
Generated: 2024-01-15T02:00:00Z
128+
Scan period: Last 7 days
129+
130+
## Summary
131+
- High priority: 2
132+
- Medium priority: 3
133+
- Low priority: 1
134+
135+
## Suggestions
136+
137+
### High Priority
138+
139+
#### `docs/docs-network/reference/cli_reference.md`
140+
141+
**Source file**: `yarn-project/aztec/src/cli/cli.ts`
142+
**Related PR**: #12345
143+
144+
**What changed**: A new `--network` CLI flag was added...
145+
146+
**Suggested updates**:
147+
- Add `--network` flag to the CLI reference table
148+
- Include description of valid network values
149+
```
150+
151+
## Using Suggestions with Claude Code
152+
153+
1. Open the generated report
154+
2. Find a suggestion you want to address
155+
3. Expand the "Copy for Claude Code" section
156+
4. Open Claude Code in the `aztec-packages` repo
157+
5. Paste the suggestion and ask Claude to make the update
158+
159+
Example prompt:
160+
```
161+
The following documentation needs updating based on source code changes:
162+
163+
Documentation file: docs/docs-network/reference/cli_reference.md
164+
Source file: yarn-project/aztec/src/cli/cli.ts
165+
Related PR: https://github.com/AztecProtocol/aztec-packages/pull/12345
166+
167+
What changed: A new --network CLI flag was added with options for selecting the target network.
168+
169+
Please make these updates to the documentation:
170+
- Add --network flag to the CLI reference table
171+
- Include description of valid network values
172+
- Add example usage showing network selection
173+
```
174+
175+
## Priority Levels
176+
177+
| Priority | Criteria |
178+
|----------|----------|
179+
| **High** | Breaking changes, new required parameters, security-related changes, major feature additions |
180+
| **Medium** | New optional features, changed defaults, significant API changes |
181+
| **Low** | Minor changes, internal refactoring visible to users, cosmetic changes |
182+
183+
## Cost Estimation
184+
185+
- **Model**: Claude Sonnet
186+
- **Per suggestion**: ~$0.015
187+
- **Typical nightly run**: 5-15 suggestions
188+
- **Estimated monthly cost**: $2-10
189+
190+
## Expanding Coverage
191+
192+
To make this tool more useful, add `references` frontmatter to more documentation files in `aztec-packages`. The tool scans all markdown files in the `docs/` directory.
193+
194+
High-value targets for adding references:
195+
- CLI reference documentation → reference the CLI source files
196+
- API documentation → reference the API implementation files
197+
- Tutorial/guide code examples → reference the example source files
198+
199+
## Architecture
200+
201+
```
202+
src/
203+
├── index.ts # Entry point, orchestrates the pipeline
204+
├── github-scanner.ts # GitHub API interactions
205+
├── reference-analyzer.ts # Staleness detection logic
206+
├── claude-client.ts # Claude API wrapper
207+
└── report-generator.ts # Markdown report generation
208+
```
209+
210+
## Troubleshooting
211+
212+
### No suggestions generated
213+
214+
- Check that docs have `references:` frontmatter
215+
- Verify the referenced files exist and have recent changes
216+
- Ensure `LOOKBACK_DAYS` covers the period of changes
217+
- Verify `BRANCH` is set correctly (default: `next` for aztec-packages)
218+
219+
### 404 errors when scanning
220+
221+
- Ensure the `BRANCH` environment variable matches the repository's main branch
222+
- For aztec-packages, use `BRANCH=next` (the default)
223+
224+
### Rate limiting
225+
226+
- GitHub API: The tool fetches commit details sequentially to avoid rate limits
227+
- Claude API: 500ms delay between suggestions
228+
229+
### Authentication errors
230+
231+
- `GITHUB_TOKEN`: For fine-grained PATs (starting with `github_pat_`), ensure "Contents: Read" permission is granted for the target repository
232+
- `ANTHROPIC_API_KEY`: Must be a valid Claude API key
233+
234+
## Future Enhancements
235+
236+
- Slack/Discord notifications for high-priority items
237+
- Dashboard showing suggestion trends
238+
- Auto-expand references by analyzing `#include_code` macros
239+
- Optional PR comments for teams that want direct integration

0 commit comments

Comments
 (0)