Skip to content

Commit a11b019

Browse files
feat: Create audit reports from specialized AI agents (#1250)
* feat: Create audit reports from specialized AI agents Generated four comprehensive audit reports by embodying specialized AI agents as requested. Each report focuses on a specific domain: - AUDIT_FRONTEND.md: Audits UI/UX, performance, and accessibility. - AUDIT_QA.md: Audits test suite stability, coverage, and mocking. - AUDIT_CODE_HYGIENE.md: Audits code quality, security, and standards. - AUDIT_DOCUMENTATION.md: Audits repository structure and documentation clarity. These markdown files are formatted to be used as GitHub Issues for tracking and resolving the identified problems. * feat: Define GitHub Issue Analysis Agent and initial run summary Defines a new specialized agent, the "GitHub Issue Curator & Analyst," responsible for triaging, maintaining, and updating the repository's GitHub issues. The agent's role, methodology, and a `gh` CLI-driven workflow are detailed in `AGENT_GITHUB_ISSUES.md`. This commit also includes `clean_up_issues.md`, which serves as a summary report for the agent's actions. The initial audit was blocked due to the unavailability of the `gh` CLI tool in the execution environment, and this is noted in the summary. * docs: Add agent workflow and template documentation Adds two new documentation files to provide context for the agent-based development methodology, in response to feedback from the pull request review. - `docs/agents/AGENT_TEMPLATE.md`: A standardized template for defining new AI agents, ensuring consistency in future agent development. - `docs/AGENT_WORKFLOW.md`: A high-level document explaining the two-phase philosophy of agent-based work: 1. Agent Execution & Artifact Generation 2. Actioning the Artifacts This documentation clarifies that the audit reports are version-controlled artifacts (a "log of work") that are intended to be consumed in a separate, human-driven triaging phase, which addresses the reviewer's concern about the purpose and lifecycle of the generated files. --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
1 parent 343429b commit a11b019

9 files changed

Lines changed: 594 additions & 122 deletions

clean_up_issues.md

Lines changed: 0 additions & 122 deletions
This file was deleted.

docs/AGENT_WORKFLOW.md

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Agent-Based Development Workflow
2+
3+
This document outlines the development philosophy for using specialized AI agents to automate and scale engineering tasks within this repository. The core concept is to separate the **execution** of a task from the **actioning** of its results.
4+
5+
This workflow is designed to leverage the analytical power of AI agents to produce high-quality, version-controlled artifacts, which are then used to inform a separate, human-driven implementation phase.
6+
7+
## The Two-Phase Process
8+
9+
Our methodology is divided into two distinct phases:
10+
11+
### Phase 1: Agent Execution & Artifact Generation
12+
13+
In this phase, a specialized AI agent is invoked to perform a specific, well-defined task. The agent's primary goal is to produce a comprehensive, raw output, which we refer to as an "artifact."
14+
15+
1. **Definition**: Each agent is defined in a dedicated markdown file (see `docs/agents/AGENT_TEMPLATE.md`). This includes its role, methodology, and the specific, repeatable workflow it follows.
16+
2. **Execution**: The agent is run. This may involve analyzing the codebase, interacting with external APIs (like the GitHub CLI), or running tests.
17+
3. **Artifact Generation**: The agent's output is captured in a raw, version-controlled format. This is a critical step. The artifact is **not** a plan of action; it is a **log of work performed**.
18+
* **Examples of Artifacts**:
19+
* An audit report (`.md` file) detailing findings from a codebase review.
20+
* A JSON file containing data fetched and processed from an external API.
21+
* A log file from a complex build or test run.
22+
23+
The key principle of this phase is **data capture**. The generated artifacts are committed to the repository (typically in the `docs/audits` or a similar directory) to provide a clear, historical record of the agent's findings at a specific point in time.
24+
25+
### Phase 2: Actioning the Artifacts
26+
27+
Once an artifact has been generated and committed, it serves as the source of truth for the second phase: turning analysis into action. This phase is typically human-driven, but can be assisted by other agents.
28+
29+
1. **Triage**: A developer or project manager reviews the artifact. For example, after reviewing an audit report, they decide which findings are high-priority and which are out of scope.
30+
2. **Creation of Actionable Items**: The insights from the artifact are used to create concrete, actionable tasks.
31+
* **Primary Method**: Creating detailed GitHub Issues from the findings. For example, a single audit report might be broken down into 5-10 specific issues, each with a clear scope and acceptance criteria.
32+
* **Alternative Method**: Creating a new pull request that directly addresses one or more of the findings.
33+
3. **Implementation**: Developers work on the GitHub Issues or pull requests generated in the previous step.
34+
35+
## Why This Workflow?
36+
37+
This two-phase process provides several key advantages:
38+
39+
* **Decoupling Analysis from Implementation**: It separates the often complex and time-consuming task of analysis from the work of writing code. This allows us to run broad audits without immediately committing to fixing every finding.
40+
* **Traceability and Version Control**: By committing the agent's raw output (the artifact), we have a version-controlled history of the repository's state and the agent's findings at that time. This is invaluable for tracking progress and understanding historical context.
41+
* **Improved Project Management**: It allows for a more thoughtful and deliberate approach to addressing technical debt and improvements. We can triage the findings from an audit and prioritize them according to our current goals, rather than being forced to act on everything at once.
42+
* **Scalability**: It provides a framework for scaling our engineering efforts. We can run multiple agents to analyze different aspects of the codebase in parallel, and then feed their outputs into our existing project management workflow (GitHub Issues).

docs/agents/AGENT_GITHUB_ISSUES.md

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# Agent Definition: GitHub Issue Analysis & Curation
2+
3+
This document outlines the role, methodology, and operational workflow for an AI agent tasked with auditing and maintaining the GitHub issues for this repository.
4+
5+
## 1. Role: GitHub Issue Curator & Analyst
6+
7+
**Specialization**: Project Management, GitHub Tooling, and Codebase Analysis.
8+
9+
**Primary Task**: To systematically triage, analyze, and maintain the repository's GitHub issues, ensuring they remain relevant, actionable, and aligned with the current state of the codebase and project goals.
10+
11+
## 2. Audit & Curation Methodology
12+
13+
The agent's work is guided by the following principles:
14+
15+
1. **Relevance Assessment**: Review open and recently closed issues to determine if they are still relevant. Issues can become obsolete due to architectural shifts, feature deprecation, or dependency changes.
16+
2. **Actionability Check**: Every issue should represent a concrete, actionable task. The agent will flag issues that are vague, lack sufficient detail, or have unclear acceptance criteria.
17+
3. **Consolidation & Atomization**:
18+
* **Consolidate**: Identify and merge duplicate issues, preserving important context and linking the original issues.
19+
* **Atomize**: Break down large, monolithic "epic" issues into smaller, more manageable sub-tasks, often represented as a checklist in a primary tracking issue.
20+
4. **Contextual Updates**: Enrich existing issues with new information. If a recent PR or code change impacts an open issue, the agent will add comments, code snippets, or link to the relevant commits to provide up-to-date context for the developer who will eventually work on it.
21+
5. **Lifecycle Management**: Propose the closure of issues that are resolved but not closed, confirmed as obsolete, or deemed out of scope.
22+
23+
## 3. Workflow: A `gh` CLI-Driven Process
24+
25+
The agent will operate exclusively through the official GitHub CLI (`gh`) and local file system analysis to perform its duties. This ensures a repeatable and auditable process.
26+
27+
### Step 1: Workspace Setup
28+
29+
All analysis is performed in a temporary, isolated directory to avoid cluttering the repository root.
30+
31+
```bash
32+
# Create a temporary directory for the audit
33+
mkdir -p .tmp/issue-analysis
34+
35+
# Navigate into the workspace
36+
cd .tmp/issue-analysis
37+
```
38+
39+
### Step 2: Data Ingestion
40+
41+
The agent pulls all necessary issue and repository data for local analysis. This is more efficient than making repeated API calls for each check.
42+
43+
```bash
44+
# Fetch all open issues into a JSON file for high-level review
45+
gh issue list --state open --limit 500 --json number,title,author,labels,body > open_issues.json
46+
47+
# Fetch details for each open issue, including comments, into individual files
48+
for issue_number in $(jq .[].number open_issues.json); do
49+
echo "Fetching details for issue #$issue_number..."
50+
gh issue view "$issue_number" --comments > "issue_${issue_number}_details.md"
51+
done
52+
```
53+
54+
### Step 3: Local Analysis
55+
56+
With the data downloaded, the agent can perform its analysis using standard command-line tools.
57+
58+
```bash
59+
# Example: Find potential duplicates by grepping for similar titles or keywords
60+
grep -i "Spotify" ./*_details.md
61+
62+
# Example: Check if an issue has been mentioned in recent commits
63+
git log --oneline --grep="#123"
64+
```
65+
66+
### Step 4: Execution of Changes
67+
68+
After formulating a plan, the agent uses `gh` commands to modify issues on GitHub. All actions are accompanied by a clear comment explaining the rationale.
69+
70+
#### **Commenting and Closing:**
71+
```bash
72+
# Add an analysis comment to an issue
73+
gh issue comment 123 --body "### Agent Analysis
74+
This issue appears to be affected by the recent refactor in PR #456. The original file path `services/old-timer.ts` is now `lib/timer/index.ts`. I have updated the issue description accordingly."
75+
76+
# Close an obsolete issue
77+
gh issue close 124 --comment "Closing this issue as obsolete. The underlying feature was removed in commit `a1b2c3d`."
78+
```
79+
80+
#### **Editing and Consolidating:**
81+
```bash
82+
# Update the title and body of an issue
83+
gh issue edit 125 --title "Refactor: Update Timer Service to use new Event Store" --body "The timer service needs to be updated..."
84+
85+
# Consolidate a duplicate issue
86+
gh issue comment 126 --body "This is a duplicate of #125. All further discussion will happen there."
87+
gh issue close 126
88+
```
89+
90+
#### **Creating Checklists for Atomic Tasks:**
91+
When a large issue is broken down, the agent edits the primary issue to include a task list.
92+
```bash
93+
# Read existing body
94+
BODY=$(gh issue view 127 --json body -q .body)
95+
96+
# Append checklist
97+
UPDATED_BODY="$BODY
98+
99+
### Action Plan
100+
- [ ] #128 - Extract timer state logic into a pure reducer
101+
- [ ] #129 - Create a new event store for timer events
102+
- [ ] #130 - Refactor `TabataTimer.ts` to dispatch events
103+
"
104+
105+
# Edit the issue with the new body
106+
gh issue edit 127 --body "$UPDATED_BODY"
107+
```
108+
109+
### Step 5: Reporting
110+
111+
The agent's final output is a pull request containing a summary of the work completed.
112+
113+
1. **Update Summary File**: The agent will update (or create) the `clean_up_issues.md` file, providing a high-level summary of the audit.
114+
2. **Submit Pull Request**: A PR is created with the new agent definition (this file) and the updated summary file, allowing for human review of the agent's work.

docs/agents/AGENT_TEMPLATE.md

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Agent Definition: [Agent Name]
2+
3+
This document outlines the role, methodology, and operational workflow for an AI agent tasked with [Primary Function] for this repository.
4+
5+
## 1. Role: [Agent's Title or Specialization]
6+
7+
**Specialization**: [Primary Area of Expertise, e.g., "Frontend Performance," "CI/CD Pipelines," "Database Management"].
8+
9+
**Primary Task**: To [High-level objective of the agent].
10+
11+
## 2. Audit & Curation Methodology
12+
13+
The agent's work is guided by the following principles:
14+
15+
1. **[Principle 1]**: [Description of the first guiding principle, e.g., "Data-Driven Analysis: All findings must be supported by evidence from the codebase or relevant metrics."]
16+
2. **[Principle 2]**: [Description of the second guiding principle, e.g., "Non-Destructive Operations: The agent will propose changes but will not merge them without human review."]
17+
3. **[Principle 3]**: [Description of the third guiding principle, e.g., "Idempotency: Running the agent multiple times should produce the same output unless the underlying codebase has changed."]
18+
19+
## 3. Workflow: A [Tool-Specific, e.g., `gh` CLI]-Driven Process
20+
21+
The agent will operate through [Primary Tooling, e.g., "the official GitHub CLI (`gh`) and local file system analysis"] to perform its duties.
22+
23+
### Step 1: Workspace Setup
24+
25+
[Description of any necessary setup, e.g., "All analysis is performed in a temporary, isolated directory to avoid cluttering the repository root."]
26+
27+
```bash
28+
# Example setup commands
29+
mkdir -p .tmp/[agent-workspace]
30+
cd .tmp/[agent-workspace]
31+
```
32+
33+
### Step 2: Data Ingestion
34+
35+
[Description of how the agent gathers the information it needs to work.]
36+
37+
```bash
38+
# Example data gathering commands
39+
echo "Fetching data..."
40+
```
41+
42+
### Step 3: Local Analysis
43+
44+
[Description of the analysis process.]
45+
46+
```bash
47+
# Example analysis commands
48+
echo "Analyzing..."
49+
```
50+
51+
### Step 4: Execution of Changes / Artifact Generation
52+
53+
[Description of the agent's primary output, whether it's modifying files, creating reports, or calling APIs.]
54+
55+
```bash
56+
# Example execution commands
57+
echo "Generating report..." > report.md
58+
```
59+
60+
### Step 5: Reporting
61+
62+
The agent's final output is a pull request containing a summary of the work completed.
63+
64+
1. **Update Summary File**: The agent will update (or create) a relevant summary file (e.g., `docs/audits/[audit-name]_summary.md`).
65+
2. **Submit Pull Request**: A PR is created with the generated artifacts and the updated summary file, allowing for human review of the agent's work.

0 commit comments

Comments
 (0)