Skip to content

Commit a3d082a

Browse files
Swiftyosclaudentindle
authored
feat(backend): snapshot test responses (#10039)
This pull request introduces a comprehensive backend testing guide and adds new tests for analytics logging and various API endpoints, focusing on snapshot testing. It also includes corresponding snapshot files for these tests. Below are the most significant changes: ### Documentation Updates: * Added a detailed `TESTING.md` file to the backend, providing a guide for running tests, snapshot testing, writing API route tests, and best practices. It includes examples for mocking, fixtures, and CI/CD integration. ### Analytics Logging Tests: * Implemented tests for logging raw metrics and analytics in `analytics_test.py`, covering success scenarios, various input values, invalid requests, and complex nested data. These tests utilize snapshot testing for response validation. * Added snapshot files for analytics logging tests, including responses for success cases, various metric values, and complex analytics data. [[1]](diffhunk://#diff-654bc5aa1951008ec5c110a702279ef58709ee455ba049b9fa825fa60f7e3869R1-R3) [[2]](diffhunk://#diff-e0a434b107abc71aeffb7d7989dbfd8f466b5e53f8dea25a87937ec1b885b122R1-R3) [[3]](diffhunk://#diff-dd0bc0b72264de1a0c0d3bd0c54ad656061317f425e4de461018ca51a19171a0R1-R3) [[4]](diffhunk://#diff-63af007073db553d04988544af46930458a768544cabd08412265e0818320d11R1-R30) ### Snapshot Files for API Endpoints: * Added snapshot files for various API endpoint tests, such as: - Graph-related operations (`graphs_get_single_response`, `graphs_get_all_response`, `blocks_get_all_response`). [[1]](diffhunk://#diff-b25dba271606530cfa428c00073d7e016184a7bb22166148ab1726b3e113dda8R1-R29) [[2]](diffhunk://#diff-1054e58ec3094715660f55bfba1676d65b6833a81a91a08e90ad57922444d056R1-R31) [[3]](diffhunk://#diff-cfd403ab6f3efc89188acaf993d85e6f792108d1740c7e7149eb05efb73d918dR1-R14) - User-related operations (`auth_get_or_create_user_response`, `auth_update_email_response`). [[1]](diffhunk://#diff-49e65ab1eb6af4d0163a6c54ed10be621ce7336b2ab5d47d47679bfaefdb7059R1-R5) [[2]](diffhunk://#diff-ac1216f96878bd4356454c317473654d5d5c7c180125663b80b0b45aa5ab52cbR1-R3) - Credit-related operations (`credits_get_balance_response`, `credits_get_auto_top_up_response`, `credits_top_up_request_response`). [[1]](diffhunk://#diff-189488f8da5be74d80ac3fb7f84f1039a408573184293e9ba2e321d535c57cddR1-R3) [[2]](diffhunk://#diff-ba3c4a6853793cbed24030cdccedf966d71913451ef8eb4b2c4f426ef18ed87aR1-R4) [[3]](diffhunk://#diff-43d7daa0c82070a9b6aee88a774add8e87533e630bbccbac5a838b7a7ae56a75R1-R3) - Graph execution and deletion (`blocks_execute_response`, `graphs_delete_response`). [[1]](diffhunk://#diff-a2ade7d646ad85a2801e7ff39799a925a612548a1cdd0ed99b44dd870d1465b5R1-R12) [[2]](diffhunk://#diff-c0d1cd0a8499ee175ce3007c3a87ba5f3235ce02d38ce837560b36a44fdc4a22R1-R3)## Summary - add pytest-snapshot to backend dev requirements - snapshot server route response JSONs - mention how to update stored snapshots ## Testing - `poetry run format` - `poetry run test` ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: <!-- Put your test plan here: --> - [x] run poetry run test --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Nicholas Tindle <[email protected]>
1 parent a5ff8e8 commit a3d082a

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

71 files changed

+4917
-1716
lines changed

‎autogpt_platform/CLAUDE.md‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Repository Overview
6+
7+
AutoGPT Platform is a monorepo containing:
8+
- **Backend** (`/backend`): Python FastAPI server with async support
9+
- **Frontend** (`/frontend`): Next.js React application
10+
- **Shared Libraries** (`/autogpt_libs`): Common Python utilities
11+
12+
## Essential Commands
13+
14+
### Backend Development
15+
```bash
16+
# Install dependencies
17+
cd backend && poetry install
18+
19+
# Run database migrations
20+
poetry run prisma migrate dev
21+
22+
# Start all services (database, redis, rabbitmq)
23+
docker compose up -d
24+
25+
# Run the backend server
26+
poetry run serve
27+
28+
# Run tests
29+
poetry run test
30+
31+
# Run specific test
32+
poetry run pytest path/to/test_file.py::test_function_name
33+
34+
# Lint and format
35+
poetry run format # Black + isort
36+
poetry run lint # ruff
37+
```
38+
More details can be found in TESTING.md
39+
40+
#### Creating/Updating Snapshots
41+
42+
When you first write a test or when the expected output changes:
43+
44+
```bash
45+
poetry run pytest path/to/test.py --snapshot-update
46+
```
47+
48+
⚠️ **Important**: Always review snapshot changes before committing! Use `git diff` to verify the changes are expected.
49+
50+
51+
### Frontend Development
52+
```bash
53+
# Install dependencies
54+
cd frontend && npm install
55+
56+
# Start development server
57+
npm run dev
58+
59+
# Run E2E tests
60+
npm run test
61+
62+
# Run Storybook for component development
63+
npm run storybook
64+
65+
# Build production
66+
npm run build
67+
68+
# Type checking
69+
npm run type-check
70+
```
71+
72+
## Architecture Overview
73+
74+
### Backend Architecture
75+
- **API Layer**: FastAPI with REST and WebSocket endpoints
76+
- **Database**: PostgreSQL with Prisma ORM, includes pgvector for embeddings
77+
- **Queue System**: RabbitMQ for async task processing
78+
- **Execution Engine**: Separate executor service processes agent workflows
79+
- **Authentication**: JWT-based with Supabase integration
80+
81+
### Frontend Architecture
82+
- **Framework**: Next.js App Router with React Server Components
83+
- **State Management**: React hooks + Supabase client for real-time updates
84+
- **Workflow Builder**: Visual graph editor using @xyflow/react
85+
- **UI Components**: Radix UI primitives with Tailwind CSS styling
86+
- **Feature Flags**: LaunchDarkly integration
87+
88+
### Key Concepts
89+
1. **Agent Graphs**: Workflow definitions stored as JSON, executed by the backend
90+
2. **Blocks**: Reusable components in `/backend/blocks/` that perform specific tasks
91+
3. **Integrations**: OAuth and API connections stored per user
92+
4. **Store**: Marketplace for sharing agent templates
93+
94+
### Testing Approach
95+
- Backend uses pytest with snapshot testing for API responses
96+
- Test files are colocated with source files (`*_test.py`)
97+
- Frontend uses Playwright for E2E tests
98+
- Component testing via Storybook
99+
100+
### Database Schema
101+
Key models (defined in `/backend/schema.prisma`):
102+
- `User`: Authentication and profile data
103+
- `AgentGraph`: Workflow definitions with version control
104+
- `AgentGraphExecution`: Execution history and results
105+
- `AgentNode`: Individual nodes in a workflow
106+
- `StoreListing`: Marketplace listings for sharing agents
107+
108+
### Environment Configuration
109+
- Backend: `.env` file in `/backend`
110+
- Frontend: `.env.local` file in `/frontend`
111+
- Both require Supabase credentials and API keys for various services
112+
113+
### Common Development Tasks
114+
115+
**Adding a new block:**
116+
1. Create new file in `/backend/backend/blocks/`
117+
2. Inherit from `Block` base class
118+
3. Define input/output schemas
119+
4. Implement `run` method
120+
5. Register in block registry
121+
122+
**Modifying the API:**
123+
1. Update route in `/backend/backend/server/routers/`
124+
2. Add/update Pydantic models in same directory
125+
3. Write tests alongside the route file
126+
4. Run `poetry run test` to verify
127+
128+
**Frontend feature development:**
129+
1. Components go in `/frontend/src/components/`
130+
2. Use existing UI components from `/frontend/src/components/ui/`
131+
3. Add Storybook stories for new components
132+
4. Test with Playwright if user-facing

‎autogpt_platform/backend/TESTING.md‎

Lines changed: 237 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,237 @@
1+
# Backend Testing Guide
2+
3+
This guide covers testing practices for the AutoGPT Platform backend, with a focus on snapshot testing for API endpoints.
4+
5+
## Table of Contents
6+
- [Overview](#overview)
7+
- [Running Tests](#running-tests)
8+
- [Snapshot Testing](#snapshot-testing)
9+
- [Writing Tests for API Routes](#writing-tests-for-api-routes)
10+
- [Best Practices](#best-practices)
11+
12+
## Overview
13+
14+
The backend uses pytest for testing with the following key libraries:
15+
- `pytest` - Test framework
16+
- `pytest-asyncio` - Async test support
17+
- `pytest-mock` - Mocking support
18+
- `pytest-snapshot` - Snapshot testing for API responses
19+
20+
## Running Tests
21+
22+
### Run all tests
23+
```bash
24+
poetry run test
25+
```
26+
27+
### Run specific test file
28+
```bash
29+
poetry run pytest path/to/test_file.py
30+
```
31+
32+
### Run with verbose output
33+
```bash
34+
poetry run pytest -v
35+
```
36+
37+
### Run with coverage
38+
```bash
39+
poetry run pytest --cov=backend
40+
```
41+
42+
## Snapshot Testing
43+
44+
Snapshot testing captures the output of your code and compares it against previously saved snapshots. This is particularly useful for testing API responses.
45+
46+
### How Snapshot Testing Works
47+
48+
1. First run: Creates snapshot files in `snapshots/` directories
49+
2. Subsequent runs: Compares output against saved snapshots
50+
3. Changes detected: Test fails if output differs from snapshot
51+
52+
### Creating/Updating Snapshots
53+
54+
When you first write a test or when the expected output changes:
55+
56+
```bash
57+
poetry run pytest path/to/test.py --snapshot-update
58+
```
59+
60+
⚠️ **Important**: Always review snapshot changes before committing! Use `git diff` to verify the changes are expected.
61+
62+
### Snapshot Test Example
63+
64+
```python
65+
import json
66+
from pytest_snapshot.plugin import Snapshot
67+
68+
def test_api_endpoint(snapshot: Snapshot):
69+
response = client.get("/api/endpoint")
70+
71+
# Snapshot the response
72+
snapshot.snapshot_dir = "snapshots"
73+
snapshot.assert_match(
74+
json.dumps(response.json(), indent=2, sort_keys=True),
75+
"endpoint_response"
76+
)
77+
```
78+
79+
### Best Practices for Snapshots
80+
81+
1. **Use descriptive names**: `"user_list_response"` not `"response1"`
82+
2. **Sort JSON keys**: Ensures consistent snapshots
83+
3. **Format JSON**: Use `indent=2` for readable diffs
84+
4. **Exclude dynamic data**: Remove timestamps, IDs, etc. that change between runs
85+
86+
Example of excluding dynamic data:
87+
```python
88+
response_data = response.json()
89+
# Remove dynamic fields for snapshot
90+
response_data.pop("created_at", None)
91+
response_data.pop("id", None)
92+
93+
snapshot.snapshot_dir = "snapshots"
94+
snapshot.assert_match(
95+
json.dumps(response_data, indent=2, sort_keys=True),
96+
"static_response_data"
97+
)
98+
```
99+
100+
## Writing Tests for API Routes
101+
102+
### Basic Structure
103+
104+
```python
105+
import json
106+
import fastapi
107+
import fastapi.testclient
108+
import pytest
109+
from pytest_snapshot.plugin import Snapshot
110+
111+
from backend.server.v2.myroute import router
112+
113+
app = fastapi.FastAPI()
114+
app.include_router(router)
115+
client = fastapi.testclient.TestClient(app)
116+
117+
def test_endpoint_success(snapshot: Snapshot):
118+
response = client.get("/endpoint")
119+
assert response.status_code == 200
120+
121+
# Test specific fields
122+
data = response.json()
123+
assert data["status"] == "success"
124+
125+
# Snapshot the full response
126+
snapshot.snapshot_dir = "snapshots"
127+
snapshot.assert_match(
128+
json.dumps(data, indent=2, sort_keys=True),
129+
"endpoint_success_response"
130+
)
131+
```
132+
133+
### Testing with Authentication
134+
135+
```python
136+
def override_auth_middleware():
137+
return {"sub": "test-user-id"}
138+
139+
def override_get_user_id():
140+
return "test-user-id"
141+
142+
app.dependency_overrides[auth_middleware] = override_auth_middleware
143+
app.dependency_overrides[get_user_id] = override_get_user_id
144+
```
145+
146+
### Mocking External Services
147+
148+
```python
149+
def test_external_api_call(mocker, snapshot):
150+
# Mock external service
151+
mock_response = {"external": "data"}
152+
mocker.patch(
153+
"backend.services.external_api.call",
154+
return_value=mock_response
155+
)
156+
157+
response = client.post("/api/process")
158+
assert response.status_code == 200
159+
160+
snapshot.snapshot_dir = "snapshots"
161+
snapshot.assert_match(
162+
json.dumps(response.json(), indent=2, sort_keys=True),
163+
"process_with_external_response"
164+
)
165+
```
166+
167+
## Best Practices
168+
169+
### 1. Test Organization
170+
- Place tests next to the code: `routes.py` → `routes_test.py`
171+
- Use descriptive test names: `test_create_user_with_invalid_email`
172+
- Group related tests in classes when appropriate
173+
174+
### 2. Test Coverage
175+
- Test happy path and error cases
176+
- Test edge cases (empty data, invalid formats)
177+
- Test authentication and authorization
178+
179+
### 3. Snapshot Testing Guidelines
180+
- Review all snapshot changes carefully
181+
- Don't snapshot sensitive data
182+
- Keep snapshots focused and minimal
183+
- Update snapshots intentionally, not accidentally
184+
185+
### 4. Async Testing
186+
- Use regular `def` for FastAPI TestClient tests
187+
- Use `async def` with `@pytest.mark.asyncio` for testing async functions directly
188+
189+
### 5. Fixtures
190+
Create reusable fixtures for common test data:
191+
192+
```python
193+
@pytest.fixture
194+
def sample_user():
195+
return {
196+
"email": "[email protected]",
197+
"name": "Test User"
198+
}
199+
200+
def test_create_user(sample_user, snapshot):
201+
response = client.post("/users", json=sample_user)
202+
# ... test implementation
203+
```
204+
205+
## CI/CD Integration
206+
207+
The GitHub Actions workflow automatically runs tests on:
208+
- Pull requests
209+
- Pushes to main branch
210+
211+
Snapshot tests work in CI by:
212+
1. Committing snapshot files to the repository
213+
2. CI compares against committed snapshots
214+
3. Fails if snapshots don't match
215+
216+
## Troubleshooting
217+
218+
### Snapshot Mismatches
219+
- Review the diff carefully
220+
- If changes are expected: `poetry run pytest --snapshot-update`
221+
- If changes are unexpected: Fix the code causing the difference
222+
223+
### Async Test Issues
224+
- Ensure async functions use `@pytest.mark.asyncio`
225+
- Use `AsyncMock` for mocking async functions
226+
- FastAPI TestClient handles async automatically
227+
228+
### Import Errors
229+
- Check that all dependencies are in `pyproject.toml`
230+
- Run `poetry install` to ensure dependencies are installed
231+
- Verify import paths are correct
232+
233+
## Summary
234+
235+
Snapshot testing provides a powerful way to ensure API responses remain consistent. Combined with traditional assertions, it creates a robust test suite that catches regressions while remaining maintainable.
236+
237+
Remember: Good tests are as important as good code!

‎autogpt_platform/backend/backend/server/conftest.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
"""Common test fixtures for server tests."""
2+
3+
import pytest
4+
from pytest_snapshot.plugin import Snapshot
5+
6+
7+
@pytest.fixture
8+
def configured_snapshot(snapshot: Snapshot) -> Snapshot:
9+
"""Pre-configured snapshot fixture with standard settings."""
10+
snapshot.snapshot_dir = "snapshots"
11+
return snapshot
12+
13+
14+
# Test ID constants
15+
TEST_USER_ID = "test-user-id"
16+
ADMIN_USER_ID = "admin-user-id"
17+
TARGET_USER_ID = "target-user-id"

0 commit comments

Comments
 (0)