|
| 1 | +# Backend Testing Guide |
| 2 | + |
| 3 | +This guide covers testing practices for the AutoGPT Platform backend, with a focus on snapshot testing for API endpoints. |
| 4 | + |
| 5 | +## Table of Contents |
| 6 | +- [Overview](#overview) |
| 7 | +- [Running Tests](#running-tests) |
| 8 | +- [Snapshot Testing](#snapshot-testing) |
| 9 | +- [Writing Tests for API Routes](#writing-tests-for-api-routes) |
| 10 | +- [Best Practices](#best-practices) |
| 11 | + |
| 12 | +## Overview |
| 13 | + |
| 14 | +The backend uses pytest for testing with the following key libraries: |
| 15 | +- `pytest` - Test framework |
| 16 | +- `pytest-asyncio` - Async test support |
| 17 | +- `pytest-mock` - Mocking support |
| 18 | +- `pytest-snapshot` - Snapshot testing for API responses |
| 19 | + |
| 20 | +## Running Tests |
| 21 | + |
| 22 | +### Run all tests |
| 23 | +```bash |
| 24 | +poetry run test |
| 25 | +``` |
| 26 | + |
| 27 | +### Run specific test file |
| 28 | +```bash |
| 29 | +poetry run pytest path/to/test_file.py |
| 30 | +``` |
| 31 | + |
| 32 | +### Run with verbose output |
| 33 | +```bash |
| 34 | +poetry run pytest -v |
| 35 | +``` |
| 36 | + |
| 37 | +### Run with coverage |
| 38 | +```bash |
| 39 | +poetry run pytest --cov=backend |
| 40 | +``` |
| 41 | + |
| 42 | +## Snapshot Testing |
| 43 | + |
| 44 | +Snapshot testing captures the output of your code and compares it against previously saved snapshots. This is particularly useful for testing API responses. |
| 45 | + |
| 46 | +### How Snapshot Testing Works |
| 47 | + |
| 48 | +1. First run: Creates snapshot files in `snapshots/` directories |
| 49 | +2. Subsequent runs: Compares output against saved snapshots |
| 50 | +3. Changes detected: Test fails if output differs from snapshot |
| 51 | + |
| 52 | +### Creating/Updating Snapshots |
| 53 | + |
| 54 | +When you first write a test or when the expected output changes: |
| 55 | + |
| 56 | +```bash |
| 57 | +poetry run pytest path/to/test.py --snapshot-update |
| 58 | +``` |
| 59 | + |
| 60 | +⚠️ **Important**: Always review snapshot changes before committing! Use `git diff` to verify the changes are expected. |
| 61 | + |
| 62 | +### Snapshot Test Example |
| 63 | + |
| 64 | +```python |
| 65 | +import json |
| 66 | +from pytest_snapshot.plugin import Snapshot |
| 67 | + |
| 68 | +def test_api_endpoint(snapshot: Snapshot): |
| 69 | + response = client.get("/api/endpoint") |
| 70 | + |
| 71 | + # Snapshot the response |
| 72 | + snapshot.snapshot_dir = "snapshots" |
| 73 | +snapshot.assert_match( |
| 74 | + json.dumps(response.json(), indent=2, sort_keys=True), |
| 75 | + "endpoint_response" |
| 76 | + ) |
| 77 | +``` |
| 78 | + |
| 79 | +### Best Practices for Snapshots |
| 80 | + |
| 81 | +1. **Use descriptive names**: `"user_list_response"` not `"response1"` |
| 82 | +2. **Sort JSON keys**: Ensures consistent snapshots |
| 83 | +3. **Format JSON**: Use `indent=2` for readable diffs |
| 84 | +4. **Exclude dynamic data**: Remove timestamps, IDs, etc. that change between runs |
| 85 | + |
| 86 | +Example of excluding dynamic data: |
| 87 | +```python |
| 88 | +response_data = response.json() |
| 89 | +# Remove dynamic fields for snapshot |
| 90 | +response_data.pop("created_at", None) |
| 91 | +response_data.pop("id", None) |
| 92 | + |
| 93 | +snapshot.snapshot_dir = "snapshots" |
| 94 | +snapshot.assert_match( |
| 95 | + json.dumps(response_data, indent=2, sort_keys=True), |
| 96 | + "static_response_data" |
| 97 | +) |
| 98 | +``` |
| 99 | + |
| 100 | +## Writing Tests for API Routes |
| 101 | + |
| 102 | +### Basic Structure |
| 103 | + |
| 104 | +```python |
| 105 | +import json |
| 106 | +import fastapi |
| 107 | +import fastapi.testclient |
| 108 | +import pytest |
| 109 | +from pytest_snapshot.plugin import Snapshot |
| 110 | + |
| 111 | +from backend.server.v2.myroute import router |
| 112 | + |
| 113 | +app = fastapi.FastAPI() |
| 114 | +app.include_router(router) |
| 115 | +client = fastapi.testclient.TestClient(app) |
| 116 | + |
| 117 | +def test_endpoint_success(snapshot: Snapshot): |
| 118 | + response = client.get("/endpoint") |
| 119 | + assert response.status_code == 200 |
| 120 | + |
| 121 | + # Test specific fields |
| 122 | + data = response.json() |
| 123 | + assert data["status"] == "success" |
| 124 | + |
| 125 | + # Snapshot the full response |
| 126 | + snapshot.snapshot_dir = "snapshots" |
| 127 | + snapshot.assert_match( |
| 128 | + json.dumps(data, indent=2, sort_keys=True), |
| 129 | + "endpoint_success_response" |
| 130 | + ) |
| 131 | +``` |
| 132 | + |
| 133 | +### Testing with Authentication |
| 134 | + |
| 135 | +```python |
| 136 | +def override_auth_middleware(): |
| 137 | + return {"sub": "test-user-id"} |
| 138 | + |
| 139 | +def override_get_user_id(): |
| 140 | + return "test-user-id" |
| 141 | + |
| 142 | +app.dependency_overrides[auth_middleware] = override_auth_middleware |
| 143 | +app.dependency_overrides[get_user_id] = override_get_user_id |
| 144 | +``` |
| 145 | + |
| 146 | +### Mocking External Services |
| 147 | + |
| 148 | +```python |
| 149 | +def test_external_api_call(mocker, snapshot): |
| 150 | + # Mock external service |
| 151 | + mock_response = {"external": "data"} |
| 152 | + mocker.patch( |
| 153 | + "backend.services.external_api.call", |
| 154 | + return_value=mock_response |
| 155 | + ) |
| 156 | + |
| 157 | + response = client.post("/api/process") |
| 158 | + assert response.status_code == 200 |
| 159 | + |
| 160 | + snapshot.snapshot_dir = "snapshots" |
| 161 | + snapshot.assert_match( |
| 162 | + json.dumps(response.json(), indent=2, sort_keys=True), |
| 163 | + "process_with_external_response" |
| 164 | + ) |
| 165 | +``` |
| 166 | + |
| 167 | +## Best Practices |
| 168 | + |
| 169 | +### 1. Test Organization |
| 170 | +- Place tests next to the code: `routes.py` → `routes_test.py` |
| 171 | +- Use descriptive test names: `test_create_user_with_invalid_email` |
| 172 | +- Group related tests in classes when appropriate |
| 173 | + |
| 174 | +### 2. Test Coverage |
| 175 | +- Test happy path and error cases |
| 176 | +- Test edge cases (empty data, invalid formats) |
| 177 | +- Test authentication and authorization |
| 178 | + |
| 179 | +### 3. Snapshot Testing Guidelines |
| 180 | +- Review all snapshot changes carefully |
| 181 | +- Don't snapshot sensitive data |
| 182 | +- Keep snapshots focused and minimal |
| 183 | +- Update snapshots intentionally, not accidentally |
| 184 | + |
| 185 | +### 4. Async Testing |
| 186 | +- Use regular `def` for FastAPI TestClient tests |
| 187 | +- Use `async def` with `@pytest.mark.asyncio` for testing async functions directly |
| 188 | + |
| 189 | +### 5. Fixtures |
| 190 | +Create reusable fixtures for common test data: |
| 191 | + |
| 192 | +```python |
| 193 | +@pytest.fixture |
| 194 | +def sample_user(): |
| 195 | + return { |
| 196 | + |
| 197 | + "name": "Test User" |
| 198 | + } |
| 199 | + |
| 200 | +def test_create_user(sample_user, snapshot): |
| 201 | + response = client.post("/users", json=sample_user) |
| 202 | + # ... test implementation |
| 203 | +``` |
| 204 | + |
| 205 | +## CI/CD Integration |
| 206 | + |
| 207 | +The GitHub Actions workflow automatically runs tests on: |
| 208 | +- Pull requests |
| 209 | +- Pushes to main branch |
| 210 | + |
| 211 | +Snapshot tests work in CI by: |
| 212 | +1. Committing snapshot files to the repository |
| 213 | +2. CI compares against committed snapshots |
| 214 | +3. Fails if snapshots don't match |
| 215 | + |
| 216 | +## Troubleshooting |
| 217 | + |
| 218 | +### Snapshot Mismatches |
| 219 | +- Review the diff carefully |
| 220 | +- If changes are expected: `poetry run pytest --snapshot-update` |
| 221 | +- If changes are unexpected: Fix the code causing the difference |
| 222 | + |
| 223 | +### Async Test Issues |
| 224 | +- Ensure async functions use `@pytest.mark.asyncio` |
| 225 | +- Use `AsyncMock` for mocking async functions |
| 226 | +- FastAPI TestClient handles async automatically |
| 227 | + |
| 228 | +### Import Errors |
| 229 | +- Check that all dependencies are in `pyproject.toml` |
| 230 | +- Run `poetry install` to ensure dependencies are installed |
| 231 | +- Verify import paths are correct |
| 232 | + |
| 233 | +## Summary |
| 234 | + |
| 235 | +Snapshot testing provides a powerful way to ensure API responses remain consistent. Combined with traditional assertions, it creates a robust test suite that catches regressions while remaining maintainable. |
| 236 | + |
| 237 | +Remember: Good tests are as important as good code! |
0 commit comments