-
Notifications
You must be signed in to change notification settings - Fork 45.8k
feat(backend): snapshot test responses #10039
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(backend): snapshot test responses #10039
Conversation
✅ Deploy Preview for auto-gpt-docs-dev canceled.
|
PR Reviewer Guide 🔍(Review updated until commit d8b1037)Here are some key observations to aid the review process:
|
Here's the code health analysis summary for commits Analysis Summary
|
✅ Deploy Preview for auto-gpt-docs canceled.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds snapshot testing for various backend API endpoints and introduces a testing guide to standardize writing, running, and updating tests.
- Added comprehensive
TESTING.md
with instructions and best practices for pytest and snapshot testing - Introduced new snapshot-based tests and corresponding snapshot files for admin credit routes, public v1 routes, and analytics logging
- Updated dependencies (pytest-snapshot) and test code to ensure consistent API response validation
Reviewed Changes
Copilot reviewed 59 out of 59 changed files in this pull request and generated 2 comments.
Show a summary per file
File | Description |
---|---|
autogpt_platform/backend/backend/server/v2/admin/snapshots/credit_admin_routes_test/test_add_user_credits_success/admin_add_credits_success_response | Snapshot for successful admin credit addition |
autogpt_platform/backend/backend/server/v2/admin/snapshots/credit_admin_routes_test/test_add_user_credits_negative_amount/admin_add_credits_negative_response | Snapshot for admin credit deduction case |
autogpt_platform/backend/backend/server/v2/admin/credit_admin_routes_test.py | New admin credit routes tests with snapshot assertions |
autogpt_platform/backend/backend/server/routers/v1_test.py | New v1 router tests covering auth, blocks, credits, and graphs |
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_update_user_email_route/auth_update_email_response | Snapshot for update email endpoint |
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_request_top_up/credits_top_up_request_response | Snapshot for top-up request endpoint |
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_user_credits/credits_get_balance_response | Snapshot for get balance endpoint |
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_or_create_user_route/auth_get_or_create_user_response | Snapshot for get/create user endpoint |
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_graphs/graphs_get_all_response | Snapshot for listing graphs |
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_graph_blocks/blocks_get_all_response | Snapshot for listing blocks |
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_graph/graphs_get_single_response | Snapshot for fetching a single graph |
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_auto_top_up/credits_get_auto_top_up_response | Snapshot for auto top-up config |
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_execute_graph_block/blocks_execute_response | Snapshot for block execution endpoint |
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_delete_graph/graphs_delete_response | Snapshot for graph deletion endpoint |
autogpt_platform/backend/backend/server/routers/snapshots/analytics_test/test_log_raw_metric_various_values/analytics_log_metric_various_values_response | Snapshot for metric logging with various values |
autogpt_platform/backend/backend/server/routers/snapshots/analytics_test/test_log_raw_metric_success/analytics_log_metric_success_response | Snapshot for successful metric logging |
autogpt_platform/backend/backend/server/routers/snapshots/analytics_test/test_log_raw_analytics_success/analytics_log_analytics_success_response | Snapshot for successful analytics logging |
autogpt_platform/backend/backend/server/routers/snapshots/analytics_test/test_log_raw_analytics_complex_data/analytics_log_analytics_complex_data_response | Snapshot for complex analytics data |
autogpt_platform/backend/backend/server/routers/analytics_test.py | New analytics logging tests with snapshot assertions |
autogpt_platform/backend/TESTING.md | Added backend testing guide and snapshot testing documentation |
Comments suppressed due to low confidence (3)
autogpt_platform/backend/backend/server/v2/admin/credit_admin_routes_test.py:311
- The mock raises HTTPException with status code 403, but the test asserts 401. Either update the expected status code to 403 or adjust the mock to return 401 to reflect actual behavior.
assert (response.status_code == 401)
autogpt_platform/backend/backend/server/v2/admin/credit_admin_routes_test.py:108
- The negative-amount test asserts only
new_balance
but the snapshot includestransaction_key
. Add an explicit assertion forresponse_data['transaction_key']
to ensure that field is returned correctly.
assert response_data["new_balance"] == 200
autogpt_platform/backend/backend/server/v2/admin/credit_admin_routes_test.py:1
- [nitpick] This test file spans over 300 lines and covers multiple scenarios. Consider splitting it into smaller modules or grouping related tests to improve readability and maintainability.
import json
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This pull request introduces comprehensive snapshot testing for the backend, updates several test modules and utilities, and enhances the test runner configuration to ensure proper isolation of the test database.
- New snapshot files for admin credit operations and various API endpoints have been added.
- The test runner (run_tests.py) now explicitly sets test database environment variables and performs database migration resets/deployments.
- Additional test modules and fixtures have been updated to utilize pytest-snapshot, along with documentation updates in CLAUDE.md.
Reviewed Changes
Copilot reviewed 70 out of 70 changed files in this pull request and generated 1 comment.
Show a summary per file
File | Description |
---|---|
autogpt_platform/backend/snapshots/* | New snapshot files for admin credit transactions. |
autogpt_platform/backend/run_tests.py | Updated test runner to safely configure test environment and run migrations. |
autogpt_platform/backend/pyproject.toml | Added pytest-snapshot dependency. |
autogpt_platform/backend/docker-compose.test.yaml | Updated postgres-test service with default environment variable values. |
autogpt_platform/backend/backend/server/*_test.py | Numerous tests updated to include snapshot assertions and fixed timestamps for reproducibility. |
autogpt_platform/CLAUDE.md | Documentation updates on development and testing procedures. |
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
!deploy |
|
🚀 Deploying PR #10039 to development environment... |
✅ Preview Environment Deployed Successfully 📊 Deployment Summary
🔔 Please check Discord for the preview environment URLs and details. The deployment status and URLs will be posted in the AutoGPT Discord server. |
!deploy |
|
🚀 Deploying PR #10039 to development environment... |
✅ Preview Environment Deployed Successfully 📊 Deployment Summary
🔔 Please check Discord for the preview environment URLs and details. The deployment status and URLs will be posted in the AutoGPT Discord server. |
!undeploy |
🗑️ Undeploying PR #10039 from development environment... |
1 similar comment
🗑️ Undeploying PR #10039 from development environment... |
🧹 Preview Environment Cleaned Up All resources for PR #10039 have been removed:
Cleanup completed successfully. |
1 similar comment
🧹 Preview Environment Cleaned Up All resources for PR #10039 have been removed:
Cleanup completed successfully. |
Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly. |
This pull request introduces a comprehensive backend testing guide and adds new tests for analytics logging and various API endpoints, focusing on snapshot testing. It also includes corresponding snapshot files for these tests. Below are the most significant changes:
Documentation Updates:
TESTING.md
file to the backend, providing a guide for running tests, snapshot testing, writing API route tests, and best practices. It includes examples for mocking, fixtures, and CI/CD integration.Analytics Logging Tests:
analytics_test.py
, covering success scenarios, various input values, invalid requests, and complex nested data. These tests utilize snapshot testing for response validation.Snapshot Files for API Endpoints:
graphs_get_single_response
,graphs_get_all_response
,blocks_get_all_response
). [1] [2] [3]auth_get_or_create_user_response
,auth_update_email_response
). [1] [2]credits_get_balance_response
,credits_get_auto_top_up_response
,credits_top_up_request_response
). [1] [2] [3]blocks_execute_response
,graphs_delete_response
). [1] [2]## SummaryTesting
poetry run format
poetry run test
Checklist 📋
For code changes: