Skip to content

feat(backend): snapshot test responses #10039

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Jun 6, 2025

Conversation

Swiftyos
Copy link
Contributor

@Swiftyos Swiftyos commented May 26, 2025

This pull request introduces a comprehensive backend testing guide and adds new tests for analytics logging and various API endpoints, focusing on snapshot testing. It also includes corresponding snapshot files for these tests. Below are the most significant changes:

Documentation Updates:

  • Added a detailed TESTING.md file to the backend, providing a guide for running tests, snapshot testing, writing API route tests, and best practices. It includes examples for mocking, fixtures, and CI/CD integration.

Analytics Logging Tests:

  • Implemented tests for logging raw metrics and analytics in analytics_test.py, covering success scenarios, various input values, invalid requests, and complex nested data. These tests utilize snapshot testing for response validation.
  • Added snapshot files for analytics logging tests, including responses for success cases, various metric values, and complex analytics data. [1] [2] [3] [4]

Snapshot Files for API Endpoints:

  • Added snapshot files for various API endpoint tests, such as:
    • Graph-related operations (graphs_get_single_response, graphs_get_all_response, blocks_get_all_response). [1] [2] [3]
    • User-related operations (auth_get_or_create_user_response, auth_update_email_response). [1] [2]
    • Credit-related operations (credits_get_balance_response, credits_get_auto_top_up_response, credits_top_up_request_response). [1] [2] [3]
    • Graph execution and deletion (blocks_execute_response, graphs_delete_response). [1] [2]## Summary
  • add pytest-snapshot to backend dev requirements
  • snapshot server route response JSONs
  • mention how to update stored snapshots

Testing

  • poetry run format
  • poetry run test

Checklist 📋

For code changes:

  • I have clearly listed my changes in the PR description
  • I have made a test plan
  • I have tested my changes according to the test plan:
    • run poetry run test

@Swiftyos Swiftyos requested a review from a team as a code owner May 26, 2025 09:40
@Swiftyos Swiftyos requested review from Pwuts and kcze and removed request for a team May 26, 2025 09:40
@github-project-automation github-project-automation bot moved this to 🆕 Needs initial review in AutoGPT development kanban May 26, 2025
Copy link

netlify bot commented May 26, 2025

Deploy Preview for auto-gpt-docs-dev canceled.

Name Link
🔨 Latest commit c8f3793
🔍 Latest deploy log https://app.netlify.com/projects/auto-gpt-docs-dev/deploys/6842c29205c429000898f622

@github-actions github-actions bot added documentation Improvements or additions to documentation platform/backend AutoGPT Platform - Back end size/l labels May 26, 2025
Copy link

qodo-merge-pro bot commented May 26, 2025

PR Reviewer Guide 🔍

(Review updated until commit d8b1037)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Fixed Timestamp

The code uses a fixed timestamp (FIXED_NOW) for testing, which is good for deterministic tests, but make sure this approach is consistent across all tests and doesn't affect production code.

FIXED_NOW = datetime.datetime(2023, 1, 1, 0, 0, 0)
Database Reset

The test runner now resets the database before running tests. This ensures clean test state but could be disruptive if developers have local data they want to preserve. Consider adding a flag to make this optional.

run_command(["prisma", "migrate", "reset", "--force", "--skip-seed"], check=False)
# Then apply migrations
run_command(["prisma", "migrate", "deploy"])

Copy link

deepsource-io bot commented May 26, 2025

Here's the code health analysis summary for commits 705be3e..c8f3793. View details on DeepSource ↗.

Analysis Summary

AnalyzerStatusSummaryLink
DeepSource JavaScript LogoJavaScript✅ SuccessView Check ↗
DeepSource Python LogoPython✅ Success
❗ 3 occurences introduced
View Check ↗

💡 If you’re a repository administrator, you can configure the quality gates from the settings.

Copy link

netlify bot commented May 26, 2025

Deploy Preview for auto-gpt-docs canceled.

Name Link
🔨 Latest commit c8f3793
🔍 Latest deploy log https://app.netlify.com/projects/auto-gpt-docs/deploys/6842c29241da0d00080248bb

@Swiftyos Swiftyos marked this pull request as draft May 26, 2025 09:47
@Swiftyos Swiftyos requested a review from Copilot May 26, 2025 13:11
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds snapshot testing for various backend API endpoints and introduces a testing guide to standardize writing, running, and updating tests.

  • Added comprehensive TESTING.md with instructions and best practices for pytest and snapshot testing
  • Introduced new snapshot-based tests and corresponding snapshot files for admin credit routes, public v1 routes, and analytics logging
  • Updated dependencies (pytest-snapshot) and test code to ensure consistent API response validation

Reviewed Changes

Copilot reviewed 59 out of 59 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
autogpt_platform/backend/backend/server/v2/admin/snapshots/credit_admin_routes_test/test_add_user_credits_success/admin_add_credits_success_response Snapshot for successful admin credit addition
autogpt_platform/backend/backend/server/v2/admin/snapshots/credit_admin_routes_test/test_add_user_credits_negative_amount/admin_add_credits_negative_response Snapshot for admin credit deduction case
autogpt_platform/backend/backend/server/v2/admin/credit_admin_routes_test.py New admin credit routes tests with snapshot assertions
autogpt_platform/backend/backend/server/routers/v1_test.py New v1 router tests covering auth, blocks, credits, and graphs
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_update_user_email_route/auth_update_email_response Snapshot for update email endpoint
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_request_top_up/credits_top_up_request_response Snapshot for top-up request endpoint
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_user_credits/credits_get_balance_response Snapshot for get balance endpoint
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_or_create_user_route/auth_get_or_create_user_response Snapshot for get/create user endpoint
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_graphs/graphs_get_all_response Snapshot for listing graphs
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_graph_blocks/blocks_get_all_response Snapshot for listing blocks
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_graph/graphs_get_single_response Snapshot for fetching a single graph
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_auto_top_up/credits_get_auto_top_up_response Snapshot for auto top-up config
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_execute_graph_block/blocks_execute_response Snapshot for block execution endpoint
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_delete_graph/graphs_delete_response Snapshot for graph deletion endpoint
autogpt_platform/backend/backend/server/routers/snapshots/analytics_test/test_log_raw_metric_various_values/analytics_log_metric_various_values_response Snapshot for metric logging with various values
autogpt_platform/backend/backend/server/routers/snapshots/analytics_test/test_log_raw_metric_success/analytics_log_metric_success_response Snapshot for successful metric logging
autogpt_platform/backend/backend/server/routers/snapshots/analytics_test/test_log_raw_analytics_success/analytics_log_analytics_success_response Snapshot for successful analytics logging
autogpt_platform/backend/backend/server/routers/snapshots/analytics_test/test_log_raw_analytics_complex_data/analytics_log_analytics_complex_data_response Snapshot for complex analytics data
autogpt_platform/backend/backend/server/routers/analytics_test.py New analytics logging tests with snapshot assertions
autogpt_platform/backend/TESTING.md Added backend testing guide and snapshot testing documentation
Comments suppressed due to low confidence (3)

autogpt_platform/backend/backend/server/v2/admin/credit_admin_routes_test.py:311

  • The mock raises HTTPException with status code 403, but the test asserts 401. Either update the expected status code to 403 or adjust the mock to return 401 to reflect actual behavior.
assert (response.status_code == 401)

autogpt_platform/backend/backend/server/v2/admin/credit_admin_routes_test.py:108

  • The negative-amount test asserts only new_balance but the snapshot includes transaction_key. Add an explicit assertion for response_data['transaction_key'] to ensure that field is returned correctly.
assert response_data["new_balance"] == 200

autogpt_platform/backend/backend/server/v2/admin/credit_admin_routes_test.py:1

  • [nitpick] This test file spans over 300 lines and covers multiple scenarios. Consider splitting it into smaller modules or grouping related tests to improve readability and maintainability.
import json

@Swiftyos Swiftyos marked this pull request as ready for review May 26, 2025 13:24
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request introduces comprehensive snapshot testing for the backend, updates several test modules and utilities, and enhances the test runner configuration to ensure proper isolation of the test database.

  • New snapshot files for admin credit operations and various API endpoints have been added.
  • The test runner (run_tests.py) now explicitly sets test database environment variables and performs database migration resets/deployments.
  • Additional test modules and fixtures have been updated to utilize pytest-snapshot, along with documentation updates in CLAUDE.md.

Reviewed Changes

Copilot reviewed 70 out of 70 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
autogpt_platform/backend/snapshots/* New snapshot files for admin credit transactions.
autogpt_platform/backend/run_tests.py Updated test runner to safely configure test environment and run migrations.
autogpt_platform/backend/pyproject.toml Added pytest-snapshot dependency.
autogpt_platform/backend/docker-compose.test.yaml Updated postgres-test service with default environment variable values.
autogpt_platform/backend/backend/server/*_test.py Numerous tests updated to include snapshot assertions and fixed timestamps for reproducibility.
autogpt_platform/CLAUDE.md Documentation updates on development and testing procedures.

@github-actions github-actions bot added the conflicts Automatically applied to PRs with merge conflicts label Jun 4, 2025
Copy link
Contributor

github-actions bot commented Jun 4, 2025

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

@Swiftyos
Copy link
Contributor Author

Swiftyos commented Jun 5, 2025

!deploy

Copy link
Contributor

github-actions bot commented Jun 5, 2025

⚠️ Deploy skipped: This PR already has an active deployment. Use !undeploy first if you want to redeploy.

Copy link
Contributor

github-actions bot commented Jun 5, 2025

🚀 Deploying PR #10039 to development environment...

@Swiftyos
Copy link
Contributor Author

Swiftyos commented Jun 5, 2025

Preview Environment Deployed Successfully

📊 Deployment Summary

Service Status
Redis ✅ Success
RabbitMQ ✅ Success
Backend Server ✅ Success
WebSocket Server ✅ Success
Scheduler Server ✅ Success
Frontend Builder ✅ Success

🔔 Please check Discord for the preview environment URLs and details.

The deployment status and URLs will be posted in the AutoGPT Discord server.

@Swiftyos
Copy link
Contributor Author

Swiftyos commented Jun 5, 2025

!deploy

Copy link
Contributor

github-actions bot commented Jun 5, 2025

⚠️ Deploy skipped: This PR already has an active deployment. Use !undeploy first if you want to redeploy.

Copy link
Contributor

github-actions bot commented Jun 5, 2025

🚀 Deploying PR #10039 to development environment...

@Swiftyos
Copy link
Contributor Author

Swiftyos commented Jun 5, 2025

Preview Environment Deployed Successfully

📊 Deployment Summary

Service Status
Redis ✅ Success
RabbitMQ ✅ Success
Backend Server ✅ Success
WebSocket Server ✅ Success
Scheduler Server ✅ Success
Frontend Builder ✅ Success

🔔 Please check Discord for the preview environment URLs and details.

The deployment status and URLs will be posted in the AutoGPT Discord server.

@Swiftyos
Copy link
Contributor Author

Swiftyos commented Jun 5, 2025

!undeploy

Copy link
Contributor

github-actions bot commented Jun 5, 2025

🗑️ Undeploying PR #10039 from development environment...

1 similar comment
Copy link
Contributor

github-actions bot commented Jun 5, 2025

🗑️ Undeploying PR #10039 from development environment...

@Swiftyos
Copy link
Contributor Author

Swiftyos commented Jun 5, 2025

🧹 Preview Environment Cleaned Up

All resources for PR #10039 have been removed:

  • ☸️ Kubernetes namespace deleted
  • 🗃️ Database schema pr_10039 dropped

Cleanup completed successfully.

1 similar comment
@Swiftyos
Copy link
Contributor Author

Swiftyos commented Jun 5, 2025

🧹 Preview Environment Cleaned Up

All resources for PR #10039 have been removed:

  • ☸️ Kubernetes namespace deleted
  • 🗃️ Database schema pr_10039 dropped

Cleanup completed successfully.

@github-actions github-actions bot removed the conflicts Automatically applied to PRs with merge conflicts label Jun 5, 2025
Copy link
Contributor

github-actions bot commented Jun 5, 2025

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

@Swiftyos Swiftyos added this pull request to the merge queue Jun 6, 2025
@github-project-automation github-project-automation bot moved this from 🚧 Needs work to 👍🏼 Mergeable in AutoGPT development kanban Jun 6, 2025
Merged via the queue into dev with commit a3d082a Jun 6, 2025
21 of 22 checks passed
@Swiftyos Swiftyos deleted the swiftyos/add-pytest-snapshot-to-dev-dependencies branch June 6, 2025 20:55
@github-project-automation github-project-automation bot moved this from 👍🏼 Mergeable to ✅ Done in AutoGPT development kanban Jun 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
codex documentation Improvements or additions to documentation platform/backend AutoGPT Platform - Back end Review effort 3/5 size/l size/xl
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants