feat(backend): snapshot test responses #10039

Swiftyos · 2025-05-26T09:40:46Z

This pull request introduces a comprehensive backend testing guide and adds new tests for analytics logging and various API endpoints, focusing on snapshot testing. It also includes corresponding snapshot files for these tests. Below are the most significant changes:

Documentation Updates:

Added a detailed TESTING.md file to the backend, providing a guide for running tests, snapshot testing, writing API route tests, and best practices. It includes examples for mocking, fixtures, and CI/CD integration.

Analytics Logging Tests:

Implemented tests for logging raw metrics and analytics in analytics_test.py, covering success scenarios, various input values, invalid requests, and complex nested data. These tests utilize snapshot testing for response validation.
Added snapshot files for analytics logging tests, including responses for success cases, various metric values, and complex analytics data. [1] [2] [3] [4]

Snapshot Files for API Endpoints:

Added snapshot files for various API endpoint tests, such as:
- Graph-related operations (graphs_get_single_response, graphs_get_all_response, blocks_get_all_response). [1] [2] [3]
- User-related operations (auth_get_or_create_user_response, auth_update_email_response). [1] [2]
- Credit-related operations (credits_get_balance_response, credits_get_auto_top_up_response, credits_top_up_request_response). [1] [2] [3]
- Graph execution and deletion (blocks_execute_response, graphs_delete_response). [1] [2]## Summary

add pytest-snapshot to backend dev requirements
snapshot server route response JSONs
mention how to update stored snapshots

Testing

poetry run format
poetry run test

Checklist 📋

For code changes:

I have clearly listed my changes in the PR description
I have made a test plan
I have tested my changes according to the test plan:
- run poetry run test

netlify · 2025-05-26T09:40:51Z

✅ Deploy Preview for auto-gpt-docs-dev canceled.

Name	Link
🔨 Latest commit	`c8f3793`
🔍 Latest deploy log	https://app.netlify.com/projects/auto-gpt-docs-dev/deploys/6842c29205c429000898f622

qodo-merge-pro · 2025-05-26T09:41:08Z

PR Reviewer Guide 🔍

(Review updated until commit `d8b1037`)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review Fixed Timestamp The code uses a fixed timestamp (FIXED_NOW) for testing, which is good for deterministic tests, but make sure this approach is consistent across all tests and doesn't affect production code. FIXED_NOW = datetime.datetime(2023, 1, 1, 0, 0, 0) Database Reset The test runner now resets the database before running tests. This ensures clean test state but could be disruptive if developers have local data they want to preserve. Consider adding a flag to make this optional. run_command(["prisma", "migrate", "reset", "--force", "--skip-seed"], check=False) # Then apply migrations run_command(["prisma", "migrate", "deploy"])

deepsource-io · 2025-05-26T09:41:16Z

Here's the code health analysis summary for commits 705be3e..c8f3793. View details on DeepSource ↗.

Analysis Summary

Analyzer	Status	Summary	Link
JavaScript	✅ Success		View Check ↗
Python	✅ Success	❗ 3 occurences introduced	View Check ↗

💡 If you’re a repository administrator, you can configure the quality gates from the settings.

netlify · 2025-05-26T09:41:34Z

✅ Deploy Preview for auto-gpt-docs canceled.

Name	Link
🔨 Latest commit	`c8f3793`
🔍 Latest deploy log	https://app.netlify.com/projects/auto-gpt-docs/deploys/6842c29241da0d00080248bb

Copilot

Pull Request Overview

This PR adds snapshot testing for various backend API endpoints and introduces a testing guide to standardize writing, running, and updating tests.

Added comprehensive TESTING.md with instructions and best practices for pytest and snapshot testing
Introduced new snapshot-based tests and corresponding snapshot files for admin credit routes, public v1 routes, and analytics logging
Updated dependencies (pytest-snapshot) and test code to ensure consistent API response validation

Reviewed Changes

Copilot reviewed 59 out of 59 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
autogpt_platform/backend/backend/server/v2/admin/snapshots/credit_admin_routes_test/test_add_user_credits_success/admin_add_credits_success_response	Snapshot for successful admin credit addition
autogpt_platform/backend/backend/server/v2/admin/snapshots/credit_admin_routes_test/test_add_user_credits_negative_amount/admin_add_credits_negative_response	Snapshot for admin credit deduction case
autogpt_platform/backend/backend/server/v2/admin/credit_admin_routes_test.py	New admin credit routes tests with snapshot assertions
autogpt_platform/backend/backend/server/routers/v1_test.py	New v1 router tests covering auth, blocks, credits, and graphs
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_update_user_email_route/auth_update_email_response	Snapshot for update email endpoint
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_request_top_up/credits_top_up_request_response	Snapshot for top-up request endpoint
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_user_credits/credits_get_balance_response	Snapshot for get balance endpoint
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_or_create_user_route/auth_get_or_create_user_response	Snapshot for get/create user endpoint
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_graphs/graphs_get_all_response	Snapshot for listing graphs
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_graph_blocks/blocks_get_all_response	Snapshot for listing blocks
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_graph/graphs_get_single_response	Snapshot for fetching a single graph
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_get_auto_top_up/credits_get_auto_top_up_response	Snapshot for auto top-up config
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_execute_graph_block/blocks_execute_response	Snapshot for block execution endpoint
autogpt_platform/backend/backend/server/routers/snapshots/v1_test/test_delete_graph/graphs_delete_response	Snapshot for graph deletion endpoint
autogpt_platform/backend/backend/server/routers/snapshots/analytics_test/test_log_raw_metric_various_values/analytics_log_metric_various_values_response	Snapshot for metric logging with various values
autogpt_platform/backend/backend/server/routers/snapshots/analytics_test/test_log_raw_metric_success/analytics_log_metric_success_response	Snapshot for successful metric logging
autogpt_platform/backend/backend/server/routers/snapshots/analytics_test/test_log_raw_analytics_success/analytics_log_analytics_success_response	Snapshot for successful analytics logging
autogpt_platform/backend/backend/server/routers/snapshots/analytics_test/test_log_raw_analytics_complex_data/analytics_log_analytics_complex_data_response	Snapshot for complex analytics data
autogpt_platform/backend/backend/server/routers/analytics_test.py	New analytics logging tests with snapshot assertions
autogpt_platform/backend/TESTING.md	Added backend testing guide and snapshot testing documentation

Comments suppressed due to low confidence (3)

autogpt_platform/backend/backend/server/v2/admin/credit_admin_routes_test.py:311

The mock raises HTTPException with status code 403, but the test asserts 401. Either update the expected status code to 403 or adjust the mock to return 401 to reflect actual behavior.

assert (response.status_code == 401)

autogpt_platform/backend/backend/server/v2/admin/credit_admin_routes_test.py:108

The negative-amount test asserts only new_balance but the snapshot includes transaction_key. Add an explicit assertion for response_data['transaction_key'] to ensure that field is returned correctly.

assert response_data["new_balance"] == 200

autogpt_platform/backend/backend/server/v2/admin/credit_admin_routes_test.py:1

[nitpick] This test file spans over 300 lines and covers multiple scenarios. Consider splitting it into smaller modules or grouping related tests to improve readability and maintainability.

import json

autogpt_platform/backend/TESTING.md

Copilot

Pull Request Overview

This pull request introduces comprehensive snapshot testing for the backend, updates several test modules and utilities, and enhances the test runner configuration to ensure proper isolation of the test database.

New snapshot files for admin credit operations and various API endpoints have been added.
The test runner (run_tests.py) now explicitly sets test database environment variables and performs database migration resets/deployments.
Additional test modules and fixtures have been updated to utilize pytest-snapshot, along with documentation updates in CLAUDE.md.

Reviewed Changes

Copilot reviewed 70 out of 70 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
autogpt_platform/backend/snapshots/*	New snapshot files for admin credit transactions.
autogpt_platform/backend/run_tests.py	Updated test runner to safely configure test environment and run migrations.
autogpt_platform/backend/pyproject.toml	Added pytest-snapshot dependency.
autogpt_platform/backend/docker-compose.test.yaml	Updated postgres-test service with default environment variable values.
autogpt_platform/backend/backend/server/*_test.py	Numerous tests updated to include snapshot assertions and fixed timestamps for reproducibility.
autogpt_platform/CLAUDE.md	Documentation updates on development and testing procedures.

autogpt_platform/backend/backend/server/v2/store/routes_test.py

github-actions · 2025-06-04T15:39:41Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

Swiftyos · 2025-06-05T11:00:26Z

!deploy

github-actions · 2025-06-05T11:00:37Z

⚠️ Deploy skipped: This PR already has an active deployment. Use !undeploy first if you want to redeploy.

github-actions · 2025-06-05T11:00:37Z

🚀 Deploying PR #10039 to development environment...

Swiftyos · 2025-06-05T11:08:04Z

✅ Preview Environment Deployed Successfully

📊 Deployment Summary

Service	Status
Redis	✅ Success
RabbitMQ	✅ Success
Backend Server	✅ Success
WebSocket Server	✅ Success
Scheduler Server	✅ Success
Frontend Builder	✅ Success

🔔 Please check Discord for the preview environment URLs and details.

The deployment status and URLs will be posted in the AutoGPT Discord server.

Swiftyos · 2025-06-05T11:20:00Z

!deploy

github-actions · 2025-06-05T11:20:10Z

⚠️ Deploy skipped: This PR already has an active deployment. Use !undeploy first if you want to redeploy.

github-actions · 2025-06-05T11:20:10Z

🚀 Deploying PR #10039 to development environment...

Swiftyos · 2025-06-05T11:25:20Z

✅ Preview Environment Deployed Successfully

📊 Deployment Summary

Service	Status
Redis	✅ Success
RabbitMQ	✅ Success
Backend Server	✅ Success
WebSocket Server	✅ Success
Scheduler Server	✅ Success
Frontend Builder	✅ Success

🔔 Please check Discord for the preview environment URLs and details.

The deployment status and URLs will be posted in the AutoGPT Discord server.

Swiftyos · 2025-06-05T13:01:51Z

!undeploy

github-actions · 2025-06-05T13:02:02Z

🗑️ Undeploying PR #10039 from development environment...

github-actions · 2025-06-05T13:02:03Z

🗑️ Undeploying PR #10039 from development environment...

Swiftyos · 2025-06-05T13:03:11Z

🧹 Preview Environment Cleaned Up

All resources for PR #10039 have been removed:

☸️ Kubernetes namespace deleted
🗃️ Database schema pr_10039 dropped

Cleanup completed successfully.

Swiftyos · 2025-06-05T13:03:11Z

🧹 Preview Environment Cleaned Up

All resources for PR #10039 have been removed:

☸️ Kubernetes namespace deleted
🗃️ Database schema pr_10039 dropped

Cleanup completed successfully.

github-actions · 2025-06-05T15:38:50Z

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

feat(backend): snapshot test responses

43b5be6

Swiftyos requested a review from a team as a code owner May 26, 2025 09:40

Swiftyos requested review from Pwuts and kcze and removed request for a team May 26, 2025 09:40

github-project-automation bot added this to AutoGPT development kanban May 26, 2025

Swiftyos added the codex label May 26, 2025 — with ChatGPT Connector

github-project-automation bot moved this to 🆕 Needs initial review in AutoGPT development kanban May 26, 2025

github-actions bot added documentation Improvements or additions to documentation platform/backend AutoGPT Platform - Back end size/l labels May 26, 2025

qodo-merge-pro bot added the Review effort 2/5 label May 26, 2025

Swiftyos marked this pull request as draft May 26, 2025 09:47

Swiftyos added 2 commits May 26, 2025 11:47

Merge branch 'dev' into swiftyos/add-pytest-snapshot-to-dev-dependencies

62227dd

added snapshot testing

8936a42

github-actions bot added the size/xl label May 26, 2025

Swiftyos added 4 commits May 26, 2025 14:58

formatting

3dd2690

fmt

0fa20e8

Merge branch 'dev' into swiftyos/add-pytest-snapshot-to-dev-dependencies

574f4f1

update lock file

ba0f9f3

Swiftyos requested a review from Copilot May 26, 2025 13:11

Copilot AI reviewed May 26, 2025

View reviewed changes

autogpt_platform/backend/TESTING.md Show resolved Hide resolved

autogpt_platform/backend/TESTING.md Show resolved Hide resolved

Swiftyos added 2 commits May 26, 2025 15:15

updated lock file

d3bb799

updated lock file

d8b1037

Swiftyos marked this pull request as ready for review May 26, 2025 13:24

qodo-merge-pro bot removed the Review effort 2/5 label May 26, 2025

Copilot AI reviewed Jun 2, 2025

View reviewed changes

autogpt_platform/backend/backend/server/v2/store/routes_test.py Show resolved Hide resolved

update lock file

69b3c9c

github-actions bot added the conflicts Automatically applied to PRs with merge conflicts label Jun 4, 2025

ntindle added 2 commits June 5, 2025 09:49

Merge branch 'dev' into swiftyos/add-pytest-snapshot-to-dev-dependencies

99e74a4

fix: lock

1f974a0

github-actions bot removed the conflicts Automatically applied to PRs with merge conflicts label Jun 5, 2025

Swiftyos added 2 commits June 6, 2025 11:54

Merge branch 'dev' into swiftyos/add-pytest-snapshot-to-dev-dependencies

3bf2c26

added testing docs

c8f3793

ntindle approved these changes Jun 6, 2025

View reviewed changes

Swiftyos added this pull request to the merge queue Jun 6, 2025

github-project-automation bot moved this from 🚧 Needs work to 👍🏼 Mergeable in AutoGPT development kanban Jun 6, 2025

Merged via the queue into dev with commit a3d082a Jun 6, 2025
21 of 22 checks passed

Swiftyos deleted the swiftyos/add-pytest-snapshot-to-dev-dependencies branch June 6, 2025 20:55

github-project-automation bot moved this from 👍🏼 Mergeable to ✅ Done in AutoGPT development kanban Jun 6, 2025

feat(backend): snapshot test responses #10039

feat(backend): snapshot test responses #10039

Uh oh!

Conversation

Swiftyos commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation Updates:

Analytics Logging Tests:

Snapshot Files for API Endpoints:

Testing

Checklist 📋

For code changes:

Uh oh!

netlify bot commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for auto-gpt-docs-dev canceled.

Uh oh!

qodo-merge-pro bot commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Reviewer Guide 🔍

(Review updated until commit d8b1037)

Uh oh!

deepsource-io bot commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Analysis Summary

Uh oh!

netlify bot commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for auto-gpt-docs canceled.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

github-actions bot commented Jun 4, 2025

Uh oh!

Swiftyos commented Jun 5, 2025

Uh oh!

github-actions bot commented Jun 5, 2025

Uh oh!

github-actions bot commented Jun 5, 2025

Uh oh!

Swiftyos commented Jun 5, 2025

Uh oh!

Swiftyos commented Jun 5, 2025

Uh oh!

github-actions bot commented Jun 5, 2025

Uh oh!

github-actions bot commented Jun 5, 2025

Uh oh!

Swiftyos commented Jun 5, 2025

Uh oh!

Swiftyos commented Jun 5, 2025

Uh oh!

github-actions bot commented Jun 5, 2025

Uh oh!

github-actions bot commented Jun 5, 2025

Uh oh!

Swiftyos commented Jun 5, 2025

Uh oh!

Swiftyos commented Jun 5, 2025

Uh oh!

github-actions bot commented Jun 5, 2025

Uh oh!

Uh oh!

Uh oh!

Swiftyos commented May 26, 2025 •

edited

Loading

netlify bot commented May 26, 2025 •

edited

Loading

qodo-merge-pro bot commented May 26, 2025 •

edited

Loading

(Review updated until commit `d8b1037`)

deepsource-io bot commented May 26, 2025 •

edited

Loading

netlify bot commented May 26, 2025 •

edited

Loading