Skip to content

E2E_TESTING.md

Latest commit

 

History

History
165 lines (115 loc) · 5.77 KB

E2E_TESTING.md

File metadata and controls

165 lines (115 loc) · 5.77 KB
Raw

E2E Testing

Complete guide for running, monitoring, and debugging e2e tests.

Key Concept

E2E tests run against pushed code in AWS CodeBuild, not local changes. Always commit and push before triggering.

When to Run E2E Tests

  • User explicitly requests e2e tests
  • User approves e2e testing as part of a task
  • Implied when user says "fix and test ..." or "add feature ... and test"

Cloud E2E Workflow

# 1. Commit and push all changes first
git push

# 2. Trigger e2e suite
yarn cloud-e2e

# 3. Monitor (auto-retries failed builds, polls every 5 min)
yarn e2e-monitor {batchId}

Batch ID format: amplify-category-api-e2e-workflow:{UUID} — always use the full ID.

Other Commands

yarn e2e-status {batchId}    # Check status once
yarn e2e-retry {batchId}     # Retry failed builds
yarn e2e-list [limit]        # List recent batches
yarn e2e-failed {batchId}    # Show failed builds
yarn e2e-logs {buildId}      # View build logs

Split E2E Mode (two batches)

yarn cloud-e2e fires a single CodeBuild batch from codebuild_specs/e2e_workflow.yml (~183 shards). Because the batch orchestrator faults materially above ~100 simultaneously in-flight builds, there is an alternative split mode that fires TWO independent, self-contained batches against the same amplify-category-api-e2e-workflow project:

  • api+gql batch — codebuild_specs/e2e_workflow_api_gql.yml (amplify-e2e-tests + graphql-transformers-e2e-tests, ~78 shards)
  • cdk batch — codebuild_specs/e2e_workflow_cdk.yml (amplify-graphql-api-construct-tests, ~105 shards)

Each batch carries the full prep/build chain (build_linux, build_windows, test, all verify_*, publish_to_local_registry) and its own cleanup_e2e_resources, and is waved independently at SPLIT_E2E_WAVE_SIZE (95 — the peak in-flight shards per batch, just under the ~100 orchestrator fault ceiling).

# Trigger BOTH batches (prints two Batch IDs)
yarn cloud-e2e-split

# Poll both to terminal state and aggregate pass/fail
yarn wait-for-all-codebuild-split <apiGqlBatchId> <cdkBatchId>

All three specs are regenerated together by yarn split-e2e-tests, which runs a reconciliation self-check asserting the two split batches exactly cover the combined shard set (no missing, extra, or overlapping shards). The combined e2e_workflow.yml remains the project's default buildspec and is still produced for the single-batch yarn cloud-e2e path.

Monitor Behavior

The monitor auto-retries failed builds up to 10 times by default. It skips retrying build_linux, build_windows, test, and lint because failures in those are typically code-related and require fixes, not retries.

If errors persist after multiple retries or multiply as fixes are applied, ask the user for guidance.

Running Individual E2E Tests Locally

cd packages/amplify-e2e-tests
yarn e2e src/__tests__/api_1.test.ts

Local e2e tests create real AWS resources. You need valid credentials.

Authentication

The repo scripts use ada (Amazon's credential management tool), called automatically. Setup:

  1. Ensure ada and mwinit are installed
  2. Create scripts/.env: 
    E2E_ACCOUNT_PROD=<account-id>
E2E_ACCOUNT_BETA=<account-id>
  3. If you see auth errors, run mwinit

For personal AWS profiles, export credentials before running:

export AWS_PROFILE=your-profile-name

Debugging E2E Failures

1. Identify the Failing Build

yarn e2e-failed <batch-id>

2. Get Build Logs

yarn e2e-logs <build-id>

3. Simulate Locally

Run the equivalent local command to reproduce. See Development Commands for build step equivalents.

4. Check for Pre-existing Issues

git stash
git checkout main
yarn <failing-command>
git checkout <your-branch>
git stash pop

Success Criteria

A passing e2e run means 100% of tests pass with zero failures. There is no "close enough" — if any test fails, the run has failed and the failures must be investigated and fixed. Do not dismiss failures as pre-existing or infrastructure-related without verifying on main and fixing them.

Common Failure Patterns

Pattern Symptoms Action
Transient infrastructure Timeouts, credential expiration, quota errors Retry the build
Code-related Test failures, build errors, coverage threshold Fix the code, don't retry
Dependency-related Module not found, version conflicts, breaking APIs Check for major version changes, consider pinning

Build Job Types

  • build_linux / build_windows — Platform builds (not auto-retried)
  • test — Unit tests (not auto-retried)
  • lint — Linting (not auto-retried)
  • verify_dependency_licenses_extract — License verification
  • verify_api_extract — API surface verification
  • verify_yarn_lock — yarn.lock consistency
  • publish_to_local_registry — Verdaccio publish
  • graphql_e2e_tests_* — GraphQL e2e test suites (171 jobs)

Resource Cleanup

E2e tests create real AWS resources. Run periodically:

yarn cleanup-stale-resources

Troubleshooting

  • "Command failed with exit code 1" — Generic error. Read the full output for the actual message.
  • "Cannot read properties of undefined" — Usually a dependency version mismatch or breaking API change.
  • "Coverage threshold not met" — Check if pre-existing by testing on the base branch.
  • "License change detected" — Expected after dependency updates. Run yarn extract-dependency-licenses and commit.