Skip to content

test-plan.md

Latest commit

 

History

History
129 lines (91 loc) · 12.2 KB

test-plan.md

File metadata and controls

129 lines (91 loc) · 12.2 KB
Raw

Test Plan

Test Plan — Health Endpoint

Area-scoped test plan for the GET /health endpoint and Docker HEALTHCHECK integration.

Coverage Summary

3 unit tests · 2 integration tests · 2 e2e tests · 3 UX visual tests · 5 edge-case tests = 15 total. Covers all acceptance criteria for the health endpoint (AC-7: GET /health returns 200 + {"status": "ok"}), all interactive elements, all error/edge cases, and the full Mockup F flow including Docker HEALTHCHECK integration.

Health Endpoint

The GET /health endpoint serves Docker HEALTHCHECK probes and load-balancer liveness checks. It returns {"status": "ok"} with HTTP 200. It has no parameters, no persistence, and no state. The only failure mode is using a disallowed HTTP method (POST, PUT, DELETE), which returns 405.

Unit Tests

  • HEALTH-UNIT-001 (unit) — GET /health returns 200 with body {"status": "ok"}.
  • HEALTH-UNIT-002 (unit) — GET /health response has Content-Type: application/json.
  • HEALTH-UNIT-003 (unit) — POST /health returns 405 Method Not Allowed.

Unit Tests — Disallowed Methods

  • HEALTH-UNIT-004 (unit) — PUT /health returns 405 Method Not Allowed.
  • HEALTH-UNIT-005 (unit) — DELETE /health returns 405 Method Not Allowed.

Integration Tests

  • HEALTH-INT-001 (integration) — After a 405 from POST, retrying with GET returns 200 — confirms the endpoint is stateless and recoverable.
  • HEALTH-INT-002 (integration) — Dockerfile contains a valid HEALTHCHECK instruction targeting GET /health on port 8080, and the HEALTHCHECK command is executable (pure-Python urllib or curl).

E2E Tests

  • HEALTH-E2E-001 (e2e) — Smoke: start the app, send GET /health, verify 200 + {"status": "ok"}, send POST /health, verify 405, send GET /health again, verify still 200. Three-action chain exercising the full request lifecycle.
  • HEALTH-E2E-002 (e2e) — Docker HEALTHCHECK end-to-end: build the Docker image, start the container, wait for the HEALTHCHECK to report healthy, verify logs show the health check request. (Validates the full mockup-F flow from Docker HEALTHCHECK through FastAPI routing to 200 OK.)

UX Visual Tests (Mockup F)

  • HEALTH-UX-F-001 (ux) — Mockup F step 1: Docker HEALTHCHECK configuration is present and correct in the Dockerfile (interval, timeout, retries, command).
  • HEALTH-UX-F-002 (ux) — Mockup F steps 2–3: GET /health returns exactly the response shape shown in mockup — {"status": "ok"} with HTTP 200, Content-Type JSON.
  • HEALTH-UX-F-003 (ux) — Mockup F step 4: Container is marked healthy by Docker after HEALTHCHECK succeeds — the full chain from probe to healthy status works as depicted.

Edge-Case Tests

  • HEALTH-EDGE-001 (unit) — GET /health?foo=bar with query parameters still returns 200 (query params are ignored).
  • HEALTH-EDGE-002 (unit) — GET /health with a JSON request body still returns 200 (body is ignored on GET).
  • HEALTH-EDGE-003 (unit) — 100 rapid sequential GET /health requests all return 200 (stateless endpoint, no concurrency issues).
  • HEALTH-EDGE-004 (unit) — GET /healthz (common misspelling) returns 404 — the endpoint is at /health only.
  • HEALTH-EDGE-005 (unit) — GET /health/ (trailing slash) returns the expected behaviour (200 with redirect or 404 — verify it doesn't crash).

Gaps & Open Questions

  • GAP-1 Question: Should the Docker HEALTHCHECK E2E test (HEALTH-E2E-002) require Docker to be available in the test environment, or should it be a manual/CI-only test? Recommendation: Make it CI-only with a @pytest.mark.docker marker — Docker isn't always available in unit test runners, but the HEALTHCHECK integration is critical enough to verify in CI.
  • GAP-2 Question: Should we test PATCH /health and OPTIONS /health in addition to POST/PUT/DELETE? Recommendation: Test at least one other method (PATCH) to confirm FastAPI's method-disallow behaviour is consistent, but don't exhaustively test every HTTP verb — that's a FastAPI framework concern, not a feature concern.

Test Plan — Webhook Endpoint

Coverage Summary

8 unit tests · 4 integration tests · 5 E2E tests · 18 UX visual tests = 35 total covering the Webhook endpoint area (POST /webhooks/shopify/product-update, Pydantic validation, structlog output, error handling, recovery flows).

Webhook Endpoint — Core POST /webhooks/shopify/product-update

Validates incoming Shopify product update payloads, logs them via structlog, and returns 200 OK or 422 on validation failure.

E2E (full request-response lifecycle via TestClient, multi-action chains)

  • WH-E2E-001 (e2e) — Smoke test: send a valid product update, verify 200 response, then send an invalid request, verify 422, then send another valid request to confirm recovery.
  • WH-E2E-002 (e2e) — State traversal: idle → valid webhook received → validation passes → logging fires → 200 returned → connection closed. Verifies the full happy-path state machine with log output capture.
  • WH-E2E-003 (e2e) — Error path: send payload missing required id, verify 422 with error detail mentioning id, verify structlog does NOT emit a product_update log line for the failed request.
  • WH-E2E-004 (e2e) — Recovery: send an invalid payload (missing title), get 422, then immediately send a valid payload, get 200 with correct log output. Confirms service recovers cleanly from errors.
  • WH-E2E-005 (e2e) — Large payload: send a webhook with 1000 variant objects in the extra fields, verify 200 response and that the full payload appears in the log output. Tests the extra="allow" behaviour under load.

Integration (multi-component interactions)

  • WH-INT-001 (integration) — Valid payload flows through FastAPI router → Pydantic model → structlog → response. Captures actual structlog output and verifies event, product_id, title, payload fields match the input.
  • WH-INT-002 (integration) — Invalid payload (missing id) flows through FastAPI router → Pydantic validation failure → 422 response with FastAPI's standard error detail structure. Verifies error schema matches spec §4.
  • WH-INT-003 (integration) — Minimal payload (only id + title) flows through full pipeline. Verifies optional fields default to None and the log output contains only the provided fields plus None defaults.
  • WH-INT-004 (integration) — Concurrent requests: send 10 valid payloads simultaneously (using asyncio), verify all return 200 and all 10 log lines are emitted. Tests stateless handling under concurrency.

Unit (single function/component in isolation)

  • WH-UNIT-001 (unit) — POST with valid top-level fields returns 200 and body {"status": "received"}. (AC: spec §15 AC-1)
  • WH-UNIT-002 (unit) — Valid payload causes structlog to emit a log line with event="shopify.product_update", correct product_id, title, and full payload. (AC: spec §15 AC-2)
  • WH-UNIT-003 (unit) — POST without id field returns 422; error detail mentions id. (AC: spec §15 AC-3)
  • WH-UNIT-004 (unit) — POST without title field returns 422; error detail mentions title. (AC: spec §15 AC-4)
  • WH-UNIT-005 (unit) — POST with extra nested fields (variants, images) returns 200; extra fields not rejected. (AC: spec §15 AC-5)
  • WH-UNIT-006 (unit) — POST with malformed JSON body returns 422. (AC: spec §15 AC-6)
  • WH-UNIT-007 (unit) — POST with null for id returns 422; Pydantic rejects null for required int field.
  • WH-UNIT-008 (unit) — POST with string "abc" for id returns 422; Pydantic rejects type mismatch.
  • WH-UNIT-009 (unit) — POST with only id and title (no optional fields) returns 200; optional fields default to None.
  • WH-UNIT-010 (unit) — POST with empty body returns 422.
  • WH-UNIT-011 (unit) — POST without Content-Type: application/json header but with valid JSON body still returns 200 (FastAPI parses body regardless of Content-Type).
  • WH-UNIT-012 (unit) — structlog output is valid JSON and a single line per event (no duplicate lines, no non-JSON output). (Risk: implementation-plan Risk #1 — structlog + stdlib bridge misconfiguration)

UX (visual/interaction verification per mockup state)

Mockup A — Idle State

  • WH-UX-A-001 (ux) — Mockup A: On app startup, Uvicorn is listening on port 8080 and GET /health returns 200, confirming the idle/service-running state.
  • WH-UX-A-002 (ux) — Mockup A: No active connections exist on startup — the service accepts connections but has not processed any requests yet.
  • WH-UX-A-003 (ux) — Mockup A: Routes are registered (POST /webhooks/shopify/product-update and GET /health) — verify via OpenAPI schema or FastAPI router inspection that both routes exist.

Mockup B — Webhook Received

  • WH-UX-B-001 (ux) — Mockup B: Sending a POST to /webhooks/shopify/product-update with Content-Type application/json is accepted (connection established, request received).
  • WH-UX-B-002 (ux) — Mockup B: POST without Content-Type header is still accepted and routed to the handler (FastAPI parses body regardless).
  • WH-UX-B-003 (ux) — Mockup B: POST with an unsupported HTTP method (e.g., PUT) to /webhooks/shopify/product-update returns 405 Method Not Allowed.

Mockup C — Validation State

  • WH-UX-C-001 (ux) — Mockup C: Pydantic model validates required fields id (int) and title (str) — valid types accepted, invalid types rejected with 422.
  • WH-UX-C-002 (ux) — Mockup C: HMAC verification is skipped — requests without X-Shopify-Hmac-Sha256 header are accepted (per spec §16 out-of-scope).
  • WH-UX-C-003 (ux) — Mockup C: Extra nested fields (variants, images, metafields) pass through without validation — extra="allow" is honoured.

Mockup D — Active Logging

  • WH-UX-D-001 (ux) — Mockup D: structlog emits a single JSON line per webhook with event="shopify.product_update", product_id, title, and payload fields matching the spec §5 Log Output Schema.
  • WH-UX-D-002 (ux) — Mockup D: Log output includes ISO 8601 UTC timestamp and level: "info" fields.
  • WH-UX-D-003 (ux) — Mockup D: Failed validation requests (422) do NOT produce a shopify.product_update info log line — only valid webhooks are logged at info level.

Mockup E — Success/Completion

  • WH-UX-E-001 (ux) — Mockup E: After valid webhook processing, HTTP 200 OK is returned with body exactly {"status": "received"}.
  • WH-UX-E-002 (ux) — Mockup E: Response Content-Type is application/json.
  • WH-UX-E-003 (ux) — Mockup E: After response is sent, the service is ready for the next request (no lingering state or connection issues).

Mockup G — Error State

  • WH-UX-G-001 (ux) — Mockup G: Missing required field returns 422 with FastAPI standard error detail containing loc, msg, and type fields.
  • WH-UX-G-002 (ux) — Mockup G: Malformed JSON returns 422 with error detail indicating JSON decode failure.
  • WH-UX-G-003 (ux) — Mockup G: After a 422 error, the service accepts and correctly processes a subsequent valid request (error state is not sticky).

Gaps & Open Questions

  • GAP-1 Question: Should E2E tests use a real Uvicorn server or FastAPI TestClient? Recommendation: Use TestClient for all tests — the spec §12 explicitly states "All testing is done via HTTP requests using httpx/TestClient" and marks E2E browser tests as N/A. TestClient exercises the full ASGI stack without network overhead.
  • GAP-2 Question: Mockup F (Health Check) is excluded from this area-scoped run — should we still include health-check-adjacent tests like verifying the endpoint exists on app startup? Recommendation: Include WH-UX-A-001 and WH-UX-A-003 which verify health endpoint registration as part of idle-state verification, but defer dedicated health-endpoint tests to the health-check area run.
  • GAP-3 Question: The spec's §14 edge case "Very large payload (>1 MB)" has no explicit limit — should WH-E2E-005 test a specific size? Recommendation: Test with 1000 variant objects (~50-100KB) to exercise extra="allow" without hitting Uvicorn defaults. A separate performance/load test is out of scope.