Skip to content

test_methodology.md

Latest commit

 

History

History
119 lines (84 loc) · 3.91 KB

test_methodology.md

File metadata and controls

119 lines (84 loc) · 3.91 KB
Raw

Testing Methodology — WMATA Analytics App

Established: 2026-05-05
Applies to: All feature iterations beyond the initial dashboard

Philosophy

Build a thin vertical slice. Test it live. Fix what breaks. Document everything. Then expand.

We test at three levels on every iteration, in order. Each level must pass before the next begins. If anything fails, we fix it before moving on — no skipping ahead.

Three-Level Test Pyramid

        ┌─────────────────────┐
        │   L3: UI / Browser  │  ← Does it look right? Can a user use it?
        ├─────────────────────┤
        │  L2: Integration    │  ← Does data flow end-to-end through the app?
        ├─────────────────────┤
        │   L1: API / Client  │  ← Does the API return what we expect?
        └─────────────────────┘

L1 — API Client Tests (tests/run_tests.py)

Run against the live WMATA API. Verify:

  • Endpoint reachable, returns HTTP 200
  • Response contains expected top-level keys
  • Data types match documented spec (especially string fields like Min)
  • Edge cases: empty results, missing fields, unexpected values

Pass criteria: All assertions pass, no exceptions. Response shape matches spec.

L2 — Integration Tests

Run the dashboard's data-fetch code path (not the UI) and verify:

  • Data reaches the display layer in the correct shape
  • Field transformations (format_min, dedup, sorting) produce expected output
  • Demo mode fallback activates correctly when key is absent

Pass criteria: No exceptions, output matches expected structure.

L3 — Browser / UI Tests

Use Claude in Chrome to interact with the running Streamlit app. Verify:

  • UI elements render (tabs, dropdowns, tables, charts)
  • User workflow completes (select route → select stop → see arrivals)
  • Live data appears (not demo/empty state)
  • No visible Python tracebacks or Streamlit error boxes
  • Auto-refresh doesn't break the selected state

Pass criteria: Screenshot shows correct content; no error states visible.

Iteration Template

Each feature iteration follows this structure:

### Iteration N — [Feature Name]

**Goal:** One sentence.

**Test Plan:**
| # | Level | What | Pass Condition |
|---|-------|------|---------------|
| T1 | L1 | ... | ... |
| T2 | L2 | ... | ... |
| T3 | L3 | ... | ... |

**Build:** [what was changed]

**Test Results:**
| # | Result | Notes |
|---|--------|-------|
| T1 | ✅ PASS / ❌ FAIL | ... |

**Fixes Applied:** [if any]

**Status:** ✅ Complete / 🔄 In Progress / ❌ Blocked

Results are recorded in tests/test_results.md.

What We Do NOT Test

  • Internal Streamlit rendering details (widget IDs, DOM structure) — too brittle
  • WMATA API correctness (we trust their data; we test our handling of it)
  • Performance / load (out of scope for a demo app)
  • Every station or route (we test representative samples; spot-check edge cases)

Representative Test Fixtures

These are used consistently across iterations for reproducible results:

Fixture Value Why
Rail station Metro Center (A01) Multi-line, always active
Rail station (Red Line only) Friendship Heights (A08) Single-line, confirmed working
Bus route 16Y Frequent, multi-stop, runs weekends
Bus stop 1001195 Pentagon City stop, high frequency
Incident-prone line RD Currently has weekend alerts
Off-peak test time Before 6am / after 11pm May return empty predictions

Regression Check (run after every iteration)

After each new feature, verify the previous iteration's L3 test still passes:

  1. Rail predictions table still renders at Metro Center
  2. Line status panel still shows all 6 lines
  3. No new Streamlit warnings in the terminal output