Skip to content

Commit 2432e0d

Browse files
sodreclaude
andcommitted
📝 docs(adr): update ADR-121 with actual performance findings
Update ADR-121 to reflect measured CI performance: - ~20% improvement (13 min → 10 min), not the initially estimated 75%+ - Document that xdist parallelization limits fixture sharing benefits - Add "Disable xdist" as rejected alternative (26 min sequential) - Clarify the observed limitation with 4 xdist workers Issue: #253 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 3b757da commit 2432e0d

1 file changed

Lines changed: 14 additions & 5 deletions

File tree

‎docs/adr/121-module-scoped-test-fixtures.md‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@
66

77
## Context
88

9-
Integration tests against LocalStack create and destroy DynamoDB tables for each test function. With 19 tests, this results in 19 table creation/deletion cycles, each taking several seconds. The cumulative overhead causes the integration test suite to run for approximately 15 minutes, creating friction in the development workflow and slowing CI feedback loops.
9+
Integration tests against LocalStack create and destroy DynamoDB tables for each test function. With 19 tests distributed across 4 pytest-xdist workers, this results in multiple table creation/deletion cycles per worker. The cumulative overhead causes the integration test suite to run for approximately 13 minutes.
1010

11-
The tests are isolated at the data level—each test creates distinct entities and buckets—but share no infrastructure. This isolation pattern is overly conservative since tests do not interfere with each other's data when using unique entity identifiers.
11+
The tests are isolated at the data level—each test creates distinct entities and buckets—but share no infrastructure within each worker. This isolation pattern is overly conservative since tests do not interfere with each other's data when using unique entity identifiers.
1212

1313
## Decision
1414

@@ -17,22 +17,31 @@ Integration tests must use module-scoped pytest fixtures (`localstack_repo_modul
1717
## Consequences
1818

1919
**Positive:**
20-
- Integration test runtime reduced from ~15 minutes to ~68 seconds (parallel) or ~2 minutes (sequential)
20+
- Integration test runtime reduced by ~20% (from ~13 minutes to ~10 minutes with xdist)
2121
- Same isolation guarantees maintained through unique entity prefixes
2222
- Pattern aligns with E2E tests which already use class-scoped fixtures
23+
- Cleaner fixture naming convention with scope in name
2324

2425
**Negative:**
2526
- Tests must use `loop_scope="module"` on both fixtures and test markers for async compatibility
2627
- Debugging failures requires understanding that data from other tests exists in the shared table
2728
- Tests that require clean infrastructure state must use function-scoped fixtures explicitly
2829

30+
**Observed Limitation:**
31+
The improvement is modest (~20%) because CI runs with pytest-xdist (`-n auto --dist loadscope`), which distributes test modules across 4 workers. Each worker creates its own module-scoped fixtures, so the total stack count equals the worker count rather than 1.
32+
33+
Disabling xdist to maximize fixture sharing was tested and rejected—sequential execution took ~26 minutes vs ~10 minutes with xdist. Parallelization benefits outweigh stack creation overhead.
34+
2935
## Alternatives Considered
3036

37+
### Disable xdist for integration tests
38+
Rejected because: Sequential execution (~26 min) is significantly slower than parallel execution with per-worker stacks (~10 min). The parallelization benefit exceeds the stack creation overhead.
39+
3140
### Class-scoped fixtures
32-
Rejected because: Module scope maximizes sharing across test classes, reducing infrastructure creation from once-per-class to once-per-module.
41+
Rejected because: Module scope maximizes sharing across test classes within each xdist worker, reducing infrastructure creation from once-per-class to once-per-module.
3342

3443
### Session-scoped fixtures
3544
Rejected because: Session scope would share infrastructure across test modules, making test isolation harder to reason about and potentially causing cross-module interference in CI parallel execution.
3645

3746
### Per-test table creation (status quo)
38-
Rejected because: The ~15 minute runtime creates unacceptable friction for development iteration.
47+
Rejected because: The ~13 minute runtime creates friction for development iteration. Even a 20% improvement is worthwhile.

0 commit comments

Comments
 (0)