You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
📝 docs(adr): update ADR-121 with actual performance findings
Update ADR-121 to reflect measured CI performance:
- ~20% improvement (13 min → 10 min), not the initially estimated 75%+
- Document that xdist parallelization limits fixture sharing benefits
- Add "Disable xdist" as rejected alternative (26 min sequential)
- Clarify the observed limitation with 4 xdist workers
Issue: #253
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/adr/121-module-scoped-test-fixtures.md
+14-5Lines changed: 14 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,9 +6,9 @@
6
6
7
7
## Context
8
8
9
-
Integration tests against LocalStack create and destroy DynamoDB tables for each test function. With 19 tests, this results in 19 table creation/deletion cycles, each taking several seconds. The cumulative overhead causes the integration test suite to run for approximately 15 minutes, creating friction in the development workflow and slowing CI feedback loops.
9
+
Integration tests against LocalStack create and destroy DynamoDB tables for each test function. With 19 tests distributed across 4 pytest-xdist workers, this results in multiple table creation/deletion cycles per worker. The cumulative overhead causes the integration test suite to run for approximately 13 minutes.
10
10
11
-
The tests are isolated at the data level—each test creates distinct entities and buckets—but share no infrastructure. This isolation pattern is overly conservative since tests do not interfere with each other's data when using unique entity identifiers.
11
+
The tests are isolated at the data level—each test creates distinct entities and buckets—but share no infrastructure within each worker. This isolation pattern is overly conservative since tests do not interfere with each other's data when using unique entity identifiers.
12
12
13
13
## Decision
14
14
@@ -17,22 +17,31 @@ Integration tests must use module-scoped pytest fixtures (`localstack_repo_modul
17
17
## Consequences
18
18
19
19
**Positive:**
20
-
- Integration test runtime reduced from ~15 minutes to ~68 seconds (parallel) or ~2 minutes (sequential)
20
+
- Integration test runtime reduced by ~20% (from ~13 minutes to ~10 minutes with xdist)
21
21
- Same isolation guarantees maintained through unique entity prefixes
22
22
- Pattern aligns with E2E tests which already use class-scoped fixtures
23
+
- Cleaner fixture naming convention with scope in name
23
24
24
25
**Negative:**
25
26
- Tests must use `loop_scope="module"` on both fixtures and test markers for async compatibility
26
27
- Debugging failures requires understanding that data from other tests exists in the shared table
27
28
- Tests that require clean infrastructure state must use function-scoped fixtures explicitly
28
29
30
+
**Observed Limitation:**
31
+
The improvement is modest (~20%) because CI runs with pytest-xdist (`-n auto --dist loadscope`), which distributes test modules across 4 workers. Each worker creates its own module-scoped fixtures, so the total stack count equals the worker count rather than 1.
32
+
33
+
Disabling xdist to maximize fixture sharing was tested and rejected—sequential execution took ~26 minutes vs ~10 minutes with xdist. Parallelization benefits outweigh stack creation overhead.
34
+
29
35
## Alternatives Considered
30
36
37
+
### Disable xdist for integration tests
38
+
Rejected because: Sequential execution (~26 min) is significantly slower than parallel execution with per-worker stacks (~10 min). The parallelization benefit exceeds the stack creation overhead.
39
+
31
40
### Class-scoped fixtures
32
-
Rejected because: Module scope maximizes sharing across test classes, reducing infrastructure creation from once-per-class to once-per-module.
41
+
Rejected because: Module scope maximizes sharing across test classes within each xdist worker, reducing infrastructure creation from once-per-class to once-per-module.
33
42
34
43
### Session-scoped fixtures
35
44
Rejected because: Session scope would share infrastructure across test modules, making test isolation harder to reason about and potentially causing cross-module interference in CI parallel execution.
36
45
37
46
### Per-test table creation (status quo)
38
-
Rejected because: The ~15 minute runtime creates unacceptable friction for development iteration.
47
+
Rejected because: The ~13 minute runtime creates friction for development iteration. Even a 20% improvement is worthwhile.
0 commit comments