♻️ Extract shared test fixtures to tests/fixtures/ package

[sodre]

Summary

The test suite creates ~122 CloudFormation stacks per full run (one per test function or class), each taking 2-10 seconds. This makes integration and E2E tests painfully slow despite tests needing only data isolation, not infrastructure isolation. Using shared session-scoped stacks with namespace-based test isolation eliminates most stack creation overhead.

Problem

Current State

Layer	Tests	Stacks created	Stack scope	Time per stack
Unit	many	0 (moto)	—	—
Integration	85	~63	mostly function	~2-10s (LocalStack)
E2E	75	~29	mixed function/class	~2-10s (LocalStack)
Benchmark	81	~30	function (LocalStack ones)	~2-10s (LocalStack)

~122 CloudFormation stacks per full test run. Each stack creation includes CloudFormation waiter polling + optional Lambda ESM stabilization (~45s on real AWS). The majority (103/122) are function-scoped, meaning each test creates and tears down its own stack.

Root Cause

Tests only need data isolation (separate entities, buckets, configs), not infrastructure isolation (separate DynamoDB tables, Lambdas, IAM roles). Since v0.10.0, namespaces provide exactly this — every DynamoDB key is prefixed with {namespace_id}/, giving complete data isolation within a shared table.

Additional Issues

1. Fixture duplication: integration/conftest.py and e2e/conftest.py are ~93% identical (~120 lines duplicated)
2. Cross-module imports: benchmark/conftest.py imports from both, creating fragile dependencies
3. No single source of truth: Hard to know which conftest is "canonical"

Proposed Solution

Shared Stacks (Session-Scoped)

Only 3-4 distinct stack configurations are needed across all tests:

Fixture	Config	Used by
shared_minimal_stack	No aggregator, no alarms	Most integration, some e2e
shared_aggregator_stack	Aggregator enabled	Aggregator tests, some benchmarks
shared_full_stack	Aggregator + alarms	Full e2e tests

Each creates the stack once per pytest session and deletes it at teardown.

Per-Test Namespace (Function-Scoped)

@pytest.fixture
async def test_namespace(shared_minimal_stack):
    """Register a unique namespace for this test, return a scoped Repository."""
    ns_name = f"test-{uuid.uuid4().hex[:8]}"
    repo = await (
        Repository.builder(shared_minimal_stack.name, "us-east-1",
                          endpoint_url=shared_minimal_stack.endpoint_url)
        .namespace(ns_name)
        .build()
    )
    yield repo
    # No cleanup needed — session teardown deletes the whole stack

Namespace registration cost: 2 WCU per namespace (~10ms on LocalStack), cached after first call.

Package Structure

Extract all shared fixtures to tests/fixtures/:

tests/fixtures/
├── __init__.py           # Re-exports
├── moto.py               # aws_credentials, mock_dynamodb, patch_aiobotocore_response
├── names.py              # unique_name, unique_namespace factories
├── stacks.py             # Session-scoped shared stack fixtures
├── repositories.py       # Namespace-scoped repo/limiter factories
├── aws_clients.py        # Boto3 client factories
├── polling.py            # E2E polling helpers
├── capacity.py           # Benchmark CapacityCounter
└── doctest_helpers.py    # Doctest mock classes and stubs

Key design: stacks.py manages session-scoped shared stacks. repositories.py creates function-scoped namespace-isolated repos within those shared stacks.

Impact on conftest.py Files

File	Before	After
unit/conftest.py (102 lines)	Inline moto fixtures	Imports from tests.fixtures.moto
integration/conftest.py (152 lines)	Per-test stack creation	Imports shared stack + namespace fixture
e2e/conftest.py (373 lines)	Per-test stack creation	Imports shared stack + namespace fixture + polling
benchmark/conftest.py (334 lines)	Cross-imports from unit/integration	Imports from tests.fixtures directly
doctest/conftest.py (547 lines)	Manual _namespace_id hacking	Uses make_moto_repo() from fixtures

Moto Fixtures (Unchanged Strategy)

Unit tests keep using create_table() + manual namespace registration (moto doesn't support CloudFormation). They benefit from the tests/fixtures/ extraction (eliminating duplication, adding namespace awareness).

Expected Performance Improvement

Layer	Before	After	Savings
Integration (85 tests)	~63 stacks × ~5s ≈ 5 min	1 stack + 85 namespaces × ~10ms ≈ 6s	~98%
E2E (75 tests)	~29 stacks × ~5s ≈ 2.5 min	2-3 stacks + 75 namespaces × ~10ms ≈ 11s	~95%
Benchmark (81 tests)	~30 stacks × ~5s ≈ 2.5 min	1-2 stacks shared ≈ 10s	~93%

Edge Cases

• Tests needing distinct stack configs (e.g., specific Lambda timeout): still create their own stack, but these are rare
• Aggregator tests: share shared_aggregator_stack — all namespaces' stream records are processed by the same Lambda (which already extracts namespace_id from PKs via ✨ Aggregator: namespace extraction from stream records #367)
• Parallel test execution (xdist): each worker gets its own namespace within the shared stack — natural isolation

Tasks

• Create tests/fixtures/__init__.py package
• Extract moto fixtures to tests/fixtures/moto.py
• Create tests/fixtures/names.py with unique_name and unique_namespace factories
• Create tests/fixtures/stacks.py with session-scoped shared stack fixtures
• Create tests/fixtures/repositories.py with namespace-scoped repo/limiter factories
• Create tests/fixtures/aws_clients.py with boto3 client factories
• Create tests/fixtures/polling.py with E2E polling helpers
• Create tests/fixtures/capacity.py with benchmark CapacityCounter
• Create tests/fixtures/doctest_helpers.py with doctest mock classes
• Refactor tests/unit/conftest.py to import from fixtures
• Refactor tests/integration/conftest.py to use shared stack + namespace fixtures
• Refactor tests/e2e/conftest.py to use shared stack + namespace fixtures
• Refactor tests/benchmark/conftest.py to import from fixtures directly
• Refactor tests/doctest/conftest.py to use fixture helpers
• Update CLAUDE.md Testing section with fixture package documentation
• Add .claude/rules/testing.md fixture organization guidelines

Acceptance Criteria

• No duplicated fixture code between conftest files
• Integration and E2E tests use session-scoped shared stacks (not per-test stacks)
• Each test gets a unique namespace for data isolation
• All tests pass with refactored fixtures
• tests/fixtures/ is properly documented in CLAUDE.md
• Test execution time reduced by >90% for integration and E2E suites
• Parallel test execution (xdist) works with namespace isolation

Documentation Updates

CLAUDE.md Changes

Add to Testing section:

### Test Fixtures

Shared fixtures live in `tests/fixtures/` package:
- `moto.py` - Moto mocking, aiobotocore compatibility
- `names.py` - unique_name, unique_namespace factories
- `stacks.py` - Session-scoped shared CloudFormation stacks
- `repositories.py` - Namespace-scoped repo/limiter factories
- `aws_clients.py` - AWS client factories
- `polling.py` - E2E polling helpers
- `capacity.py` - Benchmark CapacityCounter

Tests use shared stacks (session-scoped) with namespace isolation (function-scoped):
- Stack creation: once per session per distinct config
- Namespace registration: once per test (~2 WCU, ~10ms)
- No cleanup needed: stack deletion wipes all namespaces

[sodre]

Baseline Measurement (2026-02-15)

Branch: refactor/170-test-fixtures @ 01ccc9dc

Test Execution Times

Suite	Tests	Pass	Fail/Error	Skip	Time
Unit	2,349	2,349	0	0	162.7s
Integration	85	84	0	1	121.4s
E2E LocalStack	19	18	0	1	235.5s
E2E AWS	23	21	1	1 xfail	373.3s
E2E Discovery (AWS)	10	0	10E	0	245.5s
Benchmark	95	63	1F + 8E	23	580.4s
Doctest	549	333	0	216	202.6s

Pre-existing Failures

• E2E AWS: test_usage_snapshots_created — 1 failure
• E2E Discovery: All 10 errors — missing policy_name_format="PowerUserPB-{}" in test_aws_discovery.py fixture causes CloudFormation ROLLBACK_COMPLETE. Will fix as part of this issue.
• Benchmark: test_cascade_concurrent_throughput — 1 failure; 8 LocalStack cascade/Lambda cold-start errors

Conftest Lines (1,571 total, excluding root)

File	Lines
tests/conftest.py	63
tests/unit/conftest.py	102
tests/integration/conftest.py	152
tests/e2e/conftest.py	373
tests/benchmark/conftest.py	334
tests/doctest/conftest.py	547

Duplication: integration vs e2e conftest

Fixture	Identical?
localstack_endpoint	✅
minimal_stack_options	✅
aggregator_stack_options	✅
full_stack_options	✅
localstack_limiter	✅
localstack_limiter_with_aggregator	✅
localstack_limiter_full	✅
sync_localstack_limiter	✅
unique_name	❌ (different prefix/length)
unique_name_class	❌ (different prefix/length)

8/10 duplicated fixtures are identical. integration/conftest.py is fully a subset of e2e/conftest.py.

Other Duplication

• _patch_aiobotocore_response — identical in unit/conftest.py and doctest/conftest.py
• benchmark/conftest.py imports from unit + integration (good pattern, but fragile cross-module dependency)

Stack Creation Overhead

Every function-scoped limiter fixture creates+deletes a CloudFormation stack per test:

• Integration: ~84 stacks × ~1.4s = ~121s
• E2E: ~18 stacks × ~12.4s = ~236s
• Total: ~102 CloudFormation stacks per LocalStack run

tests/fixtures/ package

Does not exist yet.

Environment

• LocalStack 4.13.0 (community)
• macOS Darwin 25.2.0
• Python 3.12.11, pytest 9.0.2, xdist 3.8.0
• uv run pytest with xdist enabled (8 workers)

[sodre]

Results: refactor/170-test-fixtures @ d7a3c8e5

Phase 1: Fixture extraction (8325b1aa)

Suite	Baseline	After	Δ
Unit	2,349 pass / 162.7s	2,349 pass / 162.6s	no change
Integration	84 pass, 1 skip / 121.4s	84 pass, 1 skip / 109.2s	-10%
E2E LocalStack	18 pass, 1 skip / 235.5s	18 pass, 1 skip / 235.5s	no change
E2E AWS	21 pass, 1F, 1 xfail / 373.3s	21 pass, 1F, 1 xfail / 373.3s	no change
E2E Discovery (AWS)	0 pass, 10E / 245.5s	10 pass / 105.1s	0→10 pass (bug fix)
Benchmark	63 pass, 1F+8E, 23 skip / 580.4s	72 pass, 23 skip / 229.1s	+9 pass, -60% time
Doctest	333 pass, 216 skip / 202.6s	333 pass, 216 skip / 199.1s	no change

Phase 2: FileLock shared stacks + pytest_sessionfinish cleanup (d7a3c8e5)

xdist workers share CloudFormation stacks via FileLock; cleanup runs once in the controller via pytest_sessionfinish using SyncRepository.delete_stack().

Suite	Baseline	After	Δ
Integration	84 pass, 1 skip / 121.4s	84 pass, 1 skip / 76.4s	-37%
Benchmark	72 pass, 23 skip / 229.1s	72 pass, 23 skip / 189.9s	-17%
E2E LocalStack	18 pass, 1 skip / 235.5s	39 pass, 1 skip / 436.7s	N/A (test count doubled¹)

¹ E2E test count went from 19 → 40 due to new namespace/CLI tests (#394). The 3 stack-variation tests (test_cloudformation_*, test_role_naming*) account for ~342s as they deploy/delete their own stacks by design.

Conftest Lines

File	Before	After	Δ
tests/conftest.py (root)	63	81	+29% (added pytest_sessionfinish hook)
tests/unit/conftest.py	102	56	-45%
tests/integration/conftest.py	152	64	-58%
tests/e2e/conftest.py	373	118	-68%
tests/benchmark/conftest.py	334	170	-49%
tests/doctest/conftest.py	547	326	-40%
Total conftest	1,571	815	-48%
tests/fixtures/ (new)	0	875	—

Key Gains

• Integration time -37%: FileLock shared stacks across xdist workers (1 stack vs 1-per-worker)
• Benchmark time -17%: session-scoped stacks eliminate per-test stack creation overhead (benchmarks run with xdist disabled)
• Discovery bug fixed: policy_name_format was missing in test_aws_discovery.py fixtures → 10 errors eliminated
• Benchmark errors eliminated: 8 LocalStack cascade/Lambda errors gone (shared session stacks avoid cold-start races)
• Conftest duplication eliminated: identical fixtures between integration/e2e/benchmark consolidated into tests/fixtures/
• Cross-module conftest imports eliminated: rg "from tests\.(integration|e2e|benchmark)\.conftest" tests/ returns 0
• Stack cleanup: pytest_sessionfinish hook deletes shared stacks after all workers complete using SyncRepository.delete_stack() (no orphaned stacks, no raw boto3)
• CI: integration tests now run on draft PRs for faster feedback
• No regressions: all suites match or exceed baseline pass counts