⚡ perf(benchmark): module-scope fixtures for faster benchmark runs (#171)#396
Merged
Conversation
Document the design for module-scoping benchmark fixtures to reduce setup overhead and enable steady-state warm-path measurements. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
7-task plan covering module-scoped moto fixtures, BenchmarkEntities dataclass, test file updates (throughput, latency, operations), verification, and documentation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ies (#171) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…171) Replace function-scoped sync_limiter with module-scoped benchmark_entities in all throughput benchmark tests. Tests now use pre-warmed entity IDs instead of creating entities inline, measuring steady-state performance without cold-start overhead. - TestThroughputBenchmarks: use benchmark_entities.flat for standalone ops - TestThroughputWithHierarchy: use benchmark_entities.parents/children instead of hierarchy_limiter fixture - Remove unused `from typing import Any` import Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace function-scoped sync_limiter with module-scoped benchmark_entities in all latency benchmarks. Tests now use pre-warmed entity IDs from the shared BenchmarkEntities dataclass, eliminating per-test entity creation and table setup overhead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…171) Switch TestAcquireReleaseBenchmarks, TestTransactionOverheadBenchmarks, TestCascadeOverheadBenchmarks, and TestConcurrentThroughputBenchmarks to use module-scoped benchmark_entities for steady-state measurement. Keep TestConfigLookupBenchmarks and TestOptimizationComparison on function-scoped sync_limiter / sync_limiter_no_cache for clean-state optimization comparisons. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #396 +/- ##
=======================================
Coverage 91.88% 91.88%
=======================================
Files 33 33
Lines 7435 7435
=======================================
Hits 6832 6832
Misses 603 603
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.40.
| Benchmark suite | Current: e413927 | Previous: 1cbe2e3 | Ratio |
|---|---|---|---|
tests/benchmark/test_operations.py::TestOptimizationComparison::test_stored_limits_cache_disabled |
136.49090751013046 iter/sec (stddev: 0.03293390179940827) |
229.52741614495207 iter/sec (stddev: 0.00009611893914352546) |
1.68 |
This comment was automatically generated by workflow using github-action-benchmark.
Tests only use indices 0-12. Creating 111 entities dominated module setup time (~4s). With 15 flat + 11 hierarchy = 26 entities, module setup drops to ~1.3s. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mock_dynamodb_module,benchmark_limiter) and aBenchmarkEntitiesdataclass totests/benchmark/conftest.pyso entity/table setup runs once per test file instead of once per testtest_throughput.py,test_latency.py, andtest_operations.pyto use pre-warmedbenchmark_entitiesfor warm-path measurements, while capacity and optimization-comparison tests keep function-scopedsync_limiter.claude/rules/testing.mdwith a decision table and module-scoped moto tipsdocs/plans/Test plan
uv run pytest tests/benchmark/ -o "addopts=" -v --benchmark-only— all benchmark tests passuv run pytest tests/unit/ -v— no regressions in unit testsCloses #171
🤖 Generated with Claude Code