[BUG] Fix Benchmarking duplicate estimator label collisions#299
Conversation
There was a problem hiding this comment.
Pull request overview
Fixes a Benchmarking result-collision bug where multiple estimators of the same class overwrote each other by generating deterministic, disambiguated estimator display names (with 1-based suffixes only when needed), and adds regression tests for both duplicate and unique estimator naming.
Changes:
- Add
_get_estimator_names()to generate stable estimator labels and suffix duplicates (e.g.,DummyClassifier[1],DummyClassifier[2]). - Update
Benchmarking.run()to key results by these stable labels instead of raw class names. - Add regression tests to ensure duplicate estimator classes stay distinct and unique classes keep original labels.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
pyaptamer/benchmarking/_base.py |
Introduces stable estimator naming and uses it to prevent result-row overwrites for duplicate estimator classes. |
pyaptamer/benchmarking/tests/test_benchmarking_core.py |
Adds regression coverage for duplicate-name disambiguation and unique-name preservation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
siddharth7113
left a comment
There was a problem hiding this comment.
left the comment to change the suffix structure, otherwise looks good.
|
|
||
| summary = bench.run() | ||
|
|
||
| assert ("DummyClassifier[1]", "accuracy_score") in summary.index |
There was a problem hiding this comment.
this notation wouldn't be right, and could be confusing in downstream usage, user would have to make a call something like this df.loc["DummyClassifier[1]"] , I would suggest to index along with strategy something to make it either like DummyClassifer_1 or based on Pattern should be fine as well.
|
@siddharth7113 Thanks for the feedback. I updated the duplicate-estimator label format to use underscore suffixes instead of brackets. Now duplicate estimators of the same class are labeled deterministically as Updated the regression tests accordingly; |
|
Please follow PR guidelines and make sure to install and run |
|
cc @satvshr , for your feedback |
| assert len(summary) == 2 | ||
|
|
||
|
|
||
| def test_benchmarking_preserves_unique_estimator_names(): |
There was a problem hiding this comment.
Was this also a problem which you solved or are you adding a random test
Closes #297.
What this changes
Benchmarkingresults with deterministic 1-based suffixesDummyClassifierinstances no longer overwrite one anotherValidation
pytest -q pyaptamer/benchmarking/tests/test_benchmarking_core.py