[BUG] Benchmarking collapses multiple estimators of the same class into one result row

[ashum9]

Describe the bug

Benchmarking.run() stores results in a dict keyed only by estimator.__class__.__name__, so benchmarking two estimators of the same class silently overwrites the first result with the second.

This means hyperparameter comparisons within the same estimator family are not represented correctly in the summary table.

Relevant code:

• pyaptamer/benchmarking/_base.py (est_name = estimator.__class__.__name__ and results[est_name] = est_scores)

Minimal reproduction

import numpy as np
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score
from pyaptamer.benchmarking import Benchmarking

X = np.array([[0], [1], [0], [1], [0], [1], [0], [1]])
y = np.array([0, 1, 0, 1, 0, 1, 0, 1])

bench = Benchmarking(
    estimators=[
        DummyClassifier(strategy="most_frequent"),
        DummyClassifier(strategy="stratified", random_state=0),
    ],
    metrics=[accuracy_score],
    X=X,
    y=y,
    cv=2,
)

print(bench.run())

Current behavior:

• only one DummyClassifier row is returned

Expected behavior:

• both estimators should appear in the results under distinct labels
• distinct estimator classes should keep their original class-name labels

Proposed fix

Generate stable display names for estimators before running CV, and disambiguate duplicate class names with a deterministic suffix such as [1], [2], etc.

[ashum9]

Root cause: Benchmarking.run() keys results by estimator.class.name, so the second estimator overwrites the first when classes match.
Proposed fix: generate stable display names and disambiguate duplicates deterministically (e.g. DummyClassifier[1], DummyClassifier[2]) while keeping unique classes unchanged.
Test plan: add regression test with two DummyClassifier instances and assert 2 rows.

[ashum9]

Reproduced on main:

$ python - <<'PY'
import importlib.util, pathlib
import numpy as np
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score

path = pathlib.Path("pyaptamer/benchmarking/_base.py")
spec = importlib.util.spec_from_file_location("pyaptamer_benchmarking_base", path)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
Benchmarking = mod.Benchmarking

X = np.array([[0],[1],[0],[1],[0],[1],[0],[1]])
y = np.array([0,1,0,1,0,1,0,1])

bench = Benchmarking(
    estimators=[
        DummyClassifier(strategy="most_frequent"),
        DummyClassifier(strategy="stratified", random_state=0),
    ],
    metrics=[accuracy_score],
    X=X,
    y=y,
    cv=2,
)
res = bench.run()
print(res)
print("n_rows", len(res))
PY
                                train  test
estimator       metric
DummyClassifier accuracy_score    0.5   0.5
n_rows 1

Root cause: results are keyed by estimator.__class__.__name__, so same-class estimators overwrite.

Proposed fix: disambiguate duplicate class names deterministically (e.g. DummyClassifier[1], DummyClassifier[2]) while preserving original class names when unique.

PR: #299

[siddharth7113]

cc @satvshr

[siddharth7113]

Hi @ashum9 , thanks for opening an issue , but please refrain for opening a PR for an issue before the issue has been discussed with maintainers, I am currently converting your PR to draft, once we think the issue is valid , feel free to either open a new PR or convert your draft PR for review.

[ashum9]

hello @siddharth7113 , Thanks for the clarification. I understand the workflow now and will ensure issues are discussed with maintainers before opening a PR.