BinaryClassification with name != "default" crashes

[emacha]

I don't know if this is a bug or not supposed to be implemented. But I'll ask the question here.

The premise is that I have a model that trains in more than one target. So I want to pass more than one BinaryClassification class to my data_definition.
Given the signature is classification: List[Classification] | None, I'd assume this is intended as a feature.

If I don't define a unique name for BinaryClassification, it crashes on report.run because it's not unique.

>>>   raise ValueError("More than one classification with id {}".format(classification_id))
>>>ValueError: More than one classification with id default

However, if I do define a name, it doesn't work like my example below.

import pandas as pd
from evidently import Dataset, DataDefinition, Report, BinaryClassification
from evidently.metrics import Accuracy

data = pd.DataFrame({
    "id_col": [1, 2, 3],
    "actual_1": [10.0, 20.0, 30.0],
    "predicted_1": [11.0, 19.0, 29.0],
    "actual_2": [10.0, 20.0, 30.0],
    "predicted_2": [11.0, 19.0, 29.0],
})
classification_ = [BinaryClassification(target="actual_1", prediction_labels="predicted_1", name="regression_1"), BinaryClassification(target="actual_2", prediction_labels="predicted_2", name="regression_2")]
data_definition = DataDefinition(id_column="id_col", classification=classification_)
dataset = Dataset.from_pandas(data, data_definition=data_definition)
report = Report(metrics=[Accuracy(target="actual_1", prediction="predicted_1"),])
report.run(dataset)

>>>     raise KeyError(key) from err
>>> KeyError: None

I think I traced it to this function _default_input_data_generator, that hardcodes the name to default. So when running the report it tries to slice a DataFrame with none.

[sarahshehri]

Is there "BinaryClassification" library from evidently? i didn't saw any library matrices like this in last version!

[grgkovac]

You can define classification_name for Accuracy like so:

import pandas as pd
from evidently import Dataset, DataDefinition, Report, BinaryClassification
from evidently.metrics import Accuracy

data = pd.DataFrame({
    "id_col": [1, 2, 3],
    "actual_1": [0, 1, 1],
    "predicted_1": [1, 0, 1],
    "actual_2": [1, 1, 0],
    "predicted_2": [1, 1, 1],
})
classification_ = [BinaryClassification(target="actual_1", prediction_labels="predicted_1", name="cls_1"), BinaryClassification(target="actual_2", prediction_labels="predicted_2", name="cls_2")]
data_definition = DataDefinition(id_column="id_col", classification=classification_)
dataset = Dataset.from_pandas(data, data_definition=data_definition)
report = Report(metrics=[
    Accuracy(target="actual_1", prediction="predicted_1", name="acc_1", classification_name="cls_1"), # <- name defined here
    Accuracy(target="actual_2", prediction="predicted_2", name="acc_2", classification_name="cls_2") # <- name defined here
])

report.run(dataset)

I added a pull request with better error handling: #1803