The script carps.analysis.generate_report fails on critical difference plotting if less than three optimizers are given since Friedman test requires multiple samples.
E.g. error:
ValueError: At least 3 sets of samples must be given for Friedman test, got 2.
Todo: change the generate_report function to only do reports depending on Friedman tests if there are more than two optimizers being compared.
Could be something like wrapping it in an if-statement
n_optimizers = len(df['optimizer_id'].value_counts())
if n_optimizers >= 3:
# critical difference
# ranks over time
# etc.
The script
carps.analysis.generate_reportfails on critical difference plotting if less than three optimizers are given since Friedman test requires multiple samples.E.g. error:
Todo: change the generate_report function to only do reports depending on Friedman tests if there are more than two optimizers being compared.
Could be something like wrapping it in an if-statement