we can increase minimum samples — basically assuming an 80/20 train/test split N>=2 means at the lowest count you get 1.6 in train and 0.4 in test (which is not possible). For robust evaluation we want to maximize the test counts and if you are splitting 80/20 (I think I saw 20%), then if there are 10 examples for the class 8 go to training and 2 to test. But 2 is low … we may want >= 5? which implies min_samples:
0.8/0.2 = X / 5 -> X = 4/0.2 = 20
so if you want at least 5 test examples then the minimum number of total examples for the class should be 25.
this should also improve accuracy since the rare classes are going to be hard to train/predict/evaluate.
@justaddcoffee @turbomam
env-embeddings/src/env_embeddings/rf_analysis.py
Line 492 in 0bc5adc
we can increase minimum samples — basically assuming an 80/20 train/test split N>=2 means at the lowest count you get 1.6 in train and 0.4 in test (which is not possible). For robust evaluation we want to maximize the test counts and if you are splitting 80/20 (I think I saw 20%), then if there are 10 examples for the class 8 go to training and 2 to test. But 2 is low … we may want >= 5? which implies min_samples:
0.8/0.2 = X / 5 -> X = 4/0.2 = 20
so if you want at least 5 test examples then the minimum number of total examples for the class should be 25.
this should also improve accuracy since the rare classes are going to be hard to train/predict/evaluate.
@justaddcoffee @turbomam