Right now, the Siamese Net is trained on all 5 classes which poses 2 problems that reduce performance. First, more classes mean it's easier to make mistakes which inherently lowers performance. Second, when a siamese net is trained on more than 2 classes, a set of support vectors need to be chosen. Picking a "bad" support set severly reduces accuracy, and I haven't found a method of picking a good set yet. (other than trying every vector which is time-consuming)
Right now, the Siamese Net is trained on all 5 classes which poses 2 problems that reduce performance. First, more classes mean it's easier to make mistakes which inherently lowers performance. Second, when a siamese net is trained on more than 2 classes, a set of support vectors need to be chosen. Picking a "bad" support set severly reduces accuracy, and I haven't found a method of picking a good set yet. (other than trying every vector which is time-consuming)