NaN values when training a network on large datasets

[rronan26]

Hi,

We are attempting to train a classifier network on what I believe to be a fairly large dataset (about 20,000 by 20,000 px with a scaling factor of 1.0). When we go to train the network, all seems well during the initial iterations, but during the third epoch, the given loss values begin to fluctuate massively, jumping between values ranging from 0 to 10--often with a loss difference of 5 or more between each iteration. Eventually, at a certain iteration, the loss value becomes NaN, and all subsequent iterations continue to produce NaN loss values.

I am not a machine learning expert, but I assumed the issue was a result of an exploding gradient and attempted to mitigate it by lowering the learning rate to 0.00001. Unfortunately, this yielded similar results.

It is important to note that we have been able to successfully train networks on smaller datasets and thus it is likely not an issue with our downloaded version of TagLab.

It may also be notable that many of the classes in our large dataset make up only a small (>1) percentage of the image as a whole, although I am not sure how this would have such an affect on the training process.

It would be ideal for us to be able to use our entire image to train our network so that it includes all the classes we need, as no one region of the image contains each class.

[Jordan-Pierce]

@rronan26 can you somehow provide stats on the distribution of class categories in your dataset (e.g., % of pixels for each class category)?

20k x 20k is pretty big and if most of the classes make up less than 1% of the image, that's not a lot of pixels, and might actually be the reason why you're having these issues.

As a sanity check, you already know your implementation of TagLab works and so does the training code, so it is probably the dataset.

[rronan26]

Thank you so much for the quick response! Here is the class distribution:

Having looked at the distribution again, I now realize that many of the classes take up a significantly smaller proportion of the image than I had initially thought. I think you're right in saying that this is the issue. Each of the classes that take up less than 0.1% of the entire image represent only about 2-3 individual corals. Obviously, we can't add more, but would still like to include them in our overall model for segmentation of future images. I'd imagine it would be best for us to train the network on a bunch of smaller sections of the image/other images; is it possible to compile different exported datasets into one file for training? I believe it's been said here that training the network multiple times on different datasets will just result in data being overwritten, but correct me if I'm wrong.

Is there anything else you'd suggest to solve our issue?

[maxcorsini]

The problem is that the amount of pixels in your class is very low, for many classes. A percentage of 0.05%, 0.11%, 0.03% and so on is so low that it is highly unlike that the classifier may recognize such class. And since you have many of such classes, also the training suffers, from a nuermical point of view.

One suggestion may be to throw away many of the tiles in the dataset leaving only the ones with some of the target classes and reduce the number of target classes, but if this means to throw a great part of the data you could have performance problems also in this case...

[matteo-collina]

Hello everyone,

I am jumping into the conversation as I am facing the same problem. This is how my classes look:

I have used the script on the last release to remove tiles with no data, but I still don't get even close to 1%. Do you have any recommendations on how to clean the tiles better? Any other solution?

Thanks,
Matteo

[maxcorsini]

Dear Matteo,

remove the tiles with no information may help, but in your case you have a lot of classes, so the final effect is not impactful. Moreover, the amount of such classes is, in general very low, so it is difficult to train a network successfully with this data.

IF your data coming from different zones/plots, you can try to create different classifiers for different situations, but looking at the table you provided my feeling is that, even if you split your data in sub-datasets, the amount of data for the training remains too much low to obtain reliably results (I hope to be wrong, obviously).

Best,
Massimiliano

[matteo-collina]

Thanks Massimiliano,

Yes, I have also tried to train just with Background and RedBoringSponge (the most abundant) without any luck. What is not clear to me is how the training works within Taglab. Since the orthomosaics are spllited in tiles, there are tiles where the annotated part is high compared to the background. I have always used YOLO and I was thinking that the once you have 40/70 examples of the same species on different tiles the classification should start working...but looks like is not the case. Can you please briefly clarify this?

Also, do you think trying to train the model with YOLO would make any difference? Altought I am not sure if you can transform the .pt you get from YOLO to work with Taglab, but I saw there is a way to export it.

Many thanks,
Matteo