Optimizing hyperparameters

[sdalumlarsen]

Hi SUPPORT,

I have used your system extensively on a number of volumetric datasets and I am very pleased with the result. However, I would still like to see if I can improve the denoising. Obviously, some parameters, such as the blind spot size, are VERY dependent on the nature of the data, but I was wondering if the default values for the capacity given by channel sizes, the depth and the batch size were a tradeoff between performance and training/inference time or if they actually represent an approximate optimum for performance in the face of overfitting etc. This would be for large volumetric datasets with a size of lets say (1500,1500,10000).

If you would prefer, we can communicate by email as well, I just thought any potential answers could be useful to others.

Thank you for your time and this wonderful tool.

[trose-neuro]

Hi! Let me bump this. It would be great to get some input on tuning, e.g., bs_size, patch_size, etc.

[SteveJayH]

Hi @sdalumlarsen, thank you for your interest in and for using our method. Regarding the blind spot size, please refer to my previous comment on issue #18.

Increasing the channel size creates a larger model, which may produce better denoised output but requires more memory and training/inference time. The current settings in the code are designed to balance performance with accessibility for users who may not have optimal computing environments. We often increase the model size by setting --unet_channels to [64, 128, 256, 512, 1024].

We usually keep the depth unchanged. Increasing the depth allows the model to process a larger area of the input, which might be beneficial for large images; however, to fully leverage this, the patch size would also be need to be increased. This would lead much more memory. Based on our experiments with large volumetric datasets, increasing the depth beyond the current default does not necessarily improve performance. If the results still seem insufficient after increasing the channel_size, we may consider tuning the depth further.

The batch size does not directly affect denoising performance but it affects the feasibility of training. It is advisable to use the largest batch size possible without causing GPU out-of-memory errors.

In summary, we generally follow the default settings for processing data, except for adjustments to blind spot size and unet_channels.

Let me know if you have any further questions.

[trose-neuro]

Hi Steve,

thanks a lot - this is helpful.

One further thing: we sometimes get low amplitude patch artefacts in (non trainingset) data. Is this something you have observed?

[TR: edit - I assume, this is linked to https://github.com//issues/21#issuecomment-2613736317]

[SteveJayH]

Hmm, We've rarely encountered patch artifacts.

Your observation seems relevant to that comment in issue#21.

If the patch_interval parameter is relatively large, close to the patch_size, patch artifacts may occur, especially when the patch_size is small.

As I mentioned in issue #21, I recommend setting the patch_interval to half the patch_size in the x and y axes.

[trose-neuro]

Doing this now. Also: testing different training parameters and a different training set. There could have been some background illumination parameters that did not fit to this test data well.

[trose-neuro]

We currently train with [61, 128, 128] patch_size (default). Our data is 256x256, though. Would you recommend decreasing the training patch size to 64x64 and then setting the inference patch_interval to 32x32?

[SteveJayH]

I think you don't need to decrease the patch_size and patch_interval, as doing so may not resolve the stitching artifact issue.

If the current patch artifact is due to a mismatch between training and test data, you could train another model specifically on the dataset where the artifact appears (the test data with the artifact) and check whether training on this data resolves the issue. If the artifact disappears, it may indicate that the problem stems from a large difference between the training and test data.