Performance looks similar

[John1231983]

Thanks for your implementation of batch renormalization. I saw that your performance of batch renorm and batch norm are similar. Do you check the performance with a simple network as in the paper? The paper shows a simple network with high gain between batch renorm and batch norm.

[titu1994]

The findings match the paper. The performance of batch renorm is just slightly higher than batchnorm.

Renorm is only faster to get to that high score than batchnorm.

[John1231983]

Thanks. Sorry I did not give more information. I mean the benefit in small batch size as figure 2 and figure 3. For larger batch size, I totally agree with you that they are similar. Could you measure performance in small batch size case?

[titu1994]

Hmm, I did try with batch size of 4 and 8, and the results were similar to that with larger batch sizes. Batch renorm quickly hits a high score whereas for batch norm, it takes more time.

I think I removed all the code and images for those tests during a cleanup of the codebase though.

[guylinggl]

I got similar results for batch normalization and re normalization on very small (1-2) batch sizes which is inconsistent with the results in the paper.
Thanks!