How to run VTAB on multi-GPUs?

[woxue]

Thanks for your solid work!
I have some problem with reproducing the result on your leaderboard. The suggest batch size on your paper is 512 however the biggest batch size supported on one GPU(2080 ti) is 64 (128 will cause OOM).
The tensorflow_hub seems not working well with tf.distribute.MirroredStrategy() to support multi-GPUs, leading to the error below:

RuntimeError: variable_scope module/ was unused but the corresponding name_scope was already taken.
just as mentioned in tensorflow/hub#64

Would you please give me some suggestions? (Not familiar with tf very well)