Hi,is there any guarantee not only a few of embeddings being selected?

[jiqizaisikao]

Hi,i have run your source code,it seems that it works well,but i have one question,for the weights of the embeddings are inited randomly,and in tranning it always select the nearest neighbors,so is there any guarantee that not a few of embeddings always being selected?

[hiwonjoon]

It's a good question. I do not have an answer though since I am not the authors of the paper.
In my humble opinion, describing in the conceptual level, embedding is also be trained to maximize interpretability of training examples, so it could become the most informative supports for a dataset.

[leao1995]

When I run the cifar example, it always selects a fixed subset of the embedding, so the issue indeed exists.

[bckim92]

Kaiser et al. addressed this issue in their paper (https://arxiv.org/abs/1803.03382) and they called it index collapse.