K value selection, potential test set leakage?

[javierrodenas]

I've been trying to reproduce the paper results using the provided code and checkpoints. I noticed a discrepancy between the k value produced during training and the k value stored in the official Meta-tuning checkpoints.

1. Training produces k = 0.0

When running train.py with the default parameters on miniImageNet 1-shot, the optimal k value found during validation is consistently around 0.0 throughout all 100 epochs:

  epoch 84, train 1.1611|98.90%, val 1: 0.8414, 0.8414 (k = 0.0)| MAX: 0.8414
  epoch 97, train 1.1487|98.94%, val 1: 0.8360, 0.8360 (k = 0.0)| MAX: 0.8414

2. Official checkpoint has k = 0.78

  checkpoint = torch.load('./checkpoint/Meta-tuning/miniImageNet-1-shot.pth')
  print(checkpoint['k'])  # Output: 0.78

3. k=0.78 is optimal on TEST set, not validation set

I tested different k values on the novel (test) set:

k	Test Accuracy (novel)
0.00	79.56 ± 0.75%
0.20	80.96 ± 0.70%
0.40	82.30 ± 0.66%
0.60	83.15 ± 0.63%
0.78	83.68 ± 0.62%
0.80	83.67 ± 0.63%
1.00	82.73 ± 0.66%

Question

It appears that k=0.78 was selected by evaluating on the test set rather than being determined from the training/validation process. Could you please clarify:

1. How was k=0.78 determined for the official checkpoints?
2. Is this the intended behavior, or should the code be modified to produce similar k values during training?

This is important for reproducibility and fair comparison with other methods.

Thank you for your work on this project!

[peacelwh]

Thanks for raising this. If the validation code actually uses only the original support samples (or the k-weighted branch is not actually applied), the k-sweep can become meaningless and repeatedly select k=0. This is consistent with a subtle bug.

As shown in the table, performance typically peaks at intermediate values of k and is generally insensitive to variations around that region on both the validation and test sets, rather than peaking at the extremes.

value of u	miniImageNet	tieredImageNet	CIFAR-FS	FC100
0.0	86.92	89.68	90.43	65.07
0.1	87.36	90.22	90.68	65.82
0.2	87.69	90.67	90.93	66.41
0.3	88.03	91.06	91.14	66.96
0.4	88.26	91.37	91.31	67.39
0.5	88.38	91.57	91.41	67.57
0.6	88.38	91.68	91.45	67.68
0.7	88.32	91.71	91.40	67.49
0.8	88.16	91.58	91.25	67.06
0.9	87.91	91.34	91.00	66.40
1.0	87.55	90.96	90.63	65.54