sahil2801/replit-code-instruct-glaive result

[regularfry]

Am I reading this right? Right now, https://huggingface.co/sahil2801/replit-code-instruct-glaive, a 3B model, is claiming a pass@1 score of 63.5%, significantly better than the next best model, which is a 15B model.

Is that... real? Or are we seeing an artefact of how the pass@1 test works?

Apologies if this is a stupid question, but that result just looks too good to be true.

[matthiasgeihs]

Have asked myself the same question and ran some validation experiments.
Seems like test data might have leaked into training data.
See hf discussion for more details.

[regularfry]

Hm. That's a pain. These things happen.