Unable to reproduce reported ReposVul accuracy with released inference script

[zz203]

Hi, thank you for releasing the code and models for LLMxCPG.

I am trying to reproduce the ReposVul result reported in the paper. In Table 7, the paper reports the following result on ReposVul:

• Accuracy: 0.634
• Precision: 0.542
• Recall: 0.700
• F1-score: 0.610

I followed inference/README.md and ran inference with the released LLMxCPG-D model:

CUDA_VISIBLE_DEVICES=6 python detect_inference.py reposvul \
  --base-model /path/to/Qwen/QwQ-32B-Preview \
  --model-path /path/to/QCRI/LLMxCPG-D

However, the generated metrics file inference/results/QwQ-32B-Preview_reposvul_test_metrics.tsv shows:

threshold    accuracy    precision    recall    f1      total_samples
0.307        0.5000      0.5000       1.0000    0.6667  120

The released data/reposvul_test.json contains 120 samples, balanced as 60 positive and 60 negative samples. With the default ReposVul threshold 0.307 in detect_inference.py, all samples appear to be classified as vulnerable, which leads to 0.5 accuracy and 1.0 recall.

I also tried searching for the best threshold on the saved predictions, but the best accuracy I could get was around 0.55, still lower than the 0.634 reported in the paper.

Could you please clarify the expected reproduction setup for the ReposVul result?

Specifically:

Is the released data/reposvul_test.json the same split used for the paper’s Table 7 result?
Is 0.307 the correct threshold for ReposVul?
Were the Table 7 results computed using a different threshold selection procedure, validation split, checkpoint, prompt, or inference script?
Are there any additional preprocessing or CPG slicing steps required before running detect_inference.py reposvul?
Thanks again for your work and for making the project available.