Question: Why is WebQSP test dataset limited to 200 samples in run_explore.py?

[sunxin000]

Question about WebQSP Test Dataset Size

Hi @LHRLAB,

I'm studying your KBQA-o1 implementation and noticed that the WebQSP test dataset is limited to only 200 samples in run_explore.py. I have a few questions about this design choice:

Current Implementation

In run_explore.py line 456:

if not ex:
    kbqa_data = kbqa_data[:200]  # Test mode limited to 200 samples
else:
    random.shuffle(kbqa_data)    # Explore mode uses all data

Questions

1. Performance impact: How does this 200-sample subset compare to the full WebQSP test set (which typically has ~2,032 questions)? Are there any performance differences you've observed?

2. Reproducibility: For fair comparison with other KBQA methods that use the full test set, should researchers:

   • Use the same 200-sample subset for comparison?
   • Or modify the code to use the full test set?

[LHRLAB]

Hi, thanks for your question!

The 200-sample limit in run_explore.py is mainly for quick testing — it’s randomly sampled, so performance trends are generally representative of the full WebQSP test set.

[sunxin000]

So we should modify the code to use the full test set to reproduce the results in the paper?

[LHRLAB]

Both work, and we also recommend sampling 200 cases and averaging over multiple runs — it’s efficient and gives stable results.

[sunxin000]

Got it, Thank you very much for your prompt response!!!!

[LHRLAB]

Welcome.