Unable to reproduce the results from paper using default config

[dayuyang1999]

Hi,

I was trying to reproduce the result from this paper:

| Xiaolei Wang*, Kun Zhou*, Ji-Rong Wen, Wayne Xin Zhao. Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning. KDD 2022.

In this paper, the performance of KGSF and KBRD in INSPIRED dataset is like:

However, when I use default config setting to run KGSF and KBRD on INSPIRED dataset:

python run_crslab.py --config CRSLab/config/crs/kbrd/inspired.yaml

and

python run_crslab.py --config CRSLab/config/crs/kgsf/inspired.yaml

I got much worse results on test set:
(first for KBRD, latter for KGSF)

---

I am not sure why I fail to reproduce. I noticed that the whole training process is quite short which only last 1 epoch. I think it is reasonable because INSPIRED dataset is small. Stop early should prevent overfitting.

Any clue will be helpful, thanks!

[wxl1999]

So sorry, dayu, these two configs are not well tuned. However, we do not have enough time and effort to do this valuable work. You can tune these hyperparameters yourself, and welcome your PR for this!