Will the benchmark data be open source?

[tdlhl]

Dear authors,

Thanks for your great work! I wonder if the five evaluation benchmarks of TxAgent (DrugPC, BrandPC, GenericPC, DescriptionPC, and TreatmentPC) will be released?

[Jack-ZC8]

I also have the same question, thanks!

[gasvn]

Hi, thank you for your interest. We’re hosting a NeurIPS competition this year, which is why we’re not releasing the benchmark yet. The competition will launch in the next few weeks—stay tuned!

[manglu097]

Will you make the benchmark public after the challenge? If so, when will it be released?

[manglu097]

Hello, how did you build these five benchmarks?

[gasvn]

Hi, Thank you for your interest in the competition! We’ve just made the starter kit and Haggle site public https://curebench.ai/. Feel free to reach out if you have any further questions.

Happy competing!

[June01]

Hi @gasvn,

I'm currently using txagent for the CURE-bench competition, and my test set result is around 49% on the Kaggle leaderboard. Even the top participant only achieves ~70%, which is significantly lower than the 93.8% reported in the paper.

Could you clarify whether the test set used in the competition is the same as the one in the CURE-bench paper? If so, why is there such a large performance gap between the paper's results and the competition submissions (49%-70% vs. 93.8%)?

Thanks for your insights!

[gasvn]

Hi, thank you for your interest in TxAgent and Cure‑bench! The evaluation set in the TxAgent paper is different from evaluation set in the CUREBench competition.

[June01]

Now that the NIPS competition ended a month ago, will you open-source the benchmark data soon? @gasvn

[whistle-hikhi]

Hi @gasvn,
NIPS competition ended and NeurIPS, when the benchmark dataset will be released?