Missing cite_context settings

[wwydmanski]

Hi!

First - thanks for creating Scirepeval, it's extremely useful for benchmarking new models.
I have one problem with running the benchmark, though. I think that config for cite_context is missing from scirepeval_tasks.jsonl. What are the proper settings with which I can run the task?

Best wishes,
Witold

[wwydmanski]

I also have an additional question - what's included in the S2AND mini all features task and how can I get the final evaluation metric? Do I average all of the B^3 F1 scores that I get from running the S2AND benchmark (as described in https://github.com/allenai/scirepeval/blob/main/BENCHMARKING.md#s2and), or is there another procedure?

[amanpreet692]

Hello,
Thanks for using SciRepEval!
cite_context is an alias I used in the training config file here but if you look at dataset.name it's high_influence_cite and that's the huggingface name/config of the dataset. It's present in line 17 of scirepeval_tasks.jsonl.

For s2and mini eval, once you download the data and generate the embeddings with the provided script, you can run the s2and eval script with your data directory and from the table you get after the run is complete, you want average B^3 F1 of all the values in the union row.

[wwydmanski]

Ahh, I understand it now. Thank you!