Scoring and Data for ScholarQA-CS2 Benchmark

[pangxianghe]

Hello,

I am currently working on scoring the ScholarQA-CS2 benchmark and have a few questions regarding the process:

Scoring Query-Answer Pairs: Could you please clarify the correct function to use when scoring existing query-answer pairs? Which format should I organize my answers?

Discrepancy Between Paper and Code Rubrics: I noticed a discrepancy between the rubric example in the paper (Section H.3.2) and the code found at this GitHub link. The rubric in the paper mentions the query "how the AI hallucination is linked to the AI bias," but this query is not present in the data.json file. Additionally, the rubric in the paper includes a key called 'ingredients,' which is also absent in the data.json. It appears that the data.json is based on the original ScholarQA-Bench, and I am wondering where the data for ScholarQA-CS2 can be found.

Thank you in advance for your assistance!

[amanpreet692]

Hello, thanks for using our benchmark! Please find the answers to your queries below:

• Re. the discrepancy in the data.json file, you are correct that the rubrics mentioned in the paper are different. That is because data.json is an older version that is no longer used. You can find the latest version of the rubrics (consistent with the paper) in our huggingface repo (You'll need to be logged in to browse the artifacts)
• For evaluating on a single task please refer to Running a Single Task
• For scoring existing query-answer pairs, you can use one of agent-baseline scripts that read can load answers from a static file:
  • cached_solver.py
  • memorized_solver.py

Some details about using these scripts:

• ScholarQA-CS2 requires a response to be in a json format as below:

     {
  "sections":
  [
      {
          "text": "text of section 1",
          "citations":
          {
              "id": "cite 1 of sec 1",
              "snippets":
              [
                  "evidence 1",
                  "evidence 2"
              ]
          }
      },
      {
          "text": "text of section 2",
          "citations":
          {
              "id": "cite 1 of sec 2",
              "snippets":
              [
                  "List of evidence"
              ]
          }
      }
  ]

}

• cached_solver.py expects the responses to be in this format, and the response file to be a json containing a list of question and response key pairs and can be run as below:

uv run inspect eval --solver agent_baselines/solvers/sqa/debug/cached_solver.py -S path=<json file path> -S model=<optional param for recording model usage> -T split=test -T with_search_tools=False --display=plain

• memorized_solver.py doesn't enforce the format requirement and uses the gemini-flash-2.5 model to convert the raw response into the required json format above. The file should be a json containing a list of question and answer key pairs
  and can be run as below:

uv run inspect eval --solver agent_baselines/solvers/sqa/general_memorized/memorized_solver.py@formatted_solver -S sys_name_or_path=<json file path> -T split=test -T with_search_tools=False --display=plain

For running both of these, make sure you have GOOGLE_API_KEY (For running Gemini-flash as the scorer and optionally formatter) and HF_TOKEN (For accessing the rubrics)