Skip to content

Commit f6ec54f

Browse files
justaddcoffeejustaddcoffee
committed
Commit test/eval cases
1 parent d17ad94 commit f6ec54f

1 file changed

Lines changed: 348 additions & 0 deletions

File tree

‎project/test_cases.yaml‎

Lines changed: 348 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,348 @@
1+
# SHARED TEST CASES - Single Source of Truth
2+
# Do not duplicate these - use generate_configs.py to create agent-specific configs
3+
# Updated 2025-11-25: Added custom GEval metrics per metacoder PR #37
4+
5+
description: Shared test case definitions for all agent/model evaluations
6+
note: This is the single source of truth for test cases. All generated configs use
7+
these exact cases.
8+
9+
test_cases:
10+
# TEXT EXTRACTION CASES
11+
- name: PMID_28027860_Full_Text
12+
group: Text extraction
13+
metrics:
14+
- name: CorrectnessMetric
15+
evaluation_steps:
16+
- "Check whether the actual output contains the exact text from expected output"
17+
- "The text must be verbatim, not paraphrased"
18+
- "Minor punctuation or whitespace differences are acceptable"
19+
input: What is the first sentence of section 2 in PMID:28027860?
20+
expected_output: Even though many of NFLE's core features have been clarified in
21+
the last two decades, some critical issues remain controversial.
22+
threshold: 0.9
23+
24+
- name: PMID_28027860_References
25+
group: Text extraction
26+
metrics:
27+
- name: CorrectnessMetric
28+
evaluation_steps:
29+
- "Check whether the actual output contains the exact text from expected output"
30+
- "The text must be verbatim, not paraphrased"
31+
- "Minor punctuation or whitespace differences are acceptable"
32+
input: Find the citation in the paper PMID:28027860 for the claim 'The clinical
33+
spectrum of NFLS comprises distinct paroxysmal sleep-related attacks of variable
34+
duration and complexity'. What is the title of cited paper?
35+
expected_output: The concept of paroxysmal nocturnal dystonia
36+
threshold: 0.9
37+
38+
- name: 10_1016_j_seizure_2016_11_023_Full_Text
39+
group: Text extraction
40+
metrics:
41+
- name: CorrectnessMetric
42+
evaluation_steps:
43+
- "Check whether the actual output contains the exact text from expected output"
44+
- "The text must be verbatim, not paraphrased"
45+
- "Minor punctuation or whitespace differences are acceptable"
46+
input: What is the first sentence of the last paragraph from 10.1016/j.seizure.2016.11.023,
47+
before the Conflict of Interest statement?
48+
expected_output: SHE diagnosis is primarily based on the clinical history.
49+
threshold: 0.9
50+
51+
- name: 10_1371_journal_pone_0000217_Full_Text_A
52+
group: Text extraction
53+
metrics:
54+
- name: CorrectnessMetric
55+
evaluation_steps:
56+
- "Check whether the actual output contains the exact text from expected output"
57+
- "The text must be verbatim, not paraphrased"
58+
- "Minor punctuation or whitespace differences are acceptable"
59+
input: Convert the PDF for DOI 10.1371/journal.pone.0000217 to text and tell me
60+
the first sentence of the last paragraph of the Conclusions section.
61+
expected_output: However, phenotypic complexity remains an inherently abstract metric.
62+
threshold: 0.9
63+
64+
- name: 10_1038_nature12373_Full_Text
65+
group: Text extraction
66+
metrics:
67+
- name: CorrectnessMetric
68+
evaluation_steps:
69+
- "Check whether the actual output contains the exact text from expected output"
70+
- "The text must be verbatim, not paraphrased"
71+
- "Minor punctuation or whitespace differences are acceptable"
72+
input: Get the full text content for DOI 10.1038/nature12373 and tell me the first
73+
sentence of the Methods Summary.
74+
expected_output: Nanodiamond measurement pulse sequence.
75+
threshold: 0.9
76+
77+
- name: PMC2824148_Full_Text
78+
group: Text extraction
79+
metrics:
80+
- name: CorrectnessMetric
81+
evaluation_steps:
82+
- "Check whether the actual output contains the exact text from expected output"
83+
- "The text must be verbatim, not paraphrased"
84+
- "Minor punctuation or whitespace differences are acceptable"
85+
input: What is the last sentence of PMC2824148 before the Acknowledgments?
86+
expected_output: As far as the future work is concerned, it would be interesting
87+
to study the design of efficient spaced seeds for protein sequence search (see
88+
[6]), as well as to combine spaced seeds with other techniques such as seed families
89+
[17, 20, 16] or the group hit criterion [19].
90+
threshold: 0.9
91+
92+
- name: 10_1038_s41564_022_01094_z_Full_Text_A
93+
group: Text extraction
94+
metrics:
95+
- name: CorrectnessMetric
96+
evaluation_steps:
97+
- "Check how many section headers from the expected output are present in the actual output"
98+
- "Give partial credit proportional to the percentage of correct headers"
99+
- "Headers should be in the correct order when present"
100+
- "Minor formatting differences (spacing, capitalization) are acceptable"
101+
input: What are the main section headers in PMC8975739?
102+
expected_output: Abstract,Main,Results,Discussion,Methods,Supplementary information,Acknowledgements,Author
103+
contributions,Peer review,Data availability,Code availability,Competing interests,Footnotes,References,Associated
104+
Data
105+
threshold: 0.9
106+
107+
- name: 10_1038_s41564_022_01094_z_Full_Text_B
108+
group: Text extraction
109+
metrics:
110+
- name: CorrectnessMetric
111+
evaluation_steps:
112+
- "Check whether the actual output contains the exact text from expected output"
113+
- "The text must be verbatim, not paraphrased"
114+
- "Minor punctuation or whitespace differences are acceptable"
115+
input: What is the longest section in PMID:35365791?
116+
expected_output: Methods
117+
threshold: 0.9
118+
119+
- name: PMID_32198168_nan
120+
group: Text extraction
121+
metrics:
122+
- name: CorrectnessMetric
123+
evaluation_steps:
124+
- "Check whether the actual output contains the exact text from expected output"
125+
- "The text must be verbatim, not paraphrased"
126+
- "Minor punctuation or whitespace differences are acceptable"
127+
input: What does the data management section state in PMID:32198168?
128+
expected_output: Clinical data were collected into a Microsoft Excel 2010 database
129+
that was password protected.
130+
threshold: 0.9
131+
132+
- name: PMC117972_Full_Text
133+
group: Text extraction
134+
metrics:
135+
- name: CorrectnessMetric
136+
evaluation_steps:
137+
- "Check whether the actual output contains the exact text from expected output"
138+
- "The text must be verbatim, not paraphrased"
139+
- "Minor punctuation or whitespace differences are acceptable"
140+
input: What is the first sentence of the Microarray Analysis section in PMC117972?
141+
expected_output: A global representation of the changes in expression of all the
142+
expressed sequence tags (ESTs) on the microarray is depicted in Figure 1.
143+
threshold: 0.9
144+
145+
- name: PMC4831113_Full_Text
146+
group: Text extraction
147+
metrics:
148+
- name: CorrectnessMetric
149+
evaluation_steps:
150+
- "Check whether the actual output contains the exact text from expected output"
151+
- "The text must be verbatim, not paraphrased"
152+
- "Minor punctuation or whitespace differences are acceptable"
153+
- "Do not penalize if the output also mentions the article has been retracted"
154+
input: Show the Conclusions section of PMC4831113.
155+
expected_output: On the basis of phenotypic, phylogenetic and genomic analyses,
156+
we formally propose the creation of Haemophilus massiliensis sp. nov. that contains
157+
strain FF7T (CSUR P859 = DSM 28247) which is the type strain The strain was isolated
158+
from a peritoneal fluid specimen from a 44-year-old Senegalese woman admitted
159+
to Hôpital Principal in Dakar, Senegal.
160+
threshold: 0.9
161+
162+
# METADATA CASES
163+
- name: PMID_28027860_Title
164+
group: Metadata
165+
metrics:
166+
- name: CorrectnessMetric
167+
evaluation_steps:
168+
- "Check if the title exactly matches the expected output"
169+
- "Minor punctuation or formatting differences are acceptable"
170+
- "The core title text must be identical"
171+
input: What is the title of PMID:28027860?
172+
expected_output: 'From nocturnal frontal lobe epilepsy to Sleep-Related Hypermotor
173+
Epilepsy: A 35-year diagnostic challenge'
174+
threshold: 0.9
175+
176+
- name: PMC2824148_Metadata
177+
group: Metadata
178+
metrics:
179+
- name: CorrectnessMetric
180+
evaluation_steps:
181+
- "Check if the DOI exactly matches the expected output"
182+
- "Accept the DOI with or without 'https://doi.org/' prefix"
183+
- "Minor punctuation or formatting differences are acceptable"
184+
- "The core DOI must be identical"
185+
input: What is the DOI for the final published version of PMC2824148?
186+
expected_output: 10.1142/S0219720006001977
187+
threshold: 0.9
188+
189+
- name: PMC5152751_Metadata
190+
group: Metadata
191+
metrics:
192+
- name: CorrectnessMetric
193+
evaluation_steps:
194+
- "Check if the publisher name matches the expected output"
195+
- "Accept 'Nature' or 'Nature Publishing Group' as correct"
196+
- "Minor formatting differences are acceptable"
197+
input: Which publisher published PMC5152751?
198+
expected_output: Nature
199+
threshold: 0.9
200+
201+
- name: PMC3815380_PDF
202+
group: Metadata
203+
metrics:
204+
- name: CorrectnessMetric
205+
evaluation_steps:
206+
- "Check if the editor name matches the expected output"
207+
- "Minor formatting differences are acceptable"
208+
- "The core name must be identical"
209+
input: Who was the Reviews and Special Issue editor for PMC3815380?
210+
expected_output: Juan L. Ramos
211+
threshold: 0.9
212+
213+
# TABLE / FIGURE / FIGURE LEGEND EXTRACTION CASES
214+
- name: NBK1256
215+
group: Table / Figure / Figure Legend extraction
216+
metrics:
217+
- name: CorrectnessMetric
218+
evaluation_steps:
219+
- "Check whether the actual output contains the table row data from expected output"
220+
- "The data values must match (Cone-rod dystrophy, 70%, Loss of central vision & color vision, Abnormal fundoscopic exam)"
221+
- "Allow minor formatting differences in how the row is presented (spacing, delimiters)"
222+
- "All data fields from the row must be present in the correct order"
223+
input: What is the last row of Table 2 in NBK1256?
224+
expected_output: 'Cone-rod dystrophy,70%,Loss of central vision & color vision
225+
226+
Abnormal fundoscopic exam'
227+
threshold: 0.9
228+
229+
- name: PMID_40307501_Figure_Legend
230+
group: Table / Figure / Figure Legend extraction
231+
metrics:
232+
- name: CorrectnessMetric
233+
evaluation_steps:
234+
- "Check whether the actual output contains the exact text from expected output"
235+
- "The text must be verbatim, not paraphrased"
236+
- "Minor punctuation or whitespace differences are acceptable"
237+
input: What is the first sentence of Figure 1 legend from PMID 40307501?
238+
expected_output: Proposed system for bio-accelerated weathering of ultramafic materials
239+
for carbon mineralization.
240+
threshold: 0.9
241+
242+
- name: PMID_12089011_Full_Text
243+
group: Table / Figure / Figure Legend extraction
244+
metrics:
245+
- name: CorrectnessMetric
246+
evaluation_steps:
247+
- "Check whether the actual output contains the exact text from expected output"
248+
- "The text must be verbatim, not paraphrased"
249+
- "Minor punctuation or whitespace differences are acceptable"
250+
input: What is the first footnote of Table 3 in PMID:12089011?
251+
expected_output: a. Grouping at 89% similarity
252+
threshold: 0.9
253+
254+
- name: PMC3368421_nan
255+
group: Table / Figure / Figure Legend extraction
256+
metrics:
257+
- name: CorrectnessMetric
258+
evaluation_steps:
259+
- "Check whether the actual output contains the exact text from expected output"
260+
- "The text must be verbatim, not paraphrased"
261+
- "Minor punctuation or whitespace differences are acceptable"
262+
input: What is the MIGS-22 property from Table 1?
263+
expected_output: Oxygen requirement aerobe
264+
threshold: 0.9
265+
266+
# SUPPLEMENTARY MATERIAL CASES
267+
- name: 10_1371_journal_pone_0000217_Full_Text_B
268+
group: Supplementary material
269+
metrics:
270+
- name: CorrectnessMetric
271+
evaluation_steps:
272+
- "Check if the actual output contains the number from expected output"
273+
- "Accept the number with or without additional context (e.g., '3' or 'There are 3 appendices')"
274+
- "The numeric value must be correct"
275+
input: How many appendices are in 10.1371/journal.pone.0000217?
276+
expected_output: '3'
277+
threshold: 0.9
278+
279+
- name: 10_1038_nature12373_Supplementary_Material_A
280+
group: Supplementary material
281+
metrics:
282+
- name: CorrectnessMetric
283+
evaluation_steps:
284+
- "Check if the actual output contains the filename from expected output"
285+
- "Accept the filename with or without additional context"
286+
- "The core filename must be present and identical"
287+
- "Minor formatting differences are acceptable"
288+
input: What supplementary files are available for PMC4221854?
289+
expected_output: NIHMS636072-supplement-supplemental_info.pdf
290+
threshold: 0.9
291+
292+
- name: 10_1038_nature12373_Supplementary_Material_B
293+
group: Supplementary material
294+
metrics:
295+
- name: CorrectnessMetric
296+
evaluation_steps:
297+
- "Check whether the actual output contains the exact text from expected output"
298+
- "The text must be verbatim, not paraphrased"
299+
- "Minor punctuation or whitespace differences are acceptable"
300+
input: What is the figure legend for Figure S3 in 10.1038/nature12373?
301+
expected_output: Fluorescence spectrum under green excitation of gold nanoparticles
302+
(blue data) and nanodiamonds (red data). The shaded region indicates the bandpass
303+
filter used for detection of gold. The arrows mark the respective y-axis scaling
304+
for each curve.
305+
threshold: 0.9
306+
307+
# SUMMARIZATION CASES
308+
- name: PMID_28027860_Table
309+
group: Summarization
310+
metrics:
311+
- name: CorrectnessMetric
312+
evaluation_steps:
313+
- "Check if all three levels of certainty are present in the actual output"
314+
- "The three levels are: witnessed, video-documented, Video-EEG documented"
315+
- "Allow minor formatting differences (capitalization, spacing, delimiters)"
316+
- "All three items must be present"
317+
input: According to PMID:28027860, what are the three levels of certainty for SHE?
318+
expected_output: witnessed,video-documented,Video-EEG documented
319+
threshold: 0.9
320+
321+
- name: PMC8086273_Retraction
322+
group: Summarization
323+
metrics:
324+
- name: CorrectnessMetric
325+
evaluation_steps:
326+
- "Check if the output correctly states the paper's finding (no oxidative stresses)"
327+
- "Check if the output mentions the paper is retracted"
328+
- "Check if the output indicates the results should not be trusted due to retraction"
329+
- "Allow paraphrasing as long as the key points are present"
330+
input: Based on PMID 33926573 do microbes from alkaline sulphidic tailings show
331+
oxidative stresses? Is this finding from this paper reliable?
332+
expected_output: The paper says no, microbes from alkaline sulphidic tailings do
333+
not show oxidative stresses, but it is retracted so the results should not be
334+
trusted.
335+
threshold: 0.9
336+
337+
# PUBLICATION STATUS CASES
338+
- name: PMC4831113_Retraction
339+
group: Publication status
340+
metrics:
341+
- name: CorrectnessMetric
342+
evaluation_steps:
343+
- "Check if the output indicates the paper is retracted"
344+
- "Accept 'Yes' or any affirmative statement that the paper is retracted"
345+
- "The answer must clearly indicate retraction status is positive"
346+
input: Is PMC4831113 retracted?
347+
expected_output: 'Yes'
348+
threshold: 0.9

0 commit comments

Comments
 (0)