Skip to content

Commit a1b08e6

Browse files
author
Charles Parker
committed
Adding draft version of YAML file with all test cases. Only one MCP added so far.
1 parent a240e87 commit a1b08e6

1 file changed

Lines changed: 229 additions & 0 deletions

File tree

‎project/literature_mcp_eval_config.yaml‎

Lines changed: 229 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,229 @@
1+
name: pubmed tools evals
2+
description: |
3+
Evaluations for multiple pubmed MCPs
4+
5+
coders:
6+
claude: {}
7+
8+
models:
9+
gpt-4o:
10+
provider: openai
11+
name: claude-sonnet-20250219
12+
13+
servers:
14+
pubmed:
15+
name: pubmed
16+
command: uvx
17+
args: [mcp-simple-pubmed]
18+
env:
19+
PUBMED_EMAIL: cjmungall@lbl.gov
20+
21+
cases:
22+
- name: PMID:28027860_Full_Text
23+
metrics:
24+
- CorrectnessMetric
25+
input: 'What is the first sentence of section 2 in PMID: 28027860?'
26+
expected_output: "Even though many of NFLE\u2019s core features have been clarified\
27+
\ in\nthe last two decades, some critical issues remain controversial."
28+
threshold: 0.9
29+
- name: PMID:28027860_Title
30+
metrics:
31+
- CorrectnessMetric
32+
input: What is the title of PMID:28027860
33+
expected_output: 'From nocturnal frontal lobe epilepsy to Sleep-Related Hypermotor
34+
Epilepsy: A 35-year diagnostic challenge'
35+
threshold: 0.9
36+
- name: PMID:28027860_Table
37+
metrics:
38+
- CorrectnessMetric
39+
input: According to PMID:28027860, what are the three levels of certainty for SHE
40+
expected_output: witnessed; video-documented; Video-EEG documented
41+
threshold: 0.9
42+
- name: PMID:28027860_References
43+
metrics:
44+
- CorrectnessMetric
45+
input: Find the citation in the paper PMID:28027860 for the claim "The clinical
46+
spectrum of NFLS comprises distinct paroxysmal sleep-related attacks of variable
47+
duration and complexity". What is the title of cited paper?
48+
expected_output: The concept of paroxysmal nocturnal dystonia
49+
threshold: 0.9
50+
- name: 10.1016/j.seizure.2016.11.023_Full_Text
51+
metrics:
52+
- CorrectnessMetric
53+
input: What is the last paragraph of the paper, before the conflict of interest
54+
expected_output: 'SHE diagnosis is primarily based on the clinical history. The
55+
absence of clear interictal and ictal EEG correlates does not exclude the diagnosis.
56+
Three different levels of diagnostic certainty have been identified: (1) Witnessed
57+
(possible), based on the observation by a witness of the core clinical features,
58+
but without other sources of evidence. (2) Video-documented (clinical), which
59+
involves a high quality video recording of at least one (but preferably two) stereotyped
60+
episode. (3) VEEG-documented (confirmed), which requires the video-polygraphic
61+
recording of at least one but preferably two stereotyped events with documented
62+
ictal discharge or interictal epileptiform abnormalities.'
63+
threshold: 0.9
64+
- name: https://www.ncbi.nlm.nih.gov/books/NBK1256/_HTML
65+
metrics:
66+
- CorrectnessMetric
67+
input: What are the last two rows of table 2
68+
expected_output: "Behavior disorder/\nPsychosis 10% \nAltered mentation\n\
69+
Impaired reality testing\nCone-rod\ndystrophy 70% \nLoss of central\
70+
\ vision & color vision\nAbnormal fundoscopic exam"
71+
threshold: 0.9
72+
- name: PMID:40307501_Figure_Legend
73+
metrics:
74+
- CorrectnessMetric
75+
input: What is the first sentence of figure 1 legend
76+
expected_output: Proposed system for bio-accelerated weathering of ultramafic materials
77+
for carbon mineralization
78+
threshold: 0.9
79+
- name: 10.1371/journal.pone.0000217_Full_Text
80+
metrics:
81+
- CorrectnessMetric
82+
input: Convert the PDF for DOI 10.1371/journal.pone.0000217 from Europe PMC to Markdown
83+
format using streaming processing and display the last paragraph of the Discussion
84+
section.
85+
expected_output: However, phenotypic complexity remains an inherently abstract metric.
86+
It cannot aid in identifying the specific characteristics contributing to the
87+
complexity of an organism. Instead, it addresses the complexity with which natural
88+
selection views an organism, and the complexity with which an organism is capable
89+
of generating novel phenotypic variation. For this reason, testing how phenotypic
90+
complexity compares to more traditional metrics of complexity (for example, the
91+
numbers of genes, protein interactions, or cellular pathways) may provide significant
92+
insight into biological systems. Finally, phenotypic complexity (and the resulting
93+
equilibrium drift load) affords a unique opportunity to contrast the action of
94+
natural selection between different organisms or different environments in a very
95+
general and unconstrained manner.
96+
threshold: 0.9
97+
- name: 10.1371/journal.pone.0000217_Full_Text
98+
metrics:
99+
- CorrectnessMetric
100+
input: How many appendices are in 10.1371/journal.pone.0000217?
101+
expected_output: '3'
102+
threshold: 0.9
103+
- name: 10.1038/nature12373_Full_Text
104+
metrics:
105+
- CorrectnessMetric
106+
input: Get the full text content for DOI 10.1038/nature12373 in Markdown format
107+
and show me the last paragraph of the Methods section.
108+
expected_output: Nanodiamonds and Au NPs were introduced into WS1 cells via silicon
109+
nanowire-mediated delivery15. Silicon nanowires were treated with 3-amino-propyltrimethoxysilane
110+
to present NH2 functionality on the surface, and nanodiamonds / Au NPs were subsequently
111+
attached via electrostatic binding. Afterwards, human embryonic fibroblast WS1
112+
cells were plated on the silicon nanowire substrates and cultured overnight. The
113+
cells were removed by trypsin treatment and re-plated on a glass slide with lithographically
114+
defined strip lines for ESR measurements. The samples were stained with calcein-AM
115+
and ethidium homodimer-1 for the live/dead assay.
116+
threshold: 0.9
117+
- name: 10.1038/nature12373_Supplementary_Material
118+
metrics:
119+
- CorrectnessMetric
120+
input: What supplementary files are available for PMC4221854?
121+
expected_output: NIHMS636072-supplement-supplemental_info.pdf
122+
threshold: 0.9
123+
- name: 10.1038/nature12373_Supplementary_Material
124+
metrics:
125+
- CorrectnessMetric
126+
input: What is the figure legend for sigure S3
127+
expected_output: 'Schematic Setup. Depicts the optical (excitation: green, collection:
128+
red) path and microwave
129+
130+
supply (blue). Each of the two excitation (collection) paths can be independently
131+
controlled using a combination of a galvo-mirror and a piezo. By controlling the
132+
MW switches, it is possible to interchange between
133+
134+
four different MW frequencies on sub-microsecond time scale'
135+
threshold: 0.9
136+
- name: PMC2824148_Full_Text
137+
metrics:
138+
- CorrectnessMetric
139+
input: What is the last sentence of PMC2824148 before the Acknowledgements?
140+
expected_output: As far as the future work is concerned, it would be interesting
141+
to study the design of efficient spaced seeds for protein sequence search (see
142+
[6]), as well as to combine spaced seeds with other techniques such as seed families
143+
[17, 20, 16] or the group hit criterion [19].
144+
threshold: 0.9
145+
- name: PMC2824148_Metadata
146+
metrics:
147+
- CorrectnessMetric
148+
input: 'What is the DOI for the final published version of '
149+
expected_output: 10.1142/S0219720006001977
150+
threshold: 0.9
151+
- name: 10.1038/s41564-022-01094-z_Full_Text
152+
metrics:
153+
- CorrectnessMetric
154+
input: What are the main section headers in PMC8975739?
155+
expected_output: 'Abstract
156+
157+
Main
158+
159+
Results
160+
161+
Discussion
162+
163+
Methods
164+
165+
Supplementary information
166+
167+
Acknowledgements
168+
169+
Author contributions
170+
171+
Peer review
172+
173+
Data availability
174+
175+
Code availability
176+
177+
Competing interests
178+
179+
Footnotes
180+
181+
References
182+
183+
Associated Data'
184+
threshold: 0.9
185+
- name: 10.1038/s41564-022-01094-z_Full_Text
186+
metrics:
187+
- CorrectnessMetric
188+
input: 'What is the longest section in PMID: 35365791?'
189+
expected_output: Methods
190+
threshold: 0.9
191+
- name: PMID:_12089011_Full_Text
192+
metrics:
193+
- CorrectnessMetric
194+
input: 'What is the first footnote of Table 3 in PMID: 12089011?'
195+
expected_output: a Grouping at 89% similarity
196+
threshold: 0.9
197+
- name: PMC5152751_Metadata
198+
metrics:
199+
- CorrectnessMetric
200+
input: Which publisher published PMC5152751?
201+
expected_output: Nature
202+
threshold: 0.9
203+
- name: PMC3368421_nan
204+
metrics:
205+
- CorrectnessMetric
206+
input: What is the MIGS-22 property from Table 1?
207+
expected_output: Oxygen requirement aerobe
208+
threshold: 0.9
209+
- name: PMID:_32198168_nan
210+
metrics:
211+
- CorrectnessMetric
212+
input: 'What does the data management section state in PMID: 32198168?'
213+
expected_output: Clinical data were collected into a Microsoft Excel 2010 database
214+
that was password protected.
215+
threshold: 0.9
216+
- name: PMC3815380_PDF
217+
metrics:
218+
- CorrectnessMetric
219+
input: Who was the REVIEWS AND SPECIAL ISSUE EDITOR for PMC3815380?
220+
expected_output: "Juan L. Ramos, Estaci\xF3n Experimental\ndel Zaid\xEDn, Granada,\
221+
\ Spain"
222+
threshold: 0.9
223+
- name: PMC117972_Full_Text
224+
metrics:
225+
- CorrectnessMetric
226+
input: What is the first sentence of the Microarray Analysis section in PMC117972?
227+
expected_output: 'A global representation of the changes in expression of all the
228+
expressed sequence tags (ESTs) on the microarray is depicted in Figure 1. '
229+
threshold: 0.9

0 commit comments

Comments
 (0)