Summary
Hi - a notification, not a request; nothing is needed from you.
I ran a small pre-registered survey of papers publishing a quantitative figure
about their own verification component, asking only whether a third party can
recompute that figure from the release. Fourteen papers qualified; yours is one.
Determination for this repository at commit 2876530:
COMPUTABLE - and reproduced exactly. docs/adversarial-in-vivo-samples.csv
carries 1,600 rows, 800 legitimate and 800 adversarial, with per-sample ground
truth and a per-subject decision on each. Recomputing from it: upstream recall
0.000; both hardened subjects precision = recall = 1.000. That matches the
paper.
What made the check possible is specific and worth stating: the per-sample
ground truth and the per-subject decision share an identifier, so the two can be
joined by someone who was not in the room. Twelve of the fourteen releases do
not permit that join. Yours does.
Result, the protocol committed before the data existed, and every determination
with its reason:
https://github.com/kolesnikov-arch/replication-checks/tree/main/survey-verifier-claims
If any of that is factually wrong, say so and I will correct it and attribute
the correction to you. Replies are published in full and unedited. The window
runs to 19 August 2026; I do not follow up afterwards.
Problem to solve
NA
Proposed solution
NA
Alternatives considered
No response
Impact
NA
Evidence/examples
No response
Additional information
No response
Summary
Hi - a notification, not a request; nothing is needed from you.
I ran a small pre-registered survey of papers publishing a quantitative figure
about their own verification component, asking only whether a third party can
recompute that figure from the release. Fourteen papers qualified; yours is one.
Determination for this repository at commit 2876530:
COMPUTABLE - and reproduced exactly. docs/adversarial-in-vivo-samples.csv
carries 1,600 rows, 800 legitimate and 800 adversarial, with per-sample ground
truth and a per-subject decision on each. Recomputing from it: upstream recall
0.000; both hardened subjects precision = recall = 1.000. That matches the
paper.
What made the check possible is specific and worth stating: the per-sample
ground truth and the per-subject decision share an identifier, so the two can be
joined by someone who was not in the room. Twelve of the fourteen releases do
not permit that join. Yours does.
Result, the protocol committed before the data existed, and every determination
with its reason:
https://github.com/kolesnikov-arch/replication-checks/tree/main/survey-verifier-claims
If any of that is factually wrong, say so and I will correct it and attribute
the correction to you. Replies are published in full and unedited. The window
runs to 19 August 2026; I do not follow up afterwards.
Problem to solve
NA
Proposed solution
NA
Alternatives considered
No response
Impact
NA
Evidence/examples
No response
Additional information
No response