evaluation_protocol.md

Evaluation Protocol

Tasks

Task A: Binary Detection

Input: figure image and optional caption.

Output:

{
  "decision": "authentic | manipulated",
  "confidence": 0.0
}

Metrics:

• Accuracy
• Macro-F1
• False positive rate
• False negative rate

Task B: Manipulation Type Classification

Input: figure image, caption, and manipulation taxonomy choices.

Output:

{
  "decision": "manipulated",
  "manipulation_type": "legend_color_mismatch",
  "confidence": 0.0
}

Metrics:

• Macro-F1 across manipulation types
• Per-type precision, recall, and F1
• Confusion matrix by manipulation type

Task C: Evidence Explanation

Input: figure image and caption.

Output:

{
  "decision": "manipulated",
  "manipulation_type": "caption_figure_inconsistency",
  "evidence": "The caption claims Method A is best, but the tallest bar is Method C."
}

Metrics:

• Explanation accuracy: whether the explanation identifies the correct inconsistency.
• Evidence consistency: whether the explanation supports the predicted label.
• Human-checkable evidence score: 0 for wrong, 1 for partially correct, 2 for correct and specific.

Baselines

Baseline	Description	Expected value
VLM-only	Directly ask a VLM whether the figure is manipulated	Measures default visual reasoning
VLM + checklist prompt	Ask the model to inspect axes, labels, legends, captions, and visual values before deciding	Tests prompt-level assistance
OCR/parser + rules	Extract text and metadata-like structure, then check consistency rules	Provides interpretable non-training baseline
FigCheck	Figure-type-aware checklist plus VLM or rule verification	Main lightweight method

Robustness Tests

Run the test set under simple perturbations:

• JPEG compression
• Downsampling
• Slight blur
• Screenshot-style resizing

Report whether each baseline is brittle under post-processing.

Reporting Template

Each experiment should produce:

• Overall metrics table.
• Metrics by figure type.
• Metrics by manipulation type.
• Failure examples with image, prediction, ground truth, and explanation.
• Short interpretation of the dominant failure modes.