The benchmark should keep image files and metadata separate. Metadata is the source of truth for generation, manipulation, and evaluation.
data/
raw/
synthetic/
open_access/
interim/
extracted_text/
manipulated_candidates/
processed/
images/
metadata/
samples.jsonl
splits.json
taxonomy.json
samples/
Each line should be a JSON object:
{
"sample_id": "bar_000001_m01",
"image_path": "data/processed/images/bar_000001_m01.png",
"clean_image_path": "data/processed/images/bar_000001_clean.png",
"figure_type": "bar_chart",
"is_manipulated": true,
"manipulation_type": "value_label_mismatch",
"source_type": "synthetic",
"caption": "Accuracy comparison across methods. Method C achieves the best accuracy.",
"ground_truth": {
"x_labels": ["Method A", "Method B", "Method C"],
"y_values": [0.72, 0.81, 0.88],
"displayed_labels": [0.72, 0.91, 0.88],
"claimed_best": "Method C"
},
"manipulation": {
"target": "Method B value label",
"original": "0.81",
"modified": "0.91",
"region": [310, 122, 365, 145]
},
"evidence_text": "The Method B label says 0.91, but its bar height is below Method C and near 0.81.",
"split": "test"
}- Store the clean metadata before any visual manipulation.
- Store the manipulation recipe so every manipulated sample can be regenerated.
- Keep captions as explicit fields because caption-figure inconsistency is a core task.
- Keep bounding boxes optional for MVP, but include semantic evidence text for every manipulated sample.
Recommended MVP split:
| Split | Purpose | Share |
|---|---|---|
train |
Prompt development and rule calibration | 20% |
validation |
Prompt and threshold selection | 20% |
test |
Final reporting only | 60% |
If no supervised training is performed, the names still matter: they reduce accidental prompt overfitting.
The 60% test share is intentional for the MVP because the project is primarily an evaluation benchmark, not a supervised training project. If later work trains detectors or tunes many prompts, revisit this split and consider increasing the validation share.