English · 简体中文
Detecting semantic manipulation in structured academic figures.
A lightweight benchmark, reproducible evaluation suite, and structured-extraction baseline.
| 1,000 figures | 6 manipulation types | 94.0% best test accuracy |
|---|---|---|
| Frozen, reproducible benchmark | Across 4 structured figure types | Structured parser-and-rules baseline |
Academic figures can be visually plausible while being semantically wrong: a bar may carry the wrong value label, a caption may contradict the plotted winner, or a heatmap color may disagree with its number.
DSMAF Light asks a focused question:
Can vision-language models detect internal semantic inconsistencies in academic figures, or is explicit extraction and consistency checking still necessary?
This is different from detecting whether an image is AI-generated. The project studies whether the claims inside a figure agree with its own visual evidence.
On the frozen 600-item test split, three VLM families remain weak at semantic-manipulation detection. Their raw accuracy ranges from 0.405 to 0.585, with false-negative rates often above 0.5. In contrast, the offline structured baselines reach 0.940 accuracy (parser_rules) and 0.933 (FigCheck).
The test set is intentionally imbalanced (120 clean / 480 manipulated), so the paper also reports balanced accuracy, macro-F1, MCC, FPR, and FNR rather than relying on raw accuracy alone.
| Baseline | Model group | Accuracy | Macro-F1 | FPR | FNR |
|---|---|---|---|---|---|
parser_rules |
Offline structured | 0.9400 | 0.9121 | 0.0583 | 0.0604 |
figcheck |
Offline structured | 0.9333 | 0.8989 | 0.1250 | 0.0521 |
| Best VLM run | Open-source VLM | 0.5850 | 0.5252 | 0.4250 | 0.4125 |
See the complete result table, failure analysis, and robustness results.
SciFigSem-1K is a synthetic benchmark designed to preserve exact metadata and ground truth throughout generation and manipulation.
| Property | Value |
|---|---|
| Status | Frozen on 2026-06-23 |
| Total samples | 1,000 |
| Clean / manipulated | 200 / 800 |
| Train / validation / test | 200 / 200 / 600 |
| Figure types | Bar chart, line chart, heatmap, confusion matrix |
| MVP manipulation types | 6 |
The six manipulation types are:
- Value-label mismatch
- Axis-scale manipulation
- Legend-color mismatch
- Caption-figure inconsistency
- Heatmap color-value mismatch
- Confusion-matrix inconsistency
The complete benchmark images are reproducible and intentionally excluded from normal Git history. Git tracks the frozen sample metadata, family-safe splits, checksum manifest, and representative examples under data/samples/.
Read the dataset card for construction details, intended use, limitations, and licensing status.
DSMAF Light compares four levels of reasoning:
- VLM direct prompting — classify a figure without an explicit inspection procedure.
- VLM checklist prompting — inspect values, axes, legends, captions, colors, and relations before deciding.
- Parser + rules — extract observable structure and apply deterministic consistency checks.
- FigCheck — route each figure type through specialized extractors and checks.
git clone https://github.com/QCYTSN/DSMAF_Light.git
cd DSMAF_Light
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
pip install -e ".[dev]"For OCR-assisted experiments:
pip install -e ".[ocr]"python scripts\generate_clean_dataset.py --config configs\dataset_mvp.yaml
python scripts\generate_manipulated_dataset.py --config configs\dataset_mvp.yaml
python scripts\finalize_benchmark.py --config configs\dataset_mvp.yamlpython scripts\run_extraction.py `
--split validation `
--run-id extraction-validation
python scripts\run_baseline.py `
--baseline parser_rules `
--split validation `
--observations experiments\extraction-validation\observed_figures.jsonl `
--run-id parser-rules-validation
python scripts\evaluate_run.py `
--predictions experiments\parser-rules-validation\predictions.jsonl `
--split validation `
--output-dir experiments\parser-rules-validationUse --help on any script to see its current CLI options before launching a full experiment.
Real VLM runs use OpenAI-compatible chat-completions APIs through Alibaba DashScope or SiliconFlow. Keep credentials local and set only the provider key required for the run:
$env:DASHSCOPE_API_KEY="your-key" # qwen3.7-plus
$env:SILICONFLOW_API_KEY="your-key" # Qwen3-VL-32B / GLM-4.5VAPI execution is gated behind an explicit --execute flag. Without it, the runner prepares a dry-run manifest and makes no request. Completed runs record the provider, model, prompt checksum, token usage, and parse status.
Committed result artifacts are generated from experiment metrics.json files rather than hand-transcribed values:
python scripts\build_result_tables.py
python scripts\build_figures.py- Result tables:
reports/tables/ - Result figures:
reports/figures/ - Experiment freeze policy:
reports/experiment_freeze.md - IEEE paper source:
paper/
DSMAF_Light/
├── configs/ Dataset and experiment configuration
├── data/ Metadata, samples, and reproducible data artifacts
├── docs/ Dataset, taxonomy, evaluation, and research documentation
├── experiments/ Committed run outputs and metrics
├── paper/ IEEE-format LaTeX paper source
├── reports/ Generated tables, figures, and analyses
├── scripts/ Reproducible command-line entry points
├── src/dsmaf_light/ Generation, manipulation, extraction, and baselines
└── tests/ Unit and integration tests
- Research direction
- Dataset card
- Manipulation taxonomy
- Data schema
- Evaluation protocol
- Ethics and scope
- Command reference
SciFigSem-1K currently covers synthetic, structured figures with exact metadata. It does not establish performance on arbitrary real-world scientific figures, natural images, or every form of research misconduct. The benchmark is intended for controlled evaluation and failure analysis—not as a standalone integrity verdict.
The paper source still contains placeholder author metadata and should be treated as a research draft until archival publication details are available.
Built for reproducible research on semantic consistency in academic figures.