English · 简体中文

DSMAF Light

Detecting semantic manipulation in structured academic figures.
A lightweight benchmark, reproducible evaluation suite, and structured-extraction baseline.

1,000 figures	6 manipulation types	94.0% best test accuracy
Frozen, reproducible benchmark	Across 4 structured figure types	Structured parser-and-rules baseline

The research question

Academic figures can be visually plausible while being semantically wrong: a bar may carry the wrong value label, a caption may contradict the plotted winner, or a heatmap color may disagree with its number.

DSMAF Light asks a focused question:

Can vision-language models detect internal semantic inconsistencies in academic figures, or is explicit extraction and consistency checking still necessary?

This is different from detecting whether an image is AI-generated. The project studies whether the claims inside a figure agree with its own visual evidence.

Key finding

On the frozen 600-item test split, three VLM families remain weak at semantic-manipulation detection. Their raw accuracy ranges from 0.405 to 0.585, with false-negative rates often above 0.5. In contrast, the offline structured baselines reach 0.940 accuracy (parser_rules) and 0.933 (FigCheck).

The test set is intentionally imbalanced (120 clean / 480 manipulated), so the paper also reports balanced accuracy, macro-F1, MCC, FPR, and FNR rather than relying on raw accuracy alone.

Baseline	Model group	Accuracy	Macro-F1	FPR	FNR
parser_rules	Offline structured	0.9400	0.9121	0.0583	0.0604
figcheck	Offline structured	0.9333	0.8989	0.1250	0.0521
Best VLM run	Open-source VLM	0.5850	0.5252	0.4250	0.4125

See the complete result table, failure analysis, and robustness results.

SciFigSem-1K

SciFigSem-1K is a synthetic benchmark designed to preserve exact metadata and ground truth throughout generation and manipulation.

Property	Value
Status	Frozen on 2026-06-23
Total samples	1,000
Clean / manipulated	200 / 800
Train / validation / test	200 / 200 / 600
Figure types	Bar chart, line chart, heatmap, confusion matrix
MVP manipulation types	6

The six manipulation types are:

1. Value-label mismatch
2. Axis-scale manipulation
3. Legend-color mismatch
4. Caption-figure inconsistency
5. Heatmap color-value mismatch
6. Confusion-matrix inconsistency

The complete benchmark images are reproducible and intentionally excluded from normal Git history. Git tracks the frozen sample metadata, family-safe splits, checksum manifest, and representative examples under data/samples/.

Read the dataset card for construction details, intended use, limitations, and licensing status.

Approach

DSMAF Light compares four levels of reasoning:

• VLM direct prompting — classify a figure without an explicit inspection procedure.
• VLM checklist prompting — inspect values, axes, legends, captions, colors, and relations before deciding.
• Parser + rules — extract observable structure and apply deterministic consistency checks.
• FigCheck — route each figure type through specialized extractors and checks.

Quick start

1. Install

git clone https://github.com/QCYTSN/DSMAF_Light.git
cd DSMAF_Light
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
pip install -e ".[dev]"

For OCR-assisted experiments:

pip install -e ".[ocr]"

2. Rebuild the benchmark

python scripts\generate_clean_dataset.py --config configs\dataset_mvp.yaml
python scripts\generate_manipulated_dataset.py --config configs\dataset_mvp.yaml
python scripts\finalize_benchmark.py --config configs\dataset_mvp.yaml

3. Run an offline baseline

python scripts\run_extraction.py `
  --split validation `
  --run-id extraction-validation

python scripts\run_baseline.py `
  --baseline parser_rules `
  --split validation `
  --observations experiments\extraction-validation\observed_figures.jsonl `
  --run-id parser-rules-validation

python scripts\evaluate_run.py `
  --predictions experiments\parser-rules-validation\predictions.jsonl `
  --split validation `
  --output-dir experiments\parser-rules-validation

Use --help on any script to see its current CLI options before launching a full experiment.

VLM experiments

Real VLM runs use OpenAI-compatible chat-completions APIs through Alibaba DashScope or SiliconFlow. Keep credentials local and set only the provider key required for the run:

$env:DASHSCOPE_API_KEY="your-key"       # qwen3.7-plus
$env:SILICONFLOW_API_KEY="your-key"     # Qwen3-VL-32B / GLM-4.5V

API execution is gated behind an explicit --execute flag. Without it, the runner prepares a dry-run manifest and makes no request. Completed runs record the provider, model, prompt checksum, token usage, and parse status.

Reproduce tables and figures

Committed result artifacts are generated from experiment metrics.json files rather than hand-transcribed values:

python scripts\build_result_tables.py
python scripts\build_figures.py

• Result tables: reports/tables/
• Result figures: reports/figures/
• Experiment freeze policy: reports/experiment_freeze.md
• IEEE paper source: paper/

Repository map

DSMAF_Light/
├── configs/              Dataset and experiment configuration
├── data/                 Metadata, samples, and reproducible data artifacts
├── docs/                 Dataset, taxonomy, evaluation, and research documentation
├── experiments/          Committed run outputs and metrics
├── paper/                IEEE-format LaTeX paper source
├── reports/              Generated tables, figures, and analyses
├── scripts/              Reproducible command-line entry points
├── src/dsmaf_light/      Generation, manipulation, extraction, and baselines
└── tests/                Unit and integration tests

Documentation

• Research direction
• Dataset card
• Manipulation taxonomy
• Data schema
• Evaluation protocol
• Ethics and scope
• Command reference

Scope and limitations

SciFigSem-1K currently covers synthetic, structured figures with exact metadata. It does not establish performance on arbitrary real-world scientific figures, natural images, or every form of research misconduct. The benchmark is intended for controlled evaluation and failure analysis—not as a standalone integrity verdict.

The paper source still contains placeholder author metadata and should be treated as a research draft until archival publication details are available.

---

_{Built for reproducible research on semantic consistency in academic figures.}