Benchmarking active visual search in information-dense scenes.
VisualNeedle is a 300-question benchmark for testing whether multimodal large language models can find and use decisive local visual evidence in information-dense images. Rather than treating fine-grained perception as one-shot VQA, it evaluates a sequential process: localize a relevant region, inspect it through a crop, and integrate that evidence into the final answer.
This repository contains the official evaluation harness, including reproducible diagnostic protocols, answer scoring, tool traces, and offline HTML reports.
| 300 Questions | 5 Abilities | 4 Diagnostics |
|---|---|---|
| Curated local-evidence tasks | OCR, color, entity, spatial, and occlusion | Text-only, no-tool, tool-enabled, and crop-black |
| 9 MLLMs | 56.01% Best Tool-Enabled | 63.00% Human |
| Evaluated in the paper | Strongest model result | Majority-vote baseline |
Search before answering. The agent starts with a cluttered global view, finds a reference region, and progressively narrows its crop until the decisive detail is visible.
Why this is difficult. The evidence is both small and sparse: success requires multi-step search and local-evidence integration, not only a plausible answer from the question or global scene.
| Aspect | VisualNeedle |
|---|---|
| Task | Image-grounded, multi-step visual search rather than one-shot VQA |
| Scenes | Urban street views, dense documents, shelves, bookcases, and maps |
| Evidence | A small, decisive clue that is not reliably visible at a glance |
| Core actions | Crop(bbox) to inspect local evidence, then Answer to finish the episode |
Every question satisfies three anti-shortcut requirements:
- Question-alone insufficiency: linguistic priors and lexical cues should not reveal the answer.
- Global-view insufficiency: the decisive detail should not be reliably recoverable from a single whole-image observation.
- Returned-evidence dependence: a tool-enabled answer should depend on the visual content returned by the tool, not merely on following a tool-call trajectory.
VisualNeedle separates four settings so that a final accuracy number can be interpreted as evidence-search capability rather than a mixture of language priors, global scene semantics, and tool behavior.
| Setting | Model input and interaction | What it diagnoses |
|---|---|---|
| Text-only | Question only; no image or tools | Whether wording and language priors are sufficient |
| No-tool | One full-image observation; direct answer | Whether global visual semantics resolve the local clue |
| Tool-enabled | Full image plus real local crop observations | Whether the model can localize, inspect, and integrate evidence |
| Crop-black | The same tool-use protocol, but each returned crop is replaced with a same-size black image | Whether gains from tool use rely on the returned visual evidence |
The crop-black ablation is central to the benchmark. It preserves the interaction format and crop dimensions while removing returned pixels, so its gap from tool-enabled performance measures the contribution of intermediate visual evidence rather than merely rewarding tool-call trajectories.
The paper evaluates nine mainstream MLLMs on all 300 questions.
| Evaluation setting | Best accuracy | What the result shows |
|---|---|---|
| Text-only | < 10% | The wording alone does not reveal answers reliably |
| No-tool | 17.89% | A single full-image observation is insufficient for most local clues |
| Tool-enabled | 56.01% | Cropping substantially helps, but evidence acquisition remains difficult |
| Human majority vote | 63.00% | Models retain a meaningful gap even with access to crops |
VisualNeedle covers color recognition, OCR, entity recognition, spatial relationships, and occluded-object recognition. Each item is curated so that the answer becomes clear only after the relevant region is found and inspected.
Five complementary abilities. The examples span reading, recognition, relation reasoning, and occlusion under the same sparse-local-evidence constraint.
A visible target is not enough. Each image pairs a dense scene with a small highlighted clue, illustrating why direct full-image VQA often fails.
Dataset release — coming soon.
We will release the 300-question benchmark, source images, annotations, bounding boxes, and evaluation split here. Stay tuned.
The evaluation harness is already public. Once the data release is available, pass its JSONL annotation file to --dataset-file; the expected format is documented in the evaluation guide.
Install the package and evaluate a prepared JSONL dataset with a configured model:
pip install -r requirements.txt
pip install -e .
python run_visualneedle_eval.py \
--config configs/models/think_with_images_neibu.yaml \
--model my-model \
--dataset-file path/to/visualneedle.jsonl \
--concurrency 20See the evaluation guide for installation alternatives, model configuration, the three evaluation modes, complete CLI reference, data schema, output artifacts, and library usage.
The visualneedle_eval package executes each protocol consistently across model providers and preserves enough evidence to audit a result.
| Harness responsibility | What is recorded or controlled |
|---|---|
| Model and tool setup | YAML-based model configuration, provider-specific coordinate conventions, and enabled image tools |
| Protocol execution | Standard tool-enabled/no-tool evaluation, crop-black ablation, and text-only evaluation |
| Reliable execution | Async or multiprocessing execution, bounded retries, indexed reruns, and ordered output |
| Answer scoring | Exact/normalized matching first, followed by an optional configured VLM judge for unmatched answers |
| Audit artifacts | Per-sample logs, intermediate tool images, normalized crop coordinates, token usage, structured JSON, and self-contained HTML reports |
For paper reproduction, keep the model, tool configuration, judge, and evaluation mode fixed, then report exact-match score, final score, judge-use count, and tool-use statistics together.
- Evaluation guide: model configuration, modes, full CLI reference, dataset format, image delivery modes, logs, HTML reports, retries, and Python API.
- Repository layout: entry scripts, core package, model configurations, and vendored runtime.
- Makefile shortcuts: install, test, CLI help, and configuration smoke tests.
If you use VisualNeedle in research, please cite:
@article{chen2026visualneedle,
title = {VisualNeedle: Benchmarking Active Visual Search in Information-Dense Scenes},
author = {Jingru Chen and Yiming Liu and Mingtao Chen and Sijie Chen and Richeng Xuan and Liang Yang and Zhichao Hu and Fanyang Lu},
journal = {arXiv preprint arXiv:2605.26380},
year = {2026}
}VisualNeedle is released under the Apache License 2.0. See the license file for third-party notices and attributions.