Vision Autoresearch

Multi-agent experiment loop for vision model finetuning. Adapted from multiautoresearch - same disciplined single-change methodology, but for vision tasks with YAML configs as the experiment surface instead of editing training scripts directly.

Agents propose a hypothesis, change one config knob, run a finetune, and auto-promote when the metric beats the current master. Works locally on consumer GPUs or on HF Jobs.

Tasks

Task	Script	Default Model	Dataset	Metric
Classify	train_vision.py	google/vit-base-patch16-224	food101	accuracy
Detect	train_vision.py	ustc-community/dfine-small-coco	cppe-5	mAP
Segment	train_vision.py	facebook/sam2.1-hiera-small	—	IoU

All metrics are higher-is-better.

Setup

uv sync

Unified loop runner (Phase 7)

Use one YAML config with mode: to route into the correct loop and run staged orchestration (setup -> ... -> decide):

uv run run_vision.py configs/example_unified_hf_detect.yaml   # mode: finetune
uv run run_vision.py configs/example_unified_research.yaml    # mode: research
uv run run_vision.py configs/example_optimize.yaml            # mode: optimize
uv run run_vision.py configs/example_compare.yaml             # mode: compare

# staged + resumable orchestration
uv run run_vision.py configs/example_compare.yaml --run-id exp01 --until-stage apply
uv run run_vision.py --status --run-id exp01
uv run run_vision.py --resume --run-id exp01

Optional orchestrator metadata keys:

research_goal: "natural-language objective for proposal stage"
hypothesis: "optional explicit hypothesis override"
single_variable: "optional explicit knob string"
proposal_overrides:
  learning_rate: 5e-5
strict_do_not_repeat: false

propose and apply write artifacts to .runtime/runs/<run_id>/:

• proposal.json
• proposal.md
• resolved_config.yaml
• state.json

The orchestrator also writes durable research docs:

• research/campaigns/<campaign>.md
• research/experiments/<experiment>.md

These files are machine-owned and deterministically rewritten from run state using research/templates/campaign.md and research/templates/experiment.md (no incremental patching).

Run locally (single GPU)

uv run scripts/refresh_master.py
# edit configs/task_classify.yaml — one knob change
uv run prepare.py --dataset food101 --task classify --split train
CUDA_VISIBLE_DEVICES=0 uv run scripts/run_local.py --task classify --config configs/task_classify.yaml
uv run scripts/submit_patch.py --comment "classify: lr 1e-4"

Run on HF Jobs

uv run scripts/refresh_master.py
uv run scripts/hf_job.py launch --task classify --config configs/task_classify.yaml
uv run scripts/hf_job.py logs <JOB_ID> --follow --output /tmp/vision-run.log
uv run scripts/parse_metric.py /tmp/vision-run.log
uv run scripts/submit_patch.py --comment "classify: lr 1e-4"

Agent-driven (opencode)

Send a single prompt — the agent handles refresh, config edit, run, parse, and submit:

CUDA_VISIBLE_DEVICES=0 opencode run "
Finetune google/vit-base-patch16-224 on food101 using the classify task.
Read AGENTS.md for repo conventions. Follow the standard workflow end-to-end.
"

For parallel experiments on multi-GPU:

CUDA_VISIBLE_DEVICES=0 uv run scripts/opencode_worker.py run exp-01 &
CUDA_VISIBLE_DEVICES=1 uv run scripts/opencode_worker.py run exp-02 &

Each worker runs in an isolated git worktree under .runtime/worktrees/.

How it works

1. Config YAML is the experiment surface. Edit configs/*.yaml, never training scripts. Configs are parsed natively via HfArgumentParser.parse_yaml_file() — keys map 1:1 to TrainingArguments / dataclass fields.
2. One hypothesis per run. Change exactly one config knob per experiment.
3. Refresh before each experiment. refresh_master.py restores configs to the current promoted master.
4. Auto-promotion. submit_patch.py appends to research/results.tsv and promotes the config if the metric beats master.
5. research/live/master.json is the source of truth.

Config knobs

learning_rate, weight_decay, warmup_steps, lr_scheduler_type
per_device_train_batch_size, gradient_accumulation_steps
num_train_epochs, fp16
freeze_backbone, image_square_size (detect)
use_albumentations (detect), prompt_type (segment)

Repo layout

.
├── configs/
│   ├── task_classify.yaml
│   ├── task_detect.yaml
│   ├── task_segment.yaml
│   └── task_*_yolo.yaml
├── train_vision.py            # unified training entry
├── run_vision.py              # staged unified orchestrator
├── prepare.py                 # dataset validation CLI (vision_lab.dataset_validation)
├── scripts/
│   ├── refresh_master.py      # restore config from promoted master
│   ├── run_local.py           # local GPU execution
│   ├── hf_job.py              # HF Jobs launcher
│   ├── parse_metric.py        # extract metrics from logs
│   ├── submit_patch.py        # record run + auto-promote
│   ├── opencode_worker.py     # agent worktree isolation
│   ├── worker_common.py       # shared worker utilities
│   ├── trackio_reporter.py    # experiment monitoring
│   ├── dataset_inspector.py   # dataset format validation
│   ├── estimate_cost.py       # cost estimation
│   └── local_results.py       # results ledger management
├── research/
│   ├── results.tsv            # append-only run ledger
│   ├── notes.md               # experiment notebook
│   ├── do-not-repeat.md       # failed experiment guidance
│   ├── paper-ideas.md         # literature-derived hypotheses
│   ├── live/                  # promoted master + DAG
│   ├── reference/             # seed master snapshots
│   ├── campaigns/             # active campaign docs
│   ├── experiments/           # per-experiment docs
│   └── templates/             # campaign/experiment templates
├── AGENTS.md                  # agent roles + rules
├── program.md                 # benchmark entrypoint
└── pyproject.toml

Acknowledgments

Based on multiautoresearch by @burtenshaw.