codex-autoresearcher

A dead-simple, Codex-native autoresearch harness.

Let Codex optimize anything you can measure — settings, code, algorithms — by running an autonomous loop of propose → evaluate → judge → keep-or-discard.

./research examples/leduc_settings_tuning --max-turns 1

---

How It Compares

There are three approaches to autonomous LLM-driven optimization. Each makes different tradeoffs.

Karpathy's autoresearch

karpathy/autoresearch is the original idea: give an LLM a benchmark and tell it to make the number go down. The LLM does everything — proposes changes, runs the benchmark, decides whether to keep. Beautifully minimal. One conversation, one loop.

Strength: Zero setup. One prompt, one script, start optimizing.

Limitation: The LLM is judge and jury in its own case. No structured verdicts, no durable memory across context resets, no separation between the "try things" brain and the "decide if it worked" brain. When the context window fills up, you start over.

Pi autoresearch

pi-autoresearch takes the Karpathy idea and adds real infrastructure: a pi extension with run_experiment / log_experiment tools, an autoresearch.jsonl log that survives restarts, a confidence scorer based on MAD, a live dashboard widget. The agent writes autoresearch.sh and loops autonomously.

Strength: Works for any optimization target in minutes. Confidence scoring catches noise. The agent-as-tool-user model is natural and fast for simple benchmarks.

Limitation: The same agent that proposes changes also decides whether to keep them. There's no schema-enforced verdict, no explicit acceptance basis, no structured hypothesis tracking. The benchmark script is agent-authored and agent-mutable. Memory is a markdown file the agent updates by hand. Works great for "make this test faster" — but when the evaluation costs real money, takes 10 minutes, or requires domain-specific judgment, you want more structure.

codex-autoresearcher (this repo)

This repo takes a different architectural position: the LLM is a controlled subprocess, not the loop driver.

A Python runner orchestrates the loop. Each turn spawns two separate Codex CLI processes:

1. Worker turn — Codex proposes changes and writes a structured hypothesis
2. Judge turn — a separate Codex process evaluates the evidence and returns a schema-validated JSON verdict

The runner owns everything else: Git integration, evaluation, durable memory, winner advancement, and crash recovery.

	Karpathy	Pi autoresearch	codex-autoresearcher
Who proposes	The LLM	The LLM agent	A spawned Codex worker process
Who judges	The same LLM	The same agent	A separate Codex judge process with a JSON schema
Who owns the loop	The LLM	The agent (via tools)	A Python runner (runner.py)
Verdict format	Prose	keep/discard flag	Schema-validated JSON with acceptance_basis
Hypothesis tracking	None	Markdown notes	Structured hypothesis.statement + hypothesis.reasoning per attempt
Evaluation	Agent-authored script	Agent-authored autoresearch.sh	Human-authored static evaluate.sh
Memory	Context window	autoresearch.jsonl + markdown	journal.jsonl + current_winner.json + per-attempt artifacts
Git integration	None	Auto-commit/revert	Runner-managed commits with worker-authored messages
Resume	Start over	Read markdown + jsonl	Deterministic from files; no prose needed
Multi-experiment	No	One per branch	Multiple isolated experiments per project
External projects	No	workingDir override	git_repo_path + project_root with cross-repo context
Screening	No	Optional checks.sh	Optional screen.sh as a fast pre-evaluation gate
Bug-fix keeps	Not a concept	Agent judgment	First-class: non_regression_bug_fix acceptance basis
Setup time	Minutes	Minutes	Longer — you author an experiment contract
Sweet spot	Quick experiments	Any measurable benchmark	Complex, expensive, or multi-metric optimization campaigns

When to use which:

• Karpathy — You want to try something in 5 minutes and don't care about rigor
• Pi autoresearch — You want an autonomous loop with real tooling for straightforward benchmarks
• codex-autoresearcher — You want structured judgment, hypothesis tracking, durable forensics, and the ability to run expensive evaluations where the LLM shouldn't be trusted to judge its own work

---

What It Does

One run looks like this:

1. Load an experiment directory
2. Require current_winner_settings.yaml (fail loud if missing)
3. Require git_repo_path to be clean
4. Bootstrap current_winner.json if missing by running evaluate.sh
5. Compose a worker prompt from templates + experiment context + journal memory
6. Spawn Codex as the worker — it edits files and writes worker_turn.json
7. Optionally run screen.sh as a fast pre-evaluation gate
8. Run evaluate.sh to measure the candidate
9. Compose a judge prompt with all the evidence
10. Spawn Codex as the judge — it returns a schema-validated JSON verdict
11. On keep: commit the changes, advance the winner
12. On discard/crash: restore files from Git
13. Append the result to journal.jsonl
14. Continue until stopped

---

Quick Start

git clone https://github.com/aelaguiz/codex-autoresearcher.git
cd codex-autoresearcher
uv sync

Initialize the example

The example experiment needs its own Git repo for the harness to manage:

cd examples/leduc_settings_tuning
git init && git add . && git commit -m "initial state"
cd ../..

Run one turn

./research examples/leduc_settings_tuning --max-turns 1

Run endlessly

./research examples/leduc_settings_tuning

Watch the merged log stream

./research logs

Tail one experiment's runner log

./research --tail-log examples/leduc_settings_tuning

---

On-Disk Experiment Anatomy

examples/leduc_settings_tuning/
  experiment.yaml              # the experiment contract
  goal_prompt.md               # what Codex should optimize
  judge_prompt.md              # how success is judged
  evaluate.sh                  # the static evaluator
  current_winner_settings.yaml # current best settings
  current_winner.json          # measured result of the current winner
  leduc_settings.yaml          # the editable candidate settings file
  journal.jsonl                # append-only durable memory (runtime)
  runner.log                   # human-readable event stream (runtime)
  artifacts/                   # per-attempt outputs (runtime)

What Each File Does

experiment.yaml — Machine-readable contract: project root, Git repo, Codex settings, metrics, timeouts, writable surface, commands.

goal_prompt.md — Fills the "here is your goal" section of the worker prompt.

judge_prompt.md — Fills the "how success is judged" section of both worker and judge prompts.

evaluate.sh — The only evaluator. Static experiment infrastructure. Reads a settings file, writes metrics.json. Used for both current winner bootstrap and candidate evaluation.

current_winner_settings.yaml — Required. The settings that define the current best candidate.

current_winner.json — The measured result of the current winner. Bootstrapped automatically if missing.

journal.jsonl — Append-only durable attempt history. Survives across runs until you delete it.

runner.log — Human-readable live event stream. Not a source of truth.

---

experiment.yaml

version: 1
id: my_experiment
name: "My optimization experiment"
project_root: /path/to/your/project
git_repo_path: /path/to/your/project

codex:
  bin: codex
  profile: yolo
  model: o4-mini
  reasoning_effort: low

timeouts:
  codex_turn_timeout_seconds: 900
  evaluation_timeout_seconds: 180

objective:
  primary_metric:
    name: my_metric
    direction: lower  # or higher
  secondary_metrics:
    - name: other_metric
      direction: higher

current_winner:
  output: current_winner.json
  settings_file: current_winner_settings.yaml

optimization_surface:
  settings_files:
    - /path/to/your/project/settings.yaml
  levers_and_ideas:
    - thread count
    - algorithm selection

writable_paths:
  - /path/to/your/project/settings.yaml
  - /path/to/your/project/src/  # for code-editing experiments

commands:
  screen: ./screen.sh  # optional fast pre-check
  evaluate: ./evaluate.sh

Key fields

• git_repo_path — The Git repo owning the editable files. Must be clean before the runner starts.
• writable_paths — Explicit allowed mutation surface. Can include settings files, source directories, or both.
• timeouts — Required. No defaults. Both Codex turn and evaluation timeouts must be explicit.
• commands.screen — Optional. A fast pre-evaluation gate that can reject bad candidates cheaply.

---

evaluate.sh Contract

# Current winner bootstrap:
./evaluate.sh current_winner_settings.yaml current_winner.json

# Candidate evaluation:
./evaluate.sh /path/to/candidate_settings.yaml artifacts/attempt_0007/metrics.json

• Accept a settings file path as the first argument
• Accept an output JSON path as the second argument
• Write the output file with primaryMetric and secondaryMetrics
• Exit 0 on success, nonzero on failure

---

Judge Verdict Schema

The judge returns schema-validated JSON with:

• status: keep, discard, or crash
• acceptance_basis: metric_improvement, non_regression_bug_fix, non_regression_code_quality, not_good_enough, metric_regression, evaluation_timeout, evaluation_failed, invalid_or_missing_evidence
• summary: Short explanation
• primary_metric and secondary_metrics: The measured values
• conclusion: Hypothesis resolution with structured reasoning

The explicit acceptance_basis is what makes bug-fix keeps, code-quality keeps, and metric-improvement keeps all first-class — with no ambiguity about why a candidate was accepted.

---

How To Define A New Experiment

1. Create a new directory with the required files
2. Point project_root and git_repo_path at your project
3. Write an evaluate.sh that reads settings and produces metrics
4. Define the writable surface in experiment.yaml
5. Write goal_prompt.md and judge_prompt.md
6. Create current_winner_settings.yaml with your starting configuration
7. Run: ./research your_experiment_dir --max-turns 1

The shipped examples/leduc_settings_tuning/ is a complete working reference.

---

Operator Experience

# The main command:
./research <experiment_dir> [--max-turns N]

# TTY → full-screen Textual UI
# Non-TTY → plain log lines to stdout

• Stop: press q in the UI, or Ctrl-C once for clean stop, twice for hard interrupt
• Resume: just run the same command again — it picks up from durable files
• --max-turns: for bounded testing; counts completed judged turns only

---

Architecture

codex-autoresearcher/
  src/codex_research/
    cli.py              # CLI entry point
    runner.py           # main loop orchestration
    experiment.py       # experiment loading and validation
    codex_cli.py        # Codex CLI process management
    prompt_compose.py   # prompt template filling
    judge.py            # judge verdict validation
    journal.py          # append-only memory
    metrics.py          # metric parsing
    git_ops.py          # Git commit/restore
    dashboard.py        # Textual terminal UI
    live_logs.py        # merged log streaming
    schemas.py          # typed data models
  prompts/
    work_prompt.md      # worker turn template
    judge_prompt.md     # judge turn template
    judge_verdict.schema.json
  examples/
    leduc_settings_tuning/  # complete working example
  tests/

---

License

MIT