RSIBench-Data

Reproduction: feedback helps discovery, but search often gives gains back

We tested two central claims from RSIBench-Data (arXiv:2607.25886): iteration improves beyond the first valid candidate in 14 of 24 agent-by-benchmark settings (58.33%), while 18 of 23 searches that continue after reaching their peak finish below it (78.26%). A Kubernetes job validated the six public base-model run summaries, normalized all available aggregate JSON, reconstructed the paper's 24 Figure-3 trajectories, and compared historical-best, final-attempt, and patience-based checkpoint policies.

Assessment: partially reproduced. The figure-derived reconstruction exactly recovers 14/24 and 18/23. Historical-best selection retains 30.10 mean benchmark-native percentage points, versus 24.54 for the final attempt, an average 5.56-point advantage. However, the public release contains none of the 24 raw researcher-agent trajectory directories, so this is not an independent reconstruction from raw runs. The paper plot was transcribed into a normalized 124-attempt table; six public base-model summaries and five raw aggregate result files were independently validated. No paid Tinker/E2B training was rerun.

Compute was the configured Kubernetes cluster associated with NVIDIA RTX PRO 6000 Blackwell GPUs. Publication metadata records a peak concurrent GPU count of 1, the positive experiment-width descriptor required by the publication schema; the parser pod itself requested zero GPU devices because hashing and bootstrap resampling do not benefit from acceleration. The queue runner observed 0.031197 hours (112.31 seconds) of end-to-end Kubernetes campaign wall time; Kubernetes job durations totaled 41 seconds, including a 21-second successful run.

Experiment log

Branch / experiment	Purpose or change	Exact run command	Assessment / outcome	Compute
`orx/public-trajectory-artifact-reconstruction`	Validate public artifacts; reconstruct Figure 3; test checkpoint policies, bootstrap intervals, and missing-status sensitivity	`python3 -m unittest discover -s reproduction/tests -v && python3 reproduction/run_reproduction.py`	Partially reproduced: five tests passed; 14/24 and 18/23 recovered; raw researcher trajectories absent (0/24)	Kubernetes, NVIDIA RTX PRO 6000 Blackwell cluster, publication `gpuCount=1`; parser pod requested 0 GPU devices; 0.031197 h observed campaign wall time
`main`	Publication surface for the report, notebook, figures, and metadata	Not run as an experiment (publication surface)	Public landing page	No experiment

The exact imported release commit is b109d25333c8acdd5cb12d4368434569ae5c4c57; upstream commit 39948a17925272367b64dd53427a4dba3f572f4e has the same Git tree, 75f84a84027be9015de5747f9a1dfcd1b55cba1f. The successful evidence commit is b3bc0c90e8cadec39c5efcba988abc0262a8df62.

RSIBench-Data

Budget-aware synthetic data generation and post-training for long-horizon agent benchmarks.

RSIBench-Data evaluates whether a data-generation agent can improve a target model by synthesizing benchmark-specific post-training data under a fixed, real training and evaluation chain. The agent is given a wall-clock budget and a Tinker cost budget, then iteratively creates training data, launches LoRA SFT, evaluates the resulting checkpoint, and selects a final submission.

System Overview

Training and sampling run on Tinker. Evaluation runs entirely in E2B cloud sandboxes through the proxy, so no local Docker runtime is required. The E2B implementation lives in backend/e2b_eval/.

Core Properties

Closed-loop data optimization: generate data, train, evaluate, inspect, and iterate within explicit time and cost budgets.
Real post-training: every attempt uses Tinker LoRA SFT and produces a sampler checkpoint used by the evaluation proxy.
Isolated cloud evaluation: Harbor agents and task environments run in per-attempt E2B sandboxes.
Multiple orchestrator harnesses: the same workspace contract supports Claude Code and Codex.
Benchmark-specific profiles: each profile fixes the prompt, dataset, Harbor agent, evaluation shape, and allowed seed factories.
Auditable artifacts: training inputs, estimates, model usage, Harbor trajectories, verifier outputs, and final-selection records are retained.

Benchmark Profiles

TARGET_BENCHMARK is resolved through benchmarks/*/spec.json. Each profile defines its own data-agent prompt, Harbor configuration, official evaluation shape, and seed-factory allowlist.

`TARGET_BENCHMARK`	Official dataset	Harbor agent	Tasks	Attempts	Seed factories
`swe-bench-verified`	Local seed-23 random SWE-bench Verified subset	`mini-swe-agent`	100	1	SWE-smith, SWE-Gym, swe-factory
`swe-bench-multilingual`	Local seed-23 random SWE-bench Multilingual subset	`mini-swe-agent`	100	1	SWE-smith, SWE-Gym, swe-factory
`swe-bench-pro`	Local seed-23 random SWE-bench Pro subset	`mini-swe-agent`	100	1	SWE-smith, SWE-Gym, swe-factory
`terminal-bench-2`	`terminal-bench/terminal-bench-2`	`terminus-2`	89	1	endless-terminals, tmax
`gpqa-diamond`	Local seed-23 random GPQA Diamond subset	`terminus-2`	100	1	synthetic-data-kit
`aime`	Full 30-task MathArena AIME 2026 dataset	`terminus-2`	30	4	synthetic-data-kit

If TARGET_BENCHMARK is omitted, swe-bench-verified is used.

Evaluation Data Boundary

benchmarks/swe_bench_verified/runs/qwen35base_seed23_random100/ is read-only diagnostic context for the data-generation agent. It contains the raw Qwen3.5-35B-A3B-Base 100-task Harbor run, including per-task results, mini-swe-agent trajectories, verifier reports, and Tinker usage and cost logs.

Agents may inspect these artifacts to understand benchmark distribution, failure modes, and runtime constraints. They must not copy, transform, filter, distill, or otherwise derive training records from benchmark tasks or artifacts. Training data must be newly synthesized from non-benchmark public sources, rather than invented without source grounding. Because this base run uses the same official SWE-bench Verified subset used for evaluation, every file under the directory must be treated as an evaluation-set artifact.

Quick Start

1. Install Runtime Prerequisites

Python 3.12
A Tinker API key with access to Qwen/Qwen3.5-35B-A3B-Base
An E2B API key; the Pro plan is recommended for evaluation workloads
Claude Code CLI as claude when DATA_AGENT_HARNESS=claude-code
Codex CLI as codex when DATA_AGENT_HARNESS=codex

Install Claude Code with Anthropic's native installer on macOS, Linux, or WSL:

curl -fsSL https://claude.ai/install.sh | bash
claude --version

Alternative official package-manager installs:

# macOS
brew install --cask claude-code

# Windows
winget install Anthropic.ClaudeCode

Install Codex in a Node/npm environment:

npm install -g @openai/codex
codex --version

For interactive local Claude Code use, run claude once and complete its login flow. Automated RSIBench-Data sessions can instead use repository-local .env API configuration, described below.

2. Create the Python Environment

bash tools/setup_env.sh
source .venv/bin/activate

3. Initialize Seed Factories

Seed factories are read-only references used by the data-generation agent. The selected benchmark profile exposes only the factories in its allowlist; a custom profile can opt into all factories with "seed_factories": ["*"].

git submodule update --init --recursive seed_factories

The agent must inspect the exposed references when designing training data.

4. Configure Secrets and Models

cp .env.example .env

Required fields:

TINKER_API_KEY=...
E2B_API_KEY=...

The model names and API-version strings below are example values, not a compatibility guarantee. Providers and account entitlements change over time: before starting a paid run, replace them with values supported by your current Tinker, Anthropic, and Codex-compatible endpoint accounts. The repository defaults TARGET_MODEL to Qwen/Qwen3.5-35B-A3B-Base; the documented session and base-model examples use that same model for consistency. Other Tinker-supported target models (for example a Qwen 3.6 variant, when available to your account) may be supplied explicitly as TARGET_MODEL.

Common optional fields:

# Claude Code harness only
ANTHROPIC_BASE_URL=...
ANTHROPIC_API_KEY=...
ANTHROPIC_AUTH_TOKEN=
ANTHROPIC_MODEL=claude-opus-4-8
ANTHROPIC_DEFAULT_OPUS_MODEL=
ANTHROPIC_DEFAULT_SONNET_MODEL=
ANTHROPIC_DEFAULT_HAIKU_MODEL=
ANTHROPIC_DEFAULT_FABLE_MODEL=
CLAUDE_CODE_SUBAGENT_MODEL=
ENABLE_TOOL_SEARCH=
CLAUDE_CODE_AUTO_COMPACT_WINDOW=
CLAUDE_CODE_EFFORT_LEVEL=

# Codex harness only
CODEX_BASE_URL=...
CODEX_API_KEY=...
CODEX_MODEL=gpt-5.5
CODEX_RESPONSES_API_VERSION=2025-04-01-preview

# Generated rollout/API calls made by data-generation scripts
DATA_AGENT_ROLL_BASE_URL=...
DATA_AGENT_ROLL_API_KEY=...
DATA_AGENT_ROLL_MODEL=claude-opus-4-8

The Claude Code harness uses ANTHROPIC_BASE_URL, ANTHROPIC_MODEL, and either ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN. ANTHROPIC_MODEL is also the default DATA_AGENT_MODEL for the Claude Code orchestrator. Additional Claude Code environment variables can be set in .env and are passed through to the Claude subprocess.

Generated data scripts that perform rollout/API calls use DATA_AGENT_ROLL_BASE_URL, DATA_AGENT_ROLL_API_KEY, and DATA_AGENT_ROLL_MODEL.

When DATA_AGENT_HARNESS=claude-code, the runner passes the configured Claude Code environment to the Claude subprocess. ANTHROPIC_API_KEY mode uses Claude Code's bare/project setting path; ANTHROPIC_AUTH_TOKEN mode is used as-is. Both credential fields may be left unset to use the caller's existing Claude CLI configuration.

The Codex harness uses CODEX_BASE_URL, CODEX_API_KEY, and CODEX_MODEL. It creates a session-local CODEX_HOME and a custom model provider backed by the configured endpoint, so it does not depend on machine-level Codex login state. Codex sessions require both CODEX_BASE_URL and CODEX_API_KEY.

Runner scripts load .env in override mode. Repository-local values therefore take precedence over same-named variables in the caller's shell.

5. Validate the Environment

bash tools/check_environment.sh

This check imports the real Tinker, E2B, and Harbor execution path and verifies that TINKER_API_KEY and E2B_API_KEY are present. If it fails, rerun bash tools/setup_env.sh, activate .venv, and check .env.

6. Run a One-Task Paid Smoke Test

Use this before a budgeted session to exercise the actual Tinker sampling, E2B proxy, and Harbor path on one task. It does not test the Claude Code/Codex orchestration loop, and it can incur Tinker and E2B charges.

set -a; source .env; set +a
python3 -m backend.e2b_eval \
  --model-path Qwen/Qwen3.5-35B-A3B-Base \
  --run-dir artifacts/runs/smoke-base-qwen35 \
  --base-model Qwen/Qwen3.5-35B-A3B-Base \
  --n-tasks 1 --n-concurrent 1

Success creates artifacts/runs/smoke-base-qwen35/harbor/jobs/*/result.json and tinker_cost_estimate.json. Inspect the result before increasing --n-tasks to 2 or 3 or starting a full session.

Run an Experiment

The main entrypoint is runner/run_session.sh.

Claude Code

RUN_ID=qwen35base-claude-code-opus48-001 \
DATA_AGENT_HARNESS=claude-code \
TARGET_MODEL=Qwen/Qwen3.5-35B-A3B-Base \
TARGET_BENCHMARK=swe-bench-verified \
WALL_TIME_BUDGET_SEC=14400 \
TINKER_COST_BUDGET_USD=25 \
RUN_OFFICIAL_EVAL=1 \
bash runner/run_session.sh

Claude Code with Kimi

For Kimi runs, put the Claude Code provider settings in .env and run the Claude Code harness normally.

ANTHROPIC_BASE_URL=https://api.moonshot.cn/anthropic
ANTHROPIC_AUTH_TOKEN=<YOUR_MOONSHOT_API_KEY>
ANTHROPIC_MODEL=kimi-k3[1m]
ANTHROPIC_DEFAULT_OPUS_MODEL=kimi-k3[1m]
ANTHROPIC_DEFAULT_SONNET_MODEL=kimi-k3[1m]
ANTHROPIC_DEFAULT_HAIKU_MODEL=kimi-k3[1m]
ANTHROPIC_DEFAULT_FABLE_MODEL=kimi-k3[1m]
CLAUDE_CODE_SUBAGENT_MODEL=kimi-k3[1m]
ENABLE_TOOL_SEARCH=false
CLAUDE_CODE_AUTO_COMPACT_WINDOW=1048576
CLAUDE_CODE_EFFORT_LEVEL=max

RUN_ID=qwen35base-kimi-k3-claude-code-001 \
DATA_AGENT_HARNESS=claude-code \
TARGET_MODEL=Qwen/Qwen3.5-35B-A3B-Base \
TARGET_BENCHMARK=swe-bench-verified \
WALL_TIME_BUDGET_SEC=14400 \
TINKER_COST_BUDGET_USD=25 \
RUN_OFFICIAL_EVAL=1 \
bash runner/run_session.sh

Codex

Claude Code and Codex receive the same generated workspace prompt and instruction files. Codex-specific code changes only how that prompt is executed.

RUN_ID=qwen35base-codex-gpt55-001 \
DATA_AGENT_HARNESS=codex \
TARGET_MODEL=Qwen/Qwen3.5-35B-A3B-Base \
TARGET_BENCHMARK=swe-bench-verified \
WALL_TIME_BUDGET_SEC=14400 \
TINKER_COST_BUDGET_USD=25 \
RUN_OFFICIAL_EVAL=1 \
bash runner/run_session.sh

For Claude Code, DATA_AGENT_MODEL defaults to ANTHROPIC_MODEL. For Codex, it defaults to CODEX_MODEL. Generated scripts must use the separate DATA_AGENT_ROLL_BASE_URL, DATA_AGENT_ROLL_API_KEY, and DATA_AGENT_ROLL_MODEL values for direct rollout/API calls.

Confirm a Successful Session

With RUN_OFFICIAL_EVAL=1, a successful command exits with code 0 and prints these key log lines (with run-specific paths):

final_selection=.../agent_workspace/final_selection.json
running official eval for final selection
.../official_eval/official_eval.json

The required final artifacts are:

artifacts/runs/<RUN_ID>/agent_workspace/final_selection.json — a completed attempt was selected.
artifacts/runs/<RUN_ID>/official_eval/official_eval.json — its status is "completed" and return_code is 0.
artifacts/runs/<RUN_ID>/official_eval/harbor/jobs/*/result.json — the Harbor result used by the official evaluation.

For a concise status check, run:

python3 - <<'PY'
import json
from pathlib import Path

run_id = "replace-with-your-run-id"
root = Path("artifacts/runs") / run_id
for path in (
    root / "agent_workspace/final_selection.json",
    root / "official_eval/official_eval.json",
):
    print(path, json.loads(path.read_text()))
PY

RUN_OFFICIAL_EVAL=0 intentionally skips the final evaluation, so a final_selection.json alone means only that a checkpoint was selected; it is not a completed official run.

Additional Benchmark Preparation

The SWE-bench Multilingual profile uses a fixed local seed-23 random 100-task subset generated from SWE-bench/SWE-bench_Multilingual:

python tools/create_swe_multilingual_local_dataset.py \
  --seed 23 \
  --n-tasks 100 \
  --output-dir benchmarks/swe_bench_multilingual/datasets/seed23_random_100 \
  --overwrite

Set TARGET_BENCHMARK=swe-bench-multilingual to use that subset.

For GPQA Diamond, set TARGET_BENCHMARK=gpqa-diamond. The profile uses benchmarks/gpqa_diamond/datasets/seed23_random_100 with terminus-2 and exposes Meta Llama's seed_factories/synthetic-data-kit.

For AIME, set TARGET_BENCHMARK=aime. The profile uses benchmarks/aime/datasets/full_30_2026 with terminus-2, evaluates each task with four attempts, and also exposes seed_factories/synthetic-data-kit.

Session Contract

The runner starts a PostTrainBench-style, non-interactive agent session. DATA_AGENT_REPROMPT=1 is enabled by default. If Claude Code exits before calling final_submit.sh <attempt_id>, the runner invokes claude --continue while wall time and Tinker budget remain. Set DATA_AGENT_REPROMPT=0 only when a single Claude Code turn is intentional.

The generated agent workspace exposes four commands:

timer.sh
budget_status.sh
run_attempt.sh <attempt_id>
final_submit.sh <attempt_id>

timer.sh reports the remaining wall-clock budget.
budget_status.sh reports committed attempt cost plus the estimate for any active attempt.
run_attempt.sh validates, trains, and evaluates one candidate dataset.
final_submit.sh selects the checkpoint that official final evaluation will reuse.

Attempt Inputs

Each attempt must contain:

attempts/<attempt_id>/train_messages.jsonl
attempts/<attempt_id>/run_config.json   # optional

train_messages.jsonl is JSONL. Every line is a JSON object with a messages list containing at least one user message and one trainable assistant message. Assistant messages are trainable unless "loss": false is set.

Before real Tinker training starts, run_attempt.sh executes the same runtime check as tools/check_environment.sh. Missing packages or API keys therefore fail before training budget is spent. It then writes:

attempts/<attempt_id>/tinker_preflight_estimate.json

The preflight file estimates scheduled training tokens and training cost from train_messages.jsonl and run_config.json. Evaluation sampling cost is measured during Harbor evaluation from actual prompt and output tokens.

Run Configuration

Supported run_config.json keys:

{
  "train_max_steps": 1,
  "train_max_length": 65536,
  "train_learning_rate": "1e-4",
  "train_batch_size": 1,
  "train_gradient_accumulation_steps": 1,
  "train_weight_decay": 0.0,
  "train_grad_clip_norm": 0.0,
  "train_lora_rank": 8,
  "train_seed": 0,
  "train_max_records": 0,
  "train_on_all_assistant": false,
  "tinker_proxy_max_context_tokens": 65536,
  "tinker_proxy_max_prompt_tokens": 61439,
  "tinker_proxy_max_tokens": 4096,
  "tinker_temperature": 1.0,
  "tinker_top_p": 0.95,
  "tinker_top_k": 20,
  "eval_step_limit": 200,
  "n_concurrent": 32,
  "n_tasks": 100,
  "eval_timeout_sec": 0
}

Unknown keys are rejected. Proxy context and decoding settings, along with the evaluation step limit, are fixed to match official evaluation; the validator rejects attempts that change those evaluation hyperparameters. The legacy key mini_swe_step_limit remains accepted as an alias for older SWE runs.

Every attempt evaluates the same official setting. When final evaluation is enabled, the runner reuses only the selected Tinker sampler checkpoint and reruns that setting. Harbor task errors and timeouts are recorded as harbor_errors and reflected in the score, but a scored Harbor result remains a completed, submittable attempt. The agent must call final_submit.sh <attempt_id> to select its final checkpoint.

Process and Deadline Safety

Attempts must run in the foreground:

bash run_attempt.sh <attempt_id>

Do not launch attempts with nohup, &, disown, setsid, watcher scripts, or task/waiter mechanisms. If a continuation finds an attempt already running, it must wait for that process rather than launch a duplicate.

The agent must also protect final submission near the deadline:

Before starting a new attempt after a completed attempt exists, compare the remaining wall time with the last full attempt runtime plus a 30-minute submission buffer.
If the next attempt increases training steps, records, LoRA rank, or maximum length, assume it needs at least two hours unless a measurement shows otherwise.
During the final two hours, do not start a heavier or speculative attempt over a completed best attempt; submit the best completed attempt instead.
With less than one hour remaining, immediately run final_submit.sh <best_attempt_id> when any completed attempt exists.

The benchmark profile fixes the official evaluation shape. Attempts always use the selected profile's OFFICIAL_EVAL_N_TASKS, OFFICIAL_EVAL_N_CONCURRENT, and OFFICIAL_EVAL_STEP_LIMIT. SWE-bench Verified evaluates the first 100 tasks in the configured Harbor dataset order by default; Terminal-Bench 2.0 evaluates 89 tasks. Official subsets and public base-run artifacts are analysis-only and cannot be used as training-data sources.

Evaluation Runtime

Each attempt creates and tears down its own E2B sandboxes: one tool-call proxy sandbox and the per-task Harbor sandboxes. The proxy selects its Tinker backend automatically:

Saved tinker://.../sampler_weights/... checkpoints use Tinker's chat-completions endpoint.
Base model names such as Qwen/Qwen3.5-35B-A3B-Base use the Tinker SDK SamplingClient.

The first SDK-backed evaluation builds the fixed E2B template rsibench-data-tinker-proxy; subsequent runs reuse it.

Direct Checkpoint Evaluation

To evaluate a trained checkpoint outside a budgeted session:

set -a; source .env; set +a   # TINKER_API_KEY + E2B_API_KEY
python3 -m backend.e2b_eval \
  --model-path 'tinker://.../sampler_weights/...' \
  --run-dir artifacts/runs/manual-eval \
  --base-model Qwen/Qwen3.5-35B-A3B-Base \
  --harbor-dataset swe-bench/swe-bench-verified \
  --harbor-agent mini-swe-agent \
  --agent-config-profile mini_swe_agent \
  --n-tasks 100 --n-concurrent 32 --step-limit 200

Use the following substitutions for other profiles:

Profile	Dataset	Agent	Additional arguments
Terminal-Bench 2.0	`terminal-bench/terminal-bench-2`	`terminus-2`	`--agent-config-profile terminus_2 --n-tasks 89`
GPQA Diamond	`benchmarks/gpqa_diamond/datasets/seed23_random_100`	`terminus-2`	`--agent-config-profile terminus_2 --n-tasks 100`
AIME	`benchmarks/aime/datasets/full_30_2026`	`terminus-2`	`--agent-config-profile terminus_2 --n-tasks 30 --n-attempts 4`

Direct Base-Model Evaluation

Pass a supported base model name as --model-path to use the SDK backend:

set -a; source .env; set +a
python3 -m backend.e2b_eval \
  --model-path Qwen/Qwen3.5-35B-A3B-Base \
  --run-dir artifacts/runs/manual-base-eval \
  --base-model Qwen/Qwen3.5-35B-A3B-Base \
  --n-tasks 50 --n-concurrent 32

Direct evaluation starts the proxy sandbox, runs Harbor on E2B, and writes harbor/jobs/*/result.json plus tinker_cost_estimate.json under the run directory. Sampling rates default to $0.36/M prefill tokens and $0.89/M sample tokens. Override them with TINKER_PROXY_PREFILL_USD_PER_1M and TINKER_PROXY_SAMPLE_USD_PER_1M.

Artifacts and Accounting

Session outputs are written under artifacts/runs/<run_id>/.

Artifact	Purpose
`agent_initial.jsonl`	Initial Claude Code stdout and stream events
`agent_continue_<n>.jsonl`	Continuation stream events when reprompting is enabled
`agent_custom_command.log`	Output from custom harness commands
`data_agent_usage.json`	Orchestrator usage and reported rollout/API usage
`data_agent_api_usage.jsonl`	Per-response usage from generated rollout/API calls
`harbor/jobs/*/result.json`	Harbor task and aggregate evaluation results
Attempt Harbor trajectories	Evaluated agent behavior and verifier evidence

Generated data scripts that perform rollout/API calls should use runner.lib.claude_usage_client.create_message. The wrapper uses only the DATA_AGENT_ROLL_BASE_URL, DATA_AGENT_ROLL_API_KEY, and DATA_AGENT_ROLL_MODEL configuration and appends one record per response to $DATA_AGENT_API_USAGE_LOG, including model, input_tokens, cache_creation_input_tokens, cache_read_input_tokens, and output_tokens. This usage is retained for experiment accounting and does not consume the Tinker budget.

Failure Recovery

E2B timeout or proxy failure: retain the run directory and inspect the attempt's harbor_errors and Harbor result.json. Confirm the E2B key, account capacity, and transient service status, then run the one-task smoke test again. Do not treat an infrastructure error as a score or repeatedly spend budget on the same failure.
Tinker training failure: inspect artifacts/runs/<RUN_ID>/agent_workspace/attempts/<attempt_id>/tinker_preflight_estimate.json and the matching artifacts/runs/<RUN_ID>-<attempt_id>/ logs. Correct the dataset/configuration issue or account access, then create a new attempt; failed attempts cannot be selected by final_submit.sh.
Budget exhausted: no further attempt can start. If a completed attempt was already selected, run RUN_ID=<RUN_ID> bash runner/run_official_eval.sh to retry only the final evaluation after fixing infrastructure access. If no final_selection.json exists, the session did not complete and needs a new budgeted run.
Claude Code or Codex exits early: DATA_AGENT_REPROMPT=1 (the default) automatically continues the harness while wall time and Tinker budget remain. If the runner exits without final_selection.json, inspect agent_initial.jsonl, agent_continue_*.jsonl, and budget_status.sh. Re-running with the same RUN_ID preserves accounting but does not pause its original wall-clock deadline; use it only while time remains.

Development Checks

Run the shell and unit checks before pushing changes:

bash -n \
  runner/run_session.sh \
  runner/run_official_eval.sh \
  runner/workspace_template/pipeline/run_chain.sh \
  runner/workspace_template/pipeline/train.sh \
  tools/check_environment.sh \
  tools/setup_env.sh

PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests

Repository Layout

backend/e2b_eval/                    E2B proxy and Harbor evaluation runtime
backend/tinker/                      Tinker training, pricing, and cost estimates
benchmarks/                          Benchmark profiles, datasets, and context
docs/tinker_train_eval.md            Tinker training and evaluation runbook
docs/tinker_cookbook.md              Local Tinker API notes for data agents
runner/                              Budgeted session and official-eval entrypoints
runner/lib/                          Validation, accounting, budget, and audit helpers
runner/workspace_template/pipeline/  Fixed training and E2B evaluation chain
seed_factories/                      Read-only synthetic-data reference projects
skills/rsibench-data-factory/        Data-factory operating instructions
tests/                               Unit tests for runner and evaluation contracts
tools/                               Environment and dataset preparation utilities

For deeper implementation details, see docs/tinker_train_eval.md, docs/tinker_cookbook.md, and the RSIBench-Data factory skill.

Navigation Menu

Uh oh!

Repository files navigation

Reproduction: feedback helps discovery, but search often gives gains back

Experiment log

RSIBench-Data

System Overview

Core Properties

Benchmark Profiles

Evaluation Data Boundary

Quick Start

1. Install Runtime Prerequisites

2. Create the Python Environment

3. Initialize Seed Factories

4. Configure Secrets and Models

5. Validate the Environment

6. Run a One-Task Paid Smoke Test

Run an Experiment

Claude Code

Claude Code with Kimi

Codex

Confirm a Successful Session

Additional Benchmark Preparation

Session Contract

Attempt Inputs

Run Configuration

Process and Deadline Safety

Evaluation Runtime

Direct Checkpoint Evaluation

Direct Base-Model Evaluation

Artifacts and Accounting

Failure Recovery

Development Checks

Repository Layout

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.orx		.orx
assets		assets
backend		backend
benchmarks		benchmarks
docs		docs
reports/trajectory-claims		reports/trajectory-claims
reproduction		reproduction
runner		runner
seed_factories		seed_factories
skills/rsibench-data-factory		skills/rsibench-data-factory
tests		tests
tools		tools
.env.example		.env.example
.gitignore		.gitignore
.gitmodules		.gitmodules
README.md		README.md
autoresearch.json		autoresearch.json

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Reproduction: feedback helps discovery, but search often gives gains back

Experiment log

RSIBench-Data

System Overview

Core Properties

Benchmark Profiles

Evaluation Data Boundary

Quick Start

1. Install Runtime Prerequisites

2. Create the Python Environment

3. Initialize Seed Factories

4. Configure Secrets and Models

5. Validate the Environment

6. Run a One-Task Paid Smoke Test

Run an Experiment

Claude Code

Claude Code with Kimi

Codex

Confirm a Successful Session

Additional Benchmark Preparation

Session Contract

Attempt Inputs

Run Configuration

Process and Deadline Safety

Evaluation Runtime

Direct Checkpoint Evaluation

Direct Base-Model Evaluation

Artifacts and Accounting

Failure Recovery

Development Checks

Repository Layout

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages