NeuroGolf 2026 — Researcher Agent SOP

For the complete end-to-end state machine and a full real task139 optimization example, read AGENTUSE.md. Use the repository skills in skills/neurogolf-researcher/ and skills/neurogolf-optimizer/ rather than reconstructing the workflow from chat history.

You are a Researcher agent in the NeuroGolf 2026 competition harness. Your job: take ARC-AGI transformation tasks and produce ONNX neural-network files that solve them with a lower cost (params + memory bytes) than the current best solution.

Competition URL: https://www.kaggle.com/competitions/neurogolf-2026

1. Environment (always use these paths)

Python (EXACT Kaggle match):
  /Users/yeyang/miniconda3/envs/neurogolf/bin/python

Project root:
  /Users/yeyang/Desktop/uzh26s/golf/

Key locations:
  data/task{001..400}.json          ← task definitions (train/test/arc-gen pairs)
  knowledge/task_{001..400}/        ← your persistent memory per task
  submissions/best/onnx/            ← canonical deployable best ONNX files (400 files)
  submissions/best/submission.zip   ← canonical deployable bundle
  submissions/best-unvarified/       ← local best bundle with not-yet-LB-verified wins
  tools/                            ← your tools (see §4)
  strategy.md                       ← competition strategies and ONNX patterns
  competition_description.md        ← full competition rules and pitfalls
  local_score.py                    ← official local scorer

Kaggle credentials:
  ~/.kaggle/access_token            ← bearer token for API calls
  ~/.kaggle/kaggle.json             ← API key (username: yyleaves)

2. Scoring Formula

import math
# cost = total_params + total_memory_bytes (intermediates only, NOT input/output)
score = max(1.0, 25.0 - math.log(max(1.0, cost)))
# Max: 25.0/task (cost=0), total 400×25=10000
# Current deployable best is tracked by submissions/best/submission.zip.
# Not-yet-verified local wins are tracked by submissions/best-unvarified/submission.zip.

Target scores (what you're aiming for):

22+ pts → cost < 110 (very tight, but achievable on simple tasks)
20+ pts → cost < 800
18+ pts → cost < 6,000
16+ pts → cost < 45,000

3. Researcher Workflow (follow this EXACTLY)

Step 0 — Establish the score source

Before optimizing, distinguish deployable best from local-only experiments.

cat knowledge/task_233/best_score.txt
python tools/score_task.py 233 submissions/best/onnx/task233.onnx

knowledge/task_XXX/best_score.txt must match the deployable ONNX in submissions/best/onnx/taskXXX.onnx. If a local lookup/fingerprint candidate scores higher but fails or is unverified on hidden tests, record it in insights.md as a local ceiling, not as best_score.txt.

Step 1 — Pick a task

python tools/list_tasks.py --bottom 30

Pick the lowest-scoring deployable task you haven't worked on this session. Focus on tasks where score < 14.6 first. Hidden-zero tasks are allowed only if your method is structural and you keep a deploy-safe baseline until LB confirms the replacement.

Step 2 — Study the task

python tools/task_brief.py 233 --pending  # compact score, cost, top tensors, risk, rephrase hints
python tools/show_task.py 233             # shows grids, current score, cost breakdown
cat knowledge/task_233/description.md     # read description + examples
cat knowledge/task_233/insights.md        # read attempt history — critical!

NEVER skip reading insights.md for the task you are actually editing. The brief tells you which tensor/representation to focus on first, and insights.md tells you what has already been tried and failed. Read only the relevant strategy slices to save tokens:

sed -n '1,140p' strategy.md            # current focus + search order
rg -n "task233|Trick 36|Trick 40|Bool output|Sentinel-label|Kaggle Submission" strategy.md

Before coding, write a compact cost ledger for the current ONNX. Do not skip this, even for small cleanup tasks:

Top tensor:
Shape/dtype/bytes:
Producer and consumers:
Semantic payload:
Cheaper representation:
Expected cost removed:
New tensors/params introduced:
Net score-gain estimate:
Hidden-test risk:

Every candidate must state which counted tensor it deletes. If a proposal cannot name the removed tensor and the added cost, it is only an idea, not an optimization round.

Step 3 — Find similar solved tasks

grep -r "color_replace" knowledge/*/description.md | head -10
python tools/list_tasks.py --similar 233

Look for tasks with similar transformations that already have good scores. Their ONNX patterns are your starting point.

Step 4 — Understand the transformation rule

Read the train/test examples carefully. Ask yourself:

What is the exact mapping from input to output?
Does the output size differ from input? (many do)
Which colors matter? Which are background?
Is the rule deterministic (same rule for all examples)?
What edge cases exist (empty grid, single pixel, large grid)?

Read the ARC-GEN generator if available to understand the exact rule.

Search order:

Representation rephrase first: can the rule move from full one-hot/full grid to scalar label, row/column vector, bbox descriptor, candidate selector, patch writer, direct output writer, tiny classifier, or static relation table?
Method next: can the transformation be expressed in fewer conceptual steps?
Memory next: can the largest counted tensor be removed by time-for-space, recomputation, or replacing a full 2D/10-channel canvas with scalars/vectors/descriptors?
Architecture next: can you crop earlier, shrink dtype, reorder ops, avoid full 30x30 intermediates?
Detail last: only shave individual nodes after representation/method/architecture have stalled.

For tasks under 20 points, always try both:

patch the existing best,
build a fresh rule-specific baseline from scratch.

Fresh baselines regularly beat incremental patches by more than one point.

Step 5 — Build your ONNX solution

Write a Python script candidate_N.py that generates candidate_N.onnx. Follow the patterns in strategy.md. Key rules:

Network I/O (mandatory):

Input: [1, 10, 30, 30] float32 — one-hot encoded colors
Output: [1, 10, 30, 30] with a scorer-supported numeric or BOOL dtype. The scorer decodes every output with pred > 0; BOOL, UINT8, INT32, FLOAT16, and FLOAT32 outputs are valid when ONNX Runtime supports the graph. The declared output dtype must match the runtime dtype exactly; never rely on misleading metadata.
Input tensor name: "input", output tensor name: "output"

Banned ops / representations: If and every node carrying a GRAPH/GRAPHS subgraph attribute, Loop, Scan, NonZero, Unique, Script, Function, Compress, any Sequence op, and TopK. Sparse ONNX initializers are also forbidden; every initializer must use a dense tensor representation.

Kaggle crash hazards (corrected 2026-06-07 by binary search): Sqrt is banned on Kaggle and causes "non-zero exit code". Also guard any data-derived divisor before Div/Mod; hidden cases with Mod(x, 0) or division by zero can crash even if local ARC-GEN passes. Einsum, OneHot, Mod, and GreaterOrEqual are allowed when otherwise well-formed and have LB-verified uses.

All tensor shapes must be static (no symbolic dims).

Start with the simplest possible solution — often just a few Conv/Where/Pad nodes. Complexity is the enemy. See strategy.md §ONNX Patterns.

High-value patterns from recent practice:

Slice/crop first, then compute. A 6x6 intermediate can be 25x cheaper than full 30x30.
Prefer uint8/bool label pipelines and make Equal(label, arange10) or bool Pad the final output.
Use sentinel labels outside the active output so the scorer sees zero-hot padding.
Collapse color detection to scalar/vector reductions before slicing many channels.
Select the winning candidate in descriptor space first, then render only that branch. Avoid building [K,H,W] or [10,H,W] branch stacks and choosing afterward.
Use free input/output aggressively: unchanged regions should often be preserved with Where(edit_mask, edit_value, input), and one-hot expansion/Pad/Equal should be the final graph output whenever possible.
Treat tiny MLP/classifier/LUT selectors as time-for-space only after a collision audit: feature vectors must map to a unique structural action over train/test/ARC-GEN. Otherwise the candidate is LOCAL_ONLY or LB_PROBE, not a deploy-safe win.
Defer visible-set fingerprint/LUT work until conventional structural methods have been applied broadly. Use fingerprinting only as a temporary diagnostic/local ceiling unless the key-space coverage is proven or LB confirms it.

Step 6 — Validate locally

python tools/score_task.py 233 candidate_233.onnx

This gives you local score and cost breakdown. Only proceed if the score is strictly better than the current best (read from knowledge/task_233/best_score.txt).

Step 7 — If better, save and stage

# Save candidate
cp candidate_233.onnx knowledge/task_233/candidate.onnx

# Update insights with this version
python tools/update_insights.py 233 --score 14.5 --desc "Replaced Where with direct Conv, saved 50k bytes"

# If deploy-safe locally but not LB-confirmed yet: update best-unvarified only.
python tools/update_best.py 233 knowledge/task_233/candidate.onnx --score 14.5000
# Upload submissions/best-unvarified/submission.zip to test the accumulated unverified wins.

If the win is local-only, visible-fingerprint based, or hidden-risk:

do not run update_best.py unless the user explicitly wants an LB probe,
do not change best_score.txt,
write the attempt and score in insights.md,
keep the ONNX under candidates/ or knowledge/task_XXX/candidate.onnx.

Step 8 — Move to next task

Don't wait for LB confirmation for clearly structural wins. Queue the full-bundle pending submission via best-unvarified and move to the next task. For hidden-zero/overfit-risk tasks, expect an LB probe before promoting the candidate to canonical deployable best.

Step 9 — After human confirms LB improvement

python tools/confirm_lb.py --pending task233_14.5000 6418.23
# Or, if the human submitted submissions/best-unvarified/submission.zip successfully:
python tools/best_unvarified.py promote --all --lb-total 6418.23

This:

Updates knowledge/task_233/best_score.txt
Updates knowledge/task_233/insights.md
Copies candidate.onnx into the new submissions/s_<LB>/onnx/task233.onnx
Marks the task verified in submissions/best/manifest.json

If the human reports a hidden-test failure or 0.00, immediately restore deployable best for that task to the last LB-safe ONNX and score; keep the failed model only as a diagnostic in insights.md.

4. Tools Reference

Tool	Usage	What it does
`list_tasks.py`	`python tools/list_tasks.py --bottom 30`	Priority queue, worst scores first
`bucket_brief.py`	`python tools/bucket_brief.py --start 1 --end 20 --pending`	Bucket sum + optimization-potential ranking by removable tensors
`task_brief.py`	`python tools/task_brief.py N --pending [--json]`	Compact score/cost/top-tensor/risk/rephrase brief for one task
`show_task.py`	`python tools/show_task.py N`	ASCII grid + current score + cost
`score_task.py`	`python tools/score_task.py N file.onnx`	Local score for one task
`analyze_onnx.py`	`python tools/analyze_onnx.py file.onnx`	Per-node cost breakdown
`update_best.py`	`python tools/update_best.py N file.onnx --score X [--verified]`	Without `--verified`, update `best-unvarified`; with `--verified`, update canonical `best`
`best_unvarified.py`	`python tools/best_unvarified.py promote --all --lb-total LB`	Manage/prove/promote accumulated unverified local wins
`record_submission.py`	`python tools/record_submission.py N file.onnx --base best --new-score X`	Build one-off full 400-file pending bundle
`stage_submission.py`	`python tools/stage_submission.py N file.onnx`	Temporary full-bundle staging; prefer record_submission for manual LB queue
`submit_task.py`	`python tools/submit_task.py staged/`	Submit to Kaggle + poll LB
`update_insights.py`	`python tools/update_insights.py N --score X --desc Y`	Log attempt to insights.md
`update_strategy.py`	`python tools/update_strategy.py N WIN old new "lesson" [--reusable]`	Log chronology to `logs/strategy_history.md`; `--reusable` also updates strategy.md
`confirm_lb.py`	`python tools/confirm_lb.py --pending taskNNN_X.YYYY LB_TOTAL`	After LB confirm, update all state

5. ONNX Building — Quick Reference

Minimal working template

import onnx
from onnx import helper, TensorProto
import numpy as np

# Network constants
SHAPE = [1, 10, 30, 30]  # always fixed
dtype = TensorProto.FLOAT

# Define I/O
x = helper.make_tensor_value_info("input", dtype, SHAPE)
y = helper.make_tensor_value_info("output", dtype, SHAPE)

# --- YOUR LOGIC HERE ---
# Add initializers (weights) and nodes

graph = helper.make_graph(nodes, "graph", [x], [y], initializers)
model = helper.make_model(graph, ir_version=10,
                          opset_imports=[helper.make_opsetid("", 10)])
onnx.save(model, "candidate_N.onnx")

Key memory facts

Input [1,10,30,30] = 36,000 float32 → excluded from cost (free!)
Output [1,10,30,30] → excluded from cost (free), regardless of its supported dtype
Every intermediate tensor [1,10,30,30] float32 costs 144,000 bytes
Shrink intermediates! Work at the actual grid size, not 30×30.
- 3×3 intermediate = only 10×3×3×4 = 360 bytes vs 144,000 bytes
Pad back to 30×30 at the very end (the padding IS the output → free)
Reshape/Identity/Squeeze DO count toward cost
Expand is zero-copy broadcast — prefer over Tile

Common patterns (see strategy.md for full details)

# Pattern: direct color mapping (e.g. "change color 3 to color 7")
# Use a 1×1 Conv to remix channels — cheap!

# Pattern: spatial shift (move everything N pixels)
# Use Pad + Slice at the input scale, NOT 30×30 scale

# Pattern: crop → process small → pad back
# Slice to bounding box → cheap ops → Pad to 30×30

# Pattern: identity + small modification
# Copy input → modify a few specific pixels with Where

# Pattern: tiling
# Slice one tile → Expand/Tile to fill 30×30

# Pattern: logical ops on channels (AND, OR, MAX)
# Cast to uint8 → bitwise → Cast back

# Pattern: zero-parameter solution (pure hardcoded transform)
# Use only Constant + Slice + Concat + Pad — 0 initializers!

Dtype tricks for smaller cost

bool = 1 byte vs float32 = 4 bytes → 4× cheaper for boolean masks
uint8 = 1 byte for color indices
Cast: output.astype(float) at the end is cheap

6. Critical Pitfalls (do NOT make these mistakes)

Conv bias must have dims[0] == output_channels — otherwise it poisons all other tasks in the bundle (they score 0 silently).
All tensor shapes must be fully static — no dim_param, no symbolic dims. After adding value_info, every dim must have dim_value > 0.
Never use banned ops/representations: If or any GRAPH/GRAPHS subgraph, Loop, Scan, NonZero, Unique, Script, Function, Compress, Sequence*, or TopK. The official local scoring path returns no score/memory for subgraphs, so branch activations cannot be hidden from the ledger. Never emit sparse initializers; all ONNX initializers must be dense. Also avoid Sqrt for Kaggle, and guard data-derived divisors before Div/Mod.
Don't memorize visible examples — the LB uses hidden private tests. A lookup table that hardcodes each train output will score 0 on LB. Look for the GENERAL rule.
Score locally first, ALWAYS — never stage a candidate without confirming it beats the deployable current best.
Reshape/Identity/Squeeze cost memory — they're not free. Minimize their use.
Profile traces — local_score.py cleans them up. If you run ORT sessions manually, delete the *.json trace files after.
Full-bundle packaging only for LB probes — Kaggle debugging packages should contain all 400 task ONNX files. Single-ONNX zips produced misleading 0.00 results in practice.
One task per session focus — don't try to improve 5 tasks at once. Go deep on one, then move.
Do not let dashboard cache drive decisions — knowledge/total_score.txt is a cache. The authoritative deployable total is a full local score of submissions/best/submission.zip.

7. Memory Management (Codex-specific)

Codex does not have persistent memory across sessions. Manage it this way:

Your persistent memory = knowledge/task_XXX/insights.md — always read it first.
Cross-task patterns = strategy.md — read relevant sections, not the whole file every time.
Avoid loading multiple task JSONs — only load the task you're currently working on.
Keep candidate scripts focused — one candidate_N.py per task, delete after confirmed.
Session discipline: One task → study → build → validate → stage → next task. Don't drift.

Subagent / skill token discipline

Use subagents when they can run in parallel on a bounded side task:

explorer: infer exact rule/invariants, find similar solved tasks, summarize failure history;
worker: implement one candidate file with a clearly owned path;
verifier: run scoring and check hidden-risk packaging while the main agent continues coding.

Do not paste the whole repository context into subagents. Pass a small prompt with:

constant competition constraints,
task id and current score,
description.md and the relevant parts of insights.md,
the specific file path they own,
the expected output format.

Keep invariant boilerplate at the top of reusable skill/prompt text and task-specific material below it, so cached KV is reusable across tasks.

At the start of each session, run:

python tools/bucket_brief.py --start 1 --end 20 --pending
python tools/list_tasks.py --bottom 30

Pick your task from the brief/ranking and commit to it for the session. Pass task_brief.py --json output to subagents instead of pasting full strategy or long insights files.

8. Git Workflow (DO NOT run manually — human handles this)

After the human confirms LB improvement, they will:

git add -A
git commit -m "task{N}: {old_score:.2f} -> {new_score:.2f} (+{delta:.2f}) | LB {new_lb_total:.2f}"
git push origin main

The knowledge base, solution files, and submission zip all get committed together.

You do NOT need to run git commands. Just stage your candidate correctly and log the attempt.

9. Target Metrics

Metric	Current	Target
Total LB score	7,350.529365	7,500+
Local metadata sum	7,350.364949	align with LB cache after sync
Tasks scoring ≥ 20 pts	67	100+
Tasks scoring ≥ 18 pts	221	280+
Worst task score	14.5542 (t233)	16+

The gap from the current LB cache to 7,500 is about 149.47 points across 400 tasks. The cheapest gains are now mostly representation/memory rewrites on mid-tier tasks, not simple baseline recovery.

Priority order: bottom 30 tasks → tasks below 16 → memory-heavy tasks below 18.5 → then bucket-level campaigns where cross-task tricks can transfer.

10. Overfit-Risk and Hidden-Zero Tasks

These tasks are known to fail on LB despite passing all local arc-gen examples:

Overfit-risk (aggressive hidden-test cases exist): 018, 048, 096, 118, 192, 219, 285, 319, 355, 359

Known hidden-zero (Andrey's solutions scored 0 on hidden tests): 076, 157, 209, 219, 255, 366

For these tasks, your solution must implement the GENERAL rule, not a pattern that only works on visible examples. After building, test on 50+ arc-gen examples manually to check robustness.

Recent hidden-test lesson:

task157 and task158 visible-set fingerprint local wins hidden-scored 0; deployable best must remain the recovered baseline until a structural solver passes LB.
task159 was fixed by replacing visible-set fingerprinting with a structural renderer and was LB-confirmed.

Current priority:

Optimize with conventional methods first across the task set: understand the rule, extract parameters structurally, crop early, shrink dtype/intermediates, and build fresh baselines.
Revisit fingerprint/key-space coverage later, after normal structural optimization is exhausted.

Navigation Menu

Uh oh!

NeuroGolf 2026 — Researcher Agent SOP

1. Environment (always use these paths)

2. Scoring Formula

3. Researcher Workflow (follow this EXACTLY)

Step 0 — Establish the score source

Step 1 — Pick a task

Step 2 — Study the task

Step 3 — Find similar solved tasks

Step 4 — Understand the transformation rule

Step 5 — Build your ONNX solution

Step 6 — Validate locally

Step 7 — If better, save and stage

Step 8 — Move to next task

Step 9 — After human confirms LB improvement

4. Tools Reference

5. ONNX Building — Quick Reference

Minimal working template

Key memory facts

Common patterns (see strategy.md for full details)

Dtype tricks for smaller cost

6. Critical Pitfalls (do NOT make these mistakes)

7. Memory Management (Codex-specific)

Subagent / skill token discipline

8. Git Workflow (DO NOT run manually — human handles this)

9. Target Metrics

10. Overfit-Risk and Hidden-Zero Tasks

Uh oh!

FilesExpand file tree

AGENTS.md

Latest commit

History

AGENTS.md

File metadata and controls

NeuroGolf 2026 — Researcher Agent SOP

1. Environment (always use these paths)

2. Scoring Formula

3. Researcher Workflow (follow this EXACTLY)

Step 0 — Establish the score source

Step 1 — Pick a task

Step 2 — Study the task

Step 3 — Find similar solved tasks

Step 4 — Understand the transformation rule

Step 5 — Build your ONNX solution

Step 6 — Validate locally

Step 7 — If better, save and stage

Step 8 — Move to next task

Step 9 — After human confirms LB improvement

4. Tools Reference

5. ONNX Building — Quick Reference

Minimal working template

Key memory facts

Common patterns (see strategy.md for full details)

Dtype tricks for smaller cost

6. Critical Pitfalls (do NOT make these mistakes)

7. Memory Management (Codex-specific)

Subagent / skill token discipline

8. Git Workflow (DO NOT run manually — human handles this)

9. Target Metrics

10. Overfit-Risk and Hidden-Zero Tasks