NeuroGolf 2026 — Competition Notes

Last updated: 2026-06-06
Best LB score: 6256.03 (submission s_6256_03)
Local reproduction: 6256.04 ✓ (exact match with Kaggle ORT 1.24.4)

What the Competition Is

Code-golf for neural networks. For each of 400 ARC-AGI tasks (grid transformations — rotate, crop, magnify, fill, etc.) you submit one ONNX file that correctly computes the transformation. The objective is to make each network as small/cheap as possible.

Score per task = max(1, 25 - ln(cost))
where cost = total parameters + total memory footprint (bytes) of all intermediate tensors (input and output tensors are excluded from memory).

Cost	Score	Notes
0–1	25.0	Maximum. Genuinely free tasks: 179, 241
≈10	22.7	tasks 016, 276, 309, 337
≈30	21.6	tasks 053, 113, 116, 164, 172, 210, 311, 385
≈900	18.2	—
≈12,000	15.6	—
≈163,000	13.0	—

400 tasks × 25 pts max = 10,000 theoretical maximum. Practical ceiling estimated at ~7,900–8,100 by top competitors.

Correctness: Kaggle only runs your ONNX against a private hidden test set (not the public train/test/arc-gen). You must implement the general rule, not just memorize the visible examples.

Pinned Environment (Official)

From discussion #693088 (Tony Li, 4th place, confirmed):

Python:        3.12
numpy:         2.4.4
onnx:          1.21.0
onnxruntime:   1.24.4
onnx-tool:     1.0.1   (debug/display only — NOT used in scoring since May 4)

Local setup — neurogolf conda env at /Users/yeyang/miniconda3/envs/neurogolf/:

# Create (one-time):
conda create -n neurogolf python=3.12 numpy=2.4.4 -y
/Users/yeyang/miniconda3/envs/neurogolf/bin/pip install -r requirements.txt

# Run scorer:
/Users/yeyang/miniconda3/envs/neurogolf/bin/python local_score.py \
  --submission submissions/s_XXXX_XX/submission.zip --data data/

# Score a single task:
/Users/yeyang/miniconda3/envs/neurogolf/bin/python local_score.py \
  --task 233 --onnx submissions/s_6256_03/onnx/task233.onnx --data data/

All 4 packages now exactly match Kaggle. Local score = 6256.04 vs LB 6256.03 ✓.

Data Format

401 files: task{001..400}.json + neurogolf_utils/neurogolf_utils.py (~97 MB total, Apache 2.0).

Each task JSON has three fields:

train: 2–10 input/output grid pairs (ARC-AGI original, human-crafted)
test: 1–3 input/output pairs (ARC-AGI original)
arc-gen: ~4–262 pairs (ARC-GEN-100K synthetic augmentations, for local validation)

A grid is a list-of-lists of integers 0–9 (colors). Maximum 30×30.

Network I/O is always [1, 10, 30, 30] — batch 1, one-hot 10 colors, 30×30 padded. Smaller grids embed in the top-left corner; the rest is zero-hot ("clear").

input[0][color][row][col] = 1.0   for each grid cell
output[0][color][row][col] = 1.0  for correct channel; 0 elsewhere

6 oversize tasks (inputs >30×30) — scorer silently skips those examples:
021, 055, 080, 184, 202, 366
Do not assume all arc-gen examples count for these tasks.

ONNX Constraints

Rule	Detail
Static shapes only	No symbolic dims, no missing `dim_value` after shape inference → score 0 if violated
Banned ops	`Loop`, `Scan`, `NonZero`, `Unique`, `Script`, `Function`, `Compress`, any `Sequence*` op
File size	≤ 1.44 MB per task
Opset domain	Only `""` or `"ai.onnx"` — no custom domains
Graph I/O	Exactly 1 input (named `"input"`) and 1 output (named `"output"`)
Initializer names	Cannot be `"input"` or `"output"` (would intersect I/O names → rejected)
No subgraphs	No `GRAPH`/`GRAPHS` node attributes (no functions, no local subgraphs)
Tensor uniqueness	No duplicate `value_info` entries with the same name
No `kernel_time`	Node output names containing this string are rejected
Positive dims	All `dim_value > 0` — zero or negative dims → score 0
IR version	Supported up to IR 13 (requires ORT 1.24.4)

Scoring Implementation (neurogolf_utils.py)

# Score formula
cost = memory_bytes + params
score = max(1.0, 25.0 - math.log(max(1.0, cost)))

# What counts as params:
# - All initializer elements (dims=[] scalar → 1)
# - All Constant node attribute elements (t.dims, value_floats, value_ints, value_strings)
# - Sparse initializer nnz elements

# What counts as memory:
# - All intermediate tensors (excludes the graph input and output)
# - Memory = max runtime shape across all examples × dtype bytes
# - Determined by ORT profiling trace (JSON), NOT static shape alone
# - Reshape/Flatten/Identity/Squeeze/Unsqueeze all count (not zero-cost!)

Key: memory is max across all examples for each tensor — optimize for the worst-case (largest) input example.

Critical Pitfalls (from community discussion)

⚠️ #1 — Conv Bias Length Bug (bundle-poisoning, discussion #699840)

This is the most dangerous silent bug. If a Conv node's bias tensor has dims[0] < output_channels, ORT reads stale heap memory and corrupts the scoring of all subsequent tasks in the same bundle — they silently score 0, even though they're correct when submitted alone.

# WRONG: bias has fewer elements than output channels
Conv(input, W, bias)  # where bias.shape[0] < W.shape[0]

# CORRECT: bias.shape[0] == W.shape[0]

How to detect: local_score.py will print a WARNING if it finds this pattern. Validate the full bundle, not just single files. If a task scores correctly alone but contributes 0 in the bundle — check this.

⚠️ #2 — Dynamic Shapes → Score 0

Any tensor with a dim_param (symbolic, e.g., "N") or missing dim_value → calculate_memory() returns None → score 0.

# Check locally with onnx shape inference:
graph = onnx.shape_inference.infer_shapes(model, strict_mode=True).graph
for vi in list(graph.value_info) + list(graph.output):
    for dim in vi.type.tensor_type.shape.dim:
        assert dim.HasField("dim_value"), f"Symbolic dim in {vi.name}"

⚠️ #3 — Hidden Test Scoring (not arc-gen!)

Kaggle runs your ONNX only on a private hidden test set, NOT the public train/test/arc-gen. The May 4 update notes: "scoring timeouts — now only hidden tests are executed."

Implications:

A network that memorizes train/test/arc-gen may score 0 on hidden tests
Spotting overfitting: look for N similar nodes where N = number of visible tests (= lookup table pattern — @jacekwl 3rd place)
Solutions passing all arc-gen locally can still fail hidden tests if they overfit
@AndreyYunoshev found several tasks where his solution scored 0 on Kaggle (he calls them "hidden zero" = HZ): tasks 076, 157, 209, 219, 255, 366 (for his solutions)

Overfit-risk tasks (community-reported, prone to hidden-test failures):
018, 048, 096, 118, 192, 219, 285, 319, 355, 359

⚠️ #4 — Scalar Tensors/Params Count as 1

math.prod([]) = 1, not 0. A scalar constant contributes 1 parameter.

⚠️ #5 — onnx-tool "Poisoned" Ops (for display/debug only)

onnx-tool is no longer used in scoring (eliminated May 4), but neurogolf_utils.verify_network() still calls onnx_tool.model_profile() for display. These ops cause onnx-tool to crash or return wrong results:

Op	Issue	Workaround
`Min`	`not numpy.minimum` typo → crash	Use `Greater+Cast`, or `Neg+Max+Neg`
`ArgMin`	No `value_infer`	Use `ArgMax(Neg(x))`
`Clip` (opset 10 attr-form)	Crashes	Upgrade to opset 11 with min/max as inputs
`ArgMax(keepdims=0)` + Reshape to [1]	Volume-of-scalar assert	Use `keepdims=1` or add Unsqueeze
Scalar-index `Gather` on rank-4	Phantom dim	Use rank-1 [1] index + Reshape
`BitShift`	NotImplementedError	Now safe (onnx-tool eliminated from scoring)
`ArgMin`	No profile override	Use `ArgMax(Neg(x))`

⚠️ #6 — ORT Profiling Trace Cleanup

ORT profiling writes a timestamped JSON trace per session. These accumulate to >150 GB if not deleted. local_score.py cleans them automatically. If you write custom scoring scripts, always call os.remove(session.end_profiling()).

⚠️ #7 — Bundle Ordering / Stale ORT State

A malformed ONNX anywhere in your zip can corrupt later tasks' scores in the same run (same Python process, stale ORT heap state). Always validate the complete bundle with local_score.py --submission, not just individual files.

⚠️ #8 — ARC-GEN Insoluble Samples

For some complex tasks, ARC-GEN generates inputs with ambiguous (multiple valid) outputs or same input → different outputs. These cause false failures locally. The host accepts solutions that pass the hidden private tests even if some arc-gen samples are "insoluble."

⚠️ #9 — Forum Prompt Injection (discussion #695746)

The Kaggle forum contains deliberate bait for AI agents — posts with sudo reboot, rm -rf *, fake submission.zip attachments. If you feed forum content to an agent, sanitize it first.

⚠️ #10 — Per-task vs Bundle Score Divergence (discussion #702256)

Same file scored 18.60 alone but only contributed 0.42 in the 400-task bundle. Root cause: a malformed earlier network (Conv bias bug) corrupted ORT state. Validate both ways.

Effective ONNX Design Patterns

Core principle

Top solutions are deterministic ONNX programs (not trained neural nets). The task is program synthesis — encode the transformation rule as directly as possible in tensor ops.

Memory optimization tricks

Output tensor is free. Input and output tensors are excluded from memory cost. This means Pad-to-30×30 at the end is essentially free. Pattern:

crop to small intermediate R (e.g., 3×3 or object bounding box)
← process R cheaply →
Pad R back to 30×30 (free-ish)

Keep intermediate tensors as small as possible.

Dtype matters. bool = 1 byte, uint8 = 1 byte, int32 = 4 bytes, float32 = 4 bytes. Use smaller dtypes for intermediate tensors. (Early solvers found 4× improvement by switching output to bool then casting.)

Expand is zero-copy broadcast (like PyTorch's expand_as) — no new memory allocation. Prefer over Tile where possible.

Reshape/Flatten/Identity/Squeeze/Unsqueeze DO count toward memory — they're not free. Use them sparingly.

Dynamic cropping pattern (from hengck23 #695972): If the object size varies per example but is bounded, crop to a fixed maximum size using dynamic Slice, then statically annotate the value_info shape. Since memory is max-over-examples, this is valid — ORT verifies the actual runtime shape.

Recommended ONNX ops

Op	Use for
`Conv`	Spatial filtering, pattern detection, sliding-window logic
`MaxPool`	Max aggregation over regions, morphological ops
`Where`	Conditional masking (now correctly scored)
`Gather` / `GatherND`	Color lookup tables, index remapping
`ArgMax`	Find dominant color channel
`Pad`	Expand to 30×30 (excluded from memory cost if at output)
`Reshape`, `Transpose`	Rearrange tensor structure
`Slice`	Crop regions
`OneHot`	Encode class indices back to channels
`MatMul` / `Gemm`	Batch linear transforms
`BitShift`, bitwise ops	Bit-plane encoding (cuts memory ~8×); now fully safe
`Expand`	Zero-copy broadcast

Tip from hengck23 (#694628): Don't just implement the transformation — reformat the problem so it becomes trivial. Example: reshape a flattened diagonal-pattern grid to (-1, 3) so diagonals become vertical lines, then max over axis.

Approach that works (from Andrey Yunoshev #703914, 46th place, ~6580 pts)

Understand the transformation rule (read arc-gen generator code if needed)
Write a NumPy oracle first, validate on all train/test/arc-gen
Translate to ONNX, then compress
Iterate with LLM (~5 passes per task to approach the optimum)
Agents rush to close too early — let them revisit with history

Model benchmarks (from Chet #703462, 10 task controlled experiment):

GLM-4.7: 0/10 deploy-safe wins
GLM-5.1: 6/10
Opus: 9/10 (did operator-level reformulations, solved hardest task)
Claude produces better/nicer solutions but drifts from harness instructions
Codex is more stable and grinds non-stop

Current Score Analysis (s_6256_03)

400/400 tasks scored. Local = 6256.04 = Kaggle LB 6256.03 ✓

Score distribution:

[10-12): 4 tasks   — task018, task233, task255, task366 (hardest)
[12-14): 89 tasks  — biggest improvement bucket
[14-16): 162 tasks — largest bucket overall
[16-18): 77 tasks
[18-20): 45 tasks
[20-22): 8 tasks   — very efficient
[22-25): 6 tasks   — near-optimal (cost ≤ 10)
  25.0 : 2 tasks   — task179, task241 (zero-cost, legitimately free)

Zero-cost tasks (score = 25): task179, task241 — these are legitimately free (identity or constant output).

Near-perfect (score > 22): task016, task276, task309, task337 (score 22.70, cost ≈ 10)

Bottom 20 — highest ROI for improvement

Task	Score (local)	Cost (est.)	File Size	Notes
255	10.66	1,685,209	297 KB	Hard, community also struggles
233	10.68	1,659,621	321 KB	Hard, community also struggles
366	11.36	836,850	84 KB	One of 6 oversize tasks
018	11.93	476,203	29 KB	Overfit-risk!
285	12.09	402,641	51 KB	Overfit-risk!
096	12.83	193,837	59 KB	Overfit-risk!
101	12.17	374,445	20 KB	—
076	12.29	330,083	25 KB	Hidden-zero risk (Andrey)
133	12.81	196,732	10 KB	Tony Li also struggles
158	12.83	193,436	27 KB	Tony Li also struggles
173	12.93	175,238	10 KB	—
077	13.03	157,913	8 KB	—
025	13.15	128,543	14 KB	Tony Li also struggles
398	13.66	84,295	9 KB	—
074	15.01	21,825	2 KB	Timeout-risk task
286	13.00	162,183	20 KB	Tony Li also struggles
054	13.08	130,781	6 KB	Tony Li also struggles

Tasks marked "Tony Li also struggles" are validated by 4th-place competitor as genuinely hard.
Tasks marked "Overfit-risk" are prone to failing hidden tests — re-verify before optimizing cost.
Tasks marked "Hidden-zero risk" may already fail Kaggle's hidden tests (score 0 despite local pass).

Timeout-risk tasks (whole submission ≤ 30 min)

009, 022, 070, 074, 212, 246, 335, 350, 358, 375

Community Score Benchmarks

Rank	Score	Key insight
#1 (CroDoc)	~7,700	~19.25 avg/task (~665 bytes avg cost)
#4 (Tony Li)	~7,500	101 tasks ≥20 pts; sets target at 7750
#9 (yashbhaskar)	~7,200	"Token is all you need"
#46 (Andrey)	~6,580	Most detailed public methodology post
Us	6,256	400 tasks scored
Public baseline	6,029	Chet's public notebook (kaggle.com/code/jsrdcht)

Practical ceiling: ~7,900–8,100 (multiple competitor estimates). At rank 1 (~7,700), average cost ≈ 665 bytes/task. We're at average ≈ 8,950 bytes/task — ~13× more expensive than the leader.

Key insight (@jacekwl 3rd place): averages are misleading. His median cost is ~860 bytes but average ~7,200 — dominated by a few expensive tasks. Most tasks are already very small; the worst tasks dominate the gap.

What Separates Top Solutions

From discussion and benchmark data:

Program synthesis, not trained nets. Top solutions are hard-coded ONNX transformations, often with zero learned parameters. @T.-C. Chang: big score boost from "bypassing CNNs entirely — shift vectors are constant, so compile pure functional tensor math (0 parameters)."
Read the ARC-GEN generator code (github.com/google/ARC-GEN). The generator .py for each task tells you the exact rule, fixed colors, and invariants. This eliminates guessing and overfitting.
The rule → compact expression loop beats everything else tried (training nets, "give me 10 ideas," auto-solving). Understand the rule exactly, then express it in as few ONNX nodes as possible.
Minimize intermediate tensor size. The scoring is: fewer bytes in intermediate tensors = better. Work with the smallest possible representation between input and output.
Per-task optimization depth. Each task needs ~5 agent passes to approach its optimum. Agents settle too early. Keep revisiting with history.

Submission Convention

submissions/
  s_6256_03/               ← score 6256.03, attempt 3
    submission.zip         ← always named exactly this (Kaggle requirement)
    onnx/                  ← extracted for local inspection/scoring
      task001.onnx
      ...
      task400.onnx
  s_<score>_<attempt>/     ← new submissions follow this pattern
    submission.zip

Never put the zip directly at the submissions root. Folder name encodes the score.

Metric History (important for understanding current rules)

Date	Change
Apr 15	Initial: cost = params + memory + MACs (via onnx-tool)
Apr 21	Ignore >30×30 inputs; negative memory patched; versions pinned
Apr 24	Constant params to be counted; static shapes to be enforced
Apr 28	"Metric Migration": Compress banned; static shapes strictly enforced; Constant params counted; memory = byte sum of static shapes; initializer names ≠ I/O names
Apr 30	`Where` and `ConstantOfShape` exploits patched; zero-cost → 25 pts
May 4	MACs dropped entirely. Cost = params + memory only. ORT runtime shapes replace static analysis. onnx-tool eliminated from scoring. Hidden-only testing for speed.
May 6	Scalar params count as 1. ORT trace prefix fixed (no more clobbering). Multi-input/output rejected.
May 14	`_EXCLUDED_OP_TYPES` check restored; stronger name sanitization; duplicate value_info rejected. Final stable metric.

No known remaining exploits that survive ORT 1.24.4 verification.

Quick Reference

import math

# Score from cost
def score(cost):
    return max(1.0, 25.0 - math.log(max(1.0, cost)))

# Cost needed for a target score
def cost_for(target_score):
    return math.exp(25.0 - target_score)

# Examples:
# score(1) = 25.0
# score(10) = 22.70
# score(30) = 21.60
# score(900) = 18.21
# score(9000) = 15.89  ← ~floor if you materialize the full 30x30 output
# score(12000) = 15.60

# Inspect an ONNX for cost components:
import onnx, math
model = onnx.load("task001.onnx")
params = sum(math.prod(i.dims) or 1 for i in model.graph.initializer)
print(f"Initializer params: {params}")
for node in model.graph.node:
    print(f"  {node.op_type}: inputs={list(node.input)} outputs={list(node.output)}")

Next Steps

task255 & task233 — worst tasks (score ~10.7, cost ~1.7M). Large files (300 KB). Rebuild from scratch after understanding the ARC-GEN generator.
Bottom 20 tasks — improving these 20 tasks by +3 pts each = +60 total. Most tractable gains.
Verify overfit-risk tasks (018, 048, 096, 118, 192, 219, 285, 319, 355, 359) — if any are already failing hidden tests, they're contributing 0 pts despite looking fine locally.
Read ARC-GEN generators for bottom tasks to understand the exact rule.
LLM optimization loop — for each bottom-30 task: describe the rule, show current ONNX + cost breakdown, ask for a smaller reimplementation. Repeat ~5×.
Deadline: July 15, 2026 (~5.5 weeks).

Navigation Menu

Uh oh!

NeuroGolf 2026 — Competition Notes

What the Competition Is

Pinned Environment (Official)

Data Format

ONNX Constraints

Scoring Implementation (neurogolf_utils.py)

Critical Pitfalls (from community discussion)

⚠️ #1 — Conv Bias Length Bug (bundle-poisoning, discussion #699840)

⚠️ #2 — Dynamic Shapes → Score 0

⚠️ #3 — Hidden Test Scoring (not arc-gen!)

⚠️ #4 — Scalar Tensors/Params Count as 1

⚠️ #5 — onnx-tool "Poisoned" Ops (for display/debug only)

⚠️ #6 — ORT Profiling Trace Cleanup

⚠️ #7 — Bundle Ordering / Stale ORT State

⚠️ #8 — ARC-GEN Insoluble Samples

⚠️ #9 — Forum Prompt Injection (discussion #695746)

⚠️ #10 — Per-task vs Bundle Score Divergence (discussion #702256)

Effective ONNX Design Patterns

Core principle

Memory optimization tricks

Recommended ONNX ops

Approach that works (from Andrey Yunoshev #703914, 46th place, ~6580 pts)

Current Score Analysis (s_6256_03)

Bottom 20 — highest ROI for improvement

Timeout-risk tasks (whole submission ≤ 30 min)

Community Score Benchmarks

What Separates Top Solutions

Submission Convention

Metric History (important for understanding current rules)

Quick Reference

Next Steps

Uh oh!

FilesExpand file tree

competition_description.md

Latest commit

History

competition_description.md

File metadata and controls

NeuroGolf 2026 — Competition Notes

What the Competition Is

Pinned Environment (Official)

Data Format

ONNX Constraints

Scoring Implementation (neurogolf_utils.py)

Critical Pitfalls (from community discussion)

⚠️ #1 — Conv Bias Length Bug (bundle-poisoning, discussion #699840)

⚠️ #2 — Dynamic Shapes → Score 0

⚠️ #3 — Hidden Test Scoring (not arc-gen!)

⚠️ #4 — Scalar Tensors/Params Count as 1

⚠️ #5 — onnx-tool "Poisoned" Ops (for display/debug only)

⚠️ #6 — ORT Profiling Trace Cleanup

⚠️ #7 — Bundle Ordering / Stale ORT State

⚠️ #8 — ARC-GEN Insoluble Samples

⚠️ #9 — Forum Prompt Injection (discussion #695746)

⚠️ #10 — Per-task vs Bundle Score Divergence (discussion #702256)

Effective ONNX Design Patterns

Core principle

Memory optimization tricks

Recommended ONNX ops

Approach that works (from Andrey Yunoshev #703914, 46th place, ~6580 pts)

Current Score Analysis (s_6256_03)

Bottom 20 — highest ROI for improvement

Timeout-risk tasks (whole submission ≤ 30 min)

Community Score Benchmarks

What Separates Top Solutions

Submission Convention

Metric History (important for understanding current rules)

Quick Reference

Next Steps