A hybrid solver for Kaggle's NeuroGolf 2026 competition: 400 ARC-AGI-style grid puzzles, each of which had to be solved by the smallest possible ONNX neural network. Score is cost-efficiency-weighted — a correct network only counts if it's also small, on a logarithmic curve, so a solution that's merely correct isn't enough; it has to be cheap.
The core idea: exhaust deterministic code first, and only reach for an LLM on whatever's left over — with every LLM-derived answer independently re-verified against the same scoring pipeline before it's trusted.
| Stage | Tasks solved | Score |
|---|---|---|
| Original submission | 7 / 400 | ~80 pts (rank 2539/2907) |
| Deterministic pipeline only | 20 / 400 | 239.3 pts, Kaggle-confirmed at an earlier checkpoint: 301 |
| + LLM hybrid agent (final) | 31 / 400 | 454.11 pts, rank 2399 (locally verified: 376.18 — Kaggle's actual score came in above the local estimate at every checkpoint throughout this project, a good sign the local verification pipeline was a reliable, slightly conservative proxy for the real thing) |
Of the 31 solved tasks, 20 were solved by deterministic code with zero API calls and 11 by the LLM agent — the deterministic pipeline did most of the work; the LLM picked up what was left over, exactly as designed.
(Kaggle listed the final score as provisional at the time of writing, pending a private/held-out leaderboard reveal after the deadline — standard practice, not a reflection of anything wrong with the submission.)
Asking an LLM to solve all 400 tasks directly would have been slower, more expensive, and less reliable than it sounds — free-tier API quotas are tight (as low as 20 requests/day on some models, see the journal), and an LLM's first guess at a rule is often wrong in a way that's only obvious once checked against every example, not just the ones shown in the prompt. Meanwhile, a large fraction of these puzzles turn out to be small, well-known transformation families — flips, rotations, integer upscaling, symmetry completion — that are trivial to detect and implement in plain code once you know to look for them, and cost nothing to run.
So the pipeline is a strict waterfall, not a blend:
for each task:
1. Try every deterministic pattern detector (src/generic_builders.py)
-- free, instant, zero API calls
2. If none match, hand the task to the LLM agent (src/gemini_solver.py)
-- hypothesize a rule -> verify locally -> translate to ONNX -> verify for real
3. Whichever produced a genuinely correct, small-enough network wins.
If neither did, ship an honest placeholder (scores 0, not partial credit).
Step 2 is itself two independently-verified stages, not one LLM call:
Gemini/OpenRouter: "here's the rule, here's a numpy function"
│
▼
Run the numpy function against EVERY ground-truth example (not just
the ones shown in the prompt) -- if it's wrong, send back the specific
failing example and ask again. No ONNX work happens until this is 100%.
│
▼
Gemini/OpenRouter: "here's the same rule as a static ONNX graph"
│
▼
Run the actual ONNX file through onnxruntime against every example,
using the competition's own verification code -- if it's wrong, send
back the failure and ask again.
│
▼
Only a network that passed BOTH real checks is ever kept.
This distinction mattered in practice: several "solved" tasks from earlier in the project turned out to be invalid submissions (oversized files that had never actually been checked against the competition's own size limit — see the journal for how that was found and what it changed). The lesson generalizes: "the model says it's right" and "verified against the real grading logic" are different claims, and only the second one is worth trusting.
Free-tier LLM quotas are small enough that relying on one provider isn't practical for an unattended run across hundreds of tasks. src/llm_providers.py tries a configurable chain — Gemini first, then several OpenRouter free models — falling through automatically on quota exhaustion, with per-entry cooldowns persisted to disk so a restart doesn't re-waste a request re-probing something already known to be exhausted. overnight_runner.py drives this unattended: it sleeps until the next provider is actually likely to be available instead of busy-polling.
neurogolf-agent/
├── README.md # this file
├── docs/
│ └── ENGINEERING_JOURNAL.md # the real build story -- bugs found, decisions made, and why
├── config.py # all paths/constants/limits/provider chain in one place
├── run.py # CLI entrypoint
├── overnight_runner.py # unattended runner with quota-aware sleep/resume
├── neurogolf_utils.py # official scoring/verification module (competition-provided)
├── prompts/
│ └── phase3_prompts.py # the two-stage prompt templates sent to the LLM
├── src/
│ ├── onnx_helpers.py # reusable low-level ONNX construction library
│ ├── generic_builders.py # the deterministic pattern library (no API calls)
│ ├── llm_providers.py # multi-provider chain with fallback + cooldown tracking
│ ├── gemini_solver.py # the two-stage LLM agent (numpy rule -> ONNX translation)
│ ├── sandbox.py # lightly-sandboxed execution of LLM-generated code
│ ├── task_io.py / validation.py / orchestrator.py
├── outputs/ # generated networks, manifest, per-task LLM transcripts (gitignored)
└── results/ # a committed reference snapshot (manifest + submission zip)
pip install -r requirements.txt
cp .env.example .env # fill in GEMINI_API_KEY and/or OPENROUTER_API_KEYGet the competition data (task001.json ... task400.json) into data/ — unzip the official Kaggle data archive there.
# Deterministic pipeline only -- fast, free, always safe to run first
python run.py --start 1 --end 400 --no-gemini
# Full hybrid pipeline: deterministic first, then the LLM chain for anything left
python run.py --start 1 --end 400
# Just a few tasks
python run.py --start 52 --end 52
# Unattended, quota-aware, resumes automatically across cooldowns
python overnight_runner.py --start 1 --end 400
# Re-zip outputs/submission_networks/ into submission.zip without re-solving
python run.py --package-onlyProgress saves after every single task (outputs/build_manifest.csv is rewritten each time) — always safe to stop and resume; already-solved tasks are skipped automatically unless you pass --force.
The clean description above is what the system does now. Getting there involved a genuinely instructive sequence of bugs — a silently-unenforced file size limit that had been inflating the true score for a while, a deprecated SDK, a confusing 403 that took real isolation work to trace to an account-level restriction, a sandboxing gap that broke every LLM-generated function containing an import statement, a timeout implementation that could hang the whole process on Windows, and the provider-chain/cooldown design that came out of hitting real daily quota limits in practice. All of it is written up honestly in docs/ENGINEERING_JOURNAL.md.
MIT