A reproducible comparison of the models in the OpenCode Go plan: image support (vision), two-turn tool calling, and whether a model catches a sub-agent's false success report. 13 models measured, three standard-library Python scripts, one decision at the end.
Measured 2026-08-09 against the OpenCode Go
$10/monthplan (https://opencode.ai/zen/go/v1, OpenAI-compatible). The catalog returned 25 model IDs that day:hy3,qwen3.7-plus,qwen3.8-max,kimi-k3,kimi-k2.7-code,mimo-v2.5,mimo-v2.5-pro,minimax-m2.7,minimax-m3,deepseek-v4-pro,deepseek-v4-flash,gpt-5.6-luna,grok-4.5,glm-5.2and others. Model rosters churn fast — every table here is date-stamped, and the scripts are the durable part.
Contents Why this repo exists · The three tests · Full results table · Seven findings · Round 2: building a real app · Why hy3 was chosen · Error messages decoded · FAQ · Write-ups
Picking a model for a chat is easy: try two, keep the one you like.
Picking a model to sit at the top of an agent is not. That model has to dispatch work to other agents, look at a screenshot of the app to decide whether the UI is actually fixed, and — the part nobody advertises — notice when a sub-agent reports success on work it never did. A model can be excellent at prose and fail all three.
None of that is on a pricing page. So it got measured.
The three tests below are deliberately small, dependency-free and provider-agnostic. Point them at any OpenAI-compatible endpoint and you get the same table for your own stack.
| Result | |
|---|---|
| Models in catalog (2026-08-09) | 25 |
| Models measured | 13 |
| Models that can actually see an image | 6 |
| Models that pass a two-turn tool loop | 12 of 13 |
| Models that catch a falsified success report | every model that answered at all |
| Most dangerous failure found | a model that accepts images, returns HTTP 200, and is blind without saying so |
| Chosen for the orchestrator seat | hy3 — and not for the reason the first round predicted |
| # | Script | Question it answers | Why it earns its place |
|---|---|---|---|
| 1 | tests/vision_gate.py |
Can the model see the screenshot you send it? | An agent that reviews UI work is worthless without eyes. Failure here is invisible in text benchmarks. |
| 2 | tests/tool_loop.py |
Can it emit a tool call, take the result back, and finish? | Plenty of models pass turn 1 and break on turn 2. One turn proves nothing. |
| 3 | tests/false_success.py |
Does it catch a sub-agent lying about success? | The single most expensive failure mode when one model supervises others. |
Test 3 is the one people skip. Its fixture: a coder agent returns {"ok": true, "stdout": "Fixed calc.py, done."} — and the very next line of the transcript is cat calc.py showing the bug still
there. The model is asked whether the task is done.
$ export OPENCODE_GO_API_KEY=...
$ python3 tests/vision_gate.py hy3 qwen3.7-plus mimo-v2.5
$ python3 tests/tool_loop.py hy3 minimax-m3
$ python3 tests/false_success.py kimi-k3 qwen3.8-max
Standard library only. No Pillow — the test image is generated by a hand-rolled PNG writer, so the repo runs with zero assets and zero installs.
Quotas are as published by the plan on that date, converted to requests per 5-hour window and per month. They are context, not a score.
| model | 2-turn tool loop | sees images | catches false success | req / 5h | req / month |
|---|---|---|---|---|---|
| qwen3.7-plus | ✅ | ✅ | ✅ | 4,300 | 21,600 |
| mimo-v2.5 | ✅ | ✅ | ✅ ¹ | 30,100 | 150,400 |
| kimi-k2.7-code | ✅ | ✅ | ✅ | 1,350 | 6,750 |
| minimax-m3 | <think> |
✅ | <think> |
3,200 | 16,000 |
| qwen3.8-max | ✅ | ✅ | ✅ sharpest | 160 | 810 |
| kimi-k3 | ✅ | ✅ | ✅ sharpest | 110 | 490 |
| — end of models with eyes — | |||||
| minimax-m2.7 | ✅ | ❌ blind, silently | ✅ | 3,400 | 17,000 |
| hy3 | ✅ | ❌ 400 — no endpoints support image input | ✅ | 4,300 | 21,500 |
| mimo-v2.5-pro | ✅ | ❌ 400 — same upstream message | ✅ ¹ | 3,250 | 16,300 |
| deepseek-v4-pro | ✅ direct ² | ❌ 400 — unknown variant image_url |
✅ | 3,450 | 17,150 |
| gpt-5.6-luna | ✅ | ❌ 400 — body is a completion with an empty message | — | 2,050 | 10,250 |
| deepseek-v4-flash | ✅ | ❌ 400 — unknown variant image_url |
— | 31,650 | 158,150 |
| grok-4.5 | ❌ endpoint dead ³ | — | — | 120 | 600 |
¹ returns an empty content unless max_tokens is raised — see finding 3.
² passes when called directly; breaks through a Responses→chat bridge for a reason that is not the
model's fault — see finding 7 and docs/preamble-order.md.
³ HTTP 503 "Endpoint is unavailable" on two attempts minutes apart.
The vision column was re-run with the published
tests/vision_gate.pyafter the script was written, and every verdict reproduced. Verbatim upstream error strings are inresults/2026-08-09-opencode-go.md.
1. The vision gate eliminated more than half the catalog. Only 6 of 13 measured models could see. That single property decided more than any quality judgement — before a word was written about how smart any of them are.
2. 🔴 One model is blind and does not say so. Send an image to minimax-m2.7: no HTTP 400, no
warning, no refusal. It replies, pleasantly, "Could you please provide the image so I can describe
it?" An orchestrator built on it will receive screenshots and then invent or interrogate forever
with no signal that it never saw anything. The models that return HTTP 400 are, in practice, the
safe ones. vision_gate.py reports this case as BLIND_SILENT precisely because nothing else will.
3. The mimo family returns empty content at low max_tokens. It puts everything into a
reasoning field and leaves content: null. At 400 tokens the reply is empty; at 2,000 it is good.
Anyone who evaluated mimo at a low token cap and concluded "broken model" measured their own config.
4. minimax-m3 leaks <think>…</think> straight into content — reproduced on all three tests.
Any harness that parses output will break on it. The reasoning is not in a separate field; it is in
the answer.
5. 🔴 "pro" is not a superset of "cheap". mimo-v2.5-pro is blind while plain mimo-v2.5 sees —
and the cheap one has 9× the quota. Both deepseek-v4 variants are blind. On the vision axis,
bigger and pricier was simply worse. Never infer a capability from a model's name or tier; measure
each ID.
6. Small quota is a signal of a strong model. The tightest allowances in the plan — kimi-k3
(110 req/5h) and qwen3.8-max (160) — were the two that answered the false-success test sharply:
both volunteered a cause ("hallucinated edit, or it wrote to the wrong path / wrong working
directory") and committed to reading the file themselves before declaring anything done. Every other
model that answered got the verdict right but stopped at the verdict. Providers ration what costs
them.
7. A model can be broken by the harness, not by itself. deepseek-v4-pro returned HTTP 400
through a Responses→chat bridge and ran perfectly on a direct call. Cause: the harness's system
prompt asks for a short preamble before tool calls; this model emits that preamble after the tool
call, producing assistant(tool_calls) → assistant(text) → tool, and the upstream API requires tool
calls to be immediately followed by their results. Same prompt, different ordering, one model lives
and one dies. Full write-up: docs/preamble-order.md.
Round 1 ranks capabilities. It does not rank judgement, and judgement is what an orchestrator sells. So the three plausible finalists were made to build the same real application and answer the same real questions.
The task: a Sheet Register desktop app (Rust + Tauri v2) that ingests a CSV of drawing sheets and reports duplicates and numbering gaps.
⭐ The grading trick: force a fixed stdout contract (
DUPLICATE <no>,GAP <no>,ROWS <n>). Without it, each model designs its own API, the grader calls the wrong signature and reports a failure that never happened. Fix the output shape, let them design everything else.
| deepseek-v4-pro | mimo-v2.5-pro | hy3 | |
|---|---|---|---|
| Output contract | 4/5 — printed GAP A-3, lost zero-padding |
5/5 | 5/5 |
Tests written (cargo test) |
15 | 19 | 10 |
cargo check --workspace |
0 errors, 0 warnings | 0 / 0 | 0 / 0 |
| App on launch | empty table, needs a manual Load CSV | auto-loads, plain white UI | auto-loads, dark theme + validation panel |
| Unrequested extras | sort + summary (206-line UI) | — (141-line UI) | — |
Then a live research task: four facts about tauri-apps/tauri that can only be fetched, not
recalled. Ground truth: tag tauri-v2.11.5, released 2026-07-01, 1,277 open issues,
MSRV 1.77.2.
| tag | date | open issues | MSRV | network calls | |
|---|---|---|---|---|---|
| deepseek-v4-pro | ✅ | ✅ | ❌ 1,440 | ❌ 1.90 | 4 |
| mimo-v2.5-pro | ✅ | ✅ | ❌ 1,440 | ❌ 1.90 | 4 |
| hy3 | ✅ | ✅ | ✅ 1,277 | ✅ 1.77.2 | 16 |
Both wrong answers are the same mistake: stopping at the first plausible result. 1,440 is
GitHub's open_issues_count, which includes pull requests. 1.90 is the rust-version on the dev
branch rather than on the released tag. hy3 made 4× the network calls because it did not trust its
first fetch — it queried the search API separately, read Cargo.toml at the correct tag, and then
volunteered that dev says 1.90 and explained why it chose the other number.
🔴 mimo explicitly wrote "open_issues_count includes PRs, a known quirk" — and still submitted
1,440. Knowing the trap and shipping it anyway is worse than not knowing.
Two further tasks separated them further:
- Architecture (design doc, no code, "max 600 words"): all three covered the required ground.
hy3went deepest technically. Onlydeepseek-v4-proobeyed the word limit (576 words, and it ranwc -won itself to check); the other two ran ~1,200 words — double. - Check-before-you-install (prepare a machine to build and bundle the app): none of the three
clobbered the already-installed toolchain.
hy3opened worst — it nearlyapt install-ed a list copied verbatim from the docs, including packages already present — then did the thing no other model did: it discovered a skills file already sitting in the workspace and read it, measured that the host glibc was too new for a portable bundle, and pivoted to building inside an older container, standing up a probe app to prove the pivot worked.
After round 1 the obvious pick was qwen3.7-plus: the only model that both sees images and has the
quota for all-day conversation. The vision gate looked decisive.
Round 2 reversed it, and the reversal is the most useful thing in this repo.
Vision turned out to be delegable. Judgement did not. A screenshot can be handed to a second, cheap, sighted model in one extra call. There is no equivalent trick for a model that stops at the first answer that looks right — that failure propagates into every decision it makes downstream, and it looks like confidence the whole way.
The decision, stated plainly:
| Criterion | Weight | Winner |
|---|---|---|
| Doesn't stop at the first plausible answer | decisive | hy3 (4/4 vs 2/4) |
| Architectural depth on a hard domain | high | hy3 |
| Finds and uses context already in the workspace | high | hy3 (only model that did) |
| Sustained quota for an all-day role | high | hy3 (21,500/month) |
| Sees images | delegated | qwen3.7-plus, via one extra call |
| Builds the most complete app | separate seat | deepseek-v4-pro |
| Obeys an explicit length limit | deepseek-v4-pro |
Final assignment (2026-08-09)
| Seat | Model | Rationale |
|---|---|---|
| Orchestrator | hy3 |
best research discipline and architectural depth; quota to run all day |
| Coder | deepseek-v4-flash |
fast, huge quota; blindness irrelevant for writing code |
| UI reviewer / eyes | qwen3.7-plus |
the sighted model everything visual is routed to |
| Second opinion when stuck | kimi-k3, qwen3.8-max |
sharpest judgement, too little quota for routine use |
Known cost of this choice: hy3 is blind — every visual acceptance step must be routed
elsewhere, and forgetting to route it is now the standing risk. It is also verbose and needs an
explicit length constraint in its system prompt, which is exactly the axis it lost on in round 2.
The verbatim strings these providers return, and what each one actually means. Recorded 2026-08-09;
full bodies in results/2026-08-09-opencode-go.md.
Returned inside an HTTP 400 by hy3 and mimo-v2.5-pro when sent an image_url content part.
The model is text-only. This is permanent — retrying, resizing the image, or switching from a data URL
to a hosted URL will not help. Note the mismatch: the outer status says "bad request" while the inner
status says "capability absent". Branch on the message, not the status, or your client will retry a
permanent condition forever.
Failed to deserialize the JSON body into the target type: messages[0]: unknown variant 'image_url', expected 'text'
Returned as HTTP 400 by deepseek-v4-pro and deepseek-v4-flash. Same meaning: the backend's message
schema has no image_url variant, so the model cannot accept images at all. Sending
content: "a plain string" works; sending a content-part array with an image does not.
Returned by grok-4.5 on every attempt, minutes apart. A published quota does not imply a live
endpoint. Probe before you plan around a model.
gpt-5.6-luna answers an image request with status 400 and this body:
{"id":"chatcmpl_...","object":"chat.completion","model":"gpt-5.6-luna",
"choices":[{"index":0,"message":{"role":"assistant"},"finish_reason":null}]}It parses cleanly as a completion, and the assistant message has no content key. Any client that
checks the body before the status — or treats "parses as JSON" as success — reads this as a valid
empty answer.
minimax-m2.7 returns HTTP 200 and says some variant of "I can't see the image — could you provide
it?". There is no error anywhere in the exchange. This is the failure mode this repo exists to catch;
vision_gate.py classifies it as BLIND_SILENT.
Not a model problem. A tool call was recorded with no matching tool result — typically because a user
typed over or cancelled a running tool call — and every later turn replays the broken history, so the
session 400s forever. Switching models does not help. Fix it at the proxy by synthesising a
placeholder result for dangling calls; details in docs/preamble-order.md.
Not a hardware fault and not a linker bug: on many WSL setups /tmp is tmpfs backed by RAM, and a
build that fills it kills the linker with a signal that reads like failing memory. Build on real disk.
Which models in the OpenCode Go plan support image input?
As of 2026-08-09: qwen3.7-plus, mimo-v2.5, kimi-k2.7-code, minimax-m3, qwen3.8-max and
kimi-k3. Six out of thirteen measured. Everything else either errors or — in one case — silently
pretends there was no image.
Is the "pro" version of a model always more capable than the cheap one?
No. mimo-v2.5-pro cannot accept images while plain mimo-v2.5 can, and the cheap one has nine
times the quota. Capability belongs to a model ID, not to a tier name.
Why does my model return an empty response with no error?
Check max_tokens first. Some models (the mimo family here) put their output in a reasoning
field and return content: null when the budget is tight. Raise the cap before concluding the model
is broken — and make sure your client reads reasoning, not only content.
Why does a model work on a direct API call but 400 through LiteLLM or another proxy?
Most often a message-ordering violation created by the Responses→chat translation, not a fault in the
model. Full diagnosis and fix: docs/preamble-order.md.
How do I test whether an LLM can really see an image?
Send a generated image with known shapes and colours, then check the reply for those colours — and
crucially, classify "fluent reply that asks for the image" separately from an HTTP error.
tests/vision_gate.py does this in one request per model.
How do I benchmark models that all pass my capability tests?
Give them the same real task with a fixed output contract so grading is uniform, and test live
research separately from building — the rankings invert.
See docs/benchmark-by-building.md.
Do these scripts work with providers other than OpenCode Go?
Yes. They are standard-library Python against the OpenAI chat-completions shape; pass --base-url
and any API key.
git clone https://github.com/<you>/opencode-go-compare
cd opencode-go-compare
export OPENCODE_GO_API_KEY=... # any OpenAI-compatible key
python3 tests/vision_gate.py model-a model-b
python3 tests/tool_loop.py model-a model-b
python3 tests/false_success.py model-a model-b
# other providers
python3 tests/vision_gate.py --base-url https://api.example.com/v1 gpt-xAdd --json to any script for machine-readable output.
Read the verdicts as heuristics. They are keyword classifiers over free text. UNCLEAR means
"read the raw reply yourself", not "the model failed". The raw reply is always printed.
- One measurement per model per test, taken on 2026-08-09. Model routing changes underneath a stable model ID; treat old tables as history.
- Quotas are the plan's published allowances on that date, not throughput measurements.
- Round 2 grades one application in one language. It tells you about judgement, not about breadth.
- No affiliation with any provider named here. This is one user measuring a plan they pay for.
README.md this file — the results
tests/ the three scripts, standard library only
results/2026-08-09-opencode-go.md raw run log behind the tables above
docs/ four long-form write-ups
MIT for the code. Results tables are free to reuse with a link back.