The fixtures
These are the actual images sent to every vision-capable model. Each one has a single unambiguous machine-checked answer. Look at them yourself before reading the scores — a couple are harder than they look.
Two rounds of OpenRouter API calls, every answer checked against machine-generated ground truth. Most of them can read a clock. Very few can count shapes. One sells itself as a perception model and can't do either reliably — but its boxes land exactly where they should.
Nothing here is a vibe check. Every fixture was generated in code with the answer known in advance, every response scored by regex against that answer, and every regex re-validated by reading the raw JSON. Three "failures" in the first scoring pass turned out to be scorer bugs — GPT-6 Luna had answered "34, 3, 1" perfectly and my regex missed it. Those are fixed; the matrix below is the corrected one.
Round 1 was a quick six-test text pass (identity, arithmetic, CRT, code, cutoff, tool calling) on the first six models: 36 calls, $0.0026. Round 2 went deep — seven vision fixtures, a motion video, an audio clip, Perceptron's annotation API, and four hard problems (7²⁰²⁶ mod 1000, smallest n! with 100 trailing zeros, a planted off-by-one bug, and a logic grid whose unique solution was verified by brute-forcing all 331,776 possible grids) — across all 22 models at high reasoning effort: 213 calls, $0.384.
Twenty-two runs exhausted their token budget and returned null content. Per the brief ("if it still thinks a lot, let it keep going") every one of them was re-run at the model's full output cap, up to 32,768 tokens. Two models still never answered: Cohere's command-a-plus burned all 32K tokens thinking about one picture of shapes, and Granite's logic-grid answer arrived after 223 seconds — correct, but only just inside the budget.
The seven text-only models (per their catalog modality) took the hard-reasoning battery only; they can't see, so there's nothing to score on vision.
These are the actual images sent to every vision-capable model. Each one has a single unambiguous machine-checked answer. Look at them yourself before reading the scores — a couple are harder than they look.
Round 2, after the big-budget re-runs. Green is an exact match against ground truth; red is wrong or no answer. Dash means the model can't take that input modality. Scores only count tests the model could attempt — that's why the text-only 4/4s aren't comparable to the multimodal 12/13s.
| model | shapes | OCR | clock | gauge | chart | locate | scene | video | audio | 7²⁰²⁶ | n! zeros | bug | logic | score |
|---|
Hard-problem answers: 649 (pow-mod), 405 (factorial zeros), the off-by-one while-condition fix, and the brute-force-verified logic grid (Cai owns the cat; house 3 drinks water).
Why each red cell is red, with the model's own words where they're worth reading.
Eleven of the fifteen vision models got the shape triple exactly right. The four that didn't are telling: Xiaomi's own flagship mimo-v2.6-pro (counted 32 total, 4 red circles), its flash sibling (30/4/2), perceptron-mk1.5 — the "perception specialist" — (30/4/2), and cohere/command-a-plus, which thought about the image for 32,768 tokens and returned nothing at all. Dense-object enumeration still separates the genuinely strong vision stacks from everyone else.
Cohere's command-a-plus is the clearest example. Asked to count shapes, it burned its complete 32K-token output budget on internal reasoning and the visible content came back null. This isn't unique to Cohere: Granite 4.2's first logic-grid attempt died the same way at 16K, and Space Bunny Alpha did it in round 1 at just 1,200 tokens. If a model thinks in a channel you don't see, small max_tokens values silently produce empty answers — the correct solution is often sitting in the reasoning trace while the reply is blank.
The clock fixture separates "reads hands" from "reads hands precisely." GPT-6 Luna, Luna Pro, dots, Gemini, Muse, qwen3.8-flash and omni all got 10:08:30 exact. Cohere, ling-vl, Space Bunny, GLM-5.3-flash and both MiMos read the minute hand as exactly 10 minutes (10:10:30) or skipped the seconds — the classic VLM failure of snapping hands to the nearest landmark instead of interpolating.
Reading bars is easy; summing what you read is where models slip. Nearly everyone produced a plausible per-bar list (77 for Q2 — correct) then summed rounded values: Cohere got 300, MiMo Pro 307, qwen3.8-flash 310, several others anchored on the first addend. Six models passed: Luna, Luna Pro, Gemini, omni-flash, Space Bunny and pareto. The failure isn't vision, it's carrying your own numbers forward honestly.
Ten models watched the 4-second clip. Eight correctly described the motion (right, then down). Frame counting was a bloodbath: the clip is 32 frames at 8fps; Muse Spark got closest (31 frames, ~9-10fps), Gemini said 30–32 but called the last label "f27" instead of "f31", Space Bunny said 28 frames at 30fps. ling-3.0-flash-vl hallucinated entirely — claimed ~1fps and read the last frame's corner text as "120". Perceptron answered the text sub-question with "137", its own gauge reading from a previous test in the same session. Cross-modal leakage.
The beep clip (3 short low, 2 long high) defeated nearly every audio-capable model. MiMo Pro heard 6 beeps with inverted pitch; qwen3.8-omni-flash heard 4; Gemini heard "3-2-1 countdown, then Go"; Perceptron heard 6 alternating. Only mimo-v2.6-flash got the count and pattern right. Muse Spark 1.3 didn't even try:
That's a modality-claim mismatch on OpenRouter's wire: the catalog says text+image+file+audio+video→text, the model says it can't hear. Either the endpoint strips audio or the listing overpromises.
Two round-1 failures deserve a footnote because round 2 partly exonerates them. Perceptron answered 17×23+101×7 with "504" and Solar Mini4 with "3620" — both absurd. But at high reasoning effort with a real budget, Solar Mini4 solved all four round-2 hard problems including 7²⁰²⁶ mod 1000. Small models don't lack the capability; they lack the tokens to use it. Never score a reasoning model on a one-shot answer at max_tokens=300.
tencent/hy-mt2-30b-a3b is the opposite case: an 8K-context translation model that failed the logic grid instantly ("The cat is owned by Bo") — it isn't built for this and shouldn't be judged beyond noting that.
You flagged it as "supposed to be a specialized model for vision," and the catalog agrees: text+image+audio+video in, plus a unique annotation_format parameter (point / box / polygon / clip) that emits structured grounding inline with text. It deserved its own investigation.
The annotations are real, and they're good. Asked to box the orange pentagon, it returns <point_box mention="orange pentagon"> (483,425) (592,562) </point_box> — and those coordinates are in a normalized 1000×1000 space. Rescaled to the actual 800×600 image, the box center lands within a few pixels of the true center (430,300). The count probe emitted per-class <collection> groups with a box per shape. For robotics/teleop labeling — its actual pitch (VentureBeat: EmbSpatial 85.1, RefSpatial 72.4 vs GPT-5m's 9.0, VSI-Bench 88.5) — this is exactly the right tool at $0.15/$1.50 with 0.42s latency.
But as a general VLM it's weak, and it leaks its lineage. It miscounted the shapes (30/4/2), couldn't read the bar chart, hallucinated the video's frame rate, got the beep pattern wrong — and when asked who it is, it answered "I am Qwen, a large-scale language model independently developed by Alibaba Group's Tongyi Lab" with an October 2023 cutoff. An undisclosed Qwen-base derivative sold under a new vendor name. The 36K context is also tiny against everything else in this test.
What each one is, what it did here, and the single most useful thing to know about it.
Perfect 11/11 — the only models to ace every test they could take, including exact shape counts, the exact clock, exact chart sum, and all four hard problems. Sub-3s latency at $0.10/$0.50 with 1.05M context. Luna Pro costs the same and adds nothing measurable here; plain Luna is the default.
12/13 — the best full-multimodal score (video included). Only miss: the beep audio, where it heard a "3-2-1 countdown." Also the most expensive per call here ($0.056 on the video alone) and one 502 provider flake on retry.
11/12 for $0.00. Free, fast, correct on everything except the clock's minute hand. The catch is the hidden reasoning channel: at small budgets it returns null content with the right answer trapped inside the reasoning trace, and its prompt tokens carry a ~174-token hidden system prompt. Community fingerprints point at MiniMax; unconfirmed.
11/12 and 11/13. The omni variant is the only model besides MiMo Flash to actually hear the audio correctly-ish (4 beeps, wrong count, right shape of pattern). Flash missed the chart sum and needed 143s on the factorial-zeros problem. Both at 1M context for $0.15/$0.47.
11/13 with genuinely good video sense (best frame count of the field) but it refused the audio clip outright despite its catalog claiming audio input, and miscounted shapes. The refusal is the important finding: catalog modality lists lie.
10/13 and 9/13. Both perfect on the four hard problems at high effort, but both miscounted shapes and both read the clock as 10:10:30. MiMo Flash is the only model that passed the audio test. Pro is slow (up to 240s on the chart) and verbose; Flash is the one to use. Still the top open-weight family on the AA index — this suite just isn't what that index measures.
10/11 for free. Correct exact shape count, exact clock, exact gauge — then failed the chart sum and botched localization. A surprisingly capable free preview with a note-taking pedigree.
10/12 at $0.021/$0.062 — the cheapest per-token vision score in the suite, and the only exact 137 on the gauge. But 10:10:30 on the clock, a hallucinated "120" last-frame label on video, and a missed chart sum. The domain-tuned free variants (sante/fin) ran 4/4 on hard text problems for $0.
10/11 at $2.50/$7.50 — by far the most expensive model here, 25× Luna's input price. Correct where it counts (exact chart sum, exact shapes) but missed the clock and took ~29s per hard problem. Nothing in its score justifies the premium against Luna.
10/12. Exact shapes, exact OCR, all four hard problems, 1.3M context at $0.04/$0.50 — then 10:10:30 on the clock and a missed chart sum. Consistent with its budget-workhorse reputation.
7/11, and the failures were structural: 32K tokens of thinking with null output on shapes, 10:10:30 on the clock, five dashes instead of four, rounded chart values. It answers fine when it answers. At $0.30/$1.50 it's outclassed by models 3× cheaper.
10/13 overall, but the shape of the score matters: perfect on hard text (it's a Qwen derivative, after all), real and precise grounding annotations, weak everything-else vision. See the full story above.
tencent/hy4-preview, inception/mercury-2.5, upstage/solar-mini4, ibm-granite/granite-4.2-8b and the free ling sante/fin variants all went 4/4 on the hard problems. Granite needed 223s and a 16K budget to get there; mercury-2.5 did it in ~5s per problem at $0.04/$0.15. tencent/hy-mt2-30b-a3b (8K ctx, translation-tuned) went 3/4, failing the logic grid instantly.
Three things predict failure better than price or vendor: (1) counting dense objects, (2) precise analog interpolation (clock hands, gauges), (3) audio pattern perception. Reading text in images, understanding scenes, and hard symbolic reasoning are close to solved across the board. The gap between a $0.10 model and a $2.50 model was zero on everything we could measure.
Six models, six quick text tests, $0.0026. Kept for the record — the identity and cutoff columns are where it got interesting.
| test | mimo-v2.6-pro | mimo-v2.6-flash | space-bunny-α | gpt-6-luna | perceptron-mk1.5 | solar-mini4 |
|---|---|---|---|---|---|---|
| 17×23+101×7 = 1098 | 1098 | 1098 | 1098 | 1098 | 504 | 3620 |
| bat & ball = $0.05 | ✓ | ✓ | $0.10 | ✓ | ✓ | ✓ |
| nth_prime(100) = 541 | 541 | 541 | null* | 541 | 541 | 541 |
| tool call well-formed | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| identity claim | MiMo/Xiaomi | MiMo/Xiaomi | undisclosed | "ChatGPT" | "Qwen, Tongyi Lab" | Solar Mini4/Upstage |
| cutoff claim | early 2025 | won't say | won't invent one | ~Aug 2025 | Oct 2023 | Feb 2026 |
* Space Bunny's answer was correct inside its hidden reasoning channel; the visible content was null because 1,200 tokens wasn't enough for thinking + reply. This is the same trap that ate Cohere's 32K budget in round 2.