A response to The Illusion of Thinking (Shojaee et al.), presenting four independent, adversarially different benchmark domains plus the direct Apple‑puzzle head‑to‑head that the Shojaee methodology defines.
Large Language Models (LLMs) and their reasoning variants (LRMs) have demonstrated impressive benchmark performance while concealing a structural ceiling: they generate probable answers rather than constructing and verifying correct ones. A 2025 Apple Research · NeurIPS study (Shojaee et al., “The Illusion of Thinking”) demonstrated this ceiling empirically — LRMs show complete accuracy collapse beyond moderate problem complexity, fail to execute explicit algorithms when handed them verbatim, and behave counterintuitively by reducing reasoning effort precisely when problems become hardest.
Z‑O is built on a different architectural premise: deterministic construction and verification, not probabilistic generation. Every answer Z‑O commits is either recalled from verified storage or induced at solve time and independently certified against a falsifiable check before it is emitted. The cardinal property is 0‑wrong: Z‑O does not ship an unverified answer — it abstains and pushes on.
The handicap is the point. ARC‑AGI‑2, ARC‑AGI‑3, the Putnam‑like set, and the chess prototype are all run with Z‑O’s recall, self‑evaluation, and negative‑reinforcement stack disabled — nothing banked, nothing reused. The closest LLM analogue would be removing pre‑training, which leaves a network that cannot perform the task at all, because in an LLM knowledge and reasoning are fused into the same weights. In Z‑O they are separable, and the reasoning stands on its own. All four flagship results are 0‑wrong: no fabricated or unverified answer was committed on any of them.
| Domain | Result | Certificate |
|---|---|---|
| ARC-AGI-2 static visual grids |
120/120 public eval = 100%, 0-wrong 167/167 test inputs, 0 abstentions, 1.85 s/task — recall off | Induced rule reproduces all train pairs + leave-one-pair-out CV on every test input |
| ARC-AGI-3 interactive games |
25/25 games, every level engine-certified aggregate 99.28% on the live public API — 22 games at a perfect 100 | Fresh-engine replay: the engine's own score advances |
| DeepMind Putnam-like exact-answer math |
40/41 (97.6%) from a 0/41 cold start, contamination-free fresh set, every answer independently re-verified | Substitute-back identity, exhaustive small-case, high-precision convergence, explicit construction |
| Live rated chess Lichess.org |
2126 Elo autonomous live play 1,052 W / 431 L / 405 D; stronger than 97.3% of rated players and >500 Elo above the strongest published LLM chess ratings | Legal-move verification each ply; an external rating system is the arbiter |
Every modern Large Language Model, regardless of scale or instruction tuning, produces its output by predicting the most probable next token given its training distribution. This is not a limitation of only the current models — it is the defining property of the LLM architecture itself. The consequences are structural, not correctable:
Z‑O is built on premises that differ structurally from LLMs:
The Apple paper is the primary reference for this work. Methodology: four controllable puzzle environments (Tower of Hanoi, Checker Jumping, River Crossing, Blocks World) with complexity N scaled from 1 (trivial) to ~15 (collapse). 25 samples per condition, up to 64K token budgets. Both reasoning models (Claude 3.7 Sonnet with thinking, DeepSeek‑R1) and standard models (Claude 3.7 without thinking, DeepSeek‑V3) evaluated.
Published ECE values for frontier LLMs: GPT‑4 family 0.10–0.18, Claude family 0.08–0.15, open‑source models 0.15–0.30 — all substantially above the theoretically optimal 0.00. Z‑O’s tri‑morphic state (DECLARATION / RECOLLECTION / REFUSAL) is not a calibrated probability; it is a structural property of whether a certificate passed. This is not calibration — it is architecture.
| Benchmark | Frontier LLM / LRM systems | Z‑O (memory disabled) |
|---|---|---|
| ARC-AGI-2 public eval | The benchmark was built specifically because ARC-AGI-1 saturated; frontier LRMs score in the low single digits to low tens of percent, at substantial per-task inference cost | 120/120 = 100%, 0-wrong, 0 abstentions, 1.85 s/task, CPU-only |
| ARC-AGI-3 interactive preview set | Designed around skill-acquisition efficiency; the field is dominated by large VLM agents pre-trained on internet-scale corpora | 25/25 games, every level engine-certified, 99.28% aggregate, no pre-trained corpus |
| DeepMind Putnam-like (fresh, contamination-free) | Graded on a 0–10 proof rubric (~8.7/10 reported); exact answer-finding under a 0-wrong constraint is a different, complementary axis | 40/41 answer-bearing, each independently certified, from 0/41 cold |
| Live rated chess | Strongest published LLM chess ratings sit ~1500–1600 Elo, with recurring illegal-move failures | 2126 Elo live on Lichess, legal-move-verified every ply, top 2.7% of rated players |
Two framings. (a) The Apple‑puzzle LLM columns throughout this paper are the Shojaee et al. reported behaviors, hedged as such — not our re‑runs. (b) The Putnam‑like comparison is explicitly two different questions: DeepMind grades proof quality; Z‑O reports verified exact answers under a refusal‑permitted discipline. Neither number dominates the other; they measure complementary axes and demonstrate Z‑O’s generalization capabilities.
Everything in this paper is one loop applied recursively. Only the reduction and the certificate change per domain:
| Domain | The falsifiable check |
|---|---|
| ARC-AGI-2 | Rule reproduces all train pairs and survives leave-one-pair-out CV on every test input |
| ARC-AGI-3 | A fresh engine, replaying the plan, advances its own score |
| Putnam-like | Substitute-back identity / exhaustive small-case / high-precision convergence / explicit construction |
| Chess | Every ply legal-move-verified; the external Lichess rating and game outcomes are the arbiter |
| Apple puzzles | Plan replayed against the problem's own rules + an optimality witness (exact PDB distance or closed form) |
The coverage is delivered by a structure that is itself the contribution: a self‑organizing network of deterministic, self‑certifying units — skill‑nodes on static grids, gears on interactive games — each one a complete Flywheel for a mechanism family.
solve(...) → answer | None, self‑gates (detects its own family from the perceived state, returns None if it doesn’t fit), and certifies before returning. A non‑null return is a certificate.CPU‑only, integer‑only, no GPU, no network at solve time, no paid inference API, no tokens emitted. The ARC‑AGI‑3 scored run used a single Intel Xeon Gold 6542Y (96 cores) / 500 GB server with zero GPU — and a single game occupies a single core. Compute for a typical game is a small fraction of a US cent; the deepest‑search games are at most a few cents of single‑core time. There is no pre‑training run to amortize off‑book.
The evaluation has two tiers. Tier 1 — the four flagship domains: externally defined, externally scored, adversarially different from one another. Tier 2 — the Apple battery: the direct Shojaee et al. analog, the four controllable puzzles at scaled N, plus algorithm‑execution at scaled problem size.
| Domain | Modality | Adversary | Why it is hard to fake |
|---|---|---|---|
| ARC-AGI-2 | Static visual grids | Novel abstraction | Held-out families; 0-wrong requires CV on every test input |
| ARC-AGI-3 | Interactive games | An environment with hidden rules | The engine's own score is the arbiter |
| Putnam-like | Natural-language exact-answer math | A contamination-free fresh set | Each answer independently re-verified; cold start was 0/41 |
| Live chess | Real-time adversarial play | A live human opponent pool | An external rating system scores it, continuously |
Tower of Hanoi is the Apple paper’s centerpiece and the fairest head‑to‑head: it is a planning task (produce the ordered move sequence), not a recall or a single arithmetic result.
What Apple observed. On Hanoi with complexity scaled by N (disks), both standard models (Claude 3.7 Sonnet without thinking, DeepSeek‑V3) and reasoning models (Claude 3.7 with thinking, DeepSeek‑R1) hold high accuracy at low N and then show complete accuracy collapse to near‑zero beyond N≈7–8 (≈100–200 move solutions). Three findings make the collapse structural: (1) reasoning effort rises with N and then drops as the model nears the threshold — it gives up; (2) the first failure move typically lands at only 10–15% of the solution length; (3) providing the explicit recursive algorithm yielded “no meaningful improvement.” The models generate the move stream token‑by‑token, and a single wrong token derails the rest.
Z‑O’s approach — an autonomous forward planner from the core primitives. A deterministic forward‑planning module was built from Z‑O’s own primitives. Given no algorithm, it searches to the goal and returns the optimal (2ⁿ−1) solution, verified move‑by‑move on real pegs, for every N=3–10 (8/8 optimal, 8/8 verified).
Removing the search wall — search, induce, verify. Pure search expands ~3× per N toward the 3ⁿ state space, so search alone would eventually explode — the analog of the LRM wall. The self‑healing learning loop removes it: it searches the small stages (N=1–3), induces the recursive structure from those solved plans (splitting each at the largest‑disk pivot and verifying both halves are themselves (N−1)‑towers), then scales by the induced recurrence, self‑verifying every stage and re‑searching on any failure. Measured: N=1 through 18, all 18 stages verified, every one optimal, only 3 stages ever searched. Effort is O(N) learned compositions, not O(3ⁿ) search.
Z-O figures are measured; LLM curves are the Apple paper's reported collapse, drawn schematically. Z-O is flat at 100% across all four domains and all N tested.
| N | Optimal moves | Apple standard / reasoning LLMs | Z‑O planner + learning loop |
|---|---|---|---|
| 3 | 7 | ✓ correct | ✓ optimal, verified (searched) |
| 5 | 31 | ✓ correct | ✓ optimal, verified (searched) |
| 7 | 127 | ✓ → collapse onset | ✓ optimal, verified |
| 8 | 255 | ≈0% (collapsed) | ✓ optimal, verified |
| 10 | 1,023 | ≈0% | ✓ optimal, verified |
| 15 | 32,767 | ≈0% | ✓ optimal, verified (induced) |
| 18 | 262,143 | ≈0% | ✓ optimal, verified (induced) |
With the algorithm provided, the Apple models showed “no meaningful improvement”; Z‑O learns and verifies the recurrence itself.
The planner is generic over a Problem (states, moves, goal, distance features); a new capability is one more problem factory in the registry. The Rust module carries all four with a passing test suite:
| Apple puzzle | Z‑O result (same planner) |
|---|---|
| Tower of Hanoi | Optimal 2ⁿ−1 by search; recurrence learned, scales to N=18, no collapse |
| River Crossing | Classic 3/3 optimal (11 crossings, verified); unsolvable 4/4 → honest no_plan_found |
| Checker Jumping | Optimal N²+2N moves (verified, N=1–6) — slide/jump constraint search |
| Blocks World | Sussman anomaly solved optimally (3 moves, verified) |
Z‑O figures are measured; LLM columns are the Apple paper’s reported behavior, hedged as such.
| Instance | Optimal | Apple LLMs | Z‑O planner |
|---|---|---|---|
| 2 / 2 | 5 crossings | ✓ correct | ✓ optimal, verified |
| 3 / 3 | 11 crossings | ≈0% (collapsed) — the paper's standout: models doing 100+ Hanoi moves fail this ~11-move puzzle | ✓ optimal (11), verified |
| 4 / 4 | unsolvable | May fabricate a crossing for an impossible instance | 0-wrong: no_plan_found |
| Instance | Optimal | Apple LLMs | Z‑O planner |
|---|---|---|---|
| N = 2 | 8 | ✓ correct | ✓ optimal, verified |
| N = 3 | 15 | ✓ → collapse onset | ✓ optimal, verified |
| N = 4 | 24 | ≈0% (collapsed) | ✓ optimal, verified |
| N = 5 | 35 | ≈0% | ✓ optimal, verified |
| N = 6 | 48 | ≈0% | ✓ optimal, verified |
| Instance | Optimal | Apple LLMs | Z‑O planner |
|---|---|---|---|
| Sussman anomaly (3 blocks) | 3 moves | ✓ correct | ✓ optimal, verified |
| Moderate N | grows | ✓ → collapse onset at moderate N | ✓ optimal, verified |
| Large N | grows | ≈0% (collapsed) | ✓ optimal or honest refusal |
Collapse points scatter from ~11 moves (River Crossing) to ~180 (Hanoi) — collapse tracks the puzzle, not the length. Z‑O is verified‑optimal across the entire range of all four.
The cross‑puzzle point — collapse is not about solution length. This is the Apple paper’s most counterintuitive finding and the sharpest contrast with Z‑O. An LLM that correctly produces a 100+‑move Hanoi solution fails an ~11‑move River Crossing — failure tracks the puzzle, not the sequence length, because generation derails on whichever structure the model has not internalized. Z‑O is uniform across all four: it does not generate a move stream, so a short deceptive puzzle is no different from a long regular one — every instance returns a verified optimal plan or an honest refusal.
Verifiable and explainable plans. Every plan is certified — replayed against the problem’s own rules (an illegal move or a non‑goal final state fails the proof) with an optimality witness (an exact learned goal‑distance pattern database, or a closed‑form optimum). The certificate is independently checkable from (initial state, plan, rules) without re‑solving — a plan audited like a cite‑check, not trusted because a model produced it. A deterministic grounded explanation accompanies it, stating the strategy and the verified step trace with no LLM and no fabrication.
Beyond puzzles. The same engine generalizes via classical action‑schema (STRIPS) planning: a procedure expressed as predicates + actions + a goal plugs straight in, demonstrated on other data types such as a legal civil‑procedure schema and a clinical‑pathway schema, each returning a verified, explained plan with honest no_plan_found on an unreachable goal. These are demonstration schemas — the engine guarantees a plan obeys the supplied rules and reaches the goal.
The Apple paper’s most striking finding: even with the explicit recursive solution algorithm provided, LLMs showed “no meaningful improvement” beyond moderate complexity. This suite tests the direct analog — Z‑O executing its own deterministic rule sets at increasing problem sizes, with no API and no generation involved.
| N | Pairs | Z‑O | Ref | Match |
|---|---|---|---|---|
| 2 | 1 | 1 | 1 | ✓ |
| 5 | 10 | −11 | −11 | ✓ |
| 9 | 36 | 9 | 9 | ✓ |
| 16 | 120 | 45 | 45 | ✓ |
| 25 | 300 | 19 | 19 | ✓ |
| N | Z‑O F(N) | Reference | Match |
|---|---|---|---|
| 20 | 6,765 | 6,765 | ✓ |
| 35 | 9,227,465 | 9,227,465 | ✓ |
| 50 | 12,586,269,025 | 12,586,269,025 | ✓ |
The contrast. LLMs given the same explicit 9‑entry table are expected to degrade at N≈9–16, by direct analogy with the Hanoi algorithm‑execution failure. Z‑O’s integer arithmetic does not generate — it computes. “Algorithm execution failure” does not apply to deterministic integer operations. This result is reproducible to the bit on any run.
Z‑O’s full operation includes an extensive learning, memory, and negative‑reinforcement architecture: once a solution is certified, it is learned so a later encounter is instant (a live derivation that took minutes collapses to a fraction of a second), and Z‑O judges its own performance and updates prior solutions iteratively. For ARC‑AGI‑2, ARC‑AGI‑3, the Putnam‑like set, and chess, all of that is switched off.
The handicap is not cosmetic. The closest analogue for a large language model would be removing its pre‑training — leaving an untrained network that cannot perform the task at all, because in an LLM knowledge and reasoning are fused into the same weights. In Z‑O they are separable: the learned layer sits on top of the reasoning architecture. Stripped of the equivalent, Z‑O still returns 120/120, 25/25 at 99.28%, 40/41, and 2126 Elo. That separability is the point of running the benchmarks this way.
Learning makes Z‑O faster; it does not make Z‑O able.
For the benchmark runs specifically: Z‑O emits no tokens and touches no GPU and no paid inference API, so there is no per‑token or per‑call charge — the only cost is CPU time. A single ARC‑AGI‑3 game occupies a single core for a median ~39 s; at commodity CPU‑instance and electricity rates that is a small fraction of a US cent, and even the deepest‑search games are a few cents. These are the full cost of solving from cold, with no pre‑training run amortized off‑book.
| Verified recall (cached) | < 1 ms | N/A |
| Verified recall (cold) | ~47 ms | 1,000–5,000 ms |
| Structural resonance (cold) | ~272 ms | 2,000–10,000 ms |
| Same query, 2nd time | < 1 ms | full inference cost again |
| Query hardware | GPU cluster | commodity CPU |
| Query cost | $0.002–$0.020 | < $0.0001 |
| 1,000,000 queries | ~$10,000 | ~$50 |
| Knowledge update | $100M+ retraining | disk I/O only |
| Works offline | No | Yes |
| Runs on a Raspberry Pi | No | Yes |
The Apple paper identified three LLM performance regimes vs complexity: standard models outperform reasoning models at low N (overthinking); reasoning models hold an advantage at medium N; both collapse at high N. Z‑O does not exhibit three regimes, because its “reasoning” is construction and verification rather than token emission. Its regimes are:
There is no regime in which Z‑O produces a confident wrong answer at high complexity. Problems too hard for Z‑O are refused, not fabricated. This is the fundamental architectural distinction, and it is what the ARC‑AGI‑3 result demonstrates most cleanly: the engine’s own score is the arbiter, so a wrong plan is structurally uncommittable.
LLMs maximize coverage (always answer) at the cost of precision. Z‑O maximizes precision (verified when answering) at a coverage cost. For professional and safety‑critical applications the asymmetry is decisive: a confident wrong answer is actively harmful, while an abstention is a prompt to escalate.
This also means the four flagship scores are stronger than equal‑valued scores from an always‑answering system. A 100% on ARC‑AGI‑2 that includes 0 abstentions and CV on every test input is not the same object as a 100% obtained by guessing where uncertain. The Putnam set3_b4 case makes the point concretely: a 0‑wrong system, when it disagrees with a published key, produces a certificate — which turns a benchmark disagreement into benchmark QA rather than a silent miss.
The strongest evidence that Z‑O is general reasoning and not a per‑benchmark fit is that one loop — perceive → route → induce/plan → verify → declare‑or‑refuse — runs unchanged across radically different data sources. Only the reduction and the certificate change. These four are adversarially different by design. They differ in modality (grids, frames, natural‑language mathematics, board positions), in whether the environment is static or reactive, in whether an adversary is present, and in who does the scoring. Overfitting to any one of them buys nothing on the others. What transfers is the loop.
It is tempting to concede a set of “generation‑native” tasks to LLMs. Each turns out to be something Z‑O does differently — writing structure it can verify rather than sampling a probable continuation it cannot check. Z‑O’s writing creation module composes natural‑language prose deterministically from gripped knowledge with a self‑evaluation gate; the Z‑O Teleporter renders images, video, audio, code, and documents from compact mathematical blueprints on CPU only; a single bye analysis lens reads optical, RF, medical‑imaging, and audio modalities with no per‑domain model.
The one tradeoff that remains is coverage convenience, not capability. An LLM has an enormous frozen surface pre‑loaded and will emit a fluent answer to almost anything instantly — at the cost that the answer may be confidently wrong and is never traceable. Z‑O answers from what it can construct or has gripped, so a genuinely novel fact must be learned first — but that takes milliseconds, not a $100M retraining run, and what Z‑O returns is either verifiable or a deferral.
Every benchmark above was run with learning disabled. With it enabled, a certified solution is banked, a later encounter is instant, and the system judges and updates its own prior solutions. Learned structure — not just specific answers — propagates: after solving ARC‑3 instances the engine composes plans for never‑seen instances from recalled sub‑skills with zero search, and learned dead‑ends transfer so that a cold solve costing 50 attempts becomes a warm solve costing 1.
The compounding effect is real, but it is deliberately excluded from every number in §5 so that those numbers measure reasoning alone. The enabled system is strictly faster and strictly no less correct — re‑certification is never skipped.
| Evidence | Status | Key result |
|---|---|---|
| ARC-AGI-2 public eval | ✓ Measured | 120/120 = 100%, 167/167 test inputs, 0 abstentions, 1.85 s/task, recall OFF |
| ARC-AGI-3 preview set (live public API) | ✓ Measured | 25/25 games, every level engine-certified, 99.28% aggregate; lf52 10/10 with use_store=False |
| DeepMind Putnam-like fresh set | ✓ Measured | 40/41 (97.6%) from 0/41 cold; 1 flagged benchmark discrepancy (set3_b4) reported, not counted |
| Live rated chess (Lichess.org) | ✓ Measured (live, ongoing) | 2126 Elo, 1,052 W / 431 L / 405 D, top 2.7% — rating is live and moves |
| Apple puzzles — all four | ✓ Measured | One engine: Hanoi optimal + learned recurrence to N=18; River Crossing 3/3 optimal, 4/4 honest refusal; Checker Jumping optimal N²+2N; Blocks World Sussman optimal. No collapse. Rust suite passes |
| Algorithm execution | ✓ Measured | 96/96 = 100.0% across all sizes and rule sets |
| LLM head-to-head re-runs | ○ Pending API keys | scripts/zrb_llm_baseline.py runs identical inputs through a live LLM; until then, LLM figures are the published literature / Apple-reported values, hedged as such |
The Illusion of Thinking Apple Research NeurIPS 2025 paper exposed a fundamental ceiling in LLM reasoning: not a limitation to be engineered away, but a structural consequence of probabilistic generation. When an architecture generates probable answers, it will generate probable‑but‑wrong answers at precisely the moments accuracy matters most — high complexity, adversarial prompts, out‑of‑distribution questions.
Z‑O does not encounter that ceiling, because it does not generate. It constructs a candidate and subjects it to a falsifiable check, then declares or abstains. The evidence is four externally scored, adversarially different domains on one unchanged loop — all with recall, learning, and negative reinforcement switched off:
And on the Apple paper’s own battery: all four puzzles, one engine, no collapse — Hanoi optimal by search with no algorithm given, recurrence learned and verified to N=18; River Crossing optimal, honest refusal when unsolvable; Checker Jumping optimal N²+2N; Blocks World Sussman optimal — where standard and reasoning LLMs collapse to ≈0% beyond N≈7–8. And 96/96 = 100% algorithm execution across all sizes, the direct contrast with “no meaningful improvement.”
The four domains differ in modality, in whether the world reacts, in whether an adversary is present, and in who does the scoring. No single trick spans them. What spans them is the loop, and the discipline that the loop enforces: nothing is committed that has not been independently verified.
“The Illusion of Thinking” demonstrated what a generation‑first architecture cannot do. These four results demonstrate what a verification‑first architecture can.
| Benchmark | Entry point | Configuration | Verification |
|---|---|---|---|
| ARC-AGI-2 | production submission harness (self-contained, no network/API, hang-proof, deterministic) | USE_RECALL = False | Verify-on-train + leave-one-pair-out CV on every test input; emit path audited free of task-ids, grid-hashes, answer lookups |
| ARC-AGI-3 | python arc3_general.py |
Scored run live against the public ARC-AGI-3 API, one scorecard per game; use_store=False for the nothing-banked runs | Fresh-engine replay; the engine's own score must advance; emit path audited free of game-id branches |
| DeepMind Putnam-like | zo_putnam_nl_*.py family reduction modules driven by the flywheel schema router; run outside the sandbox | Cold start, no stored answers | Per-family certificate (substitute-back / exhaustive small-case / high-precision convergence / construction); every worker-produced solve independently re-derived before counting |
| Live chess | autonomous Lichess client over the Rust bitboard engine | Memory, learning, negative reinforcement disabled | Per-ply legal-move verification; external Lichess rating + game outcomes |
| Apple puzzles | scripts/zrb_planner_hanoi.py (N=3–10), scripts/zrb_hanoi_learn.py (N=1–18), scripts/zrb_hanoi.py (N=1–20); planner in PureZ/literacy/zo_planner.py and multiversedb/src/planner.rs | — | Plan replayed against the problem's own rules + optimality witness (exact PDB distance or closed form) |
| Algorithm execution | scripts/zrb_suite4_algorithm.py | — | Exact integer reference comparison, 96/96 |
Comparison data and figures: holographic_store/zrb_puzzle_comparison.json (regenerated by scripts/zrb_puzzle_comparison.py); figures by scripts/zrb_puzzle_charts.py → Documentation/figures/. Aggregate: python3 scripts/zrb_aggregate.py.