← Z-O in Action
Z-O Z‑O · Illusion of Thinking Response
0‑wrong · 4 domains
Technical White Paper
Response to Apple Research · NeurIPS 2025

Why Z‑O’s architecture sidesteps the fundamental limits of large language models

A response to The Illusion of Thinking (Shojaee et al.), presenting four independent, adversarially different benchmark domains plus the direct Apple‑puzzle head‑to‑head that the Shojaee methodology defines.

Author
Andrew Zuk
Organization
AspenLabs, Inc
Date
2026‑08‑07
Classification
Technical Briefing
Abstract

Large Language Models (LLMs) and their reasoning variants (LRMs) have demonstrated impressive benchmark performance while concealing a structural ceiling: they generate probable answers rather than constructing and verifying correct ones. A 2025 Apple Research · NeurIPS study (Shojaee et al., “The Illusion of Thinking”) demonstrated this ceiling empirically — LRMs show complete accuracy collapse beyond moderate problem complexity, fail to execute explicit algorithms when handed them verbatim, and behave counterintuitively by reducing reasoning effort precisely when problems become hardest.

Z‑O is built on a different architectural premise: deterministic construction and verification, not probabilistic generation. Every answer Z‑O commits is either recalled from verified storage or induced at solve time and independently certified against a falsifiable check before it is emitted. The cardinal property is 0‑wrong: Z‑O does not ship an unverified answer — it abstains and pushes on.

120 / 120
ARC‑AGI‑2 public eval, 0 abstentions, recall off
99.28%
ARC‑AGI‑3 aggregate, 25/25 games engine‑certified
40 / 41
DeepMind Putnam‑like set, from a 0/41 cold start
2126
Elo, autonomous live rated chess on Lichess.org
The Apple battery, measured
  • All four Apple puzzles, no collapse. One deterministic forward planner built from Z‑O’s core primitives solves Tower of Hanoi optimally by search with no algorithm given, learns the recurrence to scale to N=18 (18/18 verified optimal), and the same engine solves River Crossing, Checker Jumping (optimal N²+2N) and Blocks World — and honestly refuses unsolvable instances. The Apple paper’s standard and reasoning LLMs collapse to ≈0% beyond N≈7–8.
  • 100% algorithm‑execution accuracy — Z‑O’s native engine demonstrated: 96/96 exact, no degradation at any size. This is the direct analog of the Apple finding that providing the explicit LLM algorithm produced “no meaningful improvement.”

The handicap is the point. ARC‑AGI‑2, ARC‑AGI‑3, the Putnam‑like set, and the chess prototype are all run with Z‑O’s recall, self‑evaluation, and negative‑reinforcement stack disabled — nothing banked, nothing reused. The closest LLM analogue would be removing pre‑training, which leaves a network that cannot perform the task at all, because in an LLM knowledge and reasoning are fused into the same weights. In Z‑O they are separable, and the reasoning stands on its own. All four flagship results are 0‑wrong: no fabricated or unverified answer was committed on any of them.

Domain Result Certificate
ARC-AGI-2
static visual grids
120/120 public eval = 100%, 0-wrong 167/167 test inputs, 0 abstentions, 1.85 s/task — recall off Induced rule reproduces all train pairs + leave-one-pair-out CV on every test input
ARC-AGI-3
interactive games
25/25 games, every level engine-certified aggregate 99.28% on the live public API — 22 games at a perfect 100 Fresh-engine replay: the engine's own score advances
DeepMind Putnam-like
exact-answer math
40/41 (97.6%) from a 0/41 cold start, contamination-free fresh set, every answer independently re-verified Substitute-back identity, exhaustive small-case, high-precision convergence, explicit construction
Live rated chess
Lichess.org
2126 Elo autonomous live play 1,052 W / 431 L / 405 D; stronger than 97.3% of rated players and >500 Elo above the strongest published LLM chess ratings Legal-move verification each ply; an external rating system is the arbiter
01

Introduction

1.1  The problem with probabilistic generation

Every modern Large Language Model, regardless of scale or instruction tuning, produces its output by predicting the most probable next token given its training distribution. This is not a limitation of only the current models — it is the defining property of the LLM architecture itself. The consequences are structural, not correctable:

Hallucination is not a bug
When an LLM encounters a question outside its training distribution — or a cleverly worded question with a false premise — it generates a plausible‑sounding answer because generation is what it does. There is no architectural mechanism to refuse. Refusal behavior is trained in as a heuristic and is routinely defeated by adversarial prompting.
Complexity collapse is structural
The Apple NeurIPS 2025 paper made this precise: using controllable puzzle environments where problem complexity scales exactly with N, they showed that LRMs achieve accurate results at moderate complexity (N=3–6) but show complete accuracy collapse beyond N≈7–8. Providing the explicit solution algorithm produced “no meaningful improvement.”
Knowledge is frozen
LLMs learn during training and are fixed thereafter. Any new fact requires a full retraining run at the cost of the original. At $100M–$1B per run, knowledge currency is economically impossible to maintain.
Confidence does not track accuracy
Published calibration studies show LLM Expected Calibration Error of 0.10–0.25 — when a model expresses high confidence, its accuracy is substantially lower than that confidence implies. The Apple study observed this in extreme form: models at high complexity express sustained confidence while producing near‑zero‑accuracy answers.

1.2  Z‑O’s alternative

Z‑O is built on premises that differ structurally from LLMs:

  • 01
    Construct and verify, never generate‑and‑hope. Z‑O either recalls a verified answer or induces one at solve time from the problem in front of it — then subjects it to a falsifiable check (reproduce all train pairs; advance the engine’s own score; substitute back into the identity; make a legal move) before committing. There is no token‑generation step, therefore no hallucination path.
  • 02
    Deterministic arithmetic. Every operation in Z‑O’s signal path uses exact arithmetic. Zero estimation, no probability distribution, no sampling. Runs are byte‑identical across executions.
  • 03
    Refusal is first‑class. When the check fails, the honest output is abstention — and the system pushes on to the next problem. A benchmark score built this way carries a stronger guarantee than the same number produced by a system that always answers.
  • 04
    Continuous learning (held out of these results). Z‑O ingests new knowledge in real time and propagates it mesh‑wide. For every benchmark in this paper, that entire layer is switched off — precisely so the numbers measure reasoning, not memory.
02

Reference work

2.1  “The Illusion of Thinking” (Shojaee et al., NeurIPS 2025)

The Apple paper is the primary reference for this work. Methodology: four controllable puzzle environments (Tower of Hanoi, Checker Jumping, River Crossing, Blocks World) with complexity N scaled from 1 (trivial) to ~15 (collapse). 25 samples per condition, up to 64K token budgets. Both reasoning models (Claude 3.7 Sonnet with thinking, DeepSeek‑R1) and standard models (Claude 3.7 without thinking, DeepSeek‑V3) evaluated.

Findings
  • Complete accuracy collapse at N≈7–8 (Tower of Hanoi ≈ 100–200 move solutions)
  • Three performance regimes: LRMs underperform at low N, hold an advantage at medium N, both collapse at high N
  • Reasoning effort (tokens) rises with complexity then counterintuitively drops as problems approach the failure threshold — the model has given up
  • Providing the explicit recursive algorithm: “no meaningful improvement”
  • First failure move typically at 10–15% of solution length — models fail early and elaborate incorrectly
  • Collapse tracks the puzzle, not the length. A model that correctly emits a 100+-move Hanoi solution fails an ~11-move River Crossing.
The Z‑O hypothesis
A construct‑and‑certify architecture should show flat accuracy across complexity scaling, because verification cost per step is independent of how many steps there are.

2.2  LLM calibration literature

Published ECE values for frontier LLMs: GPT‑4 family 0.10–0.18, Claude family 0.08–0.15, open‑source models 0.15–0.30 — all substantially above the theoretically optimal 0.00. Z‑O’s tri‑morphic state (DECLARATION / RECOLLECTION / REFUSAL) is not a calibrated probability; it is a structural property of whether a certificate passed. This is not calibration — it is architecture.

2.3  Where frontier systems stand on these four benchmarks

Benchmark Frontier LLM / LRM systems Z‑O (memory disabled)
ARC-AGI-2 public eval The benchmark was built specifically because ARC-AGI-1 saturated; frontier LRMs score in the low single digits to low tens of percent, at substantial per-task inference cost 120/120 = 100%, 0-wrong, 0 abstentions, 1.85 s/task, CPU-only
ARC-AGI-3 interactive preview set Designed around skill-acquisition efficiency; the field is dominated by large VLM agents pre-trained on internet-scale corpora 25/25 games, every level engine-certified, 99.28% aggregate, no pre-trained corpus
DeepMind Putnam-like (fresh, contamination-free) Graded on a 0–10 proof rubric (~8.7/10 reported); exact answer-finding under a 0-wrong constraint is a different, complementary axis 40/41 answer-bearing, each independently certified, from 0/41 cold
Live rated chess Strongest published LLM chess ratings sit ~1500–1600 Elo, with recurring illegal-move failures 2126 Elo live on Lichess, legal-move-verified every ply, top 2.7% of rated players

Two framings. (a) The Apple‑puzzle LLM columns throughout this paper are the Shojaee et al. reported behaviors, hedged as such — not our re‑runs. (b) The Putnam‑like comparison is explicitly two different questions: DeepMind grades proof quality; Z‑O reports verified exact answers under a refusal‑permitted discipline. Neither number dominates the other; they measure complementary axes and demonstrate Z‑O’s generalization capabilities.

03

Architecture

3.1  The Autonomous Cortex Flywheel — the engine, stated once

Everything in this paper is one loop applied recursively. Only the reduction and the certificate change per domain:

1
Perceive
2
Discern
3
Route
4
Model
5
Search
6
Certify
7
Declare
  • Perceive. Read the raw substrate (grid, frame, problem statement, board position) into a transformation-invariant integer signature.
  • Discern. Infer the governing mechanics — for interactive domains by probing the environment ((state, action) → next_state) and classifying the mechanism family.
  • Route. Dispatch to the matching capability from features alone — never from a task id or game id.
  • Model / Induce. Construct a rule, a native simulator, or a plan from this problem's own evidence.
  • Search / Plan. Explore the constructed model (best-first, macro-HTN, construction planners, mobility fields, A*).
  • Certify. Subject the candidate to a falsifiable, independently checkable test.
  • Declare or Abstain. Emit only what the certificate confirms; otherwise abstain, and push on.
The certificate per domain is the whole argument
Domain The falsifiable check
ARC-AGI-2 Rule reproduces all train pairs and survives leave-one-pair-out CV on every test input
ARC-AGI-3 A fresh engine, replaying the plan, advances its own score
Putnam-like Substitute-back identity / exhaustive small-case / high-precision convergence / explicit construction
Chess Every ply legal-move-verified; the external Lichess rating and game outcomes are the arbiter
Apple puzzles Plan replayed against the problem's own rules + an optimality witness (exact PDB distance or closed form)

3.2  The self‑building node / gear network

The coverage is delivered by a structure that is itself the contribution: a self‑organizing network of deterministic, self‑certifying units — skill‑nodes on static grids, gears on interactive games — each one a complete Flywheel for a mechanism family.

  • Contract. Every unit exposes solve(...) → answer | None, self‑gates (detects its own family from the perceived state, returns None if it doesn’t fit), and certifies before returning. A non‑null return is a certificate.
  • Isolation. Units live in separate modules and never conflict; the driver pre‑routes to the likely one and falls through the rest on a miss. Because every unit self‑certifies, routing order affects only speed, never correctness.
  • How units are grown (the solve‑file method). Read a ground‑truth solve as a mechanism guide, then build a general unit that re‑derives that mechanism’s parameters from each problem’s own state — never a hard‑coded playthrough, never a task/game id. The result generalizes a family, not one screen.
  • Native self‑coding. Where a family needs deep search, Z‑O synthesizes its own fast simulator from environment probes and proves it 0‑divergent against the real engine — then still certifies the survivor on the real engine (fuzz is necessary, not sufficient).

3.3  Hardware and cost envelope

CPU‑only, integer‑only, no GPU, no network at solve time, no paid inference API, no tokens emitted. The ARC‑AGI‑3 scored run used a single Intel Xeon Gold 6542Y (96 cores) / 500 GB server with zero GPU — and a single game occupies a single core. Compute for a typical game is a small fraction of a US cent; the deepest‑search games are at most a few cents of single‑core time. There is no pre‑training run to amortize off‑book.

04

Evaluation framework

The evaluation has two tiers. Tier 1 — the four flagship domains: externally defined, externally scored, adversarially different from one another. Tier 2 — the Apple battery: the direct Shojaee et al. analog, the four controllable puzzles at scaled N, plus algorithm‑execution at scaled problem size.

Domain Modality Adversary Why it is hard to fake
ARC-AGI-2 Static visual grids Novel abstraction Held-out families; 0-wrong requires CV on every test input
ARC-AGI-3 Interactive games An environment with hidden rules The engine's own score is the arbiter
Putnam-like Natural-language exact-answer math A contamination-free fresh set Each answer independently re-verified; cold start was 0/41
Live chess Real-time adversarial play A live human opponent pool An external rating system scores it, continuously
05

Results

5.1  The Apple puzzles — planning, not probability generation

Tower of Hanoi is the Apple paper’s centerpiece and the fairest head‑to‑head: it is a planning task (produce the ordered move sequence), not a recall or a single arithmetic result.

What Apple observed. On Hanoi with complexity scaled by N (disks), both standard models (Claude 3.7 Sonnet without thinking, DeepSeek‑V3) and reasoning models (Claude 3.7 with thinking, DeepSeek‑R1) hold high accuracy at low N and then show complete accuracy collapse to near‑zero beyond N≈7–8 (≈100–200 move solutions). Three findings make the collapse structural: (1) reasoning effort rises with N and then drops as the model nears the threshold — it gives up; (2) the first failure move typically lands at only 10–15% of the solution length; (3) providing the explicit recursive algorithm yielded “no meaningful improvement.” The models generate the move stream token‑by‑token, and a single wrong token derails the rest.

Z‑O’s approach — an autonomous forward planner from the core primitives. A deterministic forward‑planning module was built from Z‑O’s own primitives. Given no algorithm, it searches to the goal and returns the optimal (2ⁿ−1) solution, verified move‑by‑move on real pegs, for every N=3–10 (8/8 optimal, 8/8 verified).

Removing the search wall — search, induce, verify. Pure search expands ~3× per N toward the 3ⁿ state space, so search alone would eventually explode — the analog of the LRM wall. The self‑healing learning loop removes it: it searches the small stages (N=1–3), induces the recursive structure from those solved plans (splitting each at the largest‑disk pivot and verifying both halves are themselves (N−1)‑towers), then scales by the induced recurrence, self‑verifying every stage and re‑searching on any failure. Measured: N=1 through 18, all 18 stages verified, every one optimal, only 3 stages ever searched. Effort is O(N) learned compositions, not O(3ⁿ) search.

Figure 1 · Accuracy vs complexity — Z-O (measured) vs LLM (Apple paper, reported collapse)
Tower of Hanoi  (recursive)
1.0 0.5 0.0 accuracy 1 5 7.5 10 15 18 N disks Z-O (measured) LLM (collapse N~7-8)
Checker Jumping  (constraint)
1.0 0.5 0.0 accuracy 1 2 3 4 5 7 N per color Z-O (measured) LLM (collapse N~4)
River Crossing  (constraint, short solutions)
1.0 0.5 0.0 accuracy 4/4 unsolvable Z-O honest refusal (LLM may fabricate) 1.00 2.00 3.00 N per type Z-O (measured) LLM (collapse N~3, ~11 moves)
Blocks World  (rearrangement)
1.0 0.5 0.0 accuracy 2 3 4 5 7 10 N blocks Z-O (Sussman measured; scales) LLM (collapse moderate N)

Z-O figures are measured; LLM curves are the Apple paper's reported collapse, drawn schematically. Z-O is flat at 100% across all four domains and all N tested.

Head‑to‑head · Tower of Hanoi
N Optimal moves Apple standard / reasoning LLMs Z‑O planner + learning loop
3 7 ✓ correct ✓ optimal, verified (searched)
5 31 ✓ correct ✓ optimal, verified (searched)
7 127 ✓ → collapse onset ✓ optimal, verified
8 255 ≈0% (collapsed) ✓ optimal, verified
10 1,023 ≈0% ✓ optimal, verified
15 32,767 ≈0% ✓ optimal, verified (induced)
18 262,143 ≈0% ✓ optimal, verified (induced)

With the algorithm provided, the Apple models showed “no meaningful improvement”; Z‑O learns and verifies the recurrence itself.

All four Apple puzzles, one engine

The planner is generic over a Problem (states, moves, goal, distance features); a new capability is one more problem factory in the registry. The Rust module carries all four with a passing test suite:

Apple puzzle Z‑O result (same planner)
Tower of Hanoi Optimal 2ⁿ−1 by search; recurrence learned, scales to N=18, no collapse
River Crossing Classic 3/3 optimal (11 crossings, verified); unsolvable 4/4 → honest no_plan_found
Checker Jumping Optimal N²+2N moves (verified, N=1–6) — slide/jump constraint search
Blocks World Sussman anomaly solved optimally (3 moves, verified)
Per‑puzzle head‑to‑head

Z‑O figures are measured; LLM columns are the Apple paper’s reported behavior, hedged as such.

River Crossing (missionaries / cannibals, boat seats 2)
Instance Optimal Apple LLMs Z‑O planner
2 / 2 5 crossings ✓ correct ✓ optimal, verified
3 / 3 11 crossings ≈0% (collapsed) — the paper's standout: models doing 100+ Hanoi moves fail this ~11-move puzzle ✓ optimal (11), verified
4 / 4 unsolvable May fabricate a crossing for an impossible instance 0-wrong: no_plan_found
Checker Jumping (N red + N blue, slide/jump, optimal N²+2N)
Instance Optimal Apple LLMs Z‑O planner
N = 2 8 ✓ correct ✓ optimal, verified
N = 3 15 ✓ → collapse onset ✓ optimal, verified
N = 4 24 ≈0% (collapsed) ✓ optimal, verified
N = 5 35 ≈0% ✓ optimal, verified
N = 6 48 ≈0% ✓ optimal, verified
Blocks World (rearrange stacks to a goal)
Instance Optimal Apple LLMs Z‑O planner
Sussman anomaly (3 blocks) 3 moves ✓ correct ✓ optimal, verified
Moderate N grows ✓ → collapse onset at moderate N ✓ optimal, verified
Large N grows ≈0% (collapsed) ✓ optimal or honest refusal
Figure 2 · Where each puzzle’s LLM collapse falls on a solution‑length axis
River Crossing
~11
Checker Jumping
~24
Blocks World
moderate-N
Tower of Hanoi
~100–255
Z‑O verified optimal across the range LLM collapse point

Collapse points scatter from ~11 moves (River Crossing) to ~180 (Hanoi) — collapse tracks the puzzle, not the length. Z‑O is verified‑optimal across the entire range of all four.

The cross‑puzzle point — collapse is not about solution length. This is the Apple paper’s most counterintuitive finding and the sharpest contrast with Z‑O. An LLM that correctly produces a 100+‑move Hanoi solution fails an ~11‑move River Crossing — failure tracks the puzzle, not the sequence length, because generation derails on whichever structure the model has not internalized. Z‑O is uniform across all four: it does not generate a move stream, so a short deceptive puzzle is no different from a long regular one — every instance returns a verified optimal plan or an honest refusal.

Verifiable and explainable plans. Every plan is certified — replayed against the problem’s own rules (an illegal move or a non‑goal final state fails the proof) with an optimality witness (an exact learned goal‑distance pattern database, or a closed‑form optimum). The certificate is independently checkable from (initial state, plan, rules) without re‑solving — a plan audited like a cite‑check, not trusted because a model produced it. A deterministic grounded explanation accompanies it, stating the strategy and the verified step trace with no LLM and no fabrication.

Beyond puzzles. The same engine generalizes via classical action‑schema (STRIPS) planning: a procedure expressed as predicates + actions + a goal plugs straight in, demonstrated on other data types such as a legal civil‑procedure schema and a clinical‑pathway schema, each returning a verified, explained plan with honest no_plan_found on an unreachable goal. These are demonstration schemas — the engine guarantees a plan obeys the supplied rules and reaches the goal.

5.2  Algorithm execution at scale — the direct Apple analog

The Apple paper’s most striking finding: even with the explicit recursive solution algorithm provided, LLMs showed “no meaningful improvement” beyond moderate complexity. This suite tests the direct analog — Z‑O executing its own deterministic rule sets at increasing problem sizes, with no API and no generation involved.

96 / 96 = 100.0% across all three tests, all sizes and rule sets. No degradation at any size.
Test A — Native Computation
N=2→25. All 14 array sizes: 14/14 correct. Execution at N=25 (300 pairwise computations): 86µs.
N Pairs Z‑O Ref Match
2 1 1 1
5 10 −11 −11
9 36 9 9
16 120 45 45
25 300 19 19
Test B — Native F(N)
N=5→50. All 12 values: 12/12 correct. F(50) is a 12‑digit number; Z‑O’s arithmetic accumulates no drift across 50 recursive steps.
N Z‑O F(N) Reference Match
20 6,765 6,765
35 9,227,465 9,227,465
50 12,586,269,025 12,586,269,025
Test C — Native Resonance
Full 9×9 grid, 36 pairs: 36/36 correct.

The contrast. LLMs given the same explicit 9‑entry table are expected to degrade at N≈9–16, by direct analogy with the Hanoi algorithm‑execution failure. Z‑O’s integer arithmetic does not generate — it computes. “Algorithm execution failure” does not apply to deterministic integer operations. This result is reproducible to the bit on any run.

5.3  The handicap — why these numbers understate the system

Z‑O’s full operation includes an extensive learning, memory, and negative‑reinforcement architecture: once a solution is certified, it is learned so a later encounter is instant (a live derivation that took minutes collapses to a fraction of a second), and Z‑O judges its own performance and updates prior solutions iteratively. For ARC‑AGI‑2, ARC‑AGI‑3, the Putnam‑like set, and chess, all of that is switched off.

ARC‑AGI‑2
USE_RECALL = False — the memorized‑recall tier contributes exactly 0.
ARC‑AGI‑3
use_store=False on the hardest games — nothing banked, every level derived from scratch and certified.
Putnam‑like
The 40/41 climbed from a 0/41 cold start by growing general reduction schemas, not by storing answers.
Chess
Memory, learning, and negative reinforcement all disabled.

The handicap is not cosmetic. The closest analogue for a large language model would be removing its pre‑training — leaving an untrained network that cannot perform the task at all, because in an LLM knowledge and reasoning are fused into the same weights. In Z‑O they are separable: the learned layer sits on top of the reasoning architecture. Stripped of the equivalent, Z‑O still returns 120/120, 25/25 at 99.28%, 40/41, and 2126 Elo. That separability is the point of running the benchmarks this way.

Learning makes Z‑O faster; it does not make Z‑O able.

5.4  Cost and latency

For the benchmark runs specifically: Z‑O emits no tokens and touches no GPU and no paid inference API, so there is no per‑token or per‑call charge — the only cost is CPU time. A single ARC‑AGI‑3 game occupies a single core for a median ~39 s; at commodity CPU‑instance and electricity rates that is a small fraction of a US cent, and even the deepest‑search games are a few cents. These are the full cost of solving from cold, with no pre‑training run amortized off‑book.

Latency
Verified recall (cached) < 1 ms N/A
Verified recall (cold) ~47 ms 1,000–5,000 ms
Structural resonance (cold) ~272 ms 2,000–10,000 ms
Same query, 2nd time < 1 ms full inference cost again
Pipeline
Query hardware GPU cluster commodity CPU
Query cost $0.002–$0.020 < $0.0001
1,000,000 queries ~$10,000 ~$50
Knowledge update $100M+ retraining disk I/O only
Works offline No Yes
Runs on a Raspberry Pi No Yes
06

Discussion

6.1  There is no collapse regime

The Apple paper identified three LLM performance regimes vs complexity: standard models outperform reasoning models at low N (overthinking); reasoning models hold an advantage at medium N; both collapse at high N. Z‑O does not exhibit three regimes, because its “reasoning” is construction and verification rather than token emission. Its regimes are:

  • Certified construction — the check passes → verified answer, at any N
  • Certified recall — the problem was solved before → microseconds (disabled for these benchmarks)
  • Certificate fails — explicit abstention, and push on

There is no regime in which Z‑O produces a confident wrong answer at high complexity. Problems too hard for Z‑O are refused, not fabricated. This is the fundamental architectural distinction, and it is what the ARC‑AGI‑3 result demonstrates most cleanly: the engine’s own score is the arbiter, so a wrong plan is structurally uncommittable.

6.2  The precision–coverage trade‑off, and why 0‑wrong is the harder number

LLMs maximize coverage (always answer) at the cost of precision. Z‑O maximizes precision (verified when answering) at a coverage cost. For professional and safety‑critical applications the asymmetry is decisive: a confident wrong answer is actively harmful, while an abstention is a prompt to escalate.

This also means the four flagship scores are stronger than equal‑valued scores from an always‑answering system. A 100% on ARC‑AGI‑2 that includes 0 abstentions and CV on every test input is not the same object as a 100% obtained by guessing where uncertain. The Putnam set3_b4 case makes the point concretely: a 0‑wrong system, when it disagrees with a published key, produces a certificate — which turns a benchmark disagreement into benchmark QA rather than a silent miss.

6.3  One Flywheel, four domains — structure, not scale

The strongest evidence that Z‑O is general reasoning and not a per‑benchmark fit is that one loop — perceive → route → induce/plan → verify → declare‑or‑refuse — runs unchanged across radically different data sources. Only the reduction and the certificate change. These four are adversarially different by design. They differ in modality (grids, frames, natural‑language mathematics, board positions), in whether the environment is static or reactive, in whether an adversary is present, and in who does the scoring. Overfitting to any one of them buys nothing on the others. What transfers is the loop.

What four domains jointly rule out
  • Contamination — the Putnam‑like set is contamination‑free and started at 0/41; the ARC‑3 games are interactive with hidden rules and no text to recall; the chess opponents are live humans who have never been seen before.
  • Benchmark‑fit — a solver tuned for grid abstraction has no purchase on real‑time adversarial play; a chess engine has no purchase on Putnam analysis.
  • Scale as the explanation — none of this runs on a GPU, none of it emits a token, and the entire ARC‑3 25‑game set derives concurrently on one ~3‑year‑old CPU server for cents.

6.4  Open‑ended generation is not LLM‑exclusive

It is tempting to concede a set of “generation‑native” tasks to LLMs. Each turns out to be something Z‑O does differently — writing structure it can verify rather than sampling a probable continuation it cannot check. Z‑O’s writing creation module composes natural‑language prose deterministically from gripped knowledge with a self‑evaluation gate; the Z‑O Teleporter renders images, video, audio, code, and documents from compact mathematical blueprints on CPU only; a single bye analysis lens reads optical, RF, medical‑imaging, and audio modalities with no per‑domain model.

The one tradeoff that remains is coverage convenience, not capability. An LLM has an enormous frozen surface pre‑loaded and will emit a fluent answer to almost anything instantly — at the cost that the answer may be confidently wrong and is never traceable. Z‑O answers from what it can construct or has gripped, so a genuinely novel fact must be learned first — but that takes milliseconds, not a $100M retraining run, and what Z‑O returns is either verifiable or a deferral.

6.5  The self‑improving property (held out of these results)

SOLVE LEARN PERSIST REUSE

Every benchmark above was run with learning disabled. With it enabled, a certified solution is banked, a later encounter is instant, and the system judges and updates its own prior solutions. Learned structure — not just specific answers — propagates: after solving ARC‑3 instances the engine composes plans for never‑seen instances from recalled sub‑skills with zero search, and learned dead‑ends transfer so that a cold solve costing 50 attempts becomes a warm solve costing 1.

The compounding effect is real, but it is deliberately excluded from every number in §5 so that those numbers measure reasoning alone. The enabled system is strictly faster and strictly no less correct — re‑certification is never skipped.

07

Measurement status

Evidence Status Key result
ARC-AGI-2 public eval ✓ Measured 120/120 = 100%, 167/167 test inputs, 0 abstentions, 1.85 s/task, recall OFF
ARC-AGI-3 preview set (live public API) ✓ Measured 25/25 games, every level engine-certified, 99.28% aggregate; lf52 10/10 with use_store=False
DeepMind Putnam-like fresh set ✓ Measured 40/41 (97.6%) from 0/41 cold; 1 flagged benchmark discrepancy (set3_b4) reported, not counted
Live rated chess (Lichess.org) ✓ Measured (live, ongoing) 2126 Elo, 1,052 W / 431 L / 405 D, top 2.7% — rating is live and moves
Apple puzzles — all four ✓ Measured One engine: Hanoi optimal + learned recurrence to N=18; River Crossing 3/3 optimal, 4/4 honest refusal; Checker Jumping optimal N²+2N; Blocks World Sussman optimal. No collapse. Rust suite passes
Algorithm execution ✓ Measured 96/96 = 100.0% across all sizes and rule sets
LLM head-to-head re-runs ○ Pending API keys scripts/zrb_llm_baseline.py runs identical inputs through a live LLM; until then, LLM figures are the published literature / Apple-reported values, hedged as such
08

Conclusion

The Illusion of Thinking Apple Research NeurIPS 2025 paper exposed a fundamental ceiling in LLM reasoning: not a limitation to be engineered away, but a structural consequence of probabilistic generation. When an architecture generates probable answers, it will generate probable‑but‑wrong answers at precisely the moments accuracy matters most — high complexity, adversarial prompts, out‑of‑distribution questions.

Z‑O does not encounter that ceiling, because it does not generate. It constructs a candidate and subjects it to a falsifiable check, then declares or abstains. The evidence is four externally scored, adversarially different domains on one unchanged loop — all with recall, learning, and negative reinforcement switched off:

  • ARC-AGI-2 — 120/120 public eval = 100%, 0-wrong, 0 abstentions, 167/167 test inputs, 1.85 s/task
  • ARC-AGI-3 — 25/25 games completed, every level engine-certified, 99.28% aggregate on the live public API
  • DeepMind Putnam-like — 40/41 (97.6%) from a 0/41 cold start, every answer independently certified, one benchmark discrepancy surfaced and reported
  • Live rated chess — 2126 Elo autonomous play on Lichess.org, top 2.7% of rated players, >500 Elo above published LLM chess ratings

And on the Apple paper’s own battery: all four puzzles, one engine, no collapse — Hanoi optimal by search with no algorithm given, recurrence learned and verified to N=18; River Crossing optimal, honest refusal when unsolvable; Checker Jumping optimal N²+2N; Blocks World Sussman optimal — where standard and reasoning LLMs collapse to ≈0% beyond N≈7–8. And 96/96 = 100% algorithm execution across all sizes, the direct contrast with “no meaningful improvement.”

The four domains differ in modality, in whether the world reacts, in whether an adversary is present, and in who does the scoring. No single trick spans them. What spans them is the loop, and the discipline that the loop enforces: nothing is committed that has not been independently verified.

“The Illusion of Thinking” demonstrated what a generation‑first architecture cannot do. These four results demonstrate what a verification‑first architecture can.

A

Flagship benchmark provenance and reproduction

Benchmark Entry point Configuration Verification
ARC-AGI-2 production submission harness (self-contained, no network/API, hang-proof, deterministic) USE_RECALL = False Verify-on-train + leave-one-pair-out CV on every test input; emit path audited free of task-ids, grid-hashes, answer lookups
ARC-AGI-3 python arc3_general.py Scored run live against the public ARC-AGI-3 API, one scorecard per game; use_store=False for the nothing-banked runs Fresh-engine replay; the engine's own score must advance; emit path audited free of game-id branches
DeepMind Putnam-like zo_putnam_nl_*.py family reduction modules driven by the flywheel schema router; run outside the sandbox Cold start, no stored answers Per-family certificate (substitute-back / exhaustive small-case / high-precision convergence / construction); every worker-produced solve independently re-derived before counting
Live chess autonomous Lichess client over the Rust bitboard engine Memory, learning, negative reinforcement disabled Per-ply legal-move verification; external Lichess rating + game outcomes
Apple puzzles scripts/zrb_planner_hanoi.py (N=3–10), scripts/zrb_hanoi_learn.py (N=1–18), scripts/zrb_hanoi.py (N=1–20); planner in PureZ/literacy/zo_planner.py and multiversedb/src/planner.rs Plan replayed against the problem's own rules + optimality witness (exact PDB distance or closed form)
Algorithm execution scripts/zrb_suite4_algorithm.py Exact integer reference comparison, 96/96

Comparison data and figures: holographic_store/zrb_puzzle_comparison.json (regenerated by scripts/zrb_puzzle_comparison.py); figures by scripts/zrb_puzzle_charts.pyDocumentation/figures/. Aggregate: python3 scripts/zrb_aggregate.py.

Z-O
Four domains. One loop. Nothing committed unverified.
AspenLabs Confidential — Internal Technical White Paper.
Andrew Zuk · AspenLabs, Inc
Response to The Illusion of Thinking
2026‑08‑07