how I accidentally got the top score on ARC-AGI-3 with 5.5x fewer tokens
two weeks of teaching an AI agent to solve 25 unfamiliar video games
so I saw this tweet that recommended looking into ARC-AGI, an AI benchmark on simple video games, as something to study if you’re into agent harness engineering. I love games and building harnesses, so I figured I’d check it out.
two weeks later, I had the highest publicly reported score in 5.5x fewer tokens than the next-best agent.
Each ARC-AGI-3 game is played on a 64x64 grid and contains 6-10 levels of increasing difficulty, each with a limited number of actions to beat it. an agent sees the grid (no other context), figures out how to play, and gets scored on how efficient it is relative to a human baseline (Relative Human Action Efficiency, RHAE for short).
july 5: seems simple enough
I cloned RGB-Agent (read, grep, bash) and baseline1 as reference agents. RGB was mentioned in the tweet and had the best score on the three preview games released in March; baseline1 was the current top scoring agent on the community leaderboard and operated by building a durable Python world model for each game, running inside codex.
I played through a couple of the simpler games by hand and thought “this doesn’t seem that complicated, I don’t think an LLM would need a full world model to solve it”. so my plan was to re-implement RGB-Agent using ThinHarness (my minimal agent harness framework), add a few improvements on top, and show that some subset of the 25 games could be solved more simply than baseline1 demonstrated--no executable world model, no coding agent harness, just a model in a loop with a python tool.
first run: ft09, a “lights-out” style game, with gpt 5.5 high, because RGB had already beaten it in only 78 actions. my port beat levels 1–4 in 46 actions, then got stuck on level 5 and burned 130 actions before hitting the $10 cost cap I’d set.
july 6-8: retrodict before you act
the traces showed it was spending actions to test hypotheses that were answerable with the existing log file, so I added retrodiction to the prompt: check every hypothesis against the recorded history in python before spending actions on it. additionally, every submitted action must carry an expected result, so a missed prediction cancels the rest of the batch instead of wasting additional actions. (the agent’s name, Retrodict, is based on this mechanic).
these changes improved behavior, but the agent still got stuck on level 5, which seemed consistent with RGB’s high variance and early hypothesis lock-in (as per their blog post). other games (vc33, sp80) showed the same pattern, where the agent solved early levels before hitting a wall.
after every run I had claude dig through the traces, and the issue was never a simple bug or one-line prompt change: the agent read the board correctly, retrodicted, and came up with reasonable hypotheses. the failures were subtler: hypothesis lock-in, planning through slightly incorrect models, inability to cut losses and reset. it understood the game but lost anyways.
july 8-9: vision and the first win
the fixes were all going to be substantial, and $10-20 per run meant expensive iteration. so I spent a day trying to find a cheaper model that was decent enough to iterate with. I tested qwen, minimax, grok, and sonnet, but none of them got past the first level of sp80; gpt 5.5 beat 4/6.
sp80 was a game baseline1 hadn’t fully solved; I just thought the first couple of levels would be easy based on playing it by hand. to be more systematic, I made an “easy list” of baseline1’s wins sorted by fewest actions. r11l was towards the top at 182 actions, so I ran gpt 5.5 on it before trying the cheaper models, and was shocked that the agent couldn’t even solve the first level--the traces showed that it never understood the goal (move the endpoints so the tethered purple ball reaches the purple target).
the goal felt obvious to me from the game’s appearance, so I sent a screenshot to an image model with no other context--it completely understood the mechanics and the goal. this led me to add an “image prime” step at the beginning: the first frame goes to a vision model, which writes a description, the likely goal, and a hypothesis of how to play. armed with this, the agent was able to... beat two levels. progress, but still below expectation.
next on the easy list was lp85, and the agent finally got a win! 8/8 levels in 125 actions, fewer than baseline1’s 190.
trace analysis showed hundreds of python calls re-deriving the same plumbing: parsing the log from scratch and implementing board diffs inline. so I gave it a small arclog library, a scratch/ folder to persist its own per-game helper code, and a [DIFF] block in the log listing exactly which cells changed at each step. that got r11l to 5/6, but it cost $36.51 on gpt 5.5 and the cheap models clearly weren’t strong enough, so I was worried that making any real progress would be too expensive.
july 10: a new model
then gpt 5.6 released.
perfect! cheaper access to gpt 5.5-level intelligence and higher max intelligence with sol!
but the 5.6 models were somehow worse.
5.6 terra was supposed to be a 50% cheaper gpt 5.5 equivalent, but in my initial probes it got stuck on level 2 of r11l; luna actually made it to level 3 once, but still not great. sol did well, but it was the same price as gpt 5.5, so I continued pushing on the cheaper models.
the traces revealed that they both understood the game perfectly, they were just acting super inefficiently compared to 5.5: instead of sending a whole batch of actions against a hypothesis, they sent a single action and looked at the result, over and over.
so I finally read the system prompt for myself, and it was obviously a mess. everything important lived under a single jumbled “## Method” section: tooling tips, perception warnings, epistemic method, and environment rules. I reorganized it into logical sections until it was actually comprehensible, then kicked off runs with terra and sol, same high effort as usual.
july 10-11: the top score
terra: 4/6 (crashed due to a parsing error in my runner code). sol: 6/6, my first-ever win on r11l, for $5.72. the prior gpt 5.5 run got 5/6 for $36.51.
it felt like magic.
that evening I tried 7 more games with a $12 cap each: 6 wins, and the seventh got close but hit the cap--on a game baseline1 had also failed. I let claude run the remaining 17 games overnight. by morning: 13/25 games solved, 65.13% mean RHAE, $270.34 total spent. baseline1’s amended score, documented in their repo, was 58.12%; the other highest entry on the community leaderboard was 63.1%, at a total cost of $4,788.
suddenly, I had the highest publicly reported score.
july 11: oh wait
until I realized that at 4:51 AM on july 10, 17 hours before the sol breakthrough on r11l, baseline1 had committed a v1.5 run: gpt 5.5 xhigh, 82.01% mean RHAE, 20/25 games solved.
july 12: memory loss
I had subagents analyze the traces of the 9 games that baseline1 v1.5 had beaten but Retrodict was still failing (it beat 2 games that baseline1 failed). the verdict: sol “couldn’t reliably forward-simulate hidden, temporal game dynamics”, and a full world model would help, but memory was the real blocker--I had context reset at 150k tokens, and after every reset it had to re-derive the rules of the game.
so I added playbook.md: a “short curated file holding only your durable, distilled knowledge,” carried across resets and maintained using ThinHarness’s built-in write/edit tools.
at first it was split between “Confirmed” knowledge and a “Current level” scratchpad; this helped beat two of the 9, but the agent still got stuck because it was locking in confirmed knowledge early that was actually wrong, which made solvable levels look impossible. I reframed the playbook as “Working model” / “Working memory” to prevent implied certainty, and 4 more games fell.
july 13-14: a real shot at the top score
at this point 19/25 games had a win on some version of the harness, against v1.5’s 20/25, so getting a real top score seemed possible. I ran the updated version of Retrodict overnight on the 16 games that it hadn’t tried yet, $25 cap each. when I woke up: 8 prior wins re-confirmed, the 6 never-solved games capped out, and two prior wins had regressed versus the pre-memory agent.
I re-ran the two regressions with no harness changes and they easily succeeded; I resumed three of the capped runs with more budget and two of them won.
july 14, ~7:30 PM, nine days after reading the tweet: 21/25 solved, 84.52% mean RHAE vs. baseline1’s 82.01%. the highest reported score.
july 15-16: not again...
it took some time to organize everything and put together my official submission. I made the PR, looked at the list of PRs on the leaderboard repo, and--
baseline1’s repo was untouched but their leaderboard PR had been updated the previous morning: gpt 5.6 sol xhigh, 98.97% mean RHAE, all 25 games solved. again, my score was never truly the highest.
july 16-17: world models as an escape hatch
that same night I was back at it. my hopes of taking the top spot had vanished, but that wasn’t my original goal anyways. looking at the four games my agent had never beaten, I had to admit that it seemed like an executable world model would actually be helpful for them: each game stalled on complex levels requiring long action sequences that were hard for Retrodict to test without spending actions.
the fix was enabling escalation: after 300 actions or two self-resets when stuck on a single level, a prompt directive gets sent that tells the agent to create and test an executable simulator with confirmed rules.
this was a move towards baseline1’s full world modeling approach, but as an escape hatch rather than a default. this allowed the agent to solve easy games cheaply and still beat the more involved games. three of the four remaining unsolved games were finally conquered; the fourth turned out to only need a re-run.
july 17: the second paper
the next evening, baseline1 restructured their repo around a second paper, Do Coding Agents Need Executable World Models, Simplification, and Verification to Solve ARC-AGI-3?
this paper contained a control version of the harness that only used a textual world model called twma (textual world modeling agent): no required executable simulator, durable rules in a text file. they then layered an executable world model, simplification, and replay verification on top.
on sol at max effort twma did quite well, beating 25/25 games, 95.97% RHAE, using roughly a third of the tokens of the executable world modeling agent (ewma).
july 18-19: the final run
taking my best runs on every game across all harness iterations, my ~92% RHAE was still well below 95.97%, but I was using ~2x fewer tokens than even twma.
in theory, Retrodict should have been able to at least match twma’s score; it had the executable escape hatch in addition to durable text memory. unless... I just did a worse job harness engineering?
I realized there was one lever I hadn’t pulled yet: effort. I had been using 5.6 sol on high effort, and never thought to change it since it was token efficient and doing well. baseline1 used xhigh (ewma) and max (twma).
so I did one last run of all 25 games using max-effort sol and the final version of the harness.
99.86% mean RHAE, 25/25 games solved. 660M tokens, 5.5x fewer than ewma’s 98.97% run and 1.8x fewer than twma’s 95.97% run. finally, 14 days after originally learning about ARC-AGI, I had the actual1 highest score on the benchmark.
official scorecard link, repo link
where the token gap comes from
full comparison of results:
RHAE is action efficiency, so Retrodict and ewma xhigh use a similar number of actions (7,703 vs. 8,347) for their very similar scores; nearly all of the 5.5x token gap is tokens per action: how much thinking each move costs.
for ewma, the token gap is so large because its simulator must be complete and is strictly verified: it has to correctly predict the value of all 4,096 cells in the 64x64 grid. a lot of effort is spent on being pixel-perfect rather than on core game understanding; on sb26, it spends 307 lines of code to model the cell rendering vs. 85 lines modeling the actual internal game mechanics.
Retrodict, on the other hand, only has to specify a small set of cell assertions for each action (median of 1 cell) that would falsify its hypothesis if incorrect. sb26 is the most extreme example of the inefficiency introduced by the strict world model: both baseline1 and Retrodict scored a perfect 100 in similar action counts (140 vs. 133), but baseline1 spent 68M tokens to my 5.8M--an 11.7x difference.
for twma, the token gap mostly exists because it can only play a single action per llm call, while Retrodict can batch multiple actions in a single request. this results in ~2.9x more llm calls but each call has ~0.6x as many tokens because it’s not doing as much work; 2.9x more llm calls * 0.62x tokens per call = ~1.8x more tokens overall.
an untested hypothesis
it wasn’t until july 20 that the baseline1 repo added the arxiv link for the second paper, whose conclusion left one thing untested:
it may be possible to improve the less resource-intensive textual variant enough to narrow or even reverse its performance gap with the verification variant. We did not test this possibility, so it remains a hypothesis for future work rather than a conclusion of the present study.
so it turns out that this whole time I was unintentionally testing the hypothesis of whether a resource-light harness based primarily on text could match the performance of a strictly verified one.
when I started, my pitch was that some of these games could be solved more simply than by using a full executable world model, with a minimal harness instead of a full coding agent. unexpectedly, and with the help of a new class of model intelligence, “some” became “all” and my agent ended up at the top of the leaderboard.
a few caveats:
based on the variation seen in baseline1’s new paper, the 0.89% difference is likely just noise. the real advantage here is token usage.
every game was run once on the final Retrodict harness, except tn36 and bp35; these re-runs were unintentional and are explained in the repo’s readme. baseline1 is much stricter, discarding the entire 25-game run on any irrecoverable technical failure.
everything this article describes is for the public benchmark; the official competition leaderboard on kaggle can only use open-weight models (top score of 2%), and the verified leaderboard appears to be model-only without a strong harness (top score of ~10% except for Opus 5 at ~30%).