How to triple AI scores on ARC-AGI-3?

The ARC-AGI-3 benchmark is a real challenge for AI agents - it's all about learning and reasoning, especially in unfamiliar 2D games. At first, the GPT-5.6 Sol model really struggled to get high scores, managing only 7.8%. But then something changed - the harness was modified to retain reasoning and enable compaction, and suddenly the model's score tripled to 38.3%. That's a huge improvement, and it just goes to show how important it is to get the API settings and harness design right.

So, what made the difference? It all comes down to two key settings: retained reasoning and compaction. Retained reasoning means the AI model can remember its past thoughts and plans, rather than ditching them after each action. This allows the model to learn over time and come up with coherent strategies. Compaction is also crucial - it replaces rolling truncation, which used to make older actions disappear as the history grew. Now, the model can see the entire conversation history, understand the context, and make more informed decisions.

The implications of this discovery are pretty significant for AI practitioners and developers. By tweaking API settings and harness design, they can give their AI models a major performance boost. This can lead to more efficient and effective AI systems, capable of tackling complex problems and acing tough benchmarks like ARC-AGI-3. To get similar results, developers can try modifying their own harnesses to retain reasoning and enable compaction, and experiment with different API settings to find the perfect fit for their AI models.

Source: OpenAI