Can LLMs improve their own harness?

Self-improving harnesses are a hot topic.

And one interesting question is whether LLMs can generate and improve them on their own.

They find that two full rewrites of the harness raise held-out scores by only 2.80 points, and the 95% interval crosses zero.

Each frontier model writes a complete harness from a fixed coding bootstrap, plays five development games through it, and then rewrites its whole codebase twice. Every version is scored on 20 games that never influence the rewrites.

Across 27 lineages, first harnesses range from zero to near-ceiling. The revisions add very little. Gains on the development games overstate the held-out gains, and revisions that cut inference cost often lower the score.

Human-steered systems still score higher on the same games, so better designs exist. The models just do not find them reliably on their own yet.

https://arxiv.org/abs/2610.30499

Chat with Paper: https://academy.dair.ai/papers/can-llms-improve-their-own-harness-a-controlled-study-on-arc-agi-3-2610.30499