LLMs Autononymous software harness improvement

The paper evaluates whether frontline language, models can autonomously construct and improve a software harness to enhance performance. An autonomous process may produce functioning software without preserving abilities or generalizing improvements beyond feedback tasks. Revisions may increase development performance on tasks, while cost and performance effects vary significantly and must be jointly analyzed. Where are follow-up questions? Research question and conceptual framework The paper investigates whether a frontier LLM can autonomously construct and improve the software harness through which it acts. This question is distinct from ordinary self-refinement, prompt optimization, or meta-learning. The model weights remain fixed; the mutable object is an executable program that controls perception, memory, planning, tool use, model-call allocation, recovery, and action execution. The study therefore treats the deployed agent as a coupled system consisting of a fixed model and an evolving harness. The central distinction is between four capabilities: constructing an executable harness, repairing operational failures, translating execution evidence into source-code changes, and selecting revisions that improve performance on tasks excluded from adaptation. The paper’s principal claim is that the first three capabilities do not entail the fourth. A model may produce functioning and substantially revised software while failing to preserve existing capabilities or generalize improvements beyond the tasks that generated the feedback. This framing is important because many existing approaches optimize a constrained artifact, such as a prompt, workflow, module, or predefined agent graph. In contrast, the present protocol allows the model to create and revise a complete task-specific repository. The study consequently evaluates autonomous harness engineering as a capability in its own right rather than as a by-product of model training or human-designed architecture. ARC-AGI-3 as an evaluation environment ARC-AGI-3 provides an appropriate testbed because its games require sequential interaction, exploration, latent-rule induction, and adaptation to unfamiliar mechanics. The agent receives grid observations, game state, progress information, and an action interface, but it is not given the semantics of visible objects or the conditions for completing levels. It must infer these through interaction. Figure 1: FT09 states illustrating an interactive …