Evolving Agent Interfaces in Computer Use Agents

The paper demonstrates that evolving interfaces for computer-use agents significantly improves their performance and efficiency, achieving a 3.8-fold increase in binary success on OSWorld V2 tasks while reducing task-solving cost from $1.39 to $1.10. Agent S4, a population-based evolutionary system, constructs and refines interfaces using synthetic tasks, crossover, and mutation techniques, selecting the best repositories through held-out validation tasks. The paper’s findings indicate that interface evolution facilitates both intermediate progress and complete-task reliability, supporting the agent’s efficiency in long-horizon and specialized task environments, such as scientific software. Problem formulation and central claim “Evolving Agent Interfaces” (2610.28601) argues that the interface through which a computer-use agent observes and manipulates its environment should be treated as an evolvable agent component rather than as a fixed, human-designed prerequisite. The paper targets a specific failure mode of current computer-use agents: a capable model may still be forced to perform inefficient, error-prone operations because it can access an application only through screenshots, mouse clicks, keystrokes, or a narrow collection of manually specified tools. In such settings, the model often compensates with additional inference and interaction steps instead of obtaining a more appropriate representation of the task state. The paper defines an interface as a mechanism connecting an agent to part of a computer through an action language and an observation language. A GUI interface, for example, accepts pointer and keyboard actions and returns screenshots; a shell or application API accepts structured requests and returns textual or structured observations. An Evolving Agent Interface is a reusable executable interface constructed or revised from task experience, together with supporting knowledge about its capabilities, configuration, workflows, and appropriate usage. These components are stored in a persistent repository RR. For a fixed task agent AA, the objective is to select a repository that maximizes expected task performance over a target task distribution. The important restriction is that the task agent’s model and harness remain fixed during evolution. Agent S4 changes the repository of interfaces and usage knowledge, not the underlying LLM, invocation mechanism, permissions, or task budget. The paper’s central methodological claim is that …

Explained