From VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation