FROM AIOPS TO AGENTOPS: SERVICE-LEVEL OBJECTIVES AND ERROR-BUDGET-GOVERNED AUTONOMY FOR PRODUCTION AI AGENTS
Proposes SLOs and error budgets to govern how much autonomy production LLM agents get, replacing signals built for deterministic services.
Abstract
Large language model (LLM) agents now run production workflows, including IT and network operations, yet they are usually monitored with signals designed for deterministic services: availability, latency, and error rate. An agent can satisfy all three while completing the wrong task. This paper extends AIOps-style service management to agentic systems. It contributes a layered set of service-level indicators (SLIs) covering platform health, step-level tool use, verified task outcomes, and guardrail integrity; a taxonomy of seven agent failure classes mapped to the signals that expose them; and an error-budget policy that governs an agent's autonomy as well as its release cadence. A Monte Carlo simulation of a production operations agent evaluates the approach across 2,100 injected incidents. Conventional platform monitoring detected only platform failures, firing in at most 1.7% of runs for the other six classes. Burn-rate alerting on verified task outcomes detected every class in 89.7% to 100% of runs within 24 hours, with no false alarms, but took a median of 3.5 to 10.6 hours under standard thresholds. A change-aware rule, active for six hours after a deployment or model change, cut median detection of change-induced regressions to 44 to 63 minutes at a cost of about one additional page per month. The results support layered, outcome-based service-level objectives as the operational basis for production agents.
Explained
What happened
A paper (World Journal of Advanced Engineering and Technology Sciences) proposes AgentOps: running production LLM agents with SLOs and error budgets, the way SRE teams run normal services. Only the abstract is available, so the details below come from it.
The problem: agents are usually monitored with signals built for deterministic services (availability, latency, error rate), and an agent can look perfect on all three while doing the wrong task.
It proposes:
- Layered SLIs (service-level indicators): platform health, step-level tool use, verified task outcomes, and guardrail integrity.
- A taxonomy of seven agent failure classes, each mapped to the signals that reveal it.
- An error-budget policy that controls autonomy, not only release speed: burn through the budget and the agent loses permissions (more human approval, fewer automatic actions).
A Monte Carlo simulation of an operations agent with 2,100 injected incidents:
- Conventional platform monitoring caught only platform failures, firing in at most 1.7% of runs for the other six classes.
- Burn-rate alerts on verified outcomes caught every class in 89.7% to 100% of runs within 24 hours with no false alarms, but slowly: median 3.5 to 10.6 hours.
- A change-aware rule (tighter thresholds for 6 hours after a deploy or model change) cut detection of change-induced regressions to 44 to 63 minutes, at the cost of about one extra page per month.
Why it works
Agents fail semantically: wrong tool, wrong arguments, a plausible but wrong answer, a guardrail quietly bypassed. HTTP 200 and fast responses say nothing about that. Measuring verified outcomes (did the ticket actually get resolved correctly?) catches these, and burn-rate alerting (how fast you're spending the error budget) is the standard SRE way to alert on rates without noise. Most regressions come right after a change, so tightening alerts in that window catches them fast.
The bigger pattern
This is SRE practice extended to non-deterministic systems: define success in terms of user outcomes, not process health. Tying autonomy to the error budget is a neat idea: trust is earned and can be revoked automatically.
What it means in practice
- For any agent you ship, define an outcome SLI (sampled human or automated verification of results), not just latency and errors.
- Treat model and prompt changes like deploys: version them, and watch outcome metrics more closely right after.
- Wire the error budget to permissions: when outcome quality drops, require approval for risky actions.
- Caveat: it's a simulation, not a production study.