Run agents like a service.

01 · THE CLAIMAn agent that runs every day is a service.

Services have budgets, incident reviews and people who read the logs. Agents usually have none of these. Our claim: run agents the way SRE teams run services. Decide how many failed or retried runs you accept before you stop and fix. End every incident with a written rule and a test, not "be more careful". And check one trace by hand, against the plan.

Budget the failures. Every incident leaves a rule.

02 · WHAT HAPPENEDThree quick reads, one habit.

We had three short pieces on the blog: an error budget for agent runs, every incident leaves a rule, and a trace check bootcamp. Each was a tip on its own. Together they are how we plan to run our own agents. They build on earlier write-ups: A timeout is not a failure. (a timeout that reconciles to done is not a miss), The plan is not the trace. (the trace is the evidence) and Catch the secret before it leaves. (a check on every send). The three quick reads stay on the blog as cheat sheets that link here.

03 · WHAT WE READ100% is the wrong target, and "improve" is not an action.

• Google SRE book, Embracing Risk. "100% is probably never the right reliability target: not only is it impossible to achieve, it’s typically more reliability than a service’s users want or notice." And: "The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter." And: "As long as the uptime measured is above the SLO—in other words, as long as there is error budget remaining—new releases can be pushed."
• Google SRE Workbook, Postmortem Culture: Learning from Failure. "In general, trying to change human behavior is less reliable than changing automated systems and processes." On a bad example postmortem: "The first two action items in the list use ambiguous phrases like “Improve” and “Make better.” These terms are vague and open to interpretation."

We quote Google's published books only. We apply their ideas to agent runs; the books are about services.

04 · THE CHECKCount the misses, freeze when spent, close with a rule.

Same method as the earlier write-ups: write the rule, run it in the terminal, keep the output. This is the check, copied from error_budget.py:

def is_miss(run):
    if run["end"] == "TIMED_OUT":
        return run.get("reconciled") != "DONE"
    return run["end"] in MISS


def check(runs, budget):
    misses = [r for r in runs if is_miss(r)]
    left = budget - len(misses)
    if left > 0:
        return "SHIP", left, misses
    return "FREEZE", left, misses


def incident_close(note):
    """Every incident leaves a rule and a test."""
    need = ["evidence", "rule", "test"]
    gaps = [k for k in need if not note.get(k)]
    rule = note.get("rule", "").lower()
    if rule == "be more careful":
        gaps.append("rule is vague")
    if gaps:
        return "OPEN", gaps
    if not note.get("replay_passed"):
        return "OPEN", ["replay not passed"]
    return "CLOSED", []


def trace_check(planned, ran):
    """15-minute trace check: plan vs trace."""
    extra = [s for s in ran if s not in planned]
    missing = [s for s in planned if s not in ran]
    if extra:
        return "REJECT", extra, missing
    if missing:
        return "UNKNOWN", extra, missing
    return "PASS", extra, missing

First, check the checker:

$ python3 -m unittest test_error_budget
..............
----------------------------------------------------------------------
Ran 14 tests in 0.000s

OK

Then run six made-up runs against a made-up budget, plus two incident notes and one trace:

$ python3 error_budget.py
MADE-UP runs, made-up budget of 3
  run-a  DONE      -> ok
  run-b  TIMED_OUT -> ok
  run-c  RETRIED   -> miss
  run-d  DONE      -> ok
  run-e  FAILED    -> miss
  run-f  TIMED_OUT -> miss
misses 3 of 3 -> FREEZE
stop shipping; write a rule + test per miss
incident 'be more careful' -> OPEN
incident rule + test + replay -> CLOSED
trace: extra 'upload' -> REJECT

Run-b timed out but reconciled to DONE, so it is not a miss. Run-f timed out and was paused, so it is. The budget is spent, so the check says stop shipping. An incident whose only lesson is "be more careful" stays open. The trace line is the 15-minute check from The plan is not the trace.: a tool call that was never planned is rejected.

05 · HOW THE CHECK DECIDESThe rules in one table.

06 · WHAT THIS DOES NOT PROVEHonest limits.

• We haven't picked our real budgets. The demo's budget is made up. Our real targets are unknown until we count real runs.
• A budget says when to stop, not why. Traces explain the failure. The budget only counts it.
• The incident check reads a note. It checks that a rule, a test and a passing replay are recorded. It does not run the replay itself.
• Our rules today are not formal incident notes. They are written rules that came from our articles and from the owner's corrections.
• It's a small local demo. Made-up data, 84 lines of Python. Nothing in our real agents runs it yet.

"Built" here means written rules and a written map. They are not enforced by code yet.

07 · WHAT WE'LL BUILDMonitoring & trace, runbooks & playbooks (in development).

1. An error-budget policy per kind of run: target, budget, stop rule.
2. A budget tracker on the trace dashboard, with reconciled timeouts counted as done.
3. An incident → rule playbook: evidence, rule, where it loads, a replay test.
4. A 15-minute trace drill in the Toolkit, one run at a time.

08 · TAKEAWAYBudget the failures. Every incident leaves a rule.

Retry forever and you never finish. Quit at the first error and you waste good work. A budget decides where the line is, an incident note turns each miss into a rule and a test, and a trace check proves what really ran. Feel free to challenge our thinking. When does your agent stop and call a person?

TRY IT YOURSELFTwo checks for this week.

1. Count last week's agent runs that failed or were retried. Would you have stopped shipping?
2. Take your last agent mistake. Is there a rule and a test for it, or only a note to be careful?