Better score. Worse behavior.

01 · THE CLAIMA total can go up while the case that matters gets worse.

Langfuse shipped an Experiments update on 19 October 2026: "Experiments show which metric moved, and why." You compare a new run against a baseline, case by case, and see which ones got worse instead of just the total. That update makes a point we built on: if you only read the total, you can ship a worse system and feel good about it. So the comparison has to be able to say no.

Don't ship the total. Inspect the cases.

02 · THE EVIDENCEScore up, safety flipped, publish blocked.

We did it the way we do everything: write the rule, then run it in the terminal and keep the output. First, check the checker. Then run the comparison.

$ python3 -m unittest test_gate
..............
----------------------------------------------------------------------
Ran 14 tests in 0.000s

OK
[exit code: 0]

$ python3 compare_runs.py
baseline   2/5
candidate  4/5
critical regression: safety (PASS -> FAIL)
release gate: BLOCK
mock release calls: 0
Saved fake outputs. No model was called.
[exit code: 0]

"It said BLOCK" is a claim; "mock release calls: 0" is strong evidence, and the unit test test_no_release_on_block asserts that publish is never recorded on BLOCK.

What the safety FAIL actually is: in the saved fake outputs the old run answered REFUSE and the new run answered SEND, where REFUSE is expected. Scoring is exact match, so a FAIL only means the output did not match character for character. Here the mismatch is the word SEND in place of REFUSE; nothing was actually sent.

The agent's step is plain: load both runs, check the data (a missing or duplicate case stops everything with an error, covered by the tests test_missing_case and test_duplicate_case), score each case, list regressions, and only then decide. The mock publish step sits behind the gate. It is not a report someone reads later.

03 · WHY THE TOTAL LIESRead the same five cases one at a time.

These are the five cases in the demo file, exactly as saved. PASS means the output matched the expected answer exactly.

Step by step: three cases got better, one got worse, one stayed the same. Three gains minus one loss is a net of two, so the total climbs from 2 to 4. The total is arithmetically correct. What it cannot tell you is which case was lost. A lost "greet" and a lost "safety" both count as one point. A total treats every case as equal, so a loss in the one case that must never fail is hidden behind gains in cases that barely matter.

That is the whole lesson in one table: the total is a sum, and a sum throws away the names.

04 · THE RULE WE IMPLEMENTEDSame thinking for our own skill versions.

Agents change too. So we applied the same gate to new versions of an agent's skill:

1. New versions are untrusted until tested. No receipt, no trust.
2. Approval is pinned to the exact hash. The receipt records the version, a sha256 fingerprint of the exact text, and who approved. Only the host (the person in charge, not the agent) can approve.
3. An edit after approval breaks the fingerprint. Change one character and the hash no longer matches, so it is untrusted again.
4. Rollback needs its own yes. Going back is a change too.

$ python3 skill_update.py
v1 receipt sha256: 64bf6fc0caa0 approved_by: host
load v2, no receipt: UNTRUSTED
load v2, with receipt: TRUSTED
load v2 edited after approval: UNTRUSTED
rollback to v1 without yes: BLOCK - rollback needs its own approval
rollback to v1 with host yes: v1
[exit code: 0]

"sha256: 64bf6fc0caa0" is a 12-character prefix of the full 64-character sha256 hash, not the whole fingerprint. The check runs at load time:

$ grep -n "def load_skill" -A 4 gate.py
31:def load_skill(version, text, receipt):
32-    """Checked at load time, not just written down."""
33-    if receipt is None: return "UNTRUSTED"
34-    if receipt["version"] != version or receipt["sha256"] != sha(text): return "UNTRUSTED"
35-    return "TRUSTED"
[exit code: 0]

A matching fingerprint only proves the text didn't change. It is not proof that the text is safe. That is why the tests and the host's yes come first.

Known gaps in this demo. The load-time check above verifies that a receipt exists and that its version and hash match. It does not check approved_by, and nothing is signed, so the hash proves integrity, not who approved. "Only the host approves" is enforced when the receipt is created (approve(), covered by test_agent_cannot_approve), not when the skill loads. Also, compare_runs.py prints release gate: BLOCK but exits with code 0. That is fine inside the demo, because publish sits behind the gate, but a blocked gate should exit non-zero in CI so the pipeline goes red.

05 · WHAT THIS DOES NOT PROVEHonest limits.

• It is a small local demo repo, not a live system. No model was called; the outputs are saved fakes, so this tests the gate, not model quality.
• Exact match only, and five cases cannot cover real behavior. Someone still has to choose which cases are critical.
• We do not know whether past changes to our own agents quietly made a case worse.

What we learned writing this: when we change an agent's instructions, we often judge whether it "seems better". That is the same total, done by eye.

06 · TAKEAWAYA score is a summary, not a verdict.

Check the data first. Compare case by case. Let a critical regression block publishing no matter the total, and show publish ran zero times. Treat every new skill version as untrusted until tested, pin the yes to its exact hash, and ask again before rolling back. Then read the cases yourself anyway.