SagaBench
Research

The method, in the open.

The method — counterfactual scoring against doing nothing, the replication design, and the group-level findings summarized on this site — is written up in full: The Steward's Paradox: When the AI You Put in Charge Does More Harm Than None — SagaBench: A Bit-Reproducible Benchmark for Long-Horizon AI Stewardship (Version 1.1). DOI: 10.5281/zenodo.21966877. Not yet peer-reviewed, and nothing on this site depends on it.

The runs themselves are public too: 2,415 from the first study and 960 from the second, one file each, with the preregistrations and two scripts that recompute every published rate in a few minutes — Zenodo, DOI 10.5281/zenodo.22979474, CC BY 4.0. To check the paper's arithmetic, you no longer need us.

The numbers stand on their own: 2,415 internal validation runs, zero deviations. Want to check that yourself? Download the records and recompute — or we walk your technical team through it.

New to this kind of measurement? Start with our guide to agentic AI testing over long horizons.

What we don't claim: peer-reviewed status, model rankings, or any prediction of how your agent will behave in the real world. What we do claim, you can re-run.

The finding, in one picture:

Where the tail lives

The same situations sink whichever models you test.

0%4%8%12%16%9.6%A2.9%B1.7%C7.5%D4.1%E14.5%F4.9%GSITUATIONgroup rate 6.5% [2.6–11.1]about 1 run in 15 — between 2.6% and 11.1%

One situation alone (F) carries about a third of all the bad runs. Swap out half the models and the hard situations stay hard; swap out half the situations and the model ordering falls apart (ρ = 0.75 [0.32–0.96] against ρ = 0.13 [−0.26–0.29], mirror split-half over 23 models). That asymmetry is why a single test can't find this, and why every engagement runs many fresh situations.

Per-situation rates, no intervals shown: with roughly 345 runs behind each bar, every one of these carries a wide bound, and the gaps between neighbouring bars are not something we would defend. What the picture is for is the shape — a few situations carry most of the tail.

Validation batch: 2,415 runs across 23 models and the Season 1 situation set · “worse than doing nothing” = past the m=−30 threshold vs the same situation left alone · a property of the measured group, never one model