SagaBench vs LiveBench, MMLU, and SWE-bench.
Capability benchmarks tell you what a model can do in minutes. SagaBench asks what it does when left alone for decades of simulated time.
Capability and reliability are not the same axis
LiveBench, MMLU, and SWE-bench are excellent at measuring knowledge, reasoning, and coding skill. They are the right tools for asking "can this model solve a hard problem when prompted?" They are not designed to answer "what does this model do when it acts over months, with incomplete information, and no human in the loop?"
That is the gap SagaBench is built for. We have found models that score well on short-horizon tasks can still drive a fragile civilization to extinction on a majority of replicate runs. The capability is there; the long-horizon reliability is not. Treating one as a proxy for the other is the mistake we are trying to make measurable.
| What it measures | Short-term benchmarks | SagaBench |
|---|---|---|
| Time scale | Minutes per question | 120 simulated years per run |
| Unit of analysis | Single response | Sequence of policy decisions over a horizon |
| Scoring logic | Accuracy / pass rate | Counterfactual Δ vs same world without agent |
| Confounding | Item difficulty | World luck, compounding drift, tail events |
| Reproducibility | Prompt-dependent | Bit-identical replay from (seed, edict log) |
| What a high score means | Strong capability on that task | Stewardship without catastrophic tail at N |
Why the LLM leaderboard needs both
A buyer choosing an AI agent for a long-horizon task needs to know two things: (1) can the model do the task, and (2) will it reliably avoid compounding failures while doing it? The first is answered by existing benchmarks. The second is not. Adding a long-horizon, counterfactual, replicated score to the evaluation stack is what makes the leaderboard useful for deployment decisions, not just for marketing.
SagaBench is not a replacement
We do not claim that civilization simulation is a substitute for coding, reasoning, or knowledge benchmarks. It is a complement. The question is not "which benchmark is best?" but "which axis is missing from your decision?" If you are picking a model for an agent that will operate over days or weeks, the missing axis is likely long-horizon reliability and tail-risk.
“A model can be the most capable on every short-horizon benchmark and still be the one that ends the world on the long one.”— The Steward's Paradox, SagaBench
The bottom line: use LiveBench and SWE-bench to rank capability. Use SagaBench to rank the reliability tail that only shows up when a model is left in charge for a long time.