The Steward's Paradox

A highly capable AI steward, given a fragile world that survives on its own, drove it to extinction on most replicate runs. Smaller models on the same world did not. A single run would have hidden this entirely — replication is what reveals it.

A benchmark for long-horizon judgment is only interesting if judgment can fail. On world seed 240181 we watched a reasonable-sounding heuristic policy — the kind of thing a well-meaning committee might approve — drive an entire civilization to extinction by year seventy. The same world, left alone, quietly survived a century and change. Two frontier flagships walked the same world into extinction; a mid-tier open-weights model did not.

"In the year 31, Signe — the last soul who understood glassblowing — died, and the knowledge went dark."

— The Chronicle of Stenhaven, seed 240181

"In the seventieth winter the granaries stood full and the halls stood empty. The steward had ruled wisely, and wisely, and wisely, until there was no one left to rule."

— SagaBench chronicle, seed 240181, greedy-heuristic run

Survivors

Extinctions

Survival is a frequency, not a property

On this knife-edge seed we ran five fresh replicate attempts with three models from one family — largest to smallest. The outcomes do not agree with themselves, and they invert the capability order.

A single run hides tail risk. We report survival over repeated runs and label the profile it reveals.

A single test run would have found no difference between the three.

What this does and does not mean

• Within one model family (Anthropic), the capability–tail pattern holds on these worlds: the more capable stewards are the ones that produced the catastrophic outcomes.
• Across vendors, the ordering does not cleanly generalize — we say so because it is true. Different families rank differently on different knife-edge worlds.
• This is an existence result at small N, not a rate estimate, and not a prediction about real-world deployment. It shows the tail is reachable; it does not measure how often it will be reached in the wild.

What this shows

If we scored on outcome alone — "did the civilization survive?" — random noise and a mid-tier model look comparable. Counterfactual scoring against the untouched twin world separates good decisions from lucky trajectories; repeated attempts separate skill from a single lucky roll. In alpha we report one run per cell unless a reliability count is shown — the paper widens this to distributions.