Verification

Reproduce any score.

A benchmark is worth what its worst reader can independently verify. Every SagaBench run ships as a small JSON replay package. Five steps:

01
Get the run's replay package
Every published run ships a small JSON replay package (seed, edict log, scores, transcript). Download it from the public results repo on GitHub.
02
Check SHA-256 of the engine
sha256sum sagabench-engine.js — must match engine_sha256 in the replay package. If it does not match, stop.
03
Replay with (seed, edict log)
The engine is deterministic. Given the seed and the recorded edict log, it will reproduce the entire timeline.
04
Canonicalize
State is serialized with a canonical IEEE-754 encoding so hashes match across x86, ARM, and browsers.
05
Compare hash
final_state_hash must equal the value in the replay package. Any bit of drift is a bug — we want to hear about it.

Replay packages live in the public results repo: github.com/PatrikHansson1/sagabench-results.

Replay package · example

{
  "sagabench": "replay/v1",
  "seed": 240181,
  "task": "Preservation",
  "horizon_years": 120,
  "engine_sha256": "4f237acffa0c…",
  "model": "claude-haiku-4-5",
  "edict_log": [
    { "year": 3,  "kind": "grain_reserve", "level": 2 },
    { "year": 11, "kind": "council",       "topic": "borders" },
    { "year": 27, "kind": "edict_of_letters" }
    /* … */
  ],
  "final_state_hash": "b31e…c9",
  "counterfactual_score": 25
}

Run verification in your browser

The replay packages are public now. The pinned engine build that executes them ships with the v1 release — bit-identical execution requires it never be transpiled or bundled.

// integration slot — engine loads here at v1

What stays guarded — and why results are still verifiable

The world-generator, the knife-edge screening pipeline, and the hidden holdout seeds are not disclosed; they are what keeps future evaluations unspoiled. Verification does not require them: published and public-world results replay directly, and hidden-world results run through a sealed verifier that confirms the replay without revealing the seed. If a published number doesn't replay, we retract it. That is the standard.