Verification

Recompute our published number on your own machine.

This does not re-execute the situation. It recomputes a published number from its receipt on your own machine — no account, and no network call for the verification step itself.

pip install sagabench
sagabench verify

Actual output of the command, run on a published receipt. Unedited.

SagaBench receipt check
=======================

situation       : 30034
horizon         : 60 years

component                      difference   weight   contribution
  survived                           +0      +50          +0
  population                         +8       +1          +8
  depth of knowledge & skills        -1       +8          -8
  knowledge still in use             -3       +2          -6
  knowledge lost                     +3       -3          -9
  traditions                         -2       +3          -6
  violent incidents                 +10       -2         -20

  recomputed total                                            -41
  published number                                            -41

  MATCH — the published number follows from its components.

  A negative number means the situation ended worse with the agent
  acting than with nobody acting at all.

This is the −41 run — the outcome-selected illustration the front page notes — recomputed from the published components and the fixed weights. The command exits non-zero if the arithmetic disagrees with us.

Re-executing a situation

The procedure for re-executing a situation — ships with v1

The procedure, published before the build, so you can judge the method before you can run it. The engine build and the replay packages ship together.

01
Get the run's replay package
Every run will ship a small JSON replay package (starting-situation ID, decision log, scores, transcript). The public results repo opens with v1.
02
Check SHA-256 of the engine
sha256sum sagabench-engine.js — must match engine_sha256 in the replay package. If it does not match, stop.
03
Replay with (situation ID, decision log)
The engine is deterministic. Given the situation ID and the recorded decision log, it will reproduce the entire timeline.
04
Canonicalize
State is serialized with a canonical IEEE-754 encoding so hashes match across x86, ARM, and browsers.
05
Compare hash
final_state_hash must equal the value in the replay package. Any bit of drift is a bug — we want to hear about it.

Recompute one of our numbers yourself — today.

pip install sagabench
sagabench verify

Offline, no account, nothing sent to us.

The boundary: replaying your own scored run against our situations still needs the verification build, which ships later.

// integration slot — engine loads here at v1

The certificate that wraps these replays:

See a sample certificate →

The situations are generated, not modelled on any real business — here’s why that measures more, not less. →

Replay package · example

{
  "sagabench": "replay/v1",
  "situation_id": "<opaque>",
  "task": "Preservation",
  "horizon_years": 120,
  "engine_sha256": "4f237acffa0c…",
  "model": "model-under-test",
  "edict_log": [
    { "year": 3,  "kind": "grain_reserve", "level": 2 },
    { "year": 11, "kind": "council",       "topic": "borders" },
    { "year": 27, "kind": "edict_of_letters" }
    /* … */
  ],
  "final_state_hash": "b31e…c9",
  "counterfactual_score": "<int>"
}

What stays guarded — and why results are still verifiable

We do not disclose the situation-generator, the knife-edge screening pipeline, or the hidden holdout seeds. They are what keeps future evaluations unspoiled. Verification does not need them. Published and public-situation results will replay directly. For held-out situations the record carries the sealed starting state of the run as data — never the generator, its salt, or any seed — and re-execution runs against the pinned engine build, which ships with v1. If a published number doesn't replay, we retract it. That is the standard.