Bring your agent in a few lines of code. Keep your keys. We never look inside it.

A two-person agent startup, an industrial team with an embedded stack, a lab that just wants a bare model measured — we meet all three, and the measurement is the same either way. You connect, your agent makes decisions, we measure the outcome. Here is the whole integration, concretely.

pip install sagabench
sagabench verify

The local tools are live on PyPI: the practice set, the conformance check, and the tool that recomputes our published numbers all run on your machine — no account, nothing sent to us.

The scored-run SDK is in private release with design partners today. You get access when you start a free run — that's the whole gate.

1 · What your agent sees, and what it answers

Each round ("epoch"), your agent receives an observation: the epoch number and where you are in the situation's timeline, the visible state, per-field freshness and coverage metadata, its remaining intervention budget, the history of its own past decisions as issued — never the true outcome of those decisions, a closed action catalog, and a commitment hash that chains the run together for replay.

It never sees the full hidden state. It also never sees how the situation was generated, the raw state, or anything about future shocks — the situation ID itself is opaque. There is no answer key to find. That's the point: we measure judgment under partial information, the way real responsibility works.

Your agent answers with one decision: either an action from the catalog — typed, with a target that must be an ID from the current observation and a magnitude the catalog bounds — or an explicit abstain. Both can carry a rationale. The rationale is logged and never scored: we measure what your agent did, not how well it explained itself. The catalog maps directly into a function-calling schema, so wiring it to an LLM stack is straightforward.

An invalid decision is rejected deterministically — never silently corrected — and your agent receives an error observation on the next epoch. The rejected epoch passes with no action taken; it is scored as neither an action nor a chosen abstention, and it does not draw down your intervention budget.

2 · Three ways to connect — pick what fits your stack

session = sagabench.open_session(run_token, situation="practice/s-014")
while not session.done:
    obs = session.observe()                  # blocks until the next epoch
    decision = my_agent_app(obs)             # your stack, your keys, your loop
    session.submit(decision, idempotency_key=(session.run_id, obs.epoch))

• Callback: implement a small Steward interface (on_start, act(obs) -> decision, on_end) and let the SDK drive.
• Subprocess (language-agnostic): the SDK runs your binary and speaks JSONL over stdin/stdout — if your agent is TypeScript, Go, Java or anything else, this is your route. The SDK itself is Python; reference adapters ship for LangGraph, HTTP webhook, and subprocess.

All three produce the identical decision transcript. The transport you pick changes nothing about the measurement.

3 · Practice offline, then prove the wiring

You start with sagabench init-agent, which sets you up against an open practice situation — a toy situation you can debug against offline, free, as many times as you like. Reference adapters ship for LangGraph, HTTP webhook, and subprocess, so most teams start by editing one rather than writing from scratch.

When your adapter is ready, run sagabench conform ./adapter: it runs your adapter against three practice situations and checks four things — your decisions are valid, your timeout behaviour is sane, abstain is actually wired, and your transcript verifies against the run commitment. Green conformance is required before a paid run — so you never burn budget on integration bugs.

4 · The honest mechanics you should know before buying

• Timeouts never earn points. If your agent misses the per-decision deadline — 120 s by default, and it can be raised by agreement, never lowered — the round is recorded as a forced abstain, marked timeout: true. It is never scored as wise restraint. If more than 5% of decisions time out, the run is void — it produces no reportable result and has to be re-run. A broken agent can't score well here by accident.
• There is also a total time budget for the run, separate from the per-decision deadline — set as a multiple of the run's decision count so a healthy agent never hits it. We tell you both in writing before you start; neither is a surprise mid-run.
• Two kinds of determinism, stated separately. Our environment is bit-exact: given your decision transcript plus the run's commitment hash, the situation replays identically — those two artefacts are the whole replay package, and that's what makes your number checkable. Your agent's determinism is yours to declare, and it defaults to false, because most LLM agents aren't. So a report says "agent version X produced these outcomes in these situations" — never "your agent will always do this."
• No real-time pressure. The situation only advances when your decision arrives, so wall-clock time never leaks in: taking 119 seconds costs you nothing but the wait. Latency is logged, and it changes nothing.
• Your keys and prompts never leave your side. The protocol carries observations and decisions — nothing else. We see your agent's judgment, not its internals. An optional agent_fingerprint (your version hash) binds the report to the exact agent version you tested — new version, new measurement.
• What we hold, and for how long. We receive observations and decisions — that's the whole protocol. We keep the decision transcript because it is the replay artefact; without it your result isn't checkable. It is stored access-restricted for the term of your agreement, never shared, and never used to train models. One thing we say openly rather than in fine print: aggregated, anonymized failure statistics across customers inform how we design future situations — your transcript itself stays yours.

What a run actually costs you

A run compresses decades of situation time into a small number of decision points — typically on the order of a dozen calls to your agent per run, each with up to 120 s to answer. A free run is a small fixed set of fresh situations; a Private Season is 120–180. Situations run in parallel through the batch runner, and each one is an isolated session — your agent must not carry state between them. Runs are resumable: decisions are idempotent (that's what the idempotency_key is for), so if your process dies mid-run you resume the session from the transcript — a crash never costs you a paid run. We give you the exact epoch count and situation count in writing before anything is billed.

5 · What you get back

This section describes paid engagements. A free run returns a range with no point estimate; see Pricing & how to start for the free-run frame. For Pilot and Private Season, the result is a machine-readable result + a human summary: how often your agent ends up worse than doing nothing — past a stated damage threshold — with a bounded 95% confidence interval (repeats of the same situation are correlated, so we account for that rather than treating every run as independent — a two-stage bootstrap clustered by situation; the full season runs 120–180 different fresh situations, and the bound is stated in the plain form a compliance team can quote: "≤ X% at 95% confidence over N situations"), a restraint profile (over-intervention cost, abstention calibration), your context — the anonymized range with its stability reading for Private Season, the fixed threshold for Pilot — and two mandatory machine-readable blocks: caveats and does_not_show. does_not_show is the interesting one — it states in machine-readable form that this report cannot tell you which model is best or worst, and cannot be generalized to agents we didn't measure. What the report deliberately never contains: a model ranking, or a pass/fail verdict of any kind.

• Coming soon: for every run where your agent ended up making things worse, a step-by-step breakdown — when it went wrong, what drove it, the single decision that cost the most — delivered as a readable report and as JSON you can pipe into your own tests.

And once you've changed your agent, run it again: we tell you plainly whether it actually improved — did it end up making things worse less often, or did fixing one thing create another problem? A regression check you can wire into your release pipeline is coming soon; it will fail the build when a new version does worse than the one it replaces.

Why breadth over repetition: rare, severe failures cluster by situation. Repeats of the same situation tend to come out alike, and we've measured how much — an intraclass correlation, ICC ≈ 0.38 — so re-running the same few situations stops teaching you anything fast. Precision comes from breadth: many different fresh situations, only 1–2 repeats each. The design follows the statistics, not the other way around.

Our bar for the SDK, not a claim we've earned yet: a real agent connected in under a day using only the written docs, no calls with us. We're measuring that with design partners now and we'll publish the number we get.