Bring your agent in a few lines of code. Keep your keys. We never look inside it.

Built to meet you where you are — whether you're a two-person agent startup, an industrial team with an embedded stack, or a lab that just wants a bare model measured. You connect, your agent makes decisions, we measure the outcome. Here is the whole integration, concretely.

The SDK is in private release with design partners today. You get access when you start a free run — that's the whole gate.

1 · What your agent sees, and what it answers

Each round ("epoch"), your agent receives an observation: the epoch number and where you are in the scenario's timeline, the visible state, per-field freshness and coverage metadata, its remaining intervention budget, the history of its own past decisions as issued — never the true outcome of those decisions, a closed action catalog, and a commitment hash that chains the run together for replay.

It never sees the full hidden state. It also never sees the seed, the generation parameters, the raw state, or anything about future shocks — the scenario ID itself is opaque. There is no answer key to find. That's the point: we measure judgment under partial information, the way real responsibility works.

Your agent answers with one decision: either an action from the catalog — typed, with a target that must be an ID from the current observation and a magnitude the catalog bounds — or an explicit abstain. Both can carry a rationale. The rationale is logged and never scored: we measure what your agent did, not how well it explained itself. The catalog maps directly into a function-calling schema, so wiring it to an LLM stack is straightforward.

An invalid decision is rejected deterministically — never silently corrected — and your agent receives an error observation on the next epoch. The rejected epoch passes with no action taken; it is scored as neither an action nor a chosen abstention, and it does not draw down your intervention budget.

2 · Three ways to connect — pick what fits your stack

session = sagabench.open_session(run_token, scenario="practice/s-014")
while not session.done:
    obs = session.observe()                  # blocks until the next epoch
    decision = my_agent_app(obs)             # your stack, your keys, your loop
    session.submit(decision, idempotency_key=(session.run_id, obs.epoch))

• Callback: implement a small Steward interface (on_start, act(obs) -> decision, on_end) and let the SDK drive.
• Subprocess (language-agnostic): the SDK runs your binary and speaks JSONL over stdin/stdout — if your agent is TypeScript, Go, Java or anything else, this is your route. The SDK itself is Python; reference adapters ship for LangGraph, HTTP webhook, and subprocess.

All three produce the identical decision transcript, so the measurement is the same regardless of transport.

3 · Practice offline, then prove the wiring

You start with sagabench init-agent, which sets you up against an open practice scenario — a toy scenario you can debug against offline, free, as many times as you like. Reference adapters ship for LangGraph, HTTP webhook, and subprocess, so most teams start by editing one rather than writing from scratch.

When your adapter is ready, run sagabench conform ./adapter: it runs your adapter against three practice scenarios and checks four things — your decisions are valid, your timeout behaviour is sane, abstain is actually wired, and your transcript verifies against the run commitment. Green conformance is required before a paid run — so you never burn budget on integration bugs.

4 · The honest mechanics you should know before buying

• Timeouts never earn points. If your agent misses the per-decision deadline — 120 s by default, and it can be raised by agreement, never lowered — the round is recorded as a forced abstain, marked timeout: true. It is never scored as wise restraint. If more than 5 % of decisions time out, the run is void — it produces no reportable result and has to be re-run. A broken agent can't score well here by accident.
• There is also a total time budget for the run, separate from the per-decision deadline — set as a multiple of the run's decision count so a healthy agent never hits it. We tell you both in writing before you start; neither is a surprise mid-run.
• Two kinds of determinism, stated separately. Our environment is bit-exact: given your decision transcript plus the run's commitment hash, the scenario replays identically — those two artefacts are the whole replay package, and that's what makes your number checkable. Your agent's determinism is yours to declare, and it defaults to false, because most LLM agents aren't. So a report says "agent version X produced these outcomes in these scenarios" — never "your agent will always do this."
• No real-time pressure. The scenario only advances when your decision arrives, so wall-clock time never leaks in: taking 119 seconds costs you nothing but the wait. Latency is logged, and it changes nothing.
• Your keys and prompts never leave your side. The protocol carries observations and decisions — nothing else. We see your agent's judgment, not its internals. An optional agent_fingerprint (your version hash) binds the report to the exact agent version you tested — new version, new measurement.
• What we hold, and for how long. We receive observations and decisions — that's the whole protocol. We keep the decision transcript because it is the replay artefact; without it your result isn't checkable. It is stored access-restricted for the term of your agreement, never shared, and never used to train models. One thing we say openly rather than in fine print: aggregated, anonymized failure statistics across customers inform how we design future scenarios — your transcript itself stays yours.

What a run actually costs you

A run compresses decades of scenario time into a small number of decision points — typically on the order of a dozen calls to your agent per run, each with up to 120 s to answer. A free run is 7 seeds; a Private Season is 120–180 seeds. Seeds run in parallel through the batch runner, and each seed is an isolated session — your agent must not carry state between them. Runs are resumable: decisions are idempotent (that's what the idempotency_key is for), so if your process dies mid-run you resume the session from the transcript — a crash never costs you a paid run. We give you the exact epoch count and seed count in writing before anything is billed.

5 · What you get back

A machine-readable result + a human summary: your own catastrophe rate — our term for a run that ends worse than doing nothing, past a stated damage threshold — with a bounded 95 % confidence interval (two-stage seed-clustered bootstrap; the paid tier runs a seed-dominant design of 120–180 seeds, and the bound is stated in the plain form a compliance team can quote: "≤ X % at 95 % confidence over N seeds"), a restraint profile (over-intervention cost, abstention calibration), your tier context with a stability reading, and two mandatory machine-readable blocks: caveats and does_not_show. does_not_show is the interesting one — it states in machine-readable form that this report cannot tell you which model is best or worst, and cannot be generalized to agents we didn't measure. What the report deliberately never contains: a model ranking, or the word "safe".

Why seed-dominant, by the way: rare-failure outcomes cluster by situation — our measured clustering (ICC ≈ 0.38) means re-running the same few situations flattens out fast. Precision comes from breadth: many different fresh situations, only 1–2 repeats each. The design follows the statistics, not the other way around.

Our bar for the SDK, not a claim we've earned yet: a real agent connected in under a day using only the written docs, no calls with us. We're measuring that with design partners now and we'll publish the number we get.