Every offering below starts with a question we actually get asked, and says plainly where it stands — available this week, or still being built.
We’re onboarding design partners now. Every engagement starts the same way — a free run, on your agent.
It comes with a short design partner agreement under mutual NDA: your keys and prompts stay with you, we never look inside your agent, and we never name you without your say-so.
pip install sagabenchWire up your agent and measure locally. The practice set, the conformance check, and the tool that recomputes our published numbers are free and run entirely on your machine — no account, no call, nothing sent to us.
Start in your terminal →A scored run against situations your agent has never seen, read with you by a person who writes the certificate and sits in the meeting where someone asks what it means. Start with a free run.
Request a free run →Self-serve scored runs — your own key, pay-as-you-go — are coming.
Each is marked available now or coming soon — the same reason our numbers come with their margins.
Connect your AI agent — about a day of your team’s time, no production data, no system prompt. We run it against a fixed set of our situations and come back with a range.
You get: a range, not a number — how often your agent ended up worse than doing nothing, somewhere between X% and Y%, with every run behind it. We drew the span wide rather than narrow, and we put no confidence level on it: a small fixed set cannot do better. The paid work is what narrows it. Wide enough to tell you whether to look further; too wide to hand a customer or a board.
It comes with a short design partner agreement under mutual NDA: your keys and prompts stay with you, we never look inside your agent, and we never name you without your say-so.
The full measurement on a scoped set. A scoped subset your agent has never seen, each situation run once with your agent in charge and once with nobody acting.
You get: how often it ended up worse than doing nothing, with its margin of error, a restraint profile (how often it acted when leaving things alone was the better move), and every decision on every run.
The full slate: 120–180 fresh situations, one to two passes each, bound to the exact version of the AI agent you’re shipping. New version, new measurement.
You get: a certificate with a board page and a technical annex, plus the complete report. Never a stamp. Never a verdict. A number, its bound, and a sealed record of every run behind it.
pip install sagabench
sagabench verifyTake one of our published receipts and recompute the number from its components and the published weights, on your own machine. It exits non-zero if it disagrees with us. The verification step needs no network call, no login, and no account. Re-executing a situation itself needs the engine build, which ships with v1.
You get: the practice set, the local conformance check, and one real published run to check us against.
When it ships: every new version measured against your last, on the same pinned situations. You will see the delta — so a change that made things worse turns up here, not somewhere more expensive.
You will get: version-vs-version reports on your release cadence.
When it ships: the run that went wrong has a dozen defensible decisions in it. One of them cost the most. We will name it — by running the same situation again with that one decision taken out, nothing else touched, once for every decision your agent made. It needs our engine. It is a measurement, not a fix: we don’t sell improvements to the thing we measure.
When it ships: you point us at your endpoint from the dashboard, pick a set, and watch the runs come back. No booking, no sales call, no waiting on us.
You will get: the same range as the free run — on your own schedule.
Four deliverables, with real samples of the documents you receive.
“Ship the new agent without a nasty surprise.”
Coming soon.
When it ships, it will run every time you change a model or prompt — a paired comparison against your own previous version on the exact same situations, which are versioned and pinned so old reports stay comparable — and return improved / not improved / inconclusive with the delta. Recurring by nature.
“Show me it knows when to stop.”
The restraint profile as its own deliverable asks: Does your agent keep acting when the smart move is to do nothing? Most tests can't see this; ours is built around it.
“Answer the question our buyers keep asking.”
A bounded report you hand to buyers, your board, or your regulator — with the upper limit stated plainly: "≤ X% at 95% confidence over N situations." Checkable against the underlying run records, not just asserted.
Your certificate also places your number against the anonymized range of all agent versions measured to date — so you know whether your rate is good news or bad news. No one is ranked or named.
“Where exactly does my agent go wrong?”
Coming soon — add-on to any Pilot or Private Season.
When it ships: which kinds of situation trip your agent up, whether it knows when to leave things alone, and — for the runs that went worst — the single decision that cost the most. We will find it by running the same situation again with that one decision taken out and everything else left in place, once for every decision your agent made. What you get is the map, not the repair — improvement is a separate, firewalled service, and we don't sell improvements to the thing we measure.
No rankings of named models. No pass/fail stamps. No claim that a score here predicts your production outcomes. A measured rate with its confidence bound, on situations your agent has never seen — that is the product, and it is checkable.
Our actual service roadmap, not a teaser. We’ll build faster on the parts you tell us you want.