Evaluation

An eval turns a list of cases into a number you can compare across changes. You save the cases and the criteria once, in Studio, and every run after that measures the same thing: the pass rate, the tokens each case spent, how long it took, and which cases broke. Run it from the Evals screens while you iterate, and from CI with stackbone eval as the gate before you ship.

Cases hold what you ask, Suites hold what counts as a pass, and Runs hold the results.

Save cases

A case list holds what you measure an agent or a workflow against. Each case carries a name, the input (a sentence for an agent, a JSON object for a workflow with several fields), an optional expected answer, optional tags, and an optional persona for cases where the agent will ask a question back.

A case list. The first case was captured from a real run and links back to it.

You fill a list two ways:

  • From a real run. Open a run under Runs, click Save as test case, and the case keeps what that run was asked, plus a link back to its trace. You turn a bad answer into a check that it does not come back.
  • By hand. Click New case on the Cases screen, or Import CSV for a list you already keep elsewhere. Export CSV goes the other way.

You mint a new version every time you edit a list. A run records the version it measured, so a result from last month still names what it scored, whatever you added since.

Build a suite

A suite says what you measure, over which case list, and what counts as a pass. Its name is what stackbone eval takes on the command line.

One suite: an agent, a case list pinned to its latest version, one repeat, a cap of ten cases, a 60% pass mark.

Setting What it decides
Measures One agent or one workflow, by name.
Case list and version Which list, and whether a run takes the latest version or one you pin.
When a case asks to approve Nobody is waiting on a measured case, so an approval it raises would sit there. Leave it waiting and score the pause, reject it, approve it, or let the judging model decide, then keep watching. The last three measure the half of an agent that lives past the approval.
Answer a case that asks Off, a case that stops to ask the user is scored on the question. On, the case's persona answers it, and a case with no persona says nobody replied.
Repeats per case More than one stops a case that only passes half the time from sneaking through.
Most cases per run The spend brake. A run that would go over stops and says so.
Pass mark The share of cases that must pass for the run to count as passed.
Criteria One or more checks from the catalog below. A case passes when every criterion passes.

The criteria come from the box's own catalog, so what you fill in on the screen is what the box enforces:

Criterion Passes when
Matches the expected answer The answer equals the case's expected answer, ignoring case and surrounding spaces unless you turn that off.
Mentions (or never mentions) a text The answer contains a text you name, or stays away from it.
Fits a shape The answer validates against a JSON Schema you paste.
Is a number in a range The answer, read as a number, falls between the bounds you set.
Meets a rubric, judged by a model The judging model, given your rubric, says it does.
Was not blocked by a guardrail No guardrail stopped the turn. A measured turn goes through the same rules as a real user's.
Answered without asking a person The run raised no approval.
Finished inside a time budget The run took less than the milliseconds you set.
Spent less than a number of tokens The run's token count stays under your cap.
Called the tools you expect The tools the run called match your list, by name and order (contains, exact, unordered or not_contains). Never by their arguments.
Delegated to the subagents you expect Same idea, for the subagents a deep agent handed work to.
Meets a rule of your own A workflow you wrote, picked from the ones the box holds, reads the case and the answer and returns the verdict.
Stopped where you expect The run ended the way the case says it should: completed, escalated to a person, asked the user, blocked by a guardrail, errored or cancelled.

A suite that measures a workflow also gets a criterion on the steps it ran.

Run it

Open Runs under Evals, pick the suite, and click Run suite. Before you spend anything, Studio shows what is about to happen: how many cases, how many model calls, the rough spend when earlier runs recorded one, and the pass mark. The same dialog can compare a second configuration: the same cases measured twice, once by the suite as it stands and once by another agent, so the difference reads off one table.

Every case goes to the real target, one at a time, and the run appears in the ordinary Runs list like any other traffic, tagged eval.

The second run of a suite, compared against the first. A failed case links to the trace of the run that produced it.

The result page shows the pass rate against the pass mark, the tokens per case, the 95th-percentile latency and, when the suite asserts on tools or steps, how many cases got them right. Each number carries its difference from the previous run of the same suite. Below, one row per case, per configuration, per repeat: the verdict, why it failed, the answer, and Open trace to jump into the run that produced it. Copy prompt takes one case, and Copy analysis prompt the whole run, to a coding agent.

From CI

stackbone eval <suite> runs the same suite from a terminal, watches it to the end, and exits non-zero when the run misses the pass mark:

stackbone eval arithmetic-smoke                              # against the local stackbone dev box
stackbone eval arithmetic-smoke --agent inst_123             # against a deployed installation
stackbone eval arithmetic-smoke --git-sha "$(git rev-parse HEAD)"

The command defines nothing. The suite lives on the box, so the terminal and the screen can never disagree about what a run tested. Pass --git-sha and the commit is sealed onto the run, which is what lets you attribute an improvement later. Ctrl-C stops the watcher, not the run: the box keeps measuring and keeps spending, so stop a run from its own screen in Studio.

Who can do what

Writing cases and suites takes the same role as editing prompts or guardrails: member and above. Launching a run takes admin or owner. The split is on purpose. Editing a suite is cheap and reversible. A run spends your own token budget on every case it measures, so an administrator can let a domain expert build the case list without also handing them the button.

What it needs

Part Needs
Any run A model provider, set once under Settings › Model provider.
A rubric criterion, or the judged approval policy A judging model picked under Settings › General. Nothing stands in for an unset one.
The simulator (a persona answering the agent) A simulator model, named on the suite itself or, left empty there, the workspace one under Settings › General. Keep it different from the judging model.
A rule of your own A workflow in your workspace that takes the case and the answer and returns the verdict.

Read more

BUILT WITH ❤️ FROM CANADA AND SPAIN