--- title: 'Evaluation' description: 'Score an agent or a workflow against saved cases, in Studio or from CI, and tell whether a change made it better or worse.' position: 12 --- # Evaluation > An **eval** turns a list of cases into a number you can compare across > changes. You save the cases and the criteria once, in Studio, and every run > after that measures the same thing: the pass rate, the tokens each case > spent, how long it took, and which cases broke. Run it from the **Evals** > screens while you iterate, and from CI with `stackbone eval` as the gate > before you ship. **Cases** hold what you ask, **Suites** hold what counts as a pass, and **Runs** hold the results. ## Save cases A **case list** holds what you measure an agent or a workflow against. Each case carries a name, the input (a sentence for an agent, a JSON object for a workflow with several fields), an optional expected answer, optional tags, and an optional persona for cases where the agent will ask a question back. ![The Eval cases screen: the "Arithmetic smoke" list at version 3 with three cases, their tags, and the History, Import CSV, Export CSV and New case actions.](/images/evaluation/cases.png) _A case list. The first case was captured from a real run and links back to it._ You fill a list two ways: - From a real run. Open a run under **Runs**, click **Save as test case**, and the case keeps what that run was asked, plus a link back to its trace. You turn a bad answer into a check that it does not come back. - By hand. Click **New case** on the Cases screen, or **Import CSV** for a list you already keep elsewhere. **Export CSV** goes the other way. You mint a new **version** every time you edit a list. A run records the version it measured, so a result from last month still names what it scored, whatever you added since. ## Build a suite A **suite** says what you measure, over which case list, and what counts as a pass. Its name is what `stackbone eval` takes on the command line. ![The suite editor in Studio: name, what it measures, the case list and its version pin, what to do when a case asks for approval, the simulator switch, repeats, the case cap and the pass mark.](/images/evaluation/suite-editor.png) _One suite: an agent, a case list pinned to its latest version, one repeat, a cap of ten cases, a 60% pass mark._ | Setting | What it decides | | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Measures | One agent or one workflow, by name. | | Case list and version | Which list, and whether a run takes the latest version or one you pin. | | When a case asks to approve | Nobody is waiting on a measured case, so an approval it raises would sit there. Leave it waiting and score the pause, reject it, approve it, or let the judging model decide, then keep watching. The last three measure the half of an agent that lives past the approval. | | Answer a case that asks | Off, a case that stops to ask the user is scored on the question. On, the case's persona answers it, and a case with no persona says nobody replied. | | Repeats per case | More than one stops a case that only passes half the time from sneaking through. | | Most cases per run | The spend brake. A run that would go over stops and says so. | | Pass mark | The share of cases that must pass for the run to count as passed. | | Criteria | One or more checks from the catalog below. A case passes when every criterion passes. | The criteria come from the box's own catalog, so what you fill in on the screen is what the box enforces: | Criterion | Passes when | | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | Matches the expected answer | The answer equals the case's expected answer, ignoring case and surrounding spaces unless you turn that off. | | Mentions (or never mentions) a text | The answer contains a text you name, or stays away from it. | | Fits a shape | The answer validates against a JSON Schema you paste. | | Is a number in a range | The answer, read as a number, falls between the bounds you set. | | Meets a rubric, judged by a model | The judging model, given your rubric, says it does. | | Was not blocked by a guardrail | No guardrail stopped the turn. A measured turn goes through the same rules as a real user's. | | Answered without asking a person | The run raised no approval. | | Finished inside a time budget | The run took less than the milliseconds you set. | | Spent less than a number of tokens | The run's token count stays under your cap. | | Called the tools you expect | The tools the run called match your list, by name and order (`contains`, `exact`, `unordered` or `not_contains`). Never by their arguments. | | Delegated to the subagents you expect | Same idea, for the subagents a deep agent handed work to. | | Meets a rule of your own | A workflow you wrote, picked from the ones the box holds, reads the case and the answer and returns the verdict. | | Stopped where you expect | The run ended the way the case says it should: completed, escalated to a person, asked the user, blocked by a guardrail, errored or cancelled. | A suite that measures a workflow also gets a criterion on the steps it ran. ## Run it Open **Runs** under Evals, pick the suite, and click **Run suite**. Before you spend anything, Studio shows what is about to happen: how many cases, how many model calls, the rough spend when earlier runs recorded one, and the pass mark. The same dialog can **compare a second configuration**: the same cases measured twice, once by the suite as it stands and once by another agent, so the difference reads off one table. Every case goes to the real target, one at a time, and the run appears in the ordinary Runs list like any other traffic, tagged `eval`. ![An eval run in Studio: 67% passed, tokens per case, 95th-percentile latency, and a table with one row per case showing verdict, reason, answer and a link to the trace.](/images/evaluation/eval-run.png) _The second run of a suite, compared against the first. A failed case links to the trace of the run that produced it._ The result page shows the pass rate against the pass mark, the tokens per case, the 95th-percentile latency and, when the suite asserts on tools or steps, how many cases got them right. Each number carries its difference from the previous run of the same suite. Below, one row per case, per configuration, per repeat: the verdict, why it failed, the answer, and **Open trace** to jump into the run that produced it. **Copy prompt** takes one case, and **Copy analysis prompt** the whole run, to a coding agent. ### From CI [`stackbone eval `](/docs/cli/reference/eval) runs the same suite from a terminal, watches it to the end, and exits non-zero when the run misses the pass mark: ```sh stackbone eval arithmetic-smoke # against the local stackbone dev box stackbone eval arithmetic-smoke --agent inst_123 # against a deployed installation stackbone eval arithmetic-smoke --git-sha "$(git rev-parse HEAD)" ``` The command defines nothing. The suite lives on the box, so the terminal and the screen can never disagree about what a run tested. Pass `--git-sha` and the commit is sealed onto the run, which is what lets you attribute an improvement later. Ctrl-C stops the watcher, not the run: the box keeps measuring and keeps spending, so stop a run from its own screen in Studio. ## Who can do what Writing cases and suites takes the same role as editing prompts or guardrails: `member` and above. **Launching a run** takes `admin` or `owner`. The split is on purpose. Editing a suite is cheap and reversible. A run spends your own token budget on every case it measures, so an administrator can let a domain expert build the case list without also handing them the button. ## What it needs | Part | Needs | | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Any run | A [model provider](/docs/home/features/gateway), set once under **Settings › Model provider**. | | A rubric criterion, or the judged approval policy | A **judging model** picked under **Settings › General**. Nothing stands in for an unset one. | | The simulator (a persona answering the agent) | A **simulator model**, named on the suite itself or, left empty there, the workspace one under **Settings › General**. Keep it different from the judging model. | | A rule of your own | A workflow in your workspace that takes the case and the answer and returns the verdict. | ## Read more - [How do I test an agent?](/docs/faqs/building/testing-and-evals): evals next to your own test runner. - [`stackbone eval `](/docs/cli/reference/eval): every flag and exit code. - [Governance](/docs/home/features/governance): the rest of what Studio does with a running box. - [Guardrails](/docs/sdk/humans/guardrails) and [Human-in-the-loop](/docs/sdk/humans/approval): the two things a case can run into on its way to an answer.