---
title: 'eval'
description: 'Run a saved eval suite against an agent installation and exit non-zero when it misses the pass mark'
position: 25
---

# stackbone eval

> `stackbone eval` targets a **running agent installation**. With no `--agent`
> it runs against the local-dev installation linked to the current project, so
> [`stackbone dev`](/docs/cli/reference/dev) must be running. See
> [target resolution](/docs/cli/reference/conventions#target-resolution). The
> exit code is the gate: `0` when the run clears the suite's pass mark, `1`
> when it does not. See
> [exit codes](/docs/cli/reference/conventions#exit-codes).

Run an **eval suite** you already saved in Studio, watch it to the end, and exit
non-zero if it misses the suite's pass mark. Use it as the quality gate in CI.

The command defines nothing. The suite lives on the box, and it holds what to
measure, over which case list, with which criteria, how many repeats and against
what pass mark. The terminal and Studio therefore report the same definition of
what the run tested.

```bash
stackbone eval smoke                       # against the local `stackbone dev` box
stackbone eval smoke --agent inst_123      # against a deployed installation
stackbone eval smoke --git-sha "$(git rev-parse HEAD)"
```

| Flag              | Type   | Description                                                                 |
| ----------------- | ------ | --------------------------------------------------------------------------- |
| `--agent`         | string | Installation id to target. Defaults to the local-dev install.               |
| `--timeout`       | string | Seconds to watch before handing the run back. Default `1800`.               |
| `--interval`      | string | Seconds between progress reads. Default `3`.                                |
| `--git-sha`       | string | Commit under test, sealed onto the run so you can attribute an improvement. |
| `--agent-version` | string | Agent build under test, sealed onto the run.                                |

The suite name is the one on the Eval suites screen in Studio, and it is the
only positional argument. Get the installation id for `--agent` from
[`stackbone agents list`](/docs/cli/reference/agents).

Human mode prints the suite, how many evaluations the run measured out of the
plan, the split across passed / failed / errored / stopped, the pass rate against
the pass mark, and a link to the run in Studio. If the suite's cap cut the run
short, or an operator stopped it, the output says so and marks the pass rate as
partial coverage.

Progress lines go to stderr while the command watches, so under `--json` stdout
carries exactly one envelope and a pipe stays parseable.

**Exit codes**

| Situation                                                               | `code`                 | Exit |
| ----------------------------------------------------------------------- | ---------------------- | ---- |
| The run cleared the pass mark                                           | (none)                 | `0`  |
| The run missed it, was stopped, broke, or measured less than it planned | `eval_below_threshold` | `1`  |
| No suite by that name (the message lists the real ones)                 | `not_found`            | `4`  |
| No box to talk to (`stackbone dev` is not running)                      | `dev_not_running`      | `3`  |
| Your seat lacks `evals:run` on a cloud install                          | `permission`           | `5`  |

`--timeout` running out is a separate failure: the watcher stops with exit `1`
and a message pointing at the run in Studio, which is still measuring.

**JSON payload**

```jsonc
{
  "schema_version": 1,
  "suite": {
    "id": "...",
    "name": "smoke",
    "threshold": 0.8,
    "targetKind": "workflow" /* what the suite measures: `agent` or `workflow` */,
    "targetName": "onboarding",
  },
  "run": {
    /* the full run row with every case verdict */
  },
  "counts": { "passed": 18, "failed": 2, "errored": 0, "stopped": 0 },
  "passRate": 0.9,
  "threshold": 0.8,
  "met": true,
  "reason": null /* the sentence explaining a red gate */,
  "url": "https://app.stackbone.ai/app/acme/installations/inst_123/studio/eval-runs/...",
}
```

> **Ctrl-C stops the watcher, not the run.** So does `--timeout` running out. The
> box keeps measuring and keeps spending your token budget, so stop a run from
> its own screen in Studio. The pass mark comes off the suite **at launch**:
> editing the suite mid-run does not change what that run reports.