stackbone eval

stackbone eval targets a running agent installation. With no --agent it runs against the local-dev installation linked to the current project, so stackbone dev must be running. See target resolution. The exit code is the gate: 0 when the run clears the suite's pass mark, 1 when it does not. See exit codes.

Run an eval suite you already saved in Studio, watch it to the end, and exit non-zero if it misses the suite's pass mark. Use it as the quality gate in CI.

The command defines nothing. The suite lives on the box, and it holds what to measure, over which case list, with which criteria, how many repeats and against what pass mark. The terminal and Studio therefore report the same definition of what the run tested.

stackbone eval smoke                       # against the local `stackbone dev` box
stackbone eval smoke --agent inst_123      # against a deployed installation
stackbone eval smoke --git-sha "$(git rev-parse HEAD)"

Flag	Type	Description
--agent	string	Installation id to target. Defaults to the local-dev install.
--timeout	string	Seconds to watch before handing the run back. Default 1800.
--interval	string	Seconds between progress reads. Default 3.
--git-sha	string	Commit under test, sealed onto the run so you can attribute an improvement.
--agent-version	string	Agent build under test, sealed onto the run.

The suite name is the one on the Eval suites screen in Studio, and it is the only positional argument. Get the installation id for --agent from stackbone agents list.

Human mode prints the suite, how many evaluations the run measured out of the plan, the split across passed / failed / errored / stopped, the pass rate against the pass mark, and a link to the run in Studio. If the suite's cap cut the run short, or an operator stopped it, the output says so and marks the pass rate as partial coverage.

Progress lines go to stderr while the command watches, so under --json stdout carries exactly one envelope and a pipe stays parseable.

Exit codes

Situation	code	Exit
The run cleared the pass mark	(none)	0
The run missed it, was stopped, broke, or measured less than it planned	eval_below_threshold	1
No suite by that name (the message lists the real ones)	not_found	4
No box to talk to (stackbone dev is not running)	dev_not_running	3
Your seat lacks evals:run on a cloud install	permission	5

--timeout running out is a separate failure: the watcher stops with exit 1 and a message pointing at the run in Studio, which is still measuring.

JSON payload

{
  "schema_version": 1,
  "suite": {
    "id": "...",
    "name": "smoke",
    "threshold": 0.8,
    "targetKind": "workflow" /* what the suite measures: `agent` or `workflow` */,
    "targetName": "onboarding",
  },
  "run": {
    /* the full run row with every case verdict */
  },
  "counts": { "passed": 18, "failed": 2, "errored": 0, "stopped": 0 },
  "passRate": 0.9,
  "threshold": 0.8,
  "met": true,
  "reason": null /* the sentence explaining a red gate */,
  "url": "https://app.stackbone.ai/app/acme/installations/inst_123/studio/eval-runs/...",
}

Ctrl-C stops the watcher, not the run. So does --timeout running out. The box keeps measuring and keeps spending your token budget, so stop a run from its own screen in Studio. The pass mark comes off the suite at launch: editing the suite mid-run does not change what that run reports.