--- title: 'Observability' description: 'What you can see after a run: the trace of every step, every line your code printed, where that evidence lives in your own box, how long it is kept and how it reaches the stack you already run.' position: 8 --- # Observability > Every chat turn and every workflow run leaves a **trace** the box records by > itself: the steps it took, what each one was given and returned, how long it > took and what it spent in tokens. Every line your code prints is captured, > stamped with the run and the step that printed it, and kept for 30 days. All > of it lives in your **box**, the container running your workspace, in your own > infrastructure. The control plane never receives it, and the box sends it out > only when you point it at your own collector. Two halves, one store: | Half | What it is | Where you read it | | --------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | The trace | The steps of a run, recorded by the runtime, kept in the box's Postgres. | Studio under **Runs**, `stackbone runs get`, and your APM as one trace per run over OTLP. | | The logs | The lines your tools and steps print, plus the runtime's own, run by run. | Studio under **Logs** (live tail and 30-day history), `stackbone logs tail`, the box's stdout, and your log backend over OTLP. | ## What a run records A **run** is one chat turn or one workflow run. As it goes, the runtime writes one row per thing that happened, and each row is one bar on the trace: | Step | Written when | What the row holds | | ---------- | ----------------------------------------------------------------------- | -------------------------------------------------------------------------- | | Activity | A `'use step'` function of a workflow runs. | Its name, timing and result. | | Model | An agent turn, or a call through `stackbone.ai`, reaches a model. | The model, the provider, latency, and input and output tokens. | | Tool | The model calls one of your tools. | The tool's arguments and what it returned. | | Subagent | A deep agent hands work to a subagent. | The task it was given and the reply. The subagent's own tools hang off it. | | Connection | Your code calls a connector through Stackbone Connect. | The connector, the operation, the arguments and the response. | | Approval | The run pauses for a person: `requestApproval()`, or a tool that waits. | The payload the reviewer saw and the decision that resumed it. | | Guardrail | A guardrail inspects the turn. | What it inspected and the verdict it reached. | Rows nest, so a subagent's tool calls sit under the subagent, and a workflow's delegation to an agent sits under that step. Waits between steps show as gaps on the timeline with their length. ![A workflow run's trace in Studio: three steps with the waits between them, the run input and final output on the left, and the inspector showing the first step's result as JSON.](/images/observability/workflow-trace.png) _A workflow run: three steps, a two-second wait before the last one, and what the first step returned._ Open a run under **Runs** in Studio and you get the waterfall, the list of steps, and an inspector for whichever step you click. The header carries the run's status, trigger, duration and total tokens. **Run Input** and **Final Output** sit at the top and bottom of the step list. A run that paused for a person shows the pending decision above the trace. **Copy run details** puts the whole run and its trace on the clipboard as JSON, for a ticket or a coding agent. **Sessions** groups chat turns into conversations, so a thread reads as one row with its token total and each turn opens its own trace. From a terminal, [`stackbone runs list`](/docs/cli/reference/runs#stackbone-runs-list) and [`stackbone runs get `](/docs/cli/reference/runs#stackbone-runs-get) read the same rows. You instrument nothing to get any of this. The trace is a projection of what the runtime already recorded, which is also why it survives a restart: the rows live in the box's own Postgres next to the run itself. If something is missing from a trace, it needs a new kind of step rather than a second tracer. ## Log from your code Inside a tool or a workflow step, log with plain `console.*`: ```ts async function deliver(to: string, scope: string) { 'use step'; console.info('digest delivered', { to, scope, templateId: 'digest-v3' }); return { to, scope, delivered: true }; } ``` The line shows up in Studio under **Logs** with its level, its time, its message and the fields you passed in the object, stamped with the run and the step that printed it. Open a failed step in a run and you see exactly what it printed. The screen has two tabs: **Live tail** for what is happening now, and **Stored history** for the 30 days the box kept. Filter by level, by run, by time range or by a substring, and pause the live tail to read. The run id is the one filter the URL carries, so a link you copy reopens the view filtered to that run. ![The live tail of the Logs screen in Studio: four lines printed by the steps of a workflow run, one at error level, with the level, search, trace id and run id filters above.](/images/observability/logs-tail.png) _What the steps above printed, on the live tail of the Logs screen._ Keep the message a short, stable phrase and put the variable data in an object: the fields stay filterable, a formatted string does not. The [SDK page](/docs/sdk/platform/observability) covers levels, `Error` arguments and why a log line belongs inside a step rather than in the workflow body. The CLI reads the same store: ```sh stackbone logs tail --follow # everything, live stackbone logs tail --run --follow # one run stackbone logs tail --since 15m --limit 200 # a window ``` ## What happens to a line Everything the box emits goes through one pipeline before it is stored: your `console.*` lines, the SDK's structured lines and the runtime's own events. The same code runs under `stackbone dev`, in a self-hosted box and in a deployed one, so what you debug locally is what happens in production. | The pipeline... | So that... | | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | stamps each line with the run and the step that printed it | a line is never an orphan. The step id comes from the workflow step that is executing, so a line from a tool inside an agent chat turn carries that turn's run and groups there. A line printed outside any run (a scheduled job, boot output) is stored with no run and the name of what produced it, so it stays filterable. | | masks secrets | an API key printed by mistake never reaches the store or the export. The values of your workspace's own secrets are masked exactly; bearer tokens, key prefixes and credentials inside URLs are masked by pattern. Nothing else is rewritten: an email address stays searchable. | | caps size and volume | a talkative loop degrades your logs instead of filling the disk. An oversized line is cut and marked as truncated. Past a hard number of lines per run the rest are dropped and counted, with one marker line saying how many. | | writes in batches and never blocks | a database hiccup costs you log lines, never a run. When the database was unreachable, one marker line names the interval and how many lines were lost, so a gap in the record is declared instead of looking like silence. | ## Where the evidence lives, and for how long This table is the box's retention statement: what it stores about your end users, where, for how long, who can read it and what removes it. | What | Where | How long | Who reads it | How it is deleted | | -------------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | Runs, steps, sessions, approvals | The box's own Postgres. | Until you delete them. They survive restarts and redeploys. | Every seat with `runs:read`, viewers and approvers included. | By deleting the run. | | Log lines | The box's own Postgres. | **30 days** by default, configurable between 7 and 90 in the workspace settings, within a storage budget in gigabytes. Both keys are accepted from an `owner` or an `admin` only. | `owner`, `admin` and `member`, behind `logs:read`. `viewer` and `approver` keep runs and lose the raw dump. | Automatically at the end of the window. Deletion on request is not built yet (see below). | | Prompt versions, config versions | The box's own Postgres. | Until you delete them. | Every seat with `runs:read`. | From their own screens. | Nothing on this page reaches the control plane. It holds your organization, its members and the record of where your box lives; runs, logs and payloads stay in your infrastructure. See [Where do agents actually run?](/docs/faqs/general/where-agents-run). ### Retention and the storage budget Log retention (30 days by default, 7 to 90) and the storage budget in gigabytes are two keys in the workspace settings in Studio, and the box accepts a write to either one from an `owner` or an `admin` only. A `member` writes the log lines and reads them, and cannot shorten the window they live in. One hourly job deletes the days past the window in small batches, never in one mass delete, and deletes the oldest day early when the budget is crossed, so a runaway loop degrades retention instead of taking the box down and is caught within the hour. The job appears on the **Recurring jobs** screen with its last execution, like every other timer the box runs. ### Archived into your own bucket > [!NOTE] Not yet built > Nothing in this section ships today. A day that expires is deleted, not > archived, and there are no digests and no verification pass. It is written > here as the design the compliance work will build, so you can see where the > box is going before you decide to keep a second copy yourself. When a day expires, the same job first writes it as one compressed object into the box's own object storage, under a reserved prefix laid out by organization, installation and date, records a digest of that object chained to the previous day's, and only then deletes the day's rows. Because each day's record carries the previous day's digest, a day removed or replaced is as detectable as a day edited. A weekly verification, and a button for an on-demand one, re-reads the objects and walks the chain; the settings screen shows when it last passed and states the encryption posture of the writes. Archiving is on by default wherever the box has object storage, and the screen says so plainly when it does not. The objects' lifecycle is yours; the box never expires them. ### Deleted on request, and never quietly > [!NOTE] Not yet built > Nothing in this section ships today either: no deletion surface, no subject > identifier, no legal hold and no access trail. What the box does today is the > retention window and the storage budget above. Until this lands, a deletion > request is answered by shortening the window or by dropping the rows in the > box's own Postgres. Deletion is a first-class surface on the box: by run, by date range, by the whole box, and by person when your code seals a subject identifier into the run's context, so every line captured in that run inherits it. Deletion needs `logs:read` plus the `admin` or `owner` seat. A **legal hold**, one switch in the workspace settings with the identity and timestamp of whoever set it, suspends the hourly purge, the early delete under the storage budget and every deletion surface until it is lifted; ingest, archiving and verification carry on, and the screen states that a held box will outgrow its budget. Every deletion, every export and every raw read is recorded in the box's activity trail, so erasing evidence cannot also erase the proof of the erasure. ## Ship it to the stack you already run You keep the canonical copy in the box. Getting a copy out is optional, and it never blocks a run: a collector that is down costs you a dashboard, never the evidence, and the export failure count shows on the settings screen. A deployed box always writes the same scrubbed lines as NDJSON to its standard output, so Promtail, Alloy, Vector, Fluent Bit or your cloud's agent pick them up like any other container. Under `stackbone dev` your terminal keeps showing the original `console.*` line instead, so the local console stays readable. The OTLP export turns on when you name an endpoint. Set `OTEL_EXPORTER_OTLP_ENDPOINT` on the container that runs the box (or the per-signal `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` and `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, plus `OTEL_EXPORTER_OTLP_HEADERS` for the collector's auth) and the box pushes over **OTLP/HTTP** in JSON to the collector you already run: the log lines as they happen, and, when a run finishes, that run as one trace whose root is the run and whose spans are its steps. The **trace id is the run id**, so the id you copy from Studio is the one you search for in your APM, and every exported log line carries the id of the span that printed it. Model calls carry the GenAI semantic conventions, so an LLM observability backend renders models and token counts without an exporter per vendor; the turns of one conversation share a conversation id. A run started by one of your own services with a `traceparent` header on the workflow start route is exported as part of that trace. Loki (3.0 and later) and any OpenTelemetry Collector accept this export directly; the Datadog Agent takes OTLP logs over HTTP with `otlp_config.logs.enabled`, and everything else (Phoenix, Elastic, Dynatrace) wants a Collector in front of it. The full variable table is on the [SDK page](/docs/sdk/platform/observability#send-logs-and-traces-to-your-own-stack). These are deployment settings, not workspace settings: whoever runs the container owns the collector. The settings screen shows the effective posture, endpoint set or not and the last export failure count, without offering to edit it. ## Read more - [Logging & observability](/docs/sdk/platform/observability): logging from tools and steps, the run timeline and the export contract, from the SDK's side. - [Governance](/docs/home/features/governance): the rest of what Studio shows about a running box. - [Recurring jobs](/docs/home/features/recurring-jobs): what the box runs on a timer. - [Logs and debugging](/docs/faqs/operations/logs-and-debugging): reading what a failing run did. - [`stackbone runs`](/docs/cli/reference/runs#stackbone-runs-list) and [`stackbone logs tail`](/docs/sdk/platform/observability#inspecting-runs-from-the-shell).