--- title: 'Logging & observability' description: 'Log from your tools and workflow steps with plain console.*, read the run timeline the runtime records for you, and know what the box keeps, for how long, and where it can send it.' position: 11 --- # Logging & observability > Your tools and workflow steps log with plain `console.*`. The runtime records > the timeline of every run on its own: each step, model call, tool call, > connection, approval and guardrail is a row in your box's Postgres, and Studio > draws it as a waterfall. Every line you print is captured, stamped with the > run and the step that printed it, kept for 30 days in the same Postgres, and > sent to your own stack when you set one environment variable. You wire no > tracer and no exporter in your code. ## The run timeline is automatic You instrument nothing to get the timeline. A **run** is one chat turn of a [deep agent](/docs/sdk/agents/overview) or one execution of a [durable workflow](/docs/sdk/workflows/overview). As it goes, the runtime writes one row per thing that happened, and Studio shows each row as one bar: - Workflow steps. Each `'use step'` function is a checkpoint. It appears on the run with its timing, its result and, if it failed and retried, each attempt. This is the core observability primitive of a workflow: the [event-log replay model](/docs/sdk/workflows/overview) means the run _is_ its list of steps. - Agent turns, tool calls and subagents. Every turn in a [session](/docs/sdk/agents/overview), every tool the model invoked during it and every subagent it delegated to, with inputs, outputs and timing. A subagent's own tools hang off it. - Model calls. Each call a turn makes to a model, and each call through [`stackbone.ai`](/docs/sdk/platform/ai) inside a workflow step, is its own row with the model, the provider, the latency and the input and output tokens. A model call made with your own client runs inside its step, so the runtime records the step's timing but not the call's tokens. - Connections, approvals and guardrails. A connector call through Stackbone Connect, a pause for a person and a guardrail verdict each get a row with what went in and what came out. There is no span processor to register and no exporter to configure in your code. If something is missing from a trace, add a new kind of step rather than a second tracer. ### Runs that pause A workflow that hits [`requestApproval()`](/docs/sdk/humans/approval) or a durable hook stops and shows on the timeline as **interrupted / pending** while it waits for a human. When the decision arrives, the run resumes from exactly that checkpoint and the rest of the steps appear. Read the timeline to see where a long-lived run is parked. ## Log from a tool or a workflow step You write code in two places: a deep-agent tool, and the steps of a durable workflow. In both, use `console.*`. The runtime knows which run and which step is executing, so you never thread a run id through your code. A tool, defined inline in `deep-agents//index.ts`: ```ts import { tool } from '@langchain/core/tools'; import { stackbone, z } from '@stackbone/sdk'; const lookupOrder = tool( async ({ orderId }: { orderId: string }) => { console.info('looking up order', { orderId }); const rows = await stackbone.database.select().from(orders).where(eq(orders.id, orderId)); if (rows.length === 0) { console.warn('order not found', { orderId }); } return rows[0] ?? null; }, { name: 'lookup_order', description: 'Look up an order by id.', schema: z.object({ orderId: z.string() }), }, ); ``` A durable workflow step in `workflows/.workflow.ts`: ```ts async function notifyPartner(partner: string) { 'use step'; console.warn('partner rate limited, backing off', { partner }); // ... do work ... } ``` `tool(...)` comes from the upstream [LangChain](https://js.langchain.com/docs/concepts/tools) package; `'use step'` is a [Workflow SDK directive](https://workflow-sdk.dev/docs/how-it-works/understanding-directives). The `stackbone` handle is the [ambient client](/docs/sdk/reference/overview), the same object on every surface, no construction required. Both lines land in Studio under **Logs** stamped with their run: the first with the agent turn that ran the tool, the second with the workflow run and the `notifyPartner` step. Open that step in the run's trace and you see the line next to the step's input and output. ### Pass fields as an object, not in the message Put the variable data in an object next to the message. The fields stay fields, so Studio filters on `orderId` instead of searching a formatted string: ```ts console.info('email sent', { to, templateId }); ``` is more useful later than interpolating the same value into the string: ```ts console.info('email sent to ' + to); // no structured `to` to filter on ``` Keep the message a short, stable phrase. Pass an `Error` as its own argument and its type, message and stack are kept as `exception.*` fields rather than flattened into text. ### Log inside steps, not in the workflow body A workflow function replays from its event log every time a step completes, so a `console.log` in the workflow body itself runs again on every replay and prints once per replay. Those lines are marked as coming from the workflow body and Studio collapses the repeats, but the honest place for a log line is inside a `'use step'` function or a tool, which runs once. ### Levels `console.debug`, `console.info`, `console.warn` and `console.error` map to the `debug`, `info`, `warn` and `error` levels; `console.log` is `info`. Studio and the CLI filter on the level, so use `warn` and `error` for what you will want to find in an incident and keep progress chatter at `info` or `debug`. ### If you already log with pino The capture works by patching `console`, so it reaches everything written through `console` and nothing written below it. A logger that writes straight to the file descriptor is below it. `pino` is the common case: it calls `process.stdout.write` itself, so its lines arrive at the box's stdout as raw text with **no run, no level, no fields and no masking pass**. They are not scrubbed, not correlated and not stored as rows you can filter. Point pino at the SDK's destination and every line it writes enters the same pipeline as everything else, keeping the level and the fields pino already built: ```ts import pino from 'pino'; import { createPinoDestination } from '@stackbone/sdk'; const log = pino({ level: 'debug' }, createPinoDestination()); log.info({ orderId }, 'looking up order'); ``` With no runtime around (a unit test, a script) the destination falls through to `process.stdout`, which is pino's own default, so wiring it up never silently mutes your logger. The same applies to any other logger you point at the destination: it takes a `{ write(chunk) }` sink, which is the slot pino's `destination` argument fills. ## What happens to a line Everything the box process emits goes through one pipeline before it is stored: your `console.*` lines, the SDK's own structured lines and the runtime's diagnostics. The same code runs under `stackbone dev`, in a self-hosted box and in a deployed one. | The pipeline... | So that... | | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | stamps each line with the run and the step that printed it | opening a failed step in Studio shows exactly what it printed. The step id comes from the `'use step'` function that is executing, so a workflow line carries both ids. A tool running inside an agent chat turn carries that turn's **run** and no step id yet, so those lines group at the run. A line printed outside any run (a scheduled job, boot output) is stored with no run and the name of what produced it, so it stays filterable. | | masks secrets | an API key printed by mistake never reaches the store or the export. The values of your workspace's own secrets are masked exactly; bearer tokens, key prefixes and credentials inside URLs are masked by pattern. Nothing else is rewritten: an email address stays searchable. | | caps size and volume | a talkative loop degrades your logs instead of filling the disk. An oversized line is cut and marked as truncated. Past a hard number of lines per run the rest are dropped and counted, with one marker line saying how many. | | writes in batches and never blocks | a database hiccup costs you log lines, never a run. When the database was unreachable, one marker line names the interval and how many lines were lost, so a gap in the record is declared instead of looking like silence. | | keeps lines for **30 days** | you can investigate a complaint that arrived late. Configurable between 7 and 90 days in the workspace settings, with a storage budget in gigabytes that deletes the oldest day early. Both keys are accepted from an `owner` or an `admin` only, so the seat that writes the log lines cannot shorten the window they live in. One hourly job enforces both and shows its last run on the **Recurring jobs** screen. | | serves a history beside the live tail | you find the line without exporting the table. Filters by level, run, time range and substring; text search always inside a time range. | | puts raw log reading behind `logs:read` | the read-only seats do not see the rawest data in the product. `owner`, `admin` and `member` read logs; `viewer` and `approver` keep runs, with inputs, outputs and steps. | Every line is scoped to the box it came from. One installation is one box with one Postgres, so "which workspace" is answered by which box you are looking at. After the window, a day is deleted. The compliance layer around that (a daily archive into your own bucket, deletion on request and legal hold) is designed and **not built yet**; the [Observability feature page](/docs/home/features/observability) describes it with that marked. ## Send logs and traces to your own stack You keep the canonical copy in the box. Getting a copy out is a separate, optional step, and it never blocks a run. A collector that is down costs you a dashboard, not the evidence, and the export failure count shows on the settings screen. A deployed box always writes to standard output. It emits the same scrubbed lines as NDJSON, so Promtail, Alloy, Vector, Fluent Bit or your cloud's agent pick them up like any other container. Under `stackbone dev` the terminal keeps showing your original `console.*` line instead, so the local console stays readable. Set an endpoint and OTLP export turns on. The box pushes logs and traces over OTLP/HTTP with a JSON body to the collector you already run. The standard variables apply, on the container that runs the box: | Variable | What it does | | ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_ENDPOINT` | Base URL. The box appends `/v1/logs` and `/v1/traces`. This is the whole switch: with no endpoint set, nothing is exported and the box keeps its only copy. | | `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | Full URL for logs, used as is. Set this when your backend has its own path, such as Loki's `/otlp/v1/logs`. On its own it turns on logs and leaves traces off, because a logs URL says nothing about where spans go. | | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | Full URL for traces, used as is. | | `OTEL_EXPORTER_OTLP_HEADERS` | Headers for the collector's auth, as `key1=value1,key2=value2`. | | `OTEL_EXPORTER_OTLP_PROTOCOL` | Only `http/json` is supported. Unset means `http/json`; any other value (the specification's own default, `http/protobuf`, included) logs one warning at boot and disables export entirely, rather than exporting something you did not ask for. | | `OTEL_SERVICE_NAME` | The `service.name` on every record. Unset, the box uses the agent's own name, so a collector never fills up with OTel's `unknown_service`. | | `OTEL_RESOURCE_ATTRIBUTES` | Extra resource attributes, as `key=value` pairs. The box adds `stackbone.installation_id`, `stackbone.organization_id` and the environment under **both** keys: `deployment.environment.name`, the current one, and `deployment.environment`, which older Datadog Agents and most shipped dashboards still read. Yours are applied last and may override any of them. | | `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` | Whether prompts and completions ride on a span at all. `NO_CONTENT` unless you set `SPAN_ONLY` or `SPAN_AND_EVENT`, and a value the box does not recognise reads as `NO_CONTENT` too: a typo must never be taken as consent to export somebody's prompts. | These are deployment settings, not workspace settings: whoever runs the container owns the collector. Loki (3.0 and later) and any OpenTelemetry Collector accept this export directly; the Datadog Agent takes OTLP logs over HTTP with `otlp_config.logs.enabled`. Datadog's agentless intake is not a documented target for the JSON encoding the box sends, so put an Agent or a Collector in front of it. When a run finishes, the box exports it as one trace: one root span for the run, and one span per timeline row under it, nested the way the timeline nests. The **trace id is the run id** with its dashes stripped, so the id you copy from Studio is the one you paste into your APM's search box. Log lines exported over OTLP carry the trace id and the span id of their step, so opening a slow step in your APM shows what it printed. The root span is `invoke_workflow ` for a workflow and `invoke_agent ` for a chat turn, and it carries the run's status, its trigger and the run's folded token totals. Each row under it reads like this: | The timeline row | The span it becomes | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------- | | Model call | `chat `, with `gen_ai.provider.name`, `gen_ai.request.model` and `gen_ai.usage.input_tokens` / `.output_tokens` | | Tool call | `execute_tool `, with `gen_ai.tool.name` | | Agent turn or subagent | `invoke_agent `, with `gen_ai.agent.name` | | Workflow step | `invoke_workflow `, with `gen_ai.workflow.name` | | Connector call | `connection `, tagged `openinference.span.kind=TOOL` | | Approval or hook | `hook `, tagged `openinference.span.kind=CHAIN` | | Guardrail | `guardrail `, tagged `openinference.span.kind=GUARDRAIL` | The first four use the OpenTelemetry GenAI conventions, so an LLM observability backend renders models and token counts without an exporter per vendor. The last three have no GenAI operation (the conventions model model calls, tools and agents, and a connector is none of them), so they carry one borrowed OpenInference key that says what they are instead of arriving as unlabelled spans. A row that failed carries `error.type`, and a row that was still open when the run ended is exported with its last known status rather than held back. Every span of a chat run carries the same `gen_ai.conversation.id`, the session id, so a ten-turn conversation is ten traces your backend groups as one thing. Payloads stay in the box unless you say otherwise. A span carries the names, the timings, the token counts and the status, and never the prompt or the completion. Set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` to `SPAN_ONLY` or `SPAN_AND_EVENT` to put the messages on the span as `gen_ai.input.messages` and `gen_ai.output.messages`. They go through the same masking pass as a log line and are truncated past a few thousand characters, because a trace is not a payload store. You can also continue a trace you already started. Send a W3C `traceparent` header on `POST /api/workflows/:name/start` and the run joins that trace instead of opening one of its own: every span the run exports carries your trace id, the run's root span hangs off the span you sent, and the run's log lines carry the same trace id. A malformed header is ignored and the run starts normally. The header is stored on the run, so a run that pauses for a human decision and resumes later still exports under your trace. `POST /api/workflows/:name/chat` and the agent chat routes do not honour the header yet. A run started through them is exported as the root of its own trace, with the trace id equal to the run id. ## Inspecting runs from the shell You do not need Studio open to read what happened. The CLI inspects the same runs and the same log store: ```sh # List recent runs, newest first. stackbone runs list # Inspect one run: status, steps, timing, errors. stackbone runs get # Live-tail logs across the installation. stackbone logs tail --follow # Tail only the lines for one run. stackbone logs tail --run --follow # A window, filtered. stackbone logs tail --since 15m --limit 200 --level warn --q "rate limited" # List the workflows in the workspace. stackbone workflows list ``` `stackbone logs tail` takes `--run`, `--level`, `--q` (substring) and `--trace-id` as server-side filters, and `--since`, `--until` and `--limit` on the client; `--agent ` picks the installation when it is not the local one. `stackbone runs retry ` and `stackbone runs cancel ` act on a run once you have found it. ## Local development `stackbone dev` runs the same runtime, the same run records and the same log pipeline as a deployed box: the lines go to the local Postgres the emulator already runs, and Studio's **Logs** screen reads them. Two things differ on purpose. Your terminal keeps showing the original `console.*` line rather than JSON, and no OTLP endpoint is set by default, so the dev stack stays at its three containers. Point `OTEL_EXPORTER_OTLP_ENDPOINT` at your own collector to see the export path work locally with the same switch a deployed box uses. ## Where to go next - **[Observability](/docs/home/features/observability)**: the same picture for operators, with the retention statement and the compliance layer that is still to come. - **[`stackbone.ai`](/docs/sdk/platform/ai)**: the model calls that appear as steps on the timeline, with their token usage. - **[Agents & sessions](/docs/sdk/agents/overview)**: turns, runs and the session model your tool logs are attributed to. - **[Workflows](/docs/sdk/workflows/overview)**: `'use step'` checkpoints, replay, and how a run is reconstructed from its steps. - **[Human approvals](/docs/sdk/humans/approval)**: `requestApproval()` and how a paused run reads on the timeline. - **[CLI reference](/docs/cli/reference/commands)**: `stackbone runs`, `stackbone logs tail` and `stackbone workflows`.