# Traces and telemetry

> How Fermix emits structured JSONL traces, what telemetry events exist, how logs rotate, what /health/ready reports, and how to enable the optional Opik exporter.

Fermix records what it does, so you can see and debug its activity. Every Fermix component emits structured telemetry (machine-readable records of events as they happen) through a central handler that converts events into JSONL trace files (one JSON record per line) under `~/.fermix/traces/YYYY-MM-DD/<type>.jsonl`, rotated daily (a fresh file each day). This page covers the trace types, event vocabulary, log rotation, health endpoint, and the optional Opik exporter.

## Trace files

Trace files live under `FERMIX_HOME/traces/` (default `~/.fermix/traces/`) in a directory per calendar date. Each event type gets its own file:

| File | Contents |
|------|----------|
| `llm_call.jsonl` | Provider adapter calls |
| `tool_exec.jsonl` | Capability execution outcomes |
| `agent_event.jsonl` | Agent lifecycle, plugin distribution ops, memory reviews |
| `channel_msg.jsonl` | Inbound and outbound channel messages |
| `sandbox_event.jsonl` | Sandbox (the safety boundary on file and command access) denials — the path and command decisions that were blocked (allowed decisions are not written) |
| `error.jsonl` | Captured failures with stack traces |

The handler keeps one open file handle per `{date, type}` pair and rotates on a date change. Writes are best-effort: a failed trace write is logged but never takes down the runtime.

Override the default directory with `FERMIX_TRACE_DIR`.

## Entry shape

Every line is a JSON object. The handler adds `ts` (ISO 8601 UTC), `type`, and `agent` to every entry, then recursively encodes the payload:

```json
{
  "ts": "2026-05-18T23:21:39.477Z",
  "type": "tool_exec",
  "agent": "main",
  "tool": "file_read",
  "success": true,
  "duration_ms": 12,
  "path": "/Users/me/projects/foo/README.md"
}
```

Every run carries a `session_id`. The format encodes the run kind: `main-<n>` for interactive turns, random hex for subagents (helper agents the main agent spins off for a piece of work), `cron_<job>_<ts>` for scheduled job runs, `harness_<run_id>` for coding-agent runs (see [coding agents](/docs/coding-agents)), `followup_<reminder_id>` for a reminder's post-delivery check-in run, and `session:<n>` for realtime voice calls. Subagents also carry a `parent_session` so the full delegation tree (which agent spun off which) can be reconstructed by correlating on those fields. A coding-agent run is deliberately not a child: it is its own root, and the turn or scheduled run that launched it rides along as `origin_session_id` correlation metadata, never as a `parent_session`. A reminder check-in run is likewise its own root, with no `parent_session` — the turn that stored the date closed hours or days before the reminder came due.

## Telemetry event vocabulary

Fermix emits telemetry through shared emitters, and a central handler routes most of those events into the JSONL trace files. The table below lists the events that are written to a trace file and which file each one lands in. Tool and provider events are generic — a new tool reuses `[:fermix, :tool, :exec]` and a new provider reuses `[:fermix, :provider, :call]`, so both appear in traces automatically.

| Event | Trace file | What it records |
|-------|-----------|-----------------|
| `[:fermix, :provider, :call]` | `llm_call` | A provider adapter call — `provider`, `model`, `duration_ms`, `tokens` (plus `auth_mode` on the adapters that have one: Anthropic and xAI). Non-chat calls reuse this event with a deliberately empty `tokens` map, so no token cost is attributed to them: voice-note transcriptions carry `purpose: :transcription` and a `provider` that may be a transcription-only vendor such as `deepgram`, and `generate_image` calls carry no `purpose` |
| `[:fermix, :tool, :exec]` | `tool_exec` | A capability execution — `tool`, `success`, `duration_ms`. A tool call the model makes that never executes — an unknown name, a tool not allowed for the run, or unparseable arguments — is also written here as a failed execution (`success: false`, `duration_ms: 0`) under the name the model used, so no model tool call is ever invisible in traces or Opik |
| `[:fermix, :memory, :write]` | `tool_exec` | The background memory reviewer's durable write (recorded as a tool row) |
| `[:fermix, :mcp, :inbound, :call]` | `tool_exec` | A tool call served to a connected MCP (Model Context Protocol, the standard for exposing tools to AI clients) client |
| `[:fermix, :channel, :message]` | `channel_msg` | An inbound or outbound channel message — `direction` (`:inbound` or `:outbound`) |
| `[:fermix, :sandbox, :decision]` | `sandbox_event` | A **denied** file or command decision — `capability`, `policy_class`, `reason_tag`, `resource` (allowed decisions are not written) |
| `[:fermix, :agent, :message]` | `agent_event` | A completed MainAgent turn — `iterations`, `total_tokens`, `duration_ms` |
| `[:fermix, :agent, :message_error]` | `agent_event` | A turn that failed |
| `[:fermix, :agent, :start\|:stop\|:task_start\|:task_complete]` | `agent_event` | Agent-process and subagent-task lifecycle markers |
| `[:fermix, :agent, :prompt_context\|:history]`, `[:fermix, :channel, :reply]`, `[:fermix, :capabilities, :select]` | `agent_event` | Turn-prep and delivery markers (prompt assembly, history load, reply sent, capability set chosen) |
| `[:fermix, :skill, :invoke\|:journal_write]` | `agent_event` | A skill run and its journal write |
| `[:fermix, :job, :run_start\|:run_complete\|:run_error]` | `agent_event` | A scheduled job run's lifecycle (carries `job_id` and `run_id`) |
| `[:fermix, :harness, :run_start\|:run_complete\|:run_error\|:progress]` | `agent_event` | A coding-agent run's lifecycle — carries the `run_id`, vendor, and origin; the run's own provider and tool events share its `harness_<run_id>` session id |
| `[:fermix, :memory, :review]` | `agent_event` | The background memory review ("dreaming") run closer |
| `[:fermix, :plugin, :dist]` | `agent_event` | A plugin install, uninstall, or garbage-collection op |
| `[:fermix, :tool_search, :query]` | `agent_event` | A tool-schema deferral bridge search (`match_count`; a miss is the health signal) |
| `[:fermix, :provider, :failover]` | `agent_event` | A provider failover on a transient error |
| `[:fermix, :timeout, :expired]` | `agent_event` | A fired failure deadline (`FermixCore.Timeouts`) — the timeout `name` and elapsed `ms`, correlated by `session_id` |
| `[:fermix, :channel, :stream]` | `agent_event` | Streaming-turn bookends (`stream_started` / `stream_finalized`; interim draft edits are telemetry-only and not written) |
| `[:fermix, :mcp, :inbound, :tools_listed]` | `agent_event` | An inbound MCP `tools/list` served to a client |
| `[:fermix, :soul_curation, :run_start\|:run_complete\|:run_error]` | `agent_event` | The `/soul review` bounded draft call lifecycle |
| `[:fermix, :skill_curation, :run_start\|:run_complete\|:run_error\|:proposal_actioned]` | `agent_event` | A [skill curation](/docs/skills#skill-curation) pass and each approve or deny that follows it — counts only |
| `[:fermix, :reminder, :lifecycle]` | `agent_event` | One [reminder](/docs/reminders)'s lifecycle, written as `reminder_lifecycle` with a `phase` of `materialized`, `claimed`, `delivered`, `retry_scheduled`, `failed`, `expired`, `superseded`, `cancelled`, `event_completed`, `scheduler_error`, or `followup_skipped`. Delivering a reminder calls no model, so this is the only record of what it did |
| `[:fermix, :reminder, :followup_start\|:followup_complete\|:followup_error]` | `agent_event` | One post-delivery [check-in](/docs/reminders#the-follow-up-check-in) run's lifecycle — the reminder and event ids it trails, plus an `outcome` of `sent`, `declined`, `empty`, or `delivery_failed`, or a failure `status` of `error` or `timeout`. The check-in calls a model, so it is a run of its own: its provider and tool events share its `followup_<reminder_id>` session id |
| `[:fermix, :mcp_client, :lifecycle]`, `[:fermix, :capability, :mcp_name_collision]` | `agent_event` | A phase of Fermix's own connection out to a remote tool server — `initialize`, `discover`, `ready`, `drift`, `reconnect`, `security_block`, `owner_down`, `teardown` — which is the only trace of why a [plugin](/docs/plugins) served from such a server stopped answering; and two tools whose names collapse to the same one the agent sees |
| `[:fermix, :realtime, :call_start\|session_created\|session_updated\|provider_error\|reconnect\|call_stop]` | `agent_event` | The realtime voice call lifecycle — `call_stop` always carries a typed reason (such as `cost_limit`, `max_session_duration`, or `provider_disconnected`) |
| `[:fermix, :realtime, :screen_feed_start\|:frame_sent\|:frame_dropped\|:screen_feed_stop]` | `agent_event` | The voice screen-sharing feed's lifecycle — frame events carry the bytes sent and how many unchanged captures were gated out; only the start and stop events export to Opik |

The reminder, skill-curation, and remote-tool-server events carry no content by construction: each accepts a fixed set of ids, counts, and phase names and rejects anything else. A remote server's endpoint URL, the credential and session identifier used to reach it, tool arguments and responses, the chat address a reminder was delivered to, and the messages a skill-curation pass scanned can never reach a trace file. A failure is recorded as a short class label rather than the underlying message, which stays in the log and in the error returned to the caller. A reminder's own text is the one exception, along with the message a check-in run sends: both are written only when content capture is on (see below).

Some telemetry the runtime emits is **not** written to the JSONL files — it is available only to a live telemetry subscriber (attach your own handler). These include auto-compaction (`[:fermix, :compaction, :auto]` and `:auto_skipped`, where older messages are summarized to stay within the model's context limit), slash-command dispatch and rejection (`[:fermix, :command, :received]` and `:unauthorized`), agent-loop iterations (`[:fermix, :agent, :iteration]`), unrecognized-model sightings (`[:fermix, :model_catalog, :unknown_model]`), attachment-send failures (`[:fermix, :channel, :media_send_error]`), and a MainAgent-unavailable signal seen during a restart (`[:fermix, :dispatcher, :agent_unavailable]`).

## Content capture

Prompt bodies, LLM responses, and tool inputs and outputs are written into the trace files. This is the default posture and needs no setting: bodies are captured whole, with no truncation, and a failed browser action additionally carries the profile's recent console/JS-exception buffer (the browser's recent log and error messages) in its error details.

What lands on disk is therefore the literal text of messages, model replies, and tool results. It stays on the machine: the trace directory sits inside `FERMIX_HOME` unless `FERMIX_TRACE_DIR` moves it, `FERMIX_HOME` itself is kept owner-only (mode `0700`; `fermix doctor` fails its home-permissions check and prints the `chmod` that fixes it), and nothing forwards the bodies anywhere unless the Opik exporter below is enabled. Trace files are not pruned or size-capped, so a busy day's `tool_exec.jsonl` reaches tens of megabytes with content captured; deleting old date directories is the operator's job.

To suppress it, set `FERMIX_TRACE_CONTENT=0` in the daemon (the long-running background Fermix process) environment — the same rule as `FERMIX_OPIK_ENABLED` below applies, so a shell export alone never reaches a service-managed daemon. With content suppressed, records still note that each event happened, but input and output bodies are omitted entirely rather than shortened. The variable accepts `1`/`true`/`yes` and `0`/`false`/`no`; any other value fails loud at startup.

## Log file

Standard runtime logs go to `~/.fermix/logs/fermix.log` (override with `FERMIX_LOG_FILE`). Rotation is configured at 10 MB per file, keeping up to 5 files (when a file fills up, Fermix starts a new one and discards the oldest). Every line is redacted before it is written: a formatter wraps both the file handler and the console handler, so all log output — OTP crash reports included, which no source-level redaction can see — is scanned for credential-shaped tokens and each match replaced with a `[REDACTED:<vendor>]` marker. See [auth and secrets](/docs/auth-and-secrets#log-redaction) for the vendor patterns covered. Tail the log (stream new lines as they are written) with:

```bash
fermix logs -f
```

## Inspecting traces

```bash
# List today's trace files
ls ~/.fermix/traces/$(date +%F)/

# Filter shell tool calls
jq 'select(.tool == "shell")' ~/.fermix/traces/$(date +%F)/tool_exec.jsonl

# Find slow LLM calls
jq 'select(.duration_ms > 5000)' ~/.fermix/traces/$(date +%F)/llm_call.jsonl

# Show errors
jq '.' ~/.fermix/traces/$(date +%F)/error.jsonl
```

Traces from multiple days can be queried by globbing across date directories:

```bash
jq 'select(.success == false)' ~/.fermix/traces/*/tool_exec.jsonl
```

## Health endpoint

`GET /health/ready` returns a structured JSON report covering:

- Config file path and FERMIX_HOME
- Provider status (per configured provider)
- Per-channel status (Telegram, WhatsApp, Slack, Discord, Signal)
- Memory backend process status
- Realtime voice status

`GET /health/live` returns process liveness only (no dependency checks). `/health` is a backward-compatible alias for `/health/ready`.

```bash
curl -s http://127.0.0.1:4030/health/ready | jq .
```

See [configuration](/docs/configuration) for the bind address (`FERMIX_HTTP_BIND`) and [channels](/docs/channels) for per-channel setup.

## Opik exporter

`fermix_opik` is a built-in component that exports Fermix telemetry to an [Opik](https://www.comet.com/opik) instance (Opik is an external dashboard for inspecting LLM traces and costs). It is compiled into every build but is entirely inert unless enabled: it does nothing and adds zero overhead when off.

Enable it by setting `FERMIX_OPIK_ENABLED=1` in the daemon environment:

```bash
FERMIX_OPIK_ENABLED=1 fermix run
```

When running as a system service, a shell export alone never reaches the daemon process. Set the variable and then reinstall the unit (the OS service definition: launchd on macOS, systemd on Linux) so it is captured into that definition:

```bash
# After exporting FERMIX_OPIK_ENABLED=1 in your shell
fermix service install
```

`fermix doctor` reports whether the exporter is off, enabled-but-not-bundled, or ready (with the resolved endpoint and project name).

When enabled, the exporter:

- Reads the shared provider, tool, job, and lifecycle telemetry correlated by `session_id` and `parent_session`.
- Maps each run (interactive turn, subagent, scheduled job, memory review, coding-agent run) into Opik's trace model, nesting child runs under their parent. Coding-agent runs are the exception: each is exported as its own root trace, correlated to the turn that launched it by `origin_session_id` metadata rather than nested under it, so a background run that outlives its spawning turn still exports cleanly.
- Renders LLM spans (the per-call records in Opik's view) with provider, model, and token usage for Opik's cost attribution. A new provider must register a short identifier for cost to work.
- Carries prompt and response bodies into Opik spans whole, under the same content-capture switch described above: `FERMIX_TRACE_CONTENT=0` suppresses them in Opik as well as in the JSONL files.

The exporter reads events the runtime emits regardless. Its absence or failure never affects a turn or reply.

| Environment variable | Default | Description |
|---------------------|---------|-------------|
| `FERMIX_OPIK_ENABLED` | unset (off) | Set to `1` to activate the exporter |
| `FERMIX_OPIK_BASE_URL` | `http://localhost:5173/api` (local Opik) | Override the Opik endpoint URL |
| `FERMIX_OPIK_PROJECT` | `fermix` | Override the Opik project name |
| `FERMIX_TRACE_CONTENT` | unset (content captured) | `0` suppresses prompt, response, and tool bodies in both traces and Opik spans; `1` states the default posture explicitly |
| `FERMIX_TRACE_DIR` | `~/.fermix/traces` | Override the JSONL trace output directory |
| `FERMIX_LOG_FILE` | `~/.fermix/logs/fermix.log` | Override the rotating log file path |

For webhook configuration that feeds into the observability pipeline, see [webhooks](/docs/webhooks). For all configuration keys and environment variables, see [configuration](/docs/configuration).