Traces and telemetry

Fermix records what it does, so you can see and debug its activity. Every Fermix component emits structured telemetry (machine-readable records of events as they happen) through a central handler that converts events into JSONL trace files (one JSON record per line) under ~/.fermix/traces/YYYY-MM-DD/<type>.jsonl, rotated daily (a fresh file each day). This page covers the trace types, event vocabulary, log rotation, health endpoint, and the optional Opik exporter.

Trace files

Trace files live under FERMIX_HOME/traces/ (default ~/.fermix/traces/) in a directory per calendar date. Each event type gets its own file:

File	Contents
llm_call.jsonl	Provider adapter calls
tool_exec.jsonl	Capability execution outcomes
agent_event.jsonl	Agent lifecycle, plugin distribution ops, memory reviews
channel_msg.jsonl	Inbound and outbound channel messages
sandbox_event.jsonl	Sandbox (the safety boundary on file and command access) denials — the path and command decisions that were blocked (allowed decisions are not written)
error.jsonl	Captured failures with stack traces

The handler keeps one open file handle per {date, type} pair and rotates on a date change. Writes are best-effort: a failed trace write is logged but never takes down the runtime.

Override the default directory with FERMIX_TRACE_DIR.

Entry shape

Every line is a JSON object. The handler adds ts (ISO 8601 UTC), type, and agent to every entry, then recursively encodes the payload:

{
  "ts": "2026-05-18T23:21:39.477Z",
  "type": "tool_exec",
  "agent": "main",
  "tool": "file_read",
  "success": true,
  "duration_ms": 12,
  "path": "/Users/me/projects/foo/README.md"
}

Every run carries a session_id. The format encodes the run kind: main-<n> for interactive turns, random hex for subagents (helper agents the main agent spins off for a piece of work), cron_<job>_<ts> for scheduled job runs, harness_<run_id> for coding-agent runs (see coding agents), followup_<reminder_id> for a reminder’s post-delivery check-in run, and session:<n> for realtime voice calls. Subagents also carry a parent_session so the full delegation tree (which agent spun off which) can be reconstructed by correlating on those fields. A coding-agent run is deliberately not a child: it is its own root, and the turn or scheduled run that launched it rides along as origin_session_id correlation metadata, never as a parent_session. A reminder check-in run is likewise its own root, with no parent_session — the turn that stored the date closed hours or days before the reminder came due.

Telemetry event vocabulary

Fermix emits telemetry through shared emitters, and a central handler routes most of those events into the JSONL trace files. The table below lists the events that are written to a trace file and which file each one lands in. Tool and provider events are generic — a new tool reuses [:fermix, :tool, :exec] and a new provider reuses [:fermix, :provider, :call], so both appear in traces automatically.

Event	Trace file	What it records
[:fermix, :provider, :call]	llm_call	A provider adapter call — provider, model, duration_ms, tokens (plus auth_mode on the adapters that have one: Anthropic and xAI). Non-chat calls reuse this event with a deliberately empty tokens map, so no token cost is attributed to them: voice-note transcriptions carry purpose: :transcription and a provider that may be a transcription-only vendor such as deepgram, and generate_image calls carry no purpose
[:fermix, :tool, :exec]	tool_exec	A capability execution — tool, success, duration_ms. A tool call the model makes that never executes — an unknown name, a tool not allowed for the run, or unparseable arguments — is also written here as a failed execution (success: false, duration_ms: 0) under the name the model used, so no model tool call is ever invisible in traces or Opik
[:fermix, :memory, :write]	tool_exec	The background memory reviewer’s durable write (recorded as a tool row)
[:fermix, :mcp, :inbound, :call]	tool_exec	A tool call served to a connected MCP (Model Context Protocol, the standard for exposing tools to AI clients) client
[:fermix, :channel, :message]	channel_msg	An inbound or outbound channel message — direction (:inbound or :outbound)
[:fermix, :sandbox, :decision]	sandbox_event	A denied file or command decision — capability, policy_class, reason_tag, resource (allowed decisions are not written)
[:fermix, :agent, :message]	agent_event	A completed MainAgent turn — iterations, total_tokens, duration_ms
[:fermix, :agent, :message_error]	agent_event	A turn that failed
[:fermix, :agent, :start|:stop|:task_start|:task_complete]	agent_event	Agent-process and subagent-task lifecycle markers
[:fermix, :agent, :prompt_context|:history], [:fermix, :channel, :reply], [:fermix, :capabilities, :select]	agent_event	Turn-prep and delivery markers (prompt assembly, history load, reply sent, capability set chosen)
[:fermix, :skill, :invoke|:journal_write]	agent_event	A skill run and its journal write
[:fermix, :job, :run_start|:run_complete|:run_error]	agent_event	A scheduled job run’s lifecycle (carries job_id and run_id)
[:fermix, :harness, :run_start|:run_complete|:run_error|:progress]	agent_event	A coding-agent run’s lifecycle — carries the run_id, vendor, and origin; the run’s own provider and tool events share its harness_<run_id> session id
[:fermix, :memory, :review]	agent_event	The background memory review (“dreaming”) run closer
[:fermix, :plugin, :dist]	agent_event	A plugin install, uninstall, or garbage-collection op
[:fermix, :tool_search, :query]	agent_event	A tool-schema deferral bridge search (match_count; a miss is the health signal)
[:fermix, :provider, :failover]	agent_event	A provider failover on a transient error
[:fermix, :timeout, :expired]	agent_event	A fired failure deadline (FermixCore.Timeouts) — the timeout name and elapsed ms, correlated by session_id
[:fermix, :channel, :stream]	agent_event	Streaming-turn bookends (stream_started / stream_finalized; interim draft edits are telemetry-only and not written)
[:fermix, :mcp, :inbound, :tools_listed]	agent_event	An inbound MCP tools/list served to a client
[:fermix, :soul_curation, :run_start|:run_complete|:run_error]	agent_event	The /soul review bounded draft call lifecycle
[:fermix, :skill_curation, :run_start|:run_complete|:run_error|:proposal_actioned]	agent_event	A skill curation pass and each approve or deny that follows it — counts only
[:fermix, :reminder, :lifecycle]	agent_event	One reminder’s lifecycle, written as reminder_lifecycle with a phase of materialized, claimed, delivered, retry_scheduled, failed, expired, superseded, cancelled, event_completed, scheduler_error, or followup_skipped. Delivering a reminder calls no model, so this is the only record of what it did
[:fermix, :reminder, :followup_start|:followup_complete|:followup_error]	agent_event	One post-delivery check-in run’s lifecycle — the reminder and event ids it trails, plus an outcome of sent, declined, empty, or delivery_failed, or a failure status of error or timeout. The check-in calls a model, so it is a run of its own: its provider and tool events share its followup_<reminder_id> session id
[:fermix, :mcp_client, :lifecycle], [:fermix, :capability, :mcp_name_collision]	agent_event	A phase of Fermix’s own connection out to a remote tool server — initialize, discover, ready, drift, reconnect, security_block, owner_down, teardown — which is the only trace of why a plugin served from such a server stopped answering; and two tools whose names collapse to the same one the agent sees
[:fermix, :realtime, :call_start|session_created|session_updated|provider_error|reconnect|call_stop]	agent_event	The realtime voice call lifecycle — call_stop always carries a typed reason (such as cost_limit, max_session_duration, or provider_disconnected)
[:fermix, :realtime, :screen_feed_start|:frame_sent|:frame_dropped|:screen_feed_stop]	agent_event	The voice screen-sharing feed’s lifecycle — frame events carry the bytes sent and how many unchanged captures were gated out; only the start and stop events export to Opik

The reminder, skill-curation, and remote-tool-server events carry no content by construction: each accepts a fixed set of ids, counts, and phase names and rejects anything else. A remote server’s endpoint URL, the credential and session identifier used to reach it, tool arguments and responses, the chat address a reminder was delivered to, and the messages a skill-curation pass scanned can never reach a trace file. A failure is recorded as a short class label rather than the underlying message, which stays in the log and in the error returned to the caller. A reminder’s own text is the one exception, along with the message a check-in run sends: both are written only when content capture is on (see below).

Some telemetry the runtime emits is not written to the JSONL files — it is available only to a live telemetry subscriber (attach your own handler). These include auto-compaction ([:fermix, :compaction, :auto] and :auto_skipped, where older messages are summarized to stay within the model’s context limit), slash-command dispatch and rejection ([:fermix, :command, :received] and :unauthorized), agent-loop iterations ([:fermix, :agent, :iteration]), unrecognized-model sightings ([:fermix, :model_catalog, :unknown_model]), attachment-send failures ([:fermix, :channel, :media_send_error]), and a MainAgent-unavailable signal seen during a restart ([:fermix, :dispatcher, :agent_unavailable]).

Content capture

Prompt bodies, LLM responses, and tool inputs and outputs are written into the trace files. This is the default posture and needs no setting: bodies are captured whole, with no truncation, and a failed browser action additionally carries the profile’s recent console/JS-exception buffer (the browser’s recent log and error messages) in its error details.

What lands on disk is therefore the literal text of messages, model replies, and tool results. It stays on the machine: the trace directory sits inside FERMIX_HOME unless FERMIX_TRACE_DIR moves it, FERMIX_HOME itself is kept owner-only (mode 0700; fermix doctor fails its home-permissions check and prints the chmod that fixes it), and nothing forwards the bodies anywhere unless the Opik exporter below is enabled. Trace files are not pruned or size-capped, so a busy day’s tool_exec.jsonl reaches tens of megabytes with content captured; deleting old date directories is the operator’s job.

To suppress it, set FERMIX_TRACE_CONTENT=0 in the daemon (the long-running background Fermix process) environment — the same rule as FERMIX_OPIK_ENABLED below applies, so a shell export alone never reaches a service-managed daemon. With content suppressed, records still note that each event happened, but input and output bodies are omitted entirely rather than shortened. The variable accepts 1/true/yes and 0/false/no; any other value fails loud at startup.

Log file

Standard runtime logs go to ~/.fermix/logs/fermix.log (override with FERMIX_LOG_FILE). Rotation is configured at 10 MB per file, keeping up to 5 files (when a file fills up, Fermix starts a new one and discards the oldest). Every line is redacted before it is written: a formatter wraps both the file handler and the console handler, so all log output — OTP crash reports included, which no source-level redaction can see — is scanned for credential-shaped tokens and each match replaced with a [REDACTED:<vendor>] marker. See auth and secrets for the vendor patterns covered. Tail the log (stream new lines as they are written) with:

fermix logs -f

Inspecting traces

# List today's trace files
ls ~/.fermix/traces/$(date +%F)/

# Filter shell tool calls
jq 'select(.tool == "shell")' ~/.fermix/traces/$(date +%F)/tool_exec.jsonl

# Find slow LLM calls
jq 'select(.duration_ms > 5000)' ~/.fermix/traces/$(date +%F)/llm_call.jsonl

# Show errors
jq '.' ~/.fermix/traces/$(date +%F)/error.jsonl

Traces from multiple days can be queried by globbing across date directories:

jq 'select(.success == false)' ~/.fermix/traces/*/tool_exec.jsonl

Health endpoint

GET /health/ready returns a structured JSON report covering:

• Config file path and FERMIX_HOME
• Provider status (per configured provider)
• Per-channel status (Telegram, WhatsApp, Slack, Discord, Signal)
• Memory backend process status
• Realtime voice status

GET /health/live returns process liveness only (no dependency checks). /health is a backward-compatible alias for /health/ready.

curl -s http://127.0.0.1:4030/health/ready | jq .

See configuration for the bind address (FERMIX_HTTP_BIND) and channels for per-channel setup.

Opik exporter

fermix_opik is a built-in component that exports Fermix telemetry to an Opik instance (Opik is an external dashboard for inspecting LLM traces and costs). It is compiled into every build but is entirely inert unless enabled: it does nothing and adds zero overhead when off.

Enable it by setting FERMIX_OPIK_ENABLED=1 in the daemon environment:

FERMIX_OPIK_ENABLED=1 fermix run

When running as a system service, a shell export alone never reaches the daemon process. Set the variable and then reinstall the unit (the OS service definition: launchd on macOS, systemd on Linux) so it is captured into that definition:

# After exporting FERMIX_OPIK_ENABLED=1 in your shell
fermix service install

fermix doctor reports whether the exporter is off, enabled-but-not-bundled, or ready (with the resolved endpoint and project name).

When enabled, the exporter:

• Reads the shared provider, tool, job, and lifecycle telemetry correlated by session_id and parent_session.
• Maps each run (interactive turn, subagent, scheduled job, memory review, coding-agent run) into Opik’s trace model, nesting child runs under their parent. Coding-agent runs are the exception: each is exported as its own root trace, correlated to the turn that launched it by origin_session_id metadata rather than nested under it, so a background run that outlives its spawning turn still exports cleanly.
• Renders LLM spans (the per-call records in Opik’s view) with provider, model, and token usage for Opik’s cost attribution. A new provider must register a short identifier for cost to work.
• Carries prompt and response bodies into Opik spans whole, under the same content-capture switch described above: FERMIX_TRACE_CONTENT=0 suppresses them in Opik as well as in the JSONL files.

The exporter reads events the runtime emits regardless. Its absence or failure never affects a turn or reply.

Environment variable	Default	Description
FERMIX_OPIK_ENABLED	unset (off)	Set to 1 to activate the exporter
FERMIX_OPIK_BASE_URL	http://localhost:5173/api (local Opik)	Override the Opik endpoint URL
FERMIX_OPIK_PROJECT	fermix	Override the Opik project name
FERMIX_TRACE_CONTENT	unset (content captured)	0 suppresses prompt, response, and tool bodies in both traces and Opik spans; 1 states the default posture explicitly
FERMIX_TRACE_DIR	~/.fermix/traces	Override the JSONL trace output directory
FERMIX_LOG_FILE	~/.fermix/logs/fermix.log	Override the rotating log file path

For webhook configuration that feeds into the observability pipeline, see webhooks. For all configuration keys and environment variables, see configuration.