Configure observability

Use this guide to monitor the runtime across tasks. If you only need one objective's progress or result, start with Observe agent work.

Before you start

Have a running Holon daemon, access to its configuration and control token, and a checkout of this repository for the sample assets. Install the OpenTelemetry Collector (otelcol-contrib), Prometheus, and Grafana if you intend to use all three. The commands below run from the repository root.

Plan a restart window before changing daemon configuration. Keep tokens in a credential store or mounted secret file. The example Collector listens on 0.0.0.0:4318; bind it to loopback for a local test or restrict network access before starting it.

1. Start a test collector

Use a deployment copy of the sample configuration, adjusting its bind address for your environment. Replace the path below with that prepared copy:

otelcol-contrib --config /path/to/prepared/otel-collector.yaml

The sample accepts OTLP/HTTP and sends traces to the Collector's debug exporter. For a production deployment, replace that exporter with your trace backend.

2. Enable Holon's exporter

Merge the following settings into the existing runtime.observability.otlp object in <HOLON_HOME>/config.json. Preserve unrelated configuration:

{
  "enabled": true,
  "endpoint": "http://127.0.0.1:4318/v1/traces",
  "queue_capacity": 1024,
  "batch_size": 128,
  "batch_interval_ms": 1000,
  "timeout_ms": 5000
}

For a collector that requires a bearer token:

printf '%s' "$OTLP_TOKEN" \
  | holon config credentials set --kind bearer_token --stdin otlp-collector

Add "credential_profile": "otlp-collector" to the OTLP object. Do not also set an authorization header in headers. Check configuration, then restart during the agreed window:

holon config doctor
holon daemon restart
holon daemon status

Submit a small authorized test request to an existing agent:

holon prompt --agent builder "Reply with a brief acknowledgement. Do not run commands or change files."
holon tail --agent builder --limit 5

Confirm both the agent's response and received spans in Collector output. A report appearing in Holon does not establish that trace export worked.

3. Check protected metrics

Use your daemon's actual HTTP address:

curl --fail --silent --show-error \
  -H "Authorization: Bearer $HOLON_CONTROL_TOKEN" \
  http://127.0.0.1:7878/api/control/runtime/metrics

Expect OpenMetrics text containing holon_process_uptime_seconds and ending with # EOF. Check exporter queue, failure, and drop counters after the test request. For an authenticated remote deployment, also confirm that an unauthenticated request is rejected.

4. Add Prometheus and Grafana

Prepare deployment copies of:

docs/website/assets/observability/prometheus.yaml
docs/website/assets/observability/holon-alerts.yaml
docs/website/assets/observability/grafana-dashboard.json

The Prometheus sample expects the control token at /run/secrets/holon-control-token, alert rules at /etc/prometheus/holon-alerts.yaml, and Holon at 127.0.0.1:7878. Mount or adjust those paths and address for your deployment. In containers, localhost refers to that container, so set the target to the reachable Holon service.

prometheus --config.file=/path/to/prepared/prometheus.yaml

Use the prepared configuration path, not the unmodified repository sample. Check that the Holon scrape target is healthy and that holon_process_uptime_seconds returns samples. Import the dashboard into Grafana, select the Prometheus data source, and verify that uptime and exporter panels show data. Confirm the alert rules loaded successfully. Tune thresholds after observing a representative workload.

5. Diagnose a missing signal

Symptom	Next check
No metrics response	Address, network path, authentication, and daemon status
Metrics work but no exported spans	OTLP endpoint, collector receiver, credentials, and `holon_otlp_exporter_failed_batches_total`
Export queue or drops increase	Collector throughput and `holon_otlp_exporter_queue_depth` / `holon_otlp_exporter_dropped_spans_total`
Missing locally retained traces	Retention/sampling policy and diagnostics writer failure/drop counters
Prometheus works but dashboard is empty	Data source selection, query time range, and sample timestamps
Agent task failed but telemetry is healthy	The task's report and output, using Troubleshooting

For an incident with a message or turn ID, search retained traces:

holon debug trace --search MESSAGE_OR_TURN_ID
holon debug trace TRACE_ID

Replace the placeholders with actual IDs. Local retention is independent of OTLP export, and sampling means not every trace is retained. Avoid restarting only to reset counters: capture before/after snapshots and preserve incident evidence first.

Acceptance and rollback

The setup is verified when a test turn produces an agent report, Collector spans, readable protected metrics, and dashboard samples. Record any component you did not configure or test.

If export must be disabled, set runtime.observability.otlp.enabled to false and restart in the agreed window. Collector failure should not block task execution; diagnose it as a telemetry failure rather than rerunning business actions automatically.

See the Observability reference for configuration fields, endpoint contracts, and performance snapshot fields.