Observability

v1.2.x · updated 2026-07-07

AiHummer observability has two surfaces: the gateway serves a Prometheus GET /metrics endpoint you can scrape for the basics, and it can optionally push telemetry over OTLP to an OpenTelemetry endpoint you configure. Scrape /metrics for baseline monitoring; for traces and rich metrics point AiHummer at your OTLP collector and visualise the data with the bundled Grafana dashboards.

[!NOTE] pprof (/debug/pprof) is not exposed.

The Prometheus `/metrics` endpoint

GET /metrics serves Prometheus text-format metrics with no extra setup. Only non-sensitive gauges are exposed — build info, process uptime and runtime, DB connection-pool state and a readiness gauge; no tenant data, no secrets, no per-request labels. For traces and rich metrics, use the OTLP push.

OTLP push

Set a single variable to turn telemetry on:

# gateway.env — export telemetry to your OTLP collector
AIHUMMER_OTEL_ENDPOINT=http://otel-collector:4317

With AIHUMMER_OTEL_ENDPOINT set, the gateway pushes telemetry to that collector. From there, route it to your backend (Tempo, a metrics store, logs) and into Grafana.

Panic and error handling

Error reports are never sent anywhere outside: the gateway ships no external error-tracker client and no external DSN — data about your errors never leaves your perimeter.

Panic resilience is nonetheless complete:

a panic in an HTTP handler becomes an ordinary error response (a 500 carrying an AIH-… envelope) — the process does not die and keeps serving other requests;
a panic in a background goroutine is recovered too and does not bring the gateway down.

Both cases land in the structured log — read them on the “Logs” page (below) or through journalctl.

Live logs in the admin UI

The admin UI’s “Logs” page is a live tail of the gateway journal: new lines are pulled in automatically every few seconds, with autoscroll while you are at the bottom. The toolbar has a line search and a level filter (all / errors / warnings / info / debug). Several other pages (Dashboard, Sessions, Channels) also refresh automatically, and long lists (audit, changes, notifications and so on) load page by page with a “Show more” button.

Full logs of every service still live in systemd — aihummer logs [unit] or journalctl -u aihummer-gateway.

Grafana dashboards

Ready-made Grafana dashboards ship with the release. Import them into your Grafana instance to get the operational views without building panels from scratch.

What to watch

These are the signals that tell you the system is healthy and turns are flowing:

Signal	Why it matters
Turn latency	End-to-end responsiveness of agent turns
Error rate	Failing turns / requests — the first sign of trouble
Delivery dispositions	Whether replies are actually reaching channels
Pending deliveries	Backlog of undelivered replies; sustained growth means delivery is stuck

A sustained rise in pending or repeatedly failed deliveries is the clearest early warning that delivery is backing up — watch it during rollouts and incidents.

System endpoints

Alongside OTLP, the gateway exposes small HTTP endpoints useful for probes, clocks and client diagnostics:

Method	Endpoint	Purpose
`GET`	`/metrics`	Prometheus metrics (build, runtime, DB pool, readiness)
`GET`	`/healthz`	Liveness + version
`GET`	`/readyz`	Readiness (checks Postgres; 503 if down)
`GET`	`/v1/ping`	Lightweight reachability check
`GET`	`/v1/time`	Server time
`POST`	`/v1/client-log`	Ingest client-side log events

The health and readiness probes are covered in detail under systemd & health checks.

Where to next

Probes and the production pre-flight checklist: systemd & health checks.
What to watch during a rolling upgrade: Upgrade policy.