Monitoring
What to watch, and what the signals mean.
Health endpoints
| Path | Answers |
|---|---|
/healthz | Is the process alive |
/readyz | Can it reach the database |
Use /healthz to decide whether to restart, /readyz to decide whether to send
traffic.
Metrics
The API, the worker and appd each serve Prometheus metrics at /metrics on
an internal port. The ports are not published, so scrape from a container on
the same Docker network, or publish them to a private interface yourself.
| Service | Variable | Default in compose |
|---|---|---|
| api | METRICS_LISTEN | :9464 |
| worker | WORKER_METRICS_LISTEN | :9465 |
| appd | APPD_METRICS_LISTEN | :9466, bound to its control network |
Set a variable to an empty value to turn that listener off. Nothing on these ports is authenticated.
The series worth a dashboard:
| Metric | Tells you |
|---|---|
lenshub_http_request_duration_seconds | API latency by route pattern |
lenshub_tool_call_duration_seconds, lenshub_tool_calls_total | MCP and chat tool latency and error rate, by tool |
lenshub_query_embeds_total{outcome="error"} | Searches that fell back to exact-name matches because the embedder was down or slow |
lenshub_db_pool_empty_acquire_total | Requests that waited for a database connection. A steady climb means the pool is too small |
lenshub_worker_queue_jobs | Queue depth by job type, across all workers |
lenshub_worker_queue_oldest_pending_seconds{job_type="embed"} | Embedding lag: how long new content waits before it is searchable |
lenshub_worker_queue_oldest_pending_seconds{job_type="classify"} | Classification lag: how long new content waits for its domain and tags. Grows when the LLM is unreachable |
lenshub_worker_jobs_total{outcome="failed"} | Jobs that ran out of retries |
lenshub_code_blobs{status="failed"} | Code files the code lane gave up on |
appd_builds_total, appd_egress_refusals_total | App build failures, and app or build traffic the egress proxy refused |
Labels never carry IDs, paths or hostnames, so the series count stays flat as content grows.
Worth alerting on
/readyz failing — the database is unreachable. Everything is broken.
A connector stuck in error — visible per connector. Usually an expired credential, and it fails quietly from a user's point of view: search simply stops returning anything new from that source.
Disk. The database grows with your content. Running out is the most common self-inflicted outage.
Job backlog growing without draining — indexing is not keeping up. Raise
WORKER_CONCURRENCY, or find out what is failing repeatedly. With metrics,
alert on lenshub_worker_queue_oldest_pending_seconds rather than depth:
a deep queue that drains is fine, a stale one is not.
Logs
docker compose logs -f api workerStructured JSON on the API. The worker logs each job it runs, which is where to look when a sync misbehaves.
In the interface
The Dashboard shows content counts by source, indexing progress (its Embeddings tab) and recent activity. It is the fastest way to answer "did that sync actually work".
Normal load
Steady state is light: connectors sync on their schedule, and searches are cheap. Two moments spike:
- A first sync of a large source
- Re-indexing after a model change
Both are one-off, and both end.