Observability & KPIs
Every metric Fred exports to Prometheus, the port it's on, and exactly which labels survive — plus the handful of near-misses (audit events, dead metric names, generic process stats) that look like Prometheus data but aren't.
Grafana sees one stream of three
Fred separates three kinds of signal that used to be conflated under one "KPI" umbrella. Prometheus/Grafana is only ever Stream 1 — operational health. If you're here looking for per-user, per-team, or per-agent-instance breakdowns, they live elsewhere by design, not by omission.
| Stream | Answers | Destination | Contains identity? |
|---|---|---|---|
| 1 — Operational metrics | Is the platform healthy? | Prometheus → Grafana | No — structurally excluded, see below |
| 2 — Product analytics | How is the platform used, by whom? | OpenSearch, via Fred's own authorization-scoped API | Yes, access-scoped per viewer |
| 3 — Security & audit trail | Who did what, when, with what outcome? | Structured log line → the platform's log pipeline | Yes — that's its entire purpose |
agent.tool.invocation.started / .completed, with
outcome and identity) and, separately, a Stream-1 metric
(agent.tool_latency_ms, no identity). They share a name prefix but are not the
same data — see the callout in Defined, but not on /metrics.
For per-user/per-team dashboards or an audit trail, see Access model for how that data is scoped and reached.
Source: fred docs docs/swift/platform/OBSERVABILITY-AND-AUDIT.md §2–§3
Where metrics are exposed
Metrics are not served on the API port. Each backend that emits Prometheus
metrics runs a second, dedicated HTTP server just for scraping
(prometheus_client.start_http_server), entirely separate from the service's own
request routing.
| Service | Exposes metrics | Notes |
|---|---|---|
| control-plane-backend | Yes | dedicated exporter, on by default (config-controlled) |
| knowledge-flow-backend | Yes | dedicated exporter, on by default (config-controlled) |
| fred-agents | Yes | dedicated exporter, on by default (config-controlled) |
| knowledge-flow Temporal worker | Yes | no HTTP API of its own — the exporter is all it serves |
| control-plane Temporal worker | No | no exporter of its own — and unlike the knowledge-flow worker, it currently emits no KPI events at all. agent.suspended_total / agent.suspension_cleared_total are emitted from the control-plane-backend API process instead (see row above), not the worker |
KpiPrometheusSinkConfig.enabled defaults to true in fred-core, so a
deployment exports to Prometheus unless observability.kpi.prometheus.enabled: false
is set explicitly. The checked-in dev configuration.yaml for control-plane-backend,
knowledge-flow-backend, and fred-agents all set it to false — confirm what your own
deployment's config does before assuming metrics are (or aren't) being exported.
Service, a GMP PodMonitoring, etc.) — check your own deployment's
service manifests for the actual values. The GCP/GKE reference deployment lives in the
fred-deployment-factory repository if you need a concrete example to start from.
Source: fred-core kpi/kpi_factory.py, common/structures.py (KpiPrometheusSinkConfig) · per-service main.py/context.py exporter startup
The label allow-list
One flat allow-list, shared by every metric. Any dimension not on this list is silently dropped before the metric ever touches the Prometheus client — this is an explicit, structural guarantee, not operator discipline: a new dimension needs a deliberate code change to ever appear as a label.
tool_name, status, error_code, exception_type, http_status,
template_agent_id, model, model_name, finish_reason,
runtime_id, source_runtime_id, service, env, cluster,
route, method, actor_type, file_type, agent_step
Deliberately excluded: user_id, session_id, exchange_id
(identity); team_id, agent_instance_id, agent_instance_name,
agent_id — "usage by team/agent instance" is a Stream-2 product-analytics
question with its own access-scoped answer, so it's deliberately not duplicated here as an
unscoped, platform-wide-visible label; and trace_id, correlation_id,
checkpoint_id, doc_uid, scope_id — unique per call, no
aggregate value as a label, and a pivot vector back into raw logs from a wide-audience dashboard.
/metrics — the reference below lists only what survives, not
what the call site passes in.
Source: fred-core kpi/prometheus_kpi_store.py (PROMETHEUS_ALLOWED_LABELS, _prometheus_label_dims)
Metric reference
Metric names use dots in code; Prometheus wire names sanitize every non-alphanumeric
character to _ (so agent.tool_latency_ms →
agent_tool_latency_ms on the wire, and a histogram adds the usual
_bucket/_sum/_count suffixes). Table below uses the
code-form name; apply the substitution when writing PromQL.
Agent & tool execution
| Metric | Type | Surviving labels |
|---|---|---|
| agent.tool_latency_ms | histogram | tool_name, template_agent_id; on failure also status, error_code, exception_type |
| agent.tool_failed_total | counter | tool_name, template_agent_id, status, error_code, exception_type |
| agent.turn_completed | histogram | template_agent_id, runtime_id, model_name, finish_reason (team_id, agent_instance_id/_name are sent but dropped) |
| agent.turn_error_total | counter | same as above; only emitted when the turn errors |
| agent.turn_completed_quantity_input_tokens_total agent.turn_completed_quantity_output_tokens_total agent.turn_completed_quantity_tool_count_total | counter | same labels as agent.turn_completed — auto-derived per-call from its quantities; this is where token counts actually live today, not any cost.* series |
| agent.suspended_total agent.suspension_cleared_total | counter | none informative — team_id, agent_instance_id, reason are all sent but all dropped |
API & LLM calls
| Metric | Type | Surviving labels |
|---|---|---|
| api.request_latency_ms | histogram | route, method, status, http_status, error_code, exception_type — emitted for every HTTP request to control-plane, knowledge-flow, and fred-agents alike |
| api.error_total | counter | same as above; only on non-2xx/3xx |
| error.total | counter | generic error counter emitted alongside api.error_total, same labels |
| llm.call_latency_ms | histogram | model_name only — agent_id and operation are sent but dropped |
Retrieval & ingestion
| Metric | Type | Surviving labels |
|---|---|---|
| rag.search_latency_ms | histogram | none by default — backend, index are sent but dropped; status/error_code survive when set |
| rag.search_total rag.search_error_total rag.search_hits_total rag.search_top_k_total rag.search_empty_total | counter | status |
| rag.search_hit_ratio | gauge | status |
| rag.rerank_latency_ms rag.rerank_total rag.rerank_docs_total rag.rerank_top_r_total | timer/counter | status, model |
| document.created_total document.deleted_total | counter | file_type only — source_type, team_id are sent but dropped |
| ingestion.document_duration_ms | histogram | file_type, status — source is sent but dropped |
| app.phase_latency_ms | histogram | agent_step only, and only from the ReAct/Graph runtime call sites that set it — phase is sent everywhere but always dropped; the knowledge-flow and vector-store call sites don't set agent_step at all, so those observations are effectively unlabeled |
| temporal.system.activity_queue_wait_ms | histogram | none — phase, activity_type, task_queue, workflow_type, attempt are all sent but all dropped |
| temporal.system.activity_duration_ms | histogram | status, error_code, file_type — phase, activity_type, task_queue, workflow_type, attempt, source_type, source_tag are sent but dropped |
| temporal.ingestion.documents_total | counter | status, error_code, file_type — same dims as temporal.system.activity_duration_ms above; the rest dropped |
| temporal.ingestion.workflows_total | counter | status only — task_queue, workflow_type, has_error, has_document_uid, has_filename, workflow_id_prefix are sent but dropped |
Source: fred-runtime react/middleware/tool_observability.py, tracing_kpi.py, app/agent_app.py · fred-core kpi/http_middleware.py, kpi/kpi_writer.py, kpi/kpi_phase_metric.py · knowledge-flow-backend features/metadata/service.py, features/ingestion/ingestion_controller.py, features/scheduler/kpi_utils.py · vector_search_service.py · control-plane-backend agent_instances/suspension.py
Defined, but not on /metrics
These look like Prometheus metrics — same naming convention, same codebase — and aren't. Don't build a panel against them:
agent.tool.invocation.started / .completed are
Stream-3 audit-log events, not metrics. They're written to the audit logger
and never call into the KPI/Prometheus pipeline at all.
process.memory.* / process.db_pool.* KPI gauges
are explicitly excluded from Prometheus by name-prefix check (they stay in the log/OpenSearch
pipeline). What you'll find under a similar-looking process_* name on
/metrics instead — process_resident_memory_bytes,
process_cpu_seconds_total, process_open_fds, etc. — comes from
prometheus_client's own generic process collector, and carries no DB-pool or
Fred-specific data at all.
agent.step_total, llm.rate_limit_events_total,
llm.request_latency_ms, and every cost.* field
(cost.tokens_prompt, cost.tokens_completion,
cost.tokens_total, cost.usd) are declared in code but have zero
call sites today — nothing emits them. Token counts are live today, just under a different
name: see agent.turn_completed_quantity_input_tokens_total /
_output_tokens_total above.
Scrape config & PromQL
Minimal prometheus.yml shape — swap in your deployment's actual metrics host
and port for each service (see the callout above; it's not the same value as the API port):
scrape_configs:
- job_name: fred-control-plane
static_configs:
- targets: ["<control-plane-metrics-host:port>"]
- job_name: fred-knowledge-flow
static_configs:
- targets: ["<knowledge-flow-metrics-host:port>"]
- job_name: fred-agents
static_configs:
- targets: ["<fred-agents-metrics-host:port>"]
- job_name: fred-knowledge-flow-worker
static_configs:
- targets: ["<knowledge-flow-worker-metrics-host:port>"]Example queries, matched to the verified type/label pairs above:
# p95 tool latency by tool
histogram_quantile(0.95,
sum(rate(agent_tool_latency_ms_bucket[5m])) by (le, tool_name))
# tool failure rate by tool
sum(rate(agent_tool_failed_total[5m])) by (tool_name)
/ sum(rate(agent_tool_latency_ms_count[5m])) by (tool_name)
# API error rate by route
sum(rate(api_error_total[5m])) by (route)
/ sum(rate(api_request_latency_ms_count[5m])) by (route)
# API p95 latency by route
histogram_quantile(0.95,
sum(rate(api_request_latency_ms_bucket[5m])) by (le, route))
# LLM call latency p95 by model
histogram_quantile(0.95,
sum(rate(llm_call_latency_ms_bucket[5m])) by (le, model_name))
# turn error rate by agent template
sum(rate(agent_turn_error_total[5m])) by (template_agent_id)
/ sum(rate(agent_turn_completed_count[5m])) by (template_agent_id)Types confirmed from recording calls in prometheus_kpi_store.py: "timer"/"distribution" → Histogram (use histogram_quantile on _bucket), "counter" → Counter (use rate()), "gauge" → Gauge (query directly).