/
CrowdVision · source-available, not open source · © 2026 Nicolò Ghignatti
This page covers metrics — what’s instrumented, what scrapes it, and where the gaps are. CrowdVision has no centralized logging or distributed tracing today; each service logs to its own stdout, and the only tracing in the system is agent’s optional OpenTelemetry export to Langfuse for LLM call inspection, which is unrelated to the Prometheus stack described here — see Agent Service for that.
Prometheus and Grafana are deployed twice, independently, with different configuration:
| Aspect | docker-compose (dev) | Kubernetes |
|---|---|---|
| Config source | prometheus.yml (repo root), mounted read-only into the container | k8s/monitoring/prometheus.yml’s prometheus-config ConfigMap |
| Target discovery | Static: 4 hardcoded job_name/targets entries | Kubernetes service discovery (kubernetes_sd_configs) for app pods and the mesh’s own components, a static target for istiod, plus a cadvisor job via the API server proxy |
| Grafana datasource | Not auto-provisioned — added manually via the UI | Auto-provisioned via GF_DATASOURCES_DEFAULT_* env vars pointing at http://prometheus:9090 |
| Exposed at | /prometheus/, /grafana/ (Caddy handle_path/handle) | /grafana only (its Istio HTTPRoute) — Prometheus has no gateway route, reachable only in-cluster |
| Retention | Whatever the image default is — not configured | 15 days, backed by a 5Gi PVC (--storage.tsdb.retention.time=15d) |
Both stacks load the same seven alerting rules — see Alerting rules below.
prometheus.rules.yml (repo root, docker-compose) and the alert.rules.yml key inside k8s/monitoring/prometheus.yml’s ConfigMap define the same crowdvision-alerts group — kept in sync by hand, since the two stacks have no shared config source. None of the queries filter by a job/app label: every metric name below is emitted by exactly one service today, so the metric name alone scopes the query, which is what keeps one rule set valid across both stacks despite their different target-discovery mechanisms (a docker-compose target’s job label is the service name directly; a k8s pod’s is kubernetes-pods, with the service name instead carried on an app label via labelmap).
| Alert | Fires when | for |
|---|---|---|
ServiceDown | Any scrape target’s up == 0 (covers all 4 instrumented services in both stacks, generically). | 2m |
TwinServiceHighErrorRate / ChatServiceHighErrorRate | The error-to-total HTTP request ratio exceeds 5%. | 5m |
TwinServiceHighLatency / ChatServiceHighLatency | p95 HTTP request duration exceeds 1s. | 5m |
ContractsServiceFanoutLatencyHigh | p95 telemetry_fanout_latency_ms (sensor-ingestion-to-fan-out) exceeds 500ms. | 5m |
Neither stack deploys Alertmanager, so a firing alert is visible in Prometheus’s /alerts UI and exposed as the ALERTS metric, but nothing routes it anywhere — no email, Slack, or PagerDuty notification is sent. Someone still has to be looking at the page. Wiring up Alertmanager (and a receiver) is a separate piece of work from defining the rules themselves.
dashboard has no HTTP-shaped error counter to alert on (only the fan-out latency histogram above); it would need new instrumentation (an error counter, a request-duration histogram) before a matching alert could be written.
socket is a WebSocket relay with no request/response HTTP metrics either, but it now exports two failure counters that are alertable — relay_messages_skipped_total (a publisher is emitting malformed JSON, so readings are silently not reaching browsers) and socket_connections_rejected_total (handshakes refused for a missing or malformed claims header, which spikes if the mesh stops injecting it). Neither has an alert rule written yet.
Of the 11 backend services, only 4 expose a /metrics endpoint:
| Service | Library | Key metrics |
|---|---|---|
| digital-twin (Rust) | prometheus crate | http_requests_total, http_error_requests_total, http_request_duration_seconds — labeled {method, route, status_code}, using the route template (not the raw URL) so cardinality stays bounded even with real ids in the path |
| dashboard (Rust) | prometheus crate | telemetry_fanout_latency_ms, telemetry_events_received_total, telemetry_events_published_total — business metrics specific to its sensor-data fan-out role, not generic HTTP counters |
| socket (Rust) | prometheus crate | telemetry_relayed_total, notifications_relayed_total{scope}, relay_payload_bytes_total{channel}, relay_messages_skipped_total{channel}, socket_connections_rejected_total, socket_connected_clients — no default process metrics; the nodejs_* series ended with the Rust rewrite |
| notification (Rust) | prometheus crate | http_requests_total, http_error_requests_total, http_request_duration_seconds — same {method, route, status_code} shape as digital-twin |
| chat (Rust) | prometheus crate | chat_http_requests_total, chat_http_error_requests_total, chat_http_request_duration_seconds — same {method, route, status_code} shape as digital-twin. The chat_ prefix survives the Rust rewrite because prometheus.rules.yml alerts on these exact names. No default process metrics: the last nodejs_* series, and the NodeEventLoopLagHigh alert that read them, went with the rewrite. |
agent, claims-gateway, tenancy, registry, and provisioner have no metrics instrumentation at all — no prom-client, prometheus_client, or promhttp, no /metrics route.
digital-twin, telemetry, dashboard, socket, notification, and chat carry prometheus.io/scrape: "true" / prometheus.io/port: "3000" pod annotations in their Deployment manifests, so Kubernetes’ kubernetes-pods autodiscovery job (below) picks them up automatically — no per-service Prometheus config needed. The other 5 services stay unscraped in Kubernetes until they gain a /metrics endpoint and the matching annotations.
The Kubernetes kubernetes-cadvisor job doesn’t depend on any service exposing anything — it reaches each node’s built-in cAdvisor through the Kubernetes API server proxy (/api/v1/nodes/<node>/proxy/metrics/cadvisor) and gets per-container CPU, memory, network, and disk usage for every pod in the cluster, instrumented or not. This is the only metrics signal that currently covers all 12 services uniformly — it just can’t see inside the application (request rates, error rates, latency), only its resource footprint.
Three additional scrape jobs in k8s/monitoring/prometheus.yml cover the Istio ambient mesh itself, not application code — none of this exists in the docker-compose/Caddy stack, since Caddy isn’t a real mesh.
| Job | Target | What it covers |
|---|---|---|
istiod | Static target istiod.istio-system:15014 | Control-plane health: certificate issuance (citadel_server_csr_count), xDS push status, root cert expiry. |
istio-ztunnel | Pod discovery in istio-system, filtered to app=ztunnel, scraped on :15020/stats/prometheus | Per-node L4 mTLS proxy stats (ztunnel is a DaemonSet with no Service, so this is pod discovery, not a static target). |
istio-ingress-gateway | Pod discovery in crowdvision, filtered to the gateway.networking.k8s.io/gateway-name: crowdvision-gateway label, scraped on :15090/stats/prometheus | Envoy stats for the auto-provisioned ingress gateway (envoy_server_uptime, request/connection counters). Its Service only exposes the app port and a status port, not the Envoy stats port, so this is also pod discovery rather than a Service target. |
This closes the gap where mesh traffic was effectively unobserved — request rates, error rates, and TLS/handshake behavior at the mTLS and ingress layers are now queryable in Grafana alongside application metrics. Distributed tracing (Kiali, Jaeger) is still out of scope; this is metrics only.