/

Ambitwin · source-available, not open source · © 2026 Nicolò Ghignatti

Observability

Keywords on this page
WordMeaningExample
MetricA number measured over timetelemetry_ingest_total
ScrapePrometheus reading a service’s /metricsEvery few seconds
Alert rule“Fire when this number looks bad”ServiceDown
TraceOne request followed across servicesNot built
cAdvisorBuilt-in container stats on each nodeCPU per pod

How a problem gets noticed, and what stays invisible.


The short version

SignalState
Metrics6 of 11 backend services
Alert rulesYes, but they tell nobody
LogsEach service’s stdout, not collected
TracesNone across services. agent alone sends LLM traces to Langfuse

Who exposes what

All six use the Rust prometheus crate.

ServiceMeasures
telemetrytelemetry_ingest_total, persist failures, publish outcomes, telemetry_http_*
digital-twinhttp_requests_total, http_error_requests_total, http_request_duration_seconds
notificationSame three names
chatchat_http_*
dashboardtelemetry_fanout_latency_ms, events in and out
socketMessages relayed, skipped, connections refused, clients connected

No metrics at all: agent, claims-gateway, tenancy, registry, provisioner.

HTTP metrics label the route template, never the real URL, so ids cannot explode the series.


Local vs cluster settings

Each environment has its own Prometheus and Grafana: compose starts one pair locally, the cluster runs another. Only one pair runs at a time. Their settings differ:

LocalCluster
Configprometheus.yml, prometheus.rules.ymlk8s/monitoring/prometheus.yml
Finds targetsFixed listPods annotated prometheus.io/scrape
Reach it/prometheus/, /grafana//grafana only
HistoryImage default15 days, 5 Gi

The cluster also scrapes what no code exposes:

JobShows
kubernetes-cadvisorCPU, memory, disk of every pod, the only signal covering all services
istiodControl plane health, certificates
istio-ztunnelmTLS proxy stats per node
istio-ingress-gatewayRequests at the edge

Alert rules

AlertFires when
ServiceDownA target stops answering
TwinServiceHighErrorRate, ChatServiceHighErrorRate, SensorServiceHighErrorRateOver 5% errors
TwinServiceHighLatency, ChatServiceHighLatency, SensorServiceHighLatencyp95 over 1 s
ContractsServiceFanoutLatencyHighp95 fan-out over 500 ms
SensorIngestPersistFailuresReadings fail to save
SensorIngestStalledNo accepted reading in 10 min
SensorTelemetryFanoutFailingPublishing fails

The queries name no job, so one rule set fits both stacks. They work because a metric name normally belongs to one service.

Trap: in the cluster digital-twin and notification share metric names, so the Twin rules really add up both services.


Gaps