Ambitwin · source-available, not open source · © 2026 Nicolò Ghignatti
| Word | Meaning | Example |
|---|---|---|
| Reading | One value from one room at one moment | 24.5 °C in room r2 |
| Batch | One building’s readings sent in one request | 40 readings, POST /telemetry/ingest |
| Metric | A kind of reading, handled by one plugin | temperature, airQuality |
| Plugin | One file that declares a metric: fields, unit, limits, commands | plugins/temperature.rs |
| Device kind | What a sensor physically is. One device can report several metrics | router reports 3 counts |
| Bounds | The limits of one metric, as named numbers | {"maxTemp": 28, "minTemp": 16} |
| Breach | A field past its limit | 30 °C with maxTemp 28 |
| HMAC | A signature only the holder of a key can make | x-signature on every batch |
| Device key | The key one building’s devices sign with, derived from a master key | hex(HMAC(master, "b1:3")) |
| Epoch | A counter per building. Bumping it changes the building’s key | 3 |
| Microkernel | A small core that knows nothing about the specific kinds of data. Each kind plugs in from outside | kernel/ and plugins/ |
| Tick | Everything one building reports at one moment, as one message | one Wi-Fi snapshot |
| Hypertable | A PostgreSQL table that TimescaleDB splits by time, so recent data is fast and old data compresses | readings |
| Port | What the kernel needs, in its own words | ReadingStore |
| Adapter | The code that does it with a real technology | PgReadings |
telemetry is a Rust service (Axum, PostgreSQL with TimescaleDB, Kafka, Redis) in backend/telemetry. For the role and the choices behind it, read Telemetry architecture. For storage, Telemetry Storage.
It is the door for every sensor reading. It checks, stores and hands on readings. It also owns the limits, the sensors and the device commands, and it raises an alert for every breach.
Imagine the mail room of a big building.
alerts topic on Kafka).maxTemp and minTemp”. The mail room itself does not know what a temperature is. Adding a new kind of sensor means adding one file.flowchart LR
S["sensors, simulators"] -- "POST /ingest (signed)" --> T[("telemetry<br/>the mail room")]
C["ap-collector"] -- "POST /observations" --> T
T --> PG[("PostgreSQL<br/>the archive")]
T -- "telemetry:raw" --> R["Redis, then dashboard"]
T -- "alerts" --> K["Kafka, then notification"]
T -- "wifi-observations" --> W["digital-twin"]
W -- "estimated-occupancy" --> T
Tw["digital-twin"] <-- "registration" --> TPorts and adapters around a microkernel. tests/architecture.rs fails the build when a rule breaks.
| Path | What it holds | Must not import |
|---|---|---|
src/types/ | Plain shapes: Reading, Sensor, Bounds, BoundSpec, SensorPlugin, DomainError | anything else in the crate, any IO crate |
src/kernel/ | The use cases: ingest, observations, occupancy, readings, thresholds, sensors, actions, registration, simulation, devices, authz, registry, plus ports.rs | plugins, adapters, IO crates |
src/plugins/ | temperature, air_quality, people_count, device_count, common | kernel, adapters, IO crates, another plugin |
src/adapters/driven/ | postgres/, redis_fanout, kafka_producer, twin_directory, dispatch, simulators, threshold_cache, building_cache | |
src/adapters/driving/ | http_api/ (controllers, claims, exceptions), kafka_consumer, occupancy_consumer | driven |
src/adapters/ingest_auth.rs | Signature checks, device keys | |
src/lib.rs, src/main.rs | The router. Start-up wiring | |
migrations/ | 0001 to 0007, run at start |
IO crates the kernel may never name: sqlx, redis, rdkafka, axum, reqwest, prometheus. The kernel is pure logic, so its tests run against in-memory fakes.
flowchart TD
H["adapters/driving<br/>HTTP, Kafka consumers"] --> K["kernel<br/>Ingest, Observations, Registration, ..."]
K --> T["types<br/>Reading, Bounds, SensorPlugin"]
K --> P{{"ports"}}
P -.-> A["adapters/driven<br/>Postgres, Redis, Kafka"]
M["main.rs"] --> PL["plugins<br/>temperature, airQuality, ..."]
PL --> T
M --> KThe kernel and the plugins never import each other. Only main.rs brings them together.
A plugin has to answer five questions. They are the SensorPlugin trait in types/plugin.rs:
pub trait SensorPlugin: Send + Sync {
fn key(&self) -> &'static str;
fn descriptor(&self) -> &MetricDescriptor;
fn validate(&self, payload: &Value) -> Result<Reading, Vec<String>>;
fn bounds(&self) -> &'static [BoundSpec];
fn actions(&self) -> &'static [ActionSpec] {
&[]
}
}key: its name (temperature). This is the type a reading carries.descriptor: label, unit, and the list of fields a reading must have.validate: turn raw JSON into a checked Reading, or a list of complaints.bounds: which limits it has.actions: which commands a device can get (default: none).pub enum FieldKind {
NonEmptyString,
Finite,
NonNegativeInt,
}
pub struct FieldSpec {
pub name: &'static str,
pub kind: FieldKind,
pub required: bool,
}
fn accepts(kind: FieldKind, value: &Value) -> bool {
match kind {
FieldKind::NonEmptyString => value.as_str().is_some_and(|s| !s.is_empty()),
FieldKind::Finite => value.as_f64().is_some(),
FieldKind::NonNegativeInt => value.as_f64().is_some_and(|n| n >= 0.0 && n.fract() == 0.0),
}
}A field is a name, a kind and “required or not”. check_fields runs every spec and collects messages like temperature: must be a finite number. A missing optional field (or null) is fine.
static DESCRIPTOR: MetricDescriptor = MetricDescriptor {
value_field: "temperature",
key: "temperature",
label: "Temperature",
interface_name: "ITemperature",
unit: Some("C"),
fields: &[
FieldSpec { name: "buildingId", kind: FieldKind::NonEmptyString, required: true },
FieldSpec { name: "roomId", kind: FieldKind::NonEmptyString, required: true },
FieldSpec { name: "timestamp", kind: FieldKind::Finite, required: true },
FieldSpec { name: "temperature",kind: FieldKind::Finite, required: true },
],
};What a temperature reading must contain. value_field says which field holds the value (the one stored in the value column).
static BOUNDS: &[BoundSpec] = &[
BoundSpec { key: "maxTemp", field: "temperature", label: "Temperature", unit: Some("°C"), direction: BoundDirection::Above },
BoundSpec { key: "minTemp", field: "temperature", label: "Temperature", unit: Some("°C"), direction: BoundDirection::Below },
];Two limits. key is the name of the limit in the stored bounds (maxTemp). field is the payload field it is compared with. direction says “breach when above” or “breach when below”.
fn validate(&self, payload: &Value) -> Result<Reading, Vec<String>> {
let errors = check_fields(DESCRIPTOR.fields, payload);
if !errors.is_empty() {
return Err(errors);
}
let value = payload[DESCRIPTOR.value_field].as_f64().unwrap_or_default();
Ok(reading(payload, DESCRIPTOR.key, value))
}validate is short: check the fields, and if there are no complaints, read the value and build the Reading. The actions are setTarget(target), increase(step?) and decrease(step?).
| Plugin key | Value field | Bounds (key, field, direction) | Actions | Device kind |
|---|---|---|---|---|
temperature | temperature | maxTemp high, minTemp low, both on temperature | setTarget(target), increase(step?), decrease(step?) | temperature (Thermostat) |
airQuality | indoor_aqi | maxCo2 high on co2, maxAqi high on indoor_aqi | none | airQuality |
peopleCount | peopleCount | maxPeople high | none | peopleCount |
totalDeviceCount | totalDeviceCount | none | none | router |
ratioDeviceCount | ratioDeviceCount | none | none | router |
estimatedOccupancy | estimatedOccupancy | none | none | router |
PluginRegistry::new refuses two plugins with one key.DeviceCatalog::new refuses a metric with no device, or with two. The service does not start.telemetry_schema::ALERTABLE_METRICS, or a test fails. See Telemetry Schema. That list is also how notification knows which metrics to deliver.router is one device that reports three metrics. A sensor’s actions are the union of its device’s metrics’ actions.Adding a metric = a plugin file, plus its line in plugins::all(), plus a device in plugins::devices().
sequenceDiagram
autonumber
participant M as main.rs
participant PG as PostgreSQL
participant K as Kafka
participant R as Redis
M->>M: probe_exit_code: started with --health?
M->>M: read DATABASE_URL and TELEMETRY_DEVICE_MASTER_KEY, both required
M->>PG: pool of 10, then sqlx::migrate!
M->>M: PluginRegistry::new(plugins::all())
M->>M: DeviceCatalog::new(plugins::devices())
M->>M: DeviceKeys::new(master, shared key)
M->>M: HttpDispatch::from_json(bindings.json)
M->>R: RedisFanout::connect
M->>M: HttpSimulators::from_json(SIMULATORS), Simulation::new
M->>K: KafkaEvents::connect, ensure_topics in the background
M->>M: build Ingest, Registration, AppState
M->>K: spawn kafka_consumer, group building-registrations
M->>K: spawn occupancy_consumer, group estimated-occupancy
M->>M: bind 0.0.0.0:PORT, serve telemetry::routerAny ? in this sequence stops the process: a missing key, a short key (under 32 bytes), a failed migration, a plugin clash, a bad SIMULATORS. Kafka being down does not stop it: topics are created in the background, and produces fail after 5 s until it is back.
Every field and status code: Telemetry API (OpenAPI).
The edge strips /telemetry. Three exact paths are open at the edge and checked here by signature: /telemetry/ingest, /telemetry/collector, /telemetry/observations. Any sub-path, like /telemetry/ingest/batch, falls into the gated block and gets 401. The same three are listed in the Istio no-principal rule. Every other route needs the gateway pass.
| Method | Path | Who calls | Check | What it does |
|---|---|---|---|---|
POST | /ingest | Devices, simulators | Body HMAC | Store a batch |
POST | /observations | ap-collector | Body HMAC | Count a Wi-Fi tick, pass it to Kafka |
GET | /collector?buildingId= | ap-collector | Timestamp HMAC | Routers to poll |
GET | /contracts | dashboard | none | Metric catalog |
GET | /devices | client | none | Device kinds |
GET | /health, /metrics | Probes, Prometheus | none | |
GET | /{metric}/latest?building=&roomId= | client | read | Newest reading of one room |
GET | /{metric}/entireBuilding?building= | client | read | Newest reading of each room |
GET | /{metric}/dashboard?building=&roomId=&timeRange=&start=&end=&aggMode= | client | read | Buckets over time |
GET | /connected-devices/buildings/{id} | client | read | Building device total |
GET | /thresholds/buildings/{id} | client | read | Temperature limits, building and rooms |
PUT | /thresholds/buildings/{id} | digital-twin | edit | Register name, rooms, optional maxTemperature |
GET, PATCH | /thresholds/{metric}/buildings/{id} | client | read, edit | Building bounds |
PATCH | /thresholds/{metric}/buildings/{id}/rooms | client | edit | Bounds of many rooms, all or nothing |
PATCH | /thresholds/{metric}/buildings/{id}/rooms/{roomId} | client | edit | Bounds of one room |
GET, POST | /sensors/buildings/{id} | client | read, edit | List. Apply a batch |
GET | /sensors/buildings/{id}/rooms/{roomId} | client | read | Sensors of one room |
POST | /device-keys/buildings/{id} | client | edit | New device key |
GET, PUT, DELETE | /simulation/buildings/{id} | client | read, edit, edit | Status, start, stop |
POST | /executeAction | client | edit | Send a command to a device |
read and edit in controllers.rs run before any data is touched:
sequenceDiagram
autonumber
participant C as controllers.rs
participant D as TwinDirectory
participant Tw as digital-twin
participant A as kernel/authz.rs
C->>C: GatewayClaims: header present, decodes, has sub
C->>D: domains_of(building, raw claims)
D->>D: cache younger than 60 s?
D->>Tw: GET /domain/{building}, x-gateway-claims, 5 s timeout
Tw-->>D: ["eng"]
D->>D: cache 60 s
C->>A: read: is_member_of(claims, "eng")<br/>edit: can_edit_domains(claims, ["eng"])
A-->>C: allow, or 403 Not authorized for this buildingtelemetry does not know who owns a building. It asks digital-twin, forwarding the caller’s own claims, then lets Cedar decide.
| Case | Answer |
|---|---|
No header, bad base64, no sub | 401 |
digital-twin down, slow, or non-2xx | 403. A failed lookup is never cached |
| Caller in no owning domain | 403, telemetry_authz_denials_total{action} + 1 |
edit uses Cedar with the shared bundle from backend/libs/auth-policy. A role Cedar does not know is ignored, never treated as “allow all”.
Every error is {"type": "...", "message": "..."}:
DomainError | Status | type |
|---|---|---|
Validation | 400 | Validation Error |
Rejected | 422 | Rejected Error, plus errors: [{ref, field, message}] |
NotFound | 404 | Not Found Error |
Forbidden | 403 | Forbidden Error |
BadGateway | 502 | Bad Gateway Error |
Internal | 500 | Internal Server Error. The real error only in the log |
ingest_auth.rsDevices do not have a person’s JWT. They sign each request instead.
pub fn sign(&self, body: &[u8]) -> String {
let mut mac = Hmac::<Sha256>::new_from_slice(&self.0).expect("hmac accepts keys of any length");
mac.update(body);
mac.finalize().into_bytes().iter().fold(String::new(), |mut hex, byte| {
let _ = write!(hex, "{byte:02x}");
hex
})
}HMAC-SHA256 over the body bytes, written as lowercase hex. A key shorter than 32 bytes is refused when it is made (IngestKey::new), so a weak secret fails at start.
/// The building's key at `epoch`: hex HMAC(master, "{building}:{epoch}").
pub fn derive(&self, building_id: &str, epoch: i32) -> String {
self.master.sign(format!("{building_id}:{epoch}").as_bytes())
}
pub async fn issue(&self, building_id: &str) -> anyhow::Result<Option<String>> {
let epoch = self.epochs.rotate(building_id).await?;
Ok(epoch.map(|epoch| self.derive(building_id, epoch)))
}There is one master key (TELEMETRY_DEVICE_MASTER_KEY). A building’s key is the HMAC of the master key over "{buildingId}:{epoch}". So:
device_key_epoch number.issue). The old key stops working at once, and the new key is shown once.b1 does not open b2: the epoch string is different.Example. b1 at epoch 2. Rotation sets epoch 3. A collector still signing with the epoch 2 key gets 401 until it is given the new key. b2 is not affected.
pub async fn accepts(&self, building_id: Option<&str>, message: &[u8], signature: &str) -> bool {
let signed_by = |key: &IngestKey| constant_time_eq(&key.sign(message), signature);
if self.shared.as_ref().is_some_and(signed_by) {
return true;
}
let Some(building_id) = building_id else { return false; };
match self.epochs.epoch(building_id).await {
Ok(Some(epoch)) => IngestKey::new(&self.derive(building_id, epoch)).is_ok_and(|key| signed_by(&key)),
Ok(None) => false,
Err(error) => { log::warn!("device key epoch unreadable for {building_id}: {error:?}"); false }
}
}TELEMETRY_INGEST_SECRET) signs for any building. It is for local runs and simulators only.The epoch is read from PostgreSQL on every request, never cached, so a rotated key stops working on every copy of the service at once.
fn constant_time_eq(a: &str, b: &str) -> bool {
a.len() == b.len()
&& a.bytes().zip(b.bytes()).fold(0u8, |acc, (x, y)| acc | (x ^ y)) == 0
}The comparison looks at every byte and never stops early, so the time it takes does not reveal how many leading characters of a guess were right.
pub async fn verify_signature(State(keys): State<DeviceKeys>, request: Request, next: Next) -> Result<Response, StatusCode> {
let Some(signature) = request.headers().get(SIGNATURE_HEADER).and_then(|v| v.to_str().ok()).map(str::to_owned)
else {
metrics::record_ingest("unknown", "unsigned");
return Err(StatusCode::UNAUTHORIZED);
};
let (parts, body) = request.into_parts();
let bytes = axum::body::to_bytes(body, MAX_BODY_BYTES).await.map_err(|_| StatusCode::PAYLOAD_TOO_LARGE)?;
// Read only to pick the key; the handler still validates the whole body.
let building = serde_json::from_slice::<Addressed>(&bytes).ok();
let building_id = building.as_ref().map(|b| b.building_id.as_str());
if !keys.accepts(building_id, &bytes, &signature).await {
metrics::record_ingest("unknown", "bad_signature");
return Err(StatusCode::UNAUTHORIZED);
}
Ok(next.run(Request::from_parts(parts, Body::from(bytes))).await)
}x-signature: 401, counted unsigned.1 << 20). More: 413.buildingId only to choose the key. The real handler still validates the whole body.401, counted bad_signature.flowchart TD
S{"x-signature header?"} -- no --> U1["401, ingest unsigned"]
S -- yes --> B{"body 1 MiB or less?"}
B -- no --> P["413"]
B -- yes --> Sh{"TELEMETRY_INGEST_SECRET set<br/>and it signs the body?"}
Sh -- yes --> Ok["pass on"]
Sh -- no --> Id{"body has a buildingId?"}
Id -- no --> U2["401, bad signature"]
Id -- yes --> Ep{"building registered,<br/>epoch readable?"}
Ep -- no --> U2
Ep -- yes --> K["key = hex(HMAC(master, id:epoch))"]
K --> M{"HMAC(key, body) equals header,<br/>constant-time compare?"}
M -- no --> U2
M -- yes --> Okschemas/fixtures/internal-signature.json) are checked by the Go services, so the scheme cannot drift on one side.A GET has no body, so the collector signs a made-up message instead:
pub fn collector_message(building_id: Option<&str>, timestamp: &str) -> String {
format!("GET /collector\n{}\n{timestamp}", building_id.unwrap_or_default())
}
pub fn is_fresh(timestamp: &str, now_s: i64) -> bool {
timestamp.parse::<i64>().is_ok_and(|stamped| (now_s - stamped).abs() <= MAX_SKEW_S) // 300
}The signed text is GET /collector, the building, and the time (x-timestamp). verify_collector_request accepts only a signature that is valid and fresh (within 300 s of the server’s clock). A captured request is useless after 5 minutes.
Without buildingId only the shared key passes, and it reads every building.
sequenceDiagram
autonumber
participant A as ap-collector
participant V as verify_collector_request
participant H as controllers::collector
A->>V: GET /telemetry/collector?buildingId=b1<br/>x-timestamp 1761300000, x-signature
V->>V: message = collector_message(b1, timestamp)
V->>V: is_fresh: within 300 s of now?
V->>V: DeviceKeys::accepts(b1, message, signature)
V->>H: pass on
H->>H: Sensors::for_collector: routers in rooms, by building, sorted
H-->>A: {"buildings":[{"buildingId":"b1","routers":[…]}]}Each router in the answer has sensorId, roomId, and driver, endpoint only when set. Never a login.
kernel/ingest.rsThis is the heart of the service.
pub const MAX_BATCH_READINGS: usize = 500;
pub struct Ingest {
pub registry: Arc<PluginRegistry>,
pub readings: Arc<dyn ReadingStore>,
pub thresholds: Arc<dyn ThresholdStore>,
pub fanout: Arc<dyn Fanout>,
pub alerts: Arc<dyn Alerts>,
pub buildings: Arc<dyn BuildingStore>,
pub clock: Arc<dyn Clock>,
}Seven collaborators. Six are ports. registry is the list of plugins. The clock is a port so tests can fix the time.
pub async fn accept(&self, building_id: &str, items: &[Value]) -> Result<usize, DomainError> {
if items.len() > MAX_BATCH_READINGS {
return Err(DomainError::Validation(format!("readings: must not exceed {MAX_BATCH_READINGS} per batch.")));
}
let mut errors = Vec::new();
if building_id.trim().is_empty() {
errors.push("buildingId: must be a non-empty string.".to_owned());
}
if items.is_empty() {
errors.push("readings: must not be empty.".to_owned());
}
let mut readings = Vec::with_capacity(items.len());
for (index, item) in items.iter().enumerate() {
match self.reading_from(building_id, item) {
Ok(reading) => readings.push(reading),
Err(message) => errors.push(format!("readings[{index}]: {message}")),
}
}
if !errors.is_empty() {
return Err(DomainError::Validation(errors.join(" ")));
}400. Nothing is stored, nothing is published. A batch is all or nothing.reading_fromlet Some(object) = item.as_object() else { return Err("must be an object.".to_owned()); };
let metric = object.get("type").and_then(Value::as_str).map(str::trim).filter(|m| !m.is_empty())
.ok_or_else(|| "type: must be a non-empty string.".to_owned())?;
let plugin = self.registry.get(metric).ok_or_else(|| format!("type: unknown sensor type: {metric}."))?;
let mut payload: Map<String, Value> = object.clone();
payload.remove("type");
if !building_id.trim().is_empty() {
if let Some(own) = object.get("buildingId").and_then(Value::as_str) && own != building_id {
return Err(format!("buildingId: must match the batch's building ({building_id})."));
}
payload.insert("buildingId".to_owned(), Value::from(building_id));
}
plugin.validate(&Value::Object(payload)).map_err(|messages| messages.join(" "))must be an object.type missing or blank: type: must be a non-empty string.type: unknown sensor type: humidity.type.buildingId: must match the batch's building (b1). Otherwise the batch’s buildingId is stamped in. buildingId rides on the batch, so producers stop sending the field that could disagree.plugin.validate. This is the only place the kernel meets a specific kind of sensor, and it does it through the trait.Example. A batch of 40 readings for b1. Reading 17 has no roomId. Reading 30 says "type":"humidity".
| Step | Result |
|---|---|
| Count | 40, under 500 |
| Readings 0 to 16, 18 to 29, 31 to 39 | valid |
| Reading 17 | error from the plugin, prefixed readings[17]: |
| Reading 30 | readings[30]: type: unknown sensor type: humidity. |
| Answer | 400 {"type":"Validation Error","message":"readings[17]: … readings[30]: …"}. 0 stored |
let keys: Vec<(&str, &str)> = readings.iter().map(|r| (r.metric.as_str(), r.room_id.as_str())).collect();
let (inserted, resolved) = tokio::join!(
self.readings.insert(&readings),
self.thresholds.resolve(building_id, &keys),
);
inserted?;
let resolved = resolved.unwrap_or_else(|error| {
log::error!("threshold evaluation failed: {error}");
Vec::new()
});tokio::join! runs the insert and the limit lookup at the same time. The insert is one statement with on conflict do nothing. The two outcomes are treated differently:
inserted?): the whole call fails with 500. Nothing is published and no alert goes out.threshold evaluation failed) and treated as “no limits”. The readings are stored and published, but there are no alerts for that batch.for (reading, bounds) in readings.iter().zip(resolved) {
self.raise_breach(reading, bounds).await;
}For each reading, resolved holds its limits (or nothing). resolve picks the room’s bounds when it has any, else the building’s. Room bounds replace building bounds; they are not merged.
pub fn breaches(spec: &[BoundSpec], bounds: &Bounds, payload: &Map<String, Value>) -> Vec<Breach> {
let mut found: Vec<Breach> = Vec::new();
for bound in spec {
if found.iter().any(|seen| seen.bound.field == bound.field) {
continue;
}
let threshold = bounds.get(bound.key).and_then(Value::as_f64);
let value = payload.get(bound.field).and_then(Value::as_f64);
let (Some(threshold), Some(value)) = (threshold, value) else { continue; };
let breached = match bound.direction {
BoundDirection::Above => value > threshold,
BoundDirection::Below => value < threshold,
};
if breached {
found.push(Breach { bound: *bound, value, threshold });
}
}
found
}For each limit the plugin declares, in order:
field already breached? Skip. One alert per field, and the first listed limit wins.Above: breach when value is greater than the limit. Below: breach when it is smaller.A limit compares its own payload field, not the reading’s main value. That is why air quality can limit co2 and indoor_aqi separately.
Example. Building b1 has {"maxTemp":28}. Room r2 has {"maxTemp":24,"minTemp":16}.
| Reading | Bounds used | Result |
|---|---|---|
r1, 25 °C | building: max 28 | none |
r2, 25 °C | room: max 24, min 16 | high, threshold 24 |
r2, 14 °C | room | low, threshold 16 |
r1, air quality co2 1200, indoor_aqi 160, building has maxCo2 1000, maxAqi 150 | building | 2 alerts: one on co2, one on indoor_aqi |
// A failed lookup names the alert by its ids rather than dropping it.
let names = self.buildings.names_of(&reading.building_id).await.unwrap_or_else(|error| {
log::error!("building name lookup failed: {error}");
None
});
let building_name = names.as_ref().map_or(&reading.building_id, |n| &n.name);
let room_name = names.as_ref().and_then(|n| n.rooms.get(&reading.room_id)).unwrap_or(&reading.room_id);
for breach in found {
let alert = AlertPayload { metric, field, building_id, room_id, building_name, room_name,
value: breach.value, label: breach.bound.label, unit, direction, threshold: breach.threshold, ts_ms: self.clock.now_ms() };
self.alerts.publish_breach(&alert).await;
}Each breach becomes its own alert, with the building and room names from the registration (cached 5 min). A failed name lookup, or an unregistered room, falls back to the ids: an alert with an ugly name beats no alert. The alert is queued to Kafka and not awaited.
let now_ms = self.clock.now_ms();
let events: Vec<TelemetryEvent> = readings.iter().map(|reading| TelemetryEvent::from_reading(reading, now_ms)).collect();
self.fanout.publish_telemetry(&events).await;
Ok(events.len())All the readings become one tick on telemetry:raw. The answer is 202 {"accepted":true,"readings":40}.
sequenceDiagram
autonumber
participant G as Device or simulator
participant V as ingest_auth.rs<br/>verify_signature
participant H as controllers::ingest
participant I as kernel::Ingest::accept
participant PG as PostgreSQL
participant K as Kafka
participant R as Redis
G->>V: POST /telemetry/ingest, x-signature, body
V->>V: read body, at most 1 MiB
V->>V: peek buildingId, DeviceKeys::accepts
V->>H: same bytes
H->>H: readings must be an array
H->>I: accept(buildingId, readings)
I->>I: 500 or fewer? building non-empty? list non-empty?
loop each reading
I->>I: reading_from: type, plugin, buildingId, plugin.validate
end
par
I->>PG: readings.insert, one statement, on conflict do nothing
and
I->>PG: thresholds.resolve, through the 60 s cache
end
loop each reading with bounds
I->>I: raise_breach: breaches(plugin.bounds, bounds, payload)
I->>K: publish_breach, queued, not awaited
end
I->>R: fanout.publish_telemetry, one tick on telemetry:raw
H-->>G: 202 {"accepted":true,"readings":40}| Failure | Result |
|---|---|
| Insert fails | 500. Nothing published, no alert. telemetry_ingest_persist_failures_total + 1 |
| Bounds lookup fails | Logged as threshold evaluation failed. Readings stored and published, no alerts for that batch |
| Kafka refuses an alert | Logged, counted telemetry_alerts_published_total{outcome="error"}. The batch is still 202 |
| Redis publish fails | Logged, counted. The batch is still 202 |
| Same batch sent twice | The second insert stores nothing (unique key building_id, metric, room_id, ts). Alerts and the tick are sent again |
The tick shape and its path to the browser: Telemetry Schema.
kernel/observations.rsap-collector polls the routers about every 10 seconds and sends one snapshot per building.
pub async fn accept(&self, body: &[u8]) -> Result<(), DomainError> {
let tick: WifiObservations = serde_json::from_slice(body)
.map_err(|error| DomainError::Validation(format!("wifi observations: {error}.")))?;
let problems = tick.problems();
if !problems.is_empty() {
return Err(DomainError::Validation(problems.join(" ")));
}
let sensors = self.sensors.by_building(&tick.building_id).await?;
let readings = room_readings(&tick, &sensors);
if !readings.is_empty() {
self.ingest.accept(&tick.building_id, &readings).await?;
}
Ok(self.events.publish_observations(&tick.building_id, body).await?)
}mac field would travel with them.problems() (from the shared crate) checks the tick against the contract. Any problem: 400, nothing published.Ingest path.wifi-observations, keyed by the building.fn room_readings(tick: &WifiObservations, sensors: &[Sensor]) -> Vec<Value> {
let rooms: HashMap<&str, &str> = sensors.iter()
.filter(|sensor| sensor.sensor_type == ROUTER)
.filter_map(|sensor| Some((sensor.sensor_id.as_str(), sensor.room_id.as_deref()?)))
.collect();
let mut loudest: HashMap<&str, (i32, &str)> = HashMap::new();
for phone in &tick.connected {
let Some(&room) = rooms.get(phone.router.as_str()) else { continue; };
let heard = (phone.signal, room);
loudest.entry(phone.token.as_str())
.and_modify(|best| *best = (*best).max(heard))
.or_insert(heard);
}
let mut counts: BTreeMap<&str, u64> = tick.answered.iter()
.filter_map(|router| Some((*rooms.get(router.as_str())?, 0)))
.collect();
for (_, room) in loudest.into_values() {
*counts.entry(room).or_default() += 1;
}
// … one totalDeviceCount reading per room; ratioDeviceCount too when devicesPerPerson is set
}(signal, room) pair. On equal signal the room id that sorts last wins.totalDeviceCount reading per room, stamped with the tick time.devicesPerPerson, also ratioDeviceCount = count ÷ factor, rounded up. The source says why: rounding down 0.4 of a person would report an occupied room as empty.Example. Routers r1 in lobby, r2 in hall. Both answered. Factor 2.5.
| Phone | Heard by | Counts in |
|---|---|---|
| A | r1 −70, r2 −60 | hall (louder) |
| B | r1 −55 | lobby |
| C | r1 −60 | lobby |
| D | out (outdoor router) | nowhere |
Readings: totalDeviceCount lobby 2, hall 1. ratioDeviceCount lobby ⌈2 ÷ 2.5⌉ = 1, hall ⌈1 ÷ 2.5⌉ = 1.
| Rule | Why |
|---|---|
| Unknown fields refused | The bytes go to Kafka as received. A stray mac would travel with them |
| An answered empty room is 0. An unanswered room sends nothing | A failed router did not measure its room. A 0 would look like a fresh empty room |
| Counts stored before the Kafka publish | A count that cannot be stored is 500, and the tick is not published |
| Kafka refuses | 500. The collector drops that tick |
Topic wifi-observations, retention 10 min | A tick is stale in seconds, and it is pseudonymous |
| No state between ticks | A phone on a room boundary can switch rooms each tick |
sequenceDiagram
autonumber
participant A as ap-collector
participant V as verify_signature
participant O as kernel::Observations::accept
participant S as SensorStore
participant I as Ingest::accept
participant K as Kafka
A->>V: POST /telemetry/observations, signed body
V->>O: same bytes
O->>O: from_slice into WifiObservations, unknown fields refused
O->>O: tick.problems(): token, signal, limits
O->>S: by_building(buildingId)
O->>O: room_readings(tick, sensors)
O->>I: accept(buildingId, counts), same path as a batch
O->>K: publish_observations: the received bytes, key buildingId
O-->>A: 202 {"accepted":true}digital-twindigital-twin places each phone more precisely (see Digital Twin) and sends room counts back. telemetry stores them as a separate metric.
let readings: Vec<Value> = occupancy.rooms.iter().map(|(room, count)| json!({
"type": ESTIMATED_OCCUPANCY,
"roomId": room,
"timestamp": occupancy.tick,
ESTIMATED_OCCUPANCY: count,
})).collect();
let mut stored = 0;
for batch in readings.chunks(MAX_BATCH_READINGS) {
stored += self.ingest.accept(&occupancy.building_id, batch).await?;
}Each room becomes an estimatedOccupancy reading and goes through the same Ingest, in chunks of 500. It is never added to totalDeviceCount: the two place a phone differently.
estimated-occupancy, starts at the earliest offset, commits by hand. After an outage the missed counts are read from the topic.estimatedOccupancy has no bounds and never alerts.sequenceDiagram
autonumber
participant K as Kafka estimated-occupancy
participant C as occupancy_consumer.rs
participant O as kernel::Occupancy
participant I as Ingest::accept
K->>C: EstimatedOccupancy for one building
C->>O: rooms to estimatedOccupancy readings
loop each chunk of 500
O->>I: accept(buildingId, chunk)
end
C->>K: commit the offsetkernel/registration.rsA building must exist here before its sensors can be set up. Two entries lead to one use case, Registration::register.
pub async fn register(&self, building_id: &str, payload: &Value) -> Result<(), DomainError> {
let request = RegistrationRequest::deserialize(payload)
.map_err(|error| DomainError::Validation(format!("registration payload: {error}.")))?;
if request.name.trim().is_empty() {
return Err(DomainError::Validation("name: must be a non-empty string.".to_owned()));
}
// Postgres refuses NUL in text, and a store error is retried, so it would block the topic.
if std::iter::once(&request.name)
.chain(request.rooms.iter().flat_map(|room| [&room.id, &room.name]))
.any(|text| text.contains('\0'))
{
return Err(DomainError::Validation("name, rooms: must not contain a NUL character.".to_owned()));
}
self.buildings.upsert(&RegisteredBuilding { id: building_id.to_owned(), name: request.name,
rooms: request.rooms.into_iter().map(|room| Room { id: room.id, name: room.name }).collect() }).await?;
if let Some(max_temperature) = request.max_temperature {
let mut patch = Bounds::new();
patch.insert("maxTemp".to_owned(), max_temperature.into());
self.thresholds.upsert(building_id, None, "temperature", &patch).await?;
}
Ok(())
}400-class error.maxTemperature came with it, save it as the building’s maxTemp. (Twin does not send it over Kafka today, but the field stays: it is optional, not dead.)pub async fn register_from_event(&self, building_id: &str, payload: &Value) -> anyhow::Result<bool> {
// A store outage is not the request's fault: answering `failed` would delete the twin.
let outcome = match self.register(building_id, payload).await {
Err(DomainError::Internal(error)) => return Err(error),
outcome => outcome.map_err(|error| error.to_string()),
};
let registered = outcome.is_ok();
self.events.publish_completed(building_id, outcome).await?;
Ok(registered)
}failed with the reason. digital-twin then deletes the building.Internal) is returned as an error and is not answered. The consumer does not commit, reconnects after 5 s and reads the message again. A database outage is not the building’s fault, and answering failed would delete a good building.sequenceDiagram
autonumber
participant Tw as digital-twin
participant K as Kafka
participant C as kafka_consumer.rs
participant G as Registration
participant PG as PostgreSQL
Tw->>K: building-registration-requested {buildingId, name, rooms}
K->>C: message, group building-registrations
C->>C: handle: JSON? has buildingId?
C->>G: register_from_event(buildingId, payload)
G->>G: register: parse, name not blank, no NUL character
G->>PG: upsert building and rooms
G->>K: building-registration-completed, ready or failed
C->>K: commit the offset
Tw->>G: PUT /thresholds/buildings/b1 {name, rooms, maxTemperature}
G->>PG: same upsert, plus building maxTemp when givenflowchart TD
M["request message"] --> J{"JSON with buildingId?"}
J -- no --> D["log, commit, no answer"]
J -- yes --> R{"register"}
R -- "valid, stored" --> Ready["answer ready, commit"]
R -- "invalid: no name, NUL, bad maxTemperature" --> Failed["answer failed with the reason, commit"]
R -- "store error" --> Retry["no answer, no commit<br/>reconnect after 5 s, read again"]
Ready --> P{"answer published?"}
Failed --> P
P -- no --> RetryPUT /thresholds/buildings/{id} that digital-twin calls after an edit. It is the same register.| Route | Writes | Rule |
|---|---|---|
PATCH …/buildings/{id} | Building bounds of one metric | Merged into what is stored (`bounds |
PATCH …/buildings/{id}/rooms | {roomId: bounds} for many rooms | Every entry checked before any is written |
PATCH …/rooms/{roomId} | One room |
Each patch is checked by Thresholds::checked: every key must be a bound of that metric’s plugin, and every value a number. {"maxTmp": 28} on temperature is 400 maxTmp: not a bound of temperature. Because the check asks the plugin which keys exist, a new metric needs no change here.
CachedThresholds sits in front of the store:
resolve answers from a cache of up to 10 000 entries, each at most 60 s old. One tick asks PostgreSQL once, only for the rooms not cached.b1 and metric temperature drops every cached room of b1 / temperature, because a building row is the fallback for all rooms.sequenceDiagram
autonumber
participant C as client
participant H as controllers::apply_sensors
participant S as kernel::Sensors::apply
participant PG as PostgreSQL
C->>H: POST /sensors/buildings/b1 {create, update, delete}
H->>H: edit check
H->>S: apply(b1, body)
S->>S: arrays? 200 items or fewer? refs and ids present?
S->>PG: rooms of b1 from the registration, sensors of b1
loop each create, update, delete
S->>S: check, collect ItemError
end
S->>PG: one transaction
H-->>C: 200 {"created":[{"ref":"new-1","sensorId":"6f1c…"}]}A batch of changes follows the same idea as ingest: check everything, then write everything, or nothing.
| Item | Checks |
|---|---|
| create | ref unique in the batch. name not blank. sensorType a device kind from /devices. roomId null or a room of the building |
| update | sensorId a sensor of this building, once per batch. name not blank when given. roomId as above when given |
| delete | sensorId a sensor of this building, once per batch |
400.404.422 {"errors":[{"ref","field","message"}]}, nothing written. The ref lets the screen put each message next to the right row.sensorId sent on create is ignored.roomId null means outdoors.digital-twin, under the same sensorId.sequenceDiagram
autonumber
participant C as client
participant Si as kernel::Simulation::start
participant S1 as sensor-simulator
participant S2 as aq-simulator
participant S3 as ap-simulator
C->>Si: PUT /simulation/buildings/b1
Si->>Si: any simulator configured? building registered?
Si->>Si: plan: each sensor to the simulator claiming its kind
par join_all, 5 s timeout each
Si->>S1: POST /control/start {buildingId, sensors:[temperature, peopleCount]}
and
Si->>S2: POST /control/start {buildingId, sensors:[airQuality]}
and
Si->>S3: POST /control/start {buildingId, sensors:[]}
end
Si-->>C: 204, or 502 when any failedtelemetry does not simulate anything itself. It tells each simulator what to simulate, at start, and only then.
Simulation::new refuses a kind that is not a device, or one claimed twice (that would double the readings).502.404.running when any simulator says so. One that does not answer counts as not running.The body never carries coordinates: positions are digital-twin’s. Body shape: Telemetry Schema, fixture-only shapes.
sequenceDiagram
autonumber
participant C as client
participant H as controllers::execute_action
participant A as kernel::Actions::execute
participant D as HttpDispatch
participant PG as PostgreSQL
participant Dev as Device
C->>H: POST /executeAction {actionData:{metric, buildingId, sensorId, action, arguments}}
H->>H: fields non-empty, edit check
H->>A: execute(command)
A->>A: plugin knows the action? arguments match its parameters?
A->>D: dispatch(command)
D->>PG: driver, endpoint of (buildingId, sensorId)
D->>D: binding = bindings.json[driver][action]
D->>D: rename arguments, check URL is http or https
D->>Dev: POST endpoint + path, 5 s timeout
Dev-->>D: 2xx
H-->>C: 200 {"accepted":true,"action":"setTarget"}The kernel checks the command against the plugin’s action list (the same FieldSpec checks as for readings). The adapter then turns it into a real call for that kind of device, using bindings.json.
Example. setTarget with {"target": 21} on a sensor with driver tp-simulator and endpoint http://sensor-simulator:3000. bindings.json maps target to value, so the device gets POST http://sensor-simulator:3000/control/receive with {"value": 21}.
| Case | Answer |
|---|---|
| Unknown metric or action | 404 |
| Missing or wrong argument | 400 |
| Sensor has no driver or endpoint, or no binding | 404 no binding for sensor … |
| Endpoint not http(s) | 404, logged |
| Device answers non-2xx, or is unreachable | 502 |
The sensor is found by (buildingId, sensorId), never by room, so an outdoor or moved sensor can still be reached. The endpoint must be http or https: the service posts to it, so anything else would let a stored value point it somewhere strange. More: Sensor Actions.
| Route | Answer |
|---|---|
latest | One reading. roomId missing is 400. No data is 404 |
entireBuilding | {"data": [reading per room]} |
dashboard | {"data": [{"timestamp","value"}]} |
connected-devices | {buildingId, totalDeviceCount, timestamp}. Never reported is 404 |
A reading is answered flat: building, roomId, timestamp, value, plus every plugin field at the top level. The same names as the live tick, so a page that loads history and then live data handles one shape.
dashboard query:
timeRange | Window | Bucket |
|---|---|---|
missing or 1D | last 24 h | 1 hour |
1W | last 7 days | 1 day |
1M | last 30 days | 1 day |
custom | start to end (end defaults to now). No start is 400 | 1 day |
aggMode: sum, min, max, anything else avg.
Example, connected devices. At 10:00:00 lobby reports 12, hall 5. office last reported at 09:40 (its router was removed). Answer: {"totalDeviceCount": 17, "timestamp": <10:00:00>}. office is left out: only rooms at the newest report count. The sum is done here, never by a client, and only a count adds up: never route an average through it.
| Table | Holds | Notes |
|---|---|---|
readings | building_id, room_id, metric, ts, value, payload | Hypertable. Unique on building_id, metric, room_id, ts. Compressed after 7 days, dropped after 14 |
readings_hourly | Hourly summaries | Continuous aggregate, refreshed every hour over the last 3 days |
thresholds | building_id, room_id, metric, bounds | room_id null = the building row |
sensors | sensor_id (key), building_id, room_id, name, sensor_type, driver, endpoint | room_id null = outdoors |
buildings, building_rooms | Registration: names and rooms. device_key_epoch |
Kafka topics it creates: building-registration-requested, building-registration-completed, alerts, wifi-observations (10 min).
The unique key building_id, metric, room_id, ts is what makes a repeated batch harmless to the archive: the second insert simply stores nothing.
| Variable | Default | What it is |
|---|---|---|
DATABASE_URL | required | PostgreSQL with TimescaleDB |
TELEMETRY_DEVICE_MASTER_KEY | required, 32 bytes or more | Derives every device key |
TELEMETRY_INGEST_SECRET | unset | A shared key that signs for any building. Local and simulators only |
REDIS_URL | redis://redis:6379 | Tick fan-out |
KAFKA_BROKERS | kafka:9092 | Registration, alerts, Wi-Fi, room counts |
DIGITAL_TWIN_URL | http://digital-twin:3000 | Ownership lookup |
SIMULATORS | unset | JSON {name: {url, kinds}}. Compose gives temperature, peopleCount to sensor-simulator, airQuality to aq-simulator, router to ap-simulator |
PORT | 3000 | HTTP port |
| Name | What it counts |
|---|---|
telemetry_ingest_total{metric, outcome} | Readings accepted, batches invalid, unsigned, bad_signature |
telemetry_ingest_persist_failures_total, …_persist_duration_seconds | Insert failures and time |
telemetry_threshold_breaches_total{metric, direction} | Breaches found |
telemetry_alerts_published_total{topic, outcome} | Kafka answers to alerts |
telemetry_published_total{outcome} | Redis publishes |
telemetry_registration_total{outcome}, …_duration_seconds | ready, failed, unacknowledged |
telemetry_occupancy_ticks_total | Room-count messages |
telemetry_query_duration_seconds{query} | Read times |
telemetry_authz_denials_total{action} | read, edit denials |
telemetry_http_requests_total, …_error_requests_total, …_duration_seconds | Every route |
telemetry_db_pool_connections | Pool use |
just test telemetry # unit, kernel against fakes, no database
just test telemetry-integration # tests/*.rs against throwaway TimescaleDB, Redis, Kafka| Where | What it checks |
|---|---|
src/kernel/*.rs | Every rule above against in-memory fakes: all-or-nothing batch, the 500 cap, every breach its own alert, a failed name lookup still alerts by id, Wi-Fi counting cases, sensors batch errors, simulation plans, actions |
src/adapters/ingest_auth.rs | Golden HMAC vectors shared with Go. The collector request fixture. A key per building. Rotation revokes. The shared key only when set. Short keys refused |
src/adapters/driven/*.rs | Threshold cache: one query per tick, invalidation per building and metric, capacity. Twin lookup cache and timeout. Dispatch renames. Simulator bodies match the fixture |
tests/api.rs | HTTP: /contracts equals fixtures/metric-contract.json. Every ingest-batch.json case and rejection |
tests/persistence.rs | The PostgreSQL adapters |
tests/fanout.rs | A tick reaches telemetry:raw |
tests/alerts.rs | A breach reaches alerts keyed by building and room, even before anyone subscribes |
tests/registration.rs | Ready answer. A repeat converges. Failed with reason. A bad message does not stop the consumer. A request sent before start is answered. An unpublished answer is read again |
tests/cedar_conformance.rs | Cedar decisions match the shared golden cases |
tests/architecture.rs | The layering rules in the table above |
notification mutes repeats for 5 minutes.GET has a time limit.