/

Ambitwin · source-available, not open source · © 2026 Nicolò Ghignatti

Telemetry

Keywords on this page
WordMeaningExample
ReadingOne value from one room at one moment24.5 °C in room r2
BatchOne building’s readings sent in one request40 readings, POST /telemetry/ingest
MetricA kind of reading, handled by one plugintemperature, airQuality
PluginOne file that declares a metric: fields, unit, limits, commandsplugins/temperature.rs
Device kindWhat a sensor physically is. One device can report several metricsrouter reports 3 counts
BoundsThe limits of one metric, as named numbers{"maxTemp": 28, "minTemp": 16}
BreachA field past its limit30 °C with maxTemp 28
HMACA signature only the holder of a key can makex-signature on every batch
Device keyThe key one building’s devices sign with, derived from a master keyhex(HMAC(master, "b1:3"))
EpochA counter per building. Bumping it changes the building’s key3
MicrokernelA small core that knows nothing about the specific kinds of data. Each kind plugs in from outsidekernel/ and plugins/
TickEverything one building reports at one moment, as one messageone Wi-Fi snapshot
HypertableA PostgreSQL table that TimescaleDB splits by time, so recent data is fast and old data compressesreadings
PortWhat the kernel needs, in its own wordsReadingStore
AdapterThe code that does it with a real technologyPgReadings

telemetry is a Rust service (Axum, PostgreSQL with TimescaleDB, Kafka, Redis) in backend/telemetry. For the role and the choices behind it, read Telemetry architecture. For storage, Telemetry Storage.

It is the door for every sensor reading. It checks, stores and hands on readings. It also owns the limits, the sensors and the device commands, and it raises an alert for every breach.


The idea in one minute

Imagine the mail room of a big building.

flowchart LR
    S["sensors, simulators"] -- "POST /ingest (signed)" --> T[("telemetry<br/>the mail room")]
    C["ap-collector"] -- "POST /observations" --> T
    T --> PG[("PostgreSQL<br/>the archive")]
    T -- "telemetry:raw" --> R["Redis, then dashboard"]
    T -- "alerts" --> K["Kafka, then notification"]
    T -- "wifi-observations" --> W["digital-twin"]
    W -- "estimated-occupancy" --> T
    Tw["digital-twin"] <-- "registration" --> T

The files

Ports and adapters around a microkernel. tests/architecture.rs fails the build when a rule breaks.

PathWhat it holdsMust not import
src/types/Plain shapes: Reading, Sensor, Bounds, BoundSpec, SensorPlugin, DomainErroranything else in the crate, any IO crate
src/kernel/The use cases: ingest, observations, occupancy, readings, thresholds, sensors, actions, registration, simulation, devices, authz, registry, plus ports.rsplugins, adapters, IO crates
src/plugins/temperature, air_quality, people_count, device_count, commonkernel, adapters, IO crates, another plugin
src/adapters/driven/postgres/, redis_fanout, kafka_producer, twin_directory, dispatch, simulators, threshold_cache, building_cache
src/adapters/driving/http_api/ (controllers, claims, exceptions), kafka_consumer, occupancy_consumerdriven
src/adapters/ingest_auth.rsSignature checks, device keys
src/lib.rs, src/main.rsThe router. Start-up wiring
migrations/0001 to 0007, run at start

IO crates the kernel may never name: sqlx, redis, rdkafka, axum, reqwest, prometheus. The kernel is pure logic, so its tests run against in-memory fakes.

flowchart TD
    H["adapters/driving<br/>HTTP, Kafka consumers"] --> K["kernel<br/>Ingest, Observations, Registration, ..."]
    K --> T["types<br/>Reading, Bounds, SensorPlugin"]
    K --> P{{"ports"}}
    P -.-> A["adapters/driven<br/>Postgres, Redis, Kafka"]
    M["main.rs"] --> PL["plugins<br/>temperature, airQuality, ..."]
    PL --> T
    M --> K

The kernel and the plugins never import each other. Only main.rs brings them together.


Plugins: one file per kind of sensor

A plugin has to answer five questions. They are the SensorPlugin trait in types/plugin.rs:

pub trait SensorPlugin: Send + Sync {
    fn key(&self) -> &'static str;
    fn descriptor(&self) -> &MetricDescriptor;
    fn validate(&self, payload: &Value) -> Result<Reading, Vec<String>>;
    fn bounds(&self) -> &'static [BoundSpec];

    fn actions(&self) -> &'static [ActionSpec] {
        &[]
    }
}

The field specs

pub enum FieldKind {
    NonEmptyString,
    Finite,
    NonNegativeInt,
}

pub struct FieldSpec {
    pub name: &'static str,
    pub kind: FieldKind,
    pub required: bool,
}

fn accepts(kind: FieldKind, value: &Value) -> bool {
    match kind {
        FieldKind::NonEmptyString => value.as_str().is_some_and(|s| !s.is_empty()),
        FieldKind::Finite => value.as_f64().is_some(),
        FieldKind::NonNegativeInt => value.as_f64().is_some_and(|n| n >= 0.0 && n.fract() == 0.0),
    }
}

A field is a name, a kind and “required or not”. check_fields runs every spec and collects messages like temperature: must be a finite number. A missing optional field (or null) is fine.

A whole plugin: temperature

static DESCRIPTOR: MetricDescriptor = MetricDescriptor {
    value_field: "temperature",
    key: "temperature",
    label: "Temperature",
    interface_name: "ITemperature",
    unit: Some("C"),
    fields: &[
        FieldSpec { name: "buildingId", kind: FieldKind::NonEmptyString, required: true },
        FieldSpec { name: "roomId",     kind: FieldKind::NonEmptyString, required: true },
        FieldSpec { name: "timestamp",  kind: FieldKind::Finite,         required: true },
        FieldSpec { name: "temperature",kind: FieldKind::Finite,         required: true },
    ],
};

What a temperature reading must contain. value_field says which field holds the value (the one stored in the value column).

static BOUNDS: &[BoundSpec] = &[
    BoundSpec { key: "maxTemp", field: "temperature", label: "Temperature", unit: Some("°C"), direction: BoundDirection::Above },
    BoundSpec { key: "minTemp", field: "temperature", label: "Temperature", unit: Some("°C"), direction: BoundDirection::Below },
];

Two limits. key is the name of the limit in the stored bounds (maxTemp). field is the payload field it is compared with. direction says “breach when above” or “breach when below”.

fn validate(&self, payload: &Value) -> Result<Reading, Vec<String>> {
    let errors = check_fields(DESCRIPTOR.fields, payload);
    if !errors.is_empty() {
        return Err(errors);
    }
    let value = payload[DESCRIPTOR.value_field].as_f64().unwrap_or_default();
    Ok(reading(payload, DESCRIPTOR.key, value))
}

validate is short: check the fields, and if there are no complaints, read the value and build the Reading. The actions are setTarget(target), increase(step?) and decrease(step?).

What each plugin declares

Plugin keyValue fieldBounds (key, field, direction)ActionsDevice kind
temperaturetemperaturemaxTemp high, minTemp low, both on temperaturesetTarget(target), increase(step?), decrease(step?)temperature (Thermostat)
airQualityindoor_aqimaxCo2 high on co2, maxAqi high on indoor_aqinoneairQuality
peopleCountpeopleCountmaxPeople highnonepeopleCount
totalDeviceCounttotalDeviceCountnonenonerouter
ratioDeviceCountratioDeviceCountnonenonerouter
estimatedOccupancyestimatedOccupancynonenonerouter

Adding a metric = a plugin file, plus its line in plugins::all(), plus a device in plugins::devices().


Start-up

sequenceDiagram
    autonumber
    participant M as main.rs
    participant PG as PostgreSQL
    participant K as Kafka
    participant R as Redis
    M->>M: probe_exit_code: started with --health?
    M->>M: read DATABASE_URL and TELEMETRY_DEVICE_MASTER_KEY, both required
    M->>PG: pool of 10, then sqlx::migrate!
    M->>M: PluginRegistry::new(plugins::all())
    M->>M: DeviceCatalog::new(plugins::devices())
    M->>M: DeviceKeys::new(master, shared key)
    M->>M: HttpDispatch::from_json(bindings.json)
    M->>R: RedisFanout::connect
    M->>M: HttpSimulators::from_json(SIMULATORS), Simulation::new
    M->>K: KafkaEvents::connect, ensure_topics in the background
    M->>M: build Ingest, Registration, AppState
    M->>K: spawn kafka_consumer, group building-registrations
    M->>K: spawn occupancy_consumer, group estimated-occupancy
    M->>M: bind 0.0.0.0:PORT, serve telemetry::router

Any ? in this sequence stops the process: a missing key, a short key (under 32 bytes), a failed migration, a plugin clash, a bad SIMULATORS. Kafka being down does not stop it: topics are created in the background, and produces fail after 5 s until it is back.


Routes

Every field and status code: Telemetry API (OpenAPI).

The edge strips /telemetry. Three exact paths are open at the edge and checked here by signature: /telemetry/ingest, /telemetry/collector, /telemetry/observations. Any sub-path, like /telemetry/ingest/batch, falls into the gated block and gets 401. The same three are listed in the Istio no-principal rule. Every other route needs the gateway pass.

MethodPathWho callsCheckWhat it does
POST/ingestDevices, simulatorsBody HMACStore a batch
POST/observationsap-collectorBody HMACCount a Wi-Fi tick, pass it to Kafka
GET/collector?buildingId=ap-collectorTimestamp HMACRouters to poll
GET/contractsdashboardnoneMetric catalog
GET/devicesclientnoneDevice kinds
GET/health, /metricsProbes, Prometheusnone
GET/{metric}/latest?building=&roomId=clientreadNewest reading of one room
GET/{metric}/entireBuilding?building=clientreadNewest reading of each room
GET/{metric}/dashboard?building=&roomId=&timeRange=&start=&end=&aggMode=clientreadBuckets over time
GET/connected-devices/buildings/{id}clientreadBuilding device total
GET/thresholds/buildings/{id}clientreadTemperature limits, building and rooms
PUT/thresholds/buildings/{id}digital-twineditRegister name, rooms, optional maxTemperature
GET, PATCH/thresholds/{metric}/buildings/{id}clientread, editBuilding bounds
PATCH/thresholds/{metric}/buildings/{id}/roomsclienteditBounds of many rooms, all or nothing
PATCH/thresholds/{metric}/buildings/{id}/rooms/{roomId}clienteditBounds of one room
GET, POST/sensors/buildings/{id}clientread, editList. Apply a batch
GET/sensors/buildings/{id}/rooms/{roomId}clientreadSensors of one room
POST/device-keys/buildings/{id}clienteditNew device key
GET, PUT, DELETE/simulation/buildings/{id}clientread, edit, editStatus, start, stop
POST/executeActionclienteditSend a command to a device

Access check

read and edit in controllers.rs run before any data is touched:

sequenceDiagram
    autonumber
    participant C as controllers.rs
    participant D as TwinDirectory
    participant Tw as digital-twin
    participant A as kernel/authz.rs
    C->>C: GatewayClaims: header present, decodes, has sub
    C->>D: domains_of(building, raw claims)
    D->>D: cache younger than 60 s?
    D->>Tw: GET /domain/{building}, x-gateway-claims, 5 s timeout
    Tw-->>D: ["eng"]
    D->>D: cache 60 s
    C->>A: read: is_member_of(claims, "eng")<br/>edit: can_edit_domains(claims, ["eng"])
    A-->>C: allow, or 403 Not authorized for this building

telemetry does not know who owns a building. It asks digital-twin, forwarding the caller’s own claims, then lets Cedar decide.

CaseAnswer
No header, bad base64, no sub401
digital-twin down, slow, or non-2xx403. A failed lookup is never cached
Caller in no owning domain403, telemetry_authz_denials_total{action} + 1

edit uses Cedar with the shared bundle from backend/libs/auth-policy. A role Cedar does not know is ignored, never treated as “allow all”.

Error body

Every error is {"type": "...", "message": "..."}:

DomainErrorStatustype
Validation400Validation Error
Rejected422Rejected Error, plus errors: [{ref, field, message}]
NotFound404Not Found Error
Forbidden403Forbidden Error
BadGateway502Bad Gateway Error
Internal500Internal Server Error. The real error only in the log

The seal on a batch: ingest_auth.rs

Devices do not have a person’s JWT. They sign each request instead.

A key, and how to sign

pub fn sign(&self, body: &[u8]) -> String {
    let mut mac = Hmac::<Sha256>::new_from_slice(&self.0).expect("hmac accepts keys of any length");
    mac.update(body);
    mac.finalize().into_bytes().iter().fold(String::new(), |mut hex, byte| {
        let _ = write!(hex, "{byte:02x}");
        hex
    })
}

HMAC-SHA256 over the body bytes, written as lowercase hex. A key shorter than 32 bytes is refused when it is made (IngestKey::new), so a weak secret fails at start.

One key per building, never stored

/// The building's key at `epoch`: hex HMAC(master, "{building}:{epoch}").
pub fn derive(&self, building_id: &str, epoch: i32) -> String {
    self.master.sign(format!("{building_id}:{epoch}").as_bytes())
}

pub async fn issue(&self, building_id: &str) -> anyhow::Result<Option<String>> {
    let epoch = self.epochs.rotate(building_id).await?;
    Ok(epoch.map(|epoch| self.derive(building_id, epoch)))
}

There is one master key (TELEMETRY_DEVICE_MASTER_KEY). A building’s key is the HMAC of the master key over "{buildingId}:{epoch}". So:

Example. b1 at epoch 2. Rotation sets epoch 3. A collector still signing with the epoch 2 key gets 401 until it is given the new key. b2 is not affected.

Who is accepted

pub async fn accepts(&self, building_id: Option<&str>, message: &[u8], signature: &str) -> bool {
    let signed_by = |key: &IngestKey| constant_time_eq(&key.sign(message), signature);
    if self.shared.as_ref().is_some_and(signed_by) {
        return true;
    }
    let Some(building_id) = building_id else { return false; };
    match self.epochs.epoch(building_id).await {
        Ok(Some(epoch)) => IngestKey::new(&self.derive(building_id, epoch)).is_ok_and(|key| signed_by(&key)),
        Ok(None) => false,
        Err(error) => { log::warn!("device key epoch unreadable for {building_id}: {error:?}"); false }
    }
}
  1. The optional shared key (TELEMETRY_INGEST_SECRET) signs for any building. It is for local runs and simulators only.
  2. Otherwise, work out the building’s key from its current epoch, and compare.
  3. Unknown building, or an epoch that cannot be read: not accepted.

The epoch is read from PostgreSQL on every request, never cached, so a rotated key stops working on every copy of the service at once.

fn constant_time_eq(a: &str, b: &str) -> bool {
    a.len() == b.len()
        && a.bytes().zip(b.bytes()).fold(0u8, |acc, (x, y)| acc | (x ^ y)) == 0
}

The comparison looks at every byte and never stops early, so the time it takes does not reveal how many leading characters of a guess were right.

The middleware for a batch

pub async fn verify_signature(State(keys): State<DeviceKeys>, request: Request, next: Next) -> Result<Response, StatusCode> {
    let Some(signature) = request.headers().get(SIGNATURE_HEADER).and_then(|v| v.to_str().ok()).map(str::to_owned)
    else {
        metrics::record_ingest("unknown", "unsigned");
        return Err(StatusCode::UNAUTHORIZED);
    };

    let (parts, body) = request.into_parts();
    let bytes = axum::body::to_bytes(body, MAX_BODY_BYTES).await.map_err(|_| StatusCode::PAYLOAD_TOO_LARGE)?;

    // Read only to pick the key; the handler still validates the whole body.
    let building = serde_json::from_slice::<Addressed>(&bytes).ok();
    let building_id = building.as_ref().map(|b| b.building_id.as_str());
    if !keys.accepts(building_id, &bytes, &signature).await {
        metrics::record_ingest("unknown", "bad_signature");
        return Err(StatusCode::UNAUTHORIZED);
    }

    Ok(next.run(Request::from_parts(parts, Body::from(bytes))).await)
}
  1. No x-signature: 401, counted unsigned.
  2. Read the body, at most 1 MiB (1 << 20). More: 413.
  3. Peek at buildingId only to choose the key. The real handler still validates the whole body.
  4. Wrong seal: 401, counted bad_signature.
  5. Put the very same bytes back and let the request through.
flowchart TD
    S{"x-signature header?"} -- no --> U1["401, ingest unsigned"]
    S -- yes --> B{"body 1 MiB or less?"}
    B -- no --> P["413"]
    B -- yes --> Sh{"TELEMETRY_INGEST_SECRET set<br/>and it signs the body?"}
    Sh -- yes --> Ok["pass on"]
    Sh -- no --> Id{"body has a buildingId?"}
    Id -- no --> U2["401, bad signature"]
    Id -- yes --> Ep{"building registered,<br/>epoch readable?"}
    Ep -- no --> U2
    Ep -- yes --> K["key = hex(HMAC(master, id:epoch))"]
    K --> M{"HMAC(key, body) equals header,<br/>constant-time compare?"}
    M -- no --> U2
    M -- yes --> Ok

The collector’s GET

A GET has no body, so the collector signs a made-up message instead:

pub fn collector_message(building_id: Option<&str>, timestamp: &str) -> String {
    format!("GET /collector\n{}\n{timestamp}", building_id.unwrap_or_default())
}

pub fn is_fresh(timestamp: &str, now_s: i64) -> bool {
    timestamp.parse::<i64>().is_ok_and(|stamped| (now_s - stamped).abs() <= MAX_SKEW_S)   // 300
}

The signed text is GET /collector, the building, and the time (x-timestamp). verify_collector_request accepts only a signature that is valid and fresh (within 300 s of the server’s clock). A captured request is useless after 5 minutes.

Without buildingId only the shared key passes, and it reads every building.

sequenceDiagram
    autonumber
    participant A as ap-collector
    participant V as verify_collector_request
    participant H as controllers::collector
    A->>V: GET /telemetry/collector?buildingId=b1<br/>x-timestamp 1761300000, x-signature
    V->>V: message = collector_message(b1, timestamp)
    V->>V: is_fresh: within 300 s of now?
    V->>V: DeviceKeys::accepts(b1, message, signature)
    V->>H: pass on
    H->>H: Sensors::for_collector: routers in rooms, by building, sorted
    H-->>A: {"buildings":[{"buildingId":"b1","routers":[…]}]}

Each router in the answer has sensorId, roomId, and driver, endpoint only when set. Never a login.


Ingesting a batch: kernel/ingest.rs

This is the heart of the service.

pub const MAX_BATCH_READINGS: usize = 500;

pub struct Ingest {
    pub registry: Arc<PluginRegistry>,
    pub readings: Arc<dyn ReadingStore>,
    pub thresholds: Arc<dyn ThresholdStore>,
    pub fanout: Arc<dyn Fanout>,
    pub alerts: Arc<dyn Alerts>,
    pub buildings: Arc<dyn BuildingStore>,
    pub clock: Arc<dyn Clock>,
}

Seven collaborators. Six are ports. registry is the list of plugins. The clock is a port so tests can fix the time.

Step 1: check everything first

pub async fn accept(&self, building_id: &str, items: &[Value]) -> Result<usize, DomainError> {
    if items.len() > MAX_BATCH_READINGS {
        return Err(DomainError::Validation(format!("readings: must not exceed {MAX_BATCH_READINGS} per batch.")));
    }

    let mut errors = Vec::new();
    if building_id.trim().is_empty() {
        errors.push("buildingId: must be a non-empty string.".to_owned());
    }
    if items.is_empty() {
        errors.push("readings: must not be empty.".to_owned());
    }

    let mut readings = Vec::with_capacity(items.len());
    for (index, item) in items.iter().enumerate() {
        match self.reading_from(building_id, item) {
            Ok(reading) => readings.push(reading),
            Err(message) => errors.push(format!("readings[{index}]: {message}")),
        }
    }
    if !errors.is_empty() {
        return Err(DomainError::Validation(errors.join(" ")));
    }

What happens to one reading: reading_from

let Some(object) = item.as_object() else { return Err("must be an object.".to_owned()); };

let metric = object.get("type").and_then(Value::as_str).map(str::trim).filter(|m| !m.is_empty())
    .ok_or_else(|| "type: must be a non-empty string.".to_owned())?;

let plugin = self.registry.get(metric).ok_or_else(|| format!("type: unknown sensor type: {metric}."))?;

let mut payload: Map<String, Value> = object.clone();
payload.remove("type");
if !building_id.trim().is_empty() {
    if let Some(own) = object.get("buildingId").and_then(Value::as_str) && own != building_id {
        return Err(format!("buildingId: must match the batch's building ({building_id})."));
    }
    payload.insert("buildingId".to_owned(), Value::from(building_id));
}

plugin.validate(&Value::Object(payload)).map_err(|messages| messages.join(" "))
  1. Not an object: must be an object.
  2. type missing or blank: type: must be a non-empty string.
  3. No plugin with that key: type: unknown sensor type: humidity.
  4. Copy the reading without type.
  5. If the reading names a different building from the batch: buildingId: must match the batch's building (b1). Otherwise the batch’s buildingId is stamped in. buildingId rides on the batch, so producers stop sending the field that could disagree.
  6. Hand the payload to the plugin: plugin.validate. This is the only place the kernel meets a specific kind of sensor, and it does it through the trait.

Example. A batch of 40 readings for b1. Reading 17 has no roomId. Reading 30 says "type":"humidity".

StepResult
Count40, under 500
Readings 0 to 16, 18 to 29, 31 to 39valid
Reading 17error from the plugin, prefixed readings[17]:
Reading 30readings[30]: type: unknown sensor type: humidity.
Answer400 {"type":"Validation Error","message":"readings[17]: … readings[30]: …"}. 0 stored

Step 2: store and look up the limits at the same time

let keys: Vec<(&str, &str)> = readings.iter().map(|r| (r.metric.as_str(), r.room_id.as_str())).collect();
let (inserted, resolved) = tokio::join!(
    self.readings.insert(&readings),
    self.thresholds.resolve(building_id, &keys),
);
inserted?;

let resolved = resolved.unwrap_or_else(|error| {
    log::error!("threshold evaluation failed: {error}");
    Vec::new()
});

tokio::join! runs the insert and the limit lookup at the same time. The insert is one statement with on conflict do nothing. The two outcomes are treated differently:

Step 3: breaches

for (reading, bounds) in readings.iter().zip(resolved) {
    self.raise_breach(reading, bounds).await;
}

For each reading, resolved holds its limits (or nothing). resolve picks the room’s bounds when it has any, else the building’s. Room bounds replace building bounds; they are not merged.

pub fn breaches(spec: &[BoundSpec], bounds: &Bounds, payload: &Map<String, Value>) -> Vec<Breach> {
    let mut found: Vec<Breach> = Vec::new();
    for bound in spec {
        if found.iter().any(|seen| seen.bound.field == bound.field) {
            continue;
        }
        let threshold = bounds.get(bound.key).and_then(Value::as_f64);
        let value = payload.get(bound.field).and_then(Value::as_f64);
        let (Some(threshold), Some(value)) = (threshold, value) else { continue; };
        let breached = match bound.direction {
            BoundDirection::Above => value > threshold,
            BoundDirection::Below => value < threshold,
        };
        if breached {
            found.push(Breach { bound: *bound, value, threshold });
        }
    }
    found
}

For each limit the plugin declares, in order:

A limit compares its own payload field, not the reading’s main value. That is why air quality can limit co2 and indoor_aqi separately.

Example. Building b1 has {"maxTemp":28}. Room r2 has {"maxTemp":24,"minTemp":16}.

ReadingBounds usedResult
r1, 25 °Cbuilding: max 28none
r2, 25 °Croom: max 24, min 16high, threshold 24
r2, 14 °Croomlow, threshold 16
r1, air quality co2 1200, indoor_aqi 160, building has maxCo2 1000, maxAqi 150building2 alerts: one on co2, one on indoor_aqi
// A failed lookup names the alert by its ids rather than dropping it.
let names = self.buildings.names_of(&reading.building_id).await.unwrap_or_else(|error| {
    log::error!("building name lookup failed: {error}");
    None
});
let building_name = names.as_ref().map_or(&reading.building_id, |n| &n.name);
let room_name = names.as_ref().and_then(|n| n.rooms.get(&reading.room_id)).unwrap_or(&reading.room_id);
for breach in found {
    let alert = AlertPayload { metric, field, building_id, room_id, building_name, room_name,
        value: breach.value, label: breach.bound.label, unit, direction, threshold: breach.threshold, ts_ms: self.clock.now_ms() };
    self.alerts.publish_breach(&alert).await;
}

Each breach becomes its own alert, with the building and room names from the registration (cached 5 min). A failed name lookup, or an unregistered room, falls back to the ids: an alert with an ugly name beats no alert. The alert is queued to Kafka and not awaited.

Step 4: the live tick

let now_ms = self.clock.now_ms();
let events: Vec<TelemetryEvent> = readings.iter().map(|reading| TelemetryEvent::from_reading(reading, now_ms)).collect();
self.fanout.publish_telemetry(&events).await;
Ok(events.len())

All the readings become one tick on telemetry:raw. The answer is 202 {"accepted":true,"readings":40}.

sequenceDiagram
    autonumber
    participant G as Device or simulator
    participant V as ingest_auth.rs<br/>verify_signature
    participant H as controllers::ingest
    participant I as kernel::Ingest::accept
    participant PG as PostgreSQL
    participant K as Kafka
    participant R as Redis
    G->>V: POST /telemetry/ingest, x-signature, body
    V->>V: read body, at most 1 MiB
    V->>V: peek buildingId, DeviceKeys::accepts
    V->>H: same bytes
    H->>H: readings must be an array
    H->>I: accept(buildingId, readings)
    I->>I: 500 or fewer? building non-empty? list non-empty?
    loop each reading
        I->>I: reading_from: type, plugin, buildingId, plugin.validate
    end
    par
        I->>PG: readings.insert, one statement, on conflict do nothing
    and
        I->>PG: thresholds.resolve, through the 60 s cache
    end
    loop each reading with bounds
        I->>I: raise_breach: breaches(plugin.bounds, bounds, payload)
        I->>K: publish_breach, queued, not awaited
    end
    I->>R: fanout.publish_telemetry, one tick on telemetry:raw
    H-->>G: 202 {"accepted":true,"readings":40}

What can fail

FailureResult
Insert fails500. Nothing published, no alert. telemetry_ingest_persist_failures_total + 1
Bounds lookup failsLogged as threshold evaluation failed. Readings stored and published, no alerts for that batch
Kafka refuses an alertLogged, counted telemetry_alerts_published_total{outcome="error"}. The batch is still 202
Redis publish failsLogged, counted. The batch is still 202
Same batch sent twiceThe second insert stores nothing (unique key building_id, metric, room_id, ts). Alerts and the tick are sent again

The tick shape and its path to the browser: Telemetry Schema.


Wi-Fi ticks: kernel/observations.rs

ap-collector polls the routers about every 10 seconds and sends one snapshot per building.

pub async fn accept(&self, body: &[u8]) -> Result<(), DomainError> {
    let tick: WifiObservations = serde_json::from_slice(body)
        .map_err(|error| DomainError::Validation(format!("wifi observations: {error}.")))?;
    let problems = tick.problems();
    if !problems.is_empty() {
        return Err(DomainError::Validation(problems.join(" ")));
    }
    let sensors = self.sensors.by_building(&tick.building_id).await?;
    let readings = room_readings(&tick, &sensors);
    if !readings.is_empty() {
        self.ingest.accept(&tick.building_id, &readings).await?;
    }
    Ok(self.events.publish_observations(&tick.building_id, body).await?)
}
  1. Parse the tick. Unknown fields are refused: the bytes travel on to Kafka as received, and a stray mac field would travel with them.
  2. problems() (from the shared crate) checks the tick against the contract. Any problem: 400, nothing published.
  3. Load the building’s sensors and turn the tick into counts per room.
  4. Store those counts through the normal Ingest path.
  5. Publish the received bytes (never a re-serialised copy) to wifi-observations, keyed by the building.

Counting phones per room

fn room_readings(tick: &WifiObservations, sensors: &[Sensor]) -> Vec<Value> {
    let rooms: HashMap<&str, &str> = sensors.iter()
        .filter(|sensor| sensor.sensor_type == ROUTER)
        .filter_map(|sensor| Some((sensor.sensor_id.as_str(), sensor.room_id.as_deref()?)))
        .collect();

    let mut loudest: HashMap<&str, (i32, &str)> = HashMap::new();
    for phone in &tick.connected {
        let Some(&room) = rooms.get(phone.router.as_str()) else { continue; };
        let heard = (phone.signal, room);
        loudest.entry(phone.token.as_str())
            .and_modify(|best| *best = (*best).max(heard))
            .or_insert(heard);
    }

    let mut counts: BTreeMap<&str, u64> = tick.answered.iter()
        .filter_map(|router| Some((*rooms.get(router.as_str())?, 0)))
        .collect();
    for (_, room) in loudest.into_values() {
        *counts.entry(room).or_default() += 1;
    }
    // … one totalDeviceCount reading per room; ratioDeviceCount too when devicesPerPerson is set
}
  1. Map each router that sits in a room: router id to room id. Outdoor routers and other sensors are left out.
  2. For each connected phone on a mapped router, keep its loudest hearing: the largest (signal, room) pair. On equal signal the room id that sorts last wins.
  3. Start every room whose router answered at 0.
  4. Add 1 to the room of each phone’s loudest hearing. A phone counts once.
  5. One totalDeviceCount reading per room, stamped with the tick time.
  6. When the tick has devicesPerPerson, also ratioDeviceCount = count ÷ factor, rounded up. The source says why: rounding down 0.4 of a person would report an occupied room as empty.

Example. Routers r1 in lobby, r2 in hall. Both answered. Factor 2.5.

PhoneHeard byCounts in
Ar1 −70, r2 −60hall (louder)
Br1 −55lobby
Cr1 −60lobby
Dout (outdoor router)nowhere

Readings: totalDeviceCount lobby 2, hall 1. ratioDeviceCount lobby ⌈2 ÷ 2.5⌉ = 1, hall ⌈1 ÷ 2.5⌉ = 1.

RuleWhy
Unknown fields refusedThe bytes go to Kafka as received. A stray mac would travel with them
An answered empty room is 0. An unanswered room sends nothingA failed router did not measure its room. A 0 would look like a fresh empty room
Counts stored before the Kafka publishA count that cannot be stored is 500, and the tick is not published
Kafka refuses500. The collector drops that tick
Topic wifi-observations, retention 10 minA tick is stale in seconds, and it is pseudonymous
No state between ticksA phone on a room boundary can switch rooms each tick
sequenceDiagram
    autonumber
    participant A as ap-collector
    participant V as verify_signature
    participant O as kernel::Observations::accept
    participant S as SensorStore
    participant I as Ingest::accept
    participant K as Kafka
    A->>V: POST /telemetry/observations, signed body
    V->>O: same bytes
    O->>O: from_slice into WifiObservations, unknown fields refused
    O->>O: tick.problems(): token, signal, limits
    O->>S: by_building(buildingId)
    O->>O: room_readings(tick, sensors)
    O->>I: accept(buildingId, counts), same path as a batch
    O->>K: publish_observations: the received bytes, key buildingId
    O-->>A: 202 {"accepted":true}

Room counts back from digital-twin

digital-twin places each phone more precisely (see Digital Twin) and sends room counts back. telemetry stores them as a separate metric.

let readings: Vec<Value> = occupancy.rooms.iter().map(|(room, count)| json!({
    "type": ESTIMATED_OCCUPANCY,
    "roomId": room,
    "timestamp": occupancy.tick,
    ESTIMATED_OCCUPANCY: count,
})).collect();
let mut stored = 0;
for batch in readings.chunks(MAX_BATCH_READINGS) {
    stored += self.ingest.accept(&occupancy.building_id, batch).await?;
}

Each room becomes an estimatedOccupancy reading and goes through the same Ingest, in chunks of 500. It is never added to totalDeviceCount: the two place a phone differently.

sequenceDiagram
    autonumber
    participant K as Kafka estimated-occupancy
    participant C as occupancy_consumer.rs
    participant O as kernel::Occupancy
    participant I as Ingest::accept
    K->>C: EstimatedOccupancy for one building
    C->>O: rooms to estimatedOccupancy readings
    loop each chunk of 500
        O->>I: accept(buildingId, chunk)
    end
    C->>K: commit the offset

Building registration: kernel/registration.rs

A building must exist here before its sensors can be set up. Two entries lead to one use case, Registration::register.

pub async fn register(&self, building_id: &str, payload: &Value) -> Result<(), DomainError> {
    let request = RegistrationRequest::deserialize(payload)
        .map_err(|error| DomainError::Validation(format!("registration payload: {error}.")))?;

    if request.name.trim().is_empty() {
        return Err(DomainError::Validation("name: must be a non-empty string.".to_owned()));
    }
    // Postgres refuses NUL in text, and a store error is retried, so it would block the topic.
    if std::iter::once(&request.name)
        .chain(request.rooms.iter().flat_map(|room| [&room.id, &room.name]))
        .any(|text| text.contains('\0'))
    {
        return Err(DomainError::Validation("name, rooms: must not contain a NUL character.".to_owned()));
    }

    self.buildings.upsert(&RegisteredBuilding { id: building_id.to_owned(), name: request.name,
        rooms: request.rooms.into_iter().map(|room| Room { id: room.id, name: room.name }).collect() }).await?;

    if let Some(max_temperature) = request.max_temperature {
        let mut patch = Bounds::new();
        patch.insert("maxTemp".to_owned(), max_temperature.into());
        self.thresholds.upsert(building_id, None, "temperature", &patch).await?;
    }
    Ok(())
}
  1. Parse the request. Bad shape: 400-class error.
  2. The name must not be blank.
  3. No NUL character in any name or id. PostgreSQL refuses a NUL in text, a store error would be retried for ever, and that would block the whole topic. So anything the database would refuse is checked first.
  4. Save the building and its rooms (an upsert, so registering twice gives the same rows).
  5. If maxTemperature came with it, save it as the building’s maxTemp. (Twin does not send it over Kafka today, but the field stays: it is optional, not dead.)

The Kafka path

pub async fn register_from_event(&self, building_id: &str, payload: &Value) -> anyhow::Result<bool> {
    // A store outage is not the request's fault: answering `failed` would delete the twin.
    let outcome = match self.register(building_id, payload).await {
        Err(DomainError::Internal(error)) => return Err(error),
        outcome => outcome.map_err(|error| error.to_string()),
    };
    let registered = outcome.is_ok();
    self.events.publish_completed(building_id, outcome).await?;
    Ok(registered)
}
sequenceDiagram
    autonumber
    participant Tw as digital-twin
    participant K as Kafka
    participant C as kafka_consumer.rs
    participant G as Registration
    participant PG as PostgreSQL
    Tw->>K: building-registration-requested {buildingId, name, rooms}
    K->>C: message, group building-registrations
    C->>C: handle: JSON? has buildingId?
    C->>G: register_from_event(buildingId, payload)
    G->>G: register: parse, name not blank, no NUL character
    G->>PG: upsert building and rooms
    G->>K: building-registration-completed, ready or failed
    C->>K: commit the offset
    Tw->>G: PUT /thresholds/buildings/b1 {name, rooms, maxTemperature}
    G->>PG: same upsert, plus building maxTemp when given
flowchart TD
    M["request message"] --> J{"JSON with buildingId?"}
    J -- no --> D["log, commit, no answer"]
    J -- yes --> R{"register"}
    R -- "valid, stored" --> Ready["answer ready, commit"]
    R -- "invalid: no name, NUL, bad maxTemperature" --> Failed["answer failed with the reason, commit"]
    R -- "store error" --> Retry["no answer, no commit<br/>reconnect after 5 s, read again"]
    Ready --> P{"answer published?"}
    Failed --> P
    P -- no --> Retry

Limits

RouteWritesRule
PATCH …/buildings/{id}Building bounds of one metricMerged into what is stored (`bounds
PATCH …/buildings/{id}/rooms{roomId: bounds} for many roomsEvery entry checked before any is written
PATCH …/rooms/{roomId}One room

Each patch is checked by Thresholds::checked: every key must be a bound of that metric’s plugin, and every value a number. {"maxTmp": 28} on temperature is 400 maxTmp: not a bound of temperature. Because the check asks the plugin which keys exist, a new metric needs no change here.

CachedThresholds sits in front of the store:


Sensors batch

sequenceDiagram
    autonumber
    participant C as client
    participant H as controllers::apply_sensors
    participant S as kernel::Sensors::apply
    participant PG as PostgreSQL
    C->>H: POST /sensors/buildings/b1 {create, update, delete}
    H->>H: edit check
    H->>S: apply(b1, body)
    S->>S: arrays? 200 items or fewer? refs and ids present?
    S->>PG: rooms of b1 from the registration, sensors of b1
    loop each create, update, delete
        S->>S: check, collect ItemError
    end
    S->>PG: one transaction
    H-->>C: 200 {"created":[{"ref":"new-1","sensorId":"6f1c…"}]}

A batch of changes follows the same idea as ingest: check everything, then write everything, or nothing.

ItemChecks
createref unique in the batch. name not blank. sensorType a device kind from /devices. roomId null or a room of the building
updatesensorId a sensor of this building, once per batch. name not blank when given. roomId as above when given
deletesensorId a sensor of this building, once per batch

Simulation

sequenceDiagram
    autonumber
    participant C as client
    participant Si as kernel::Simulation::start
    participant S1 as sensor-simulator
    participant S2 as aq-simulator
    participant S3 as ap-simulator
    C->>Si: PUT /simulation/buildings/b1
    Si->>Si: any simulator configured? building registered?
    Si->>Si: plan: each sensor to the simulator claiming its kind
    par join_all, 5 s timeout each
        Si->>S1: POST /control/start {buildingId, sensors:[temperature, peopleCount]}
    and
        Si->>S2: POST /control/start {buildingId, sensors:[airQuality]}
    and
        Si->>S3: POST /control/start {buildingId, sensors:[]}
    end
    Si-->>C: 204, or 502 when any failed

telemetry does not simulate anything itself. It tells each simulator what to simulate, at start, and only then.

The body never carries coordinates: positions are digital-twin’s. Body shape: Telemetry Schema, fixture-only shapes.


Device commands

sequenceDiagram
    autonumber
    participant C as client
    participant H as controllers::execute_action
    participant A as kernel::Actions::execute
    participant D as HttpDispatch
    participant PG as PostgreSQL
    participant Dev as Device
    C->>H: POST /executeAction {actionData:{metric, buildingId, sensorId, action, arguments}}
    H->>H: fields non-empty, edit check
    H->>A: execute(command)
    A->>A: plugin knows the action? arguments match its parameters?
    A->>D: dispatch(command)
    D->>PG: driver, endpoint of (buildingId, sensorId)
    D->>D: binding = bindings.json[driver][action]
    D->>D: rename arguments, check URL is http or https
    D->>Dev: POST endpoint + path, 5 s timeout
    Dev-->>D: 2xx
    H-->>C: 200 {"accepted":true,"action":"setTarget"}

The kernel checks the command against the plugin’s action list (the same FieldSpec checks as for readings). The adapter then turns it into a real call for that kind of device, using bindings.json.

Example. setTarget with {"target": 21} on a sensor with driver tp-simulator and endpoint http://sensor-simulator:3000. bindings.json maps target to value, so the device gets POST http://sensor-simulator:3000/control/receive with {"value": 21}.

CaseAnswer
Unknown metric or action404
Missing or wrong argument400
Sensor has no driver or endpoint, or no binding404 no binding for sensor …
Endpoint not http(s)404, logged
Device answers non-2xx, or is unreachable502

The sensor is found by (buildingId, sensorId), never by room, so an outdoor or moved sensor can still be reached. The endpoint must be http or https: the service posts to it, so anything else would let a stored value point it somewhere strange. More: Sensor Actions.


Reads

RouteAnswer
latestOne reading. roomId missing is 400. No data is 404
entireBuilding{"data": [reading per room]}
dashboard{"data": [{"timestamp","value"}]}
connected-devices{buildingId, totalDeviceCount, timestamp}. Never reported is 404

A reading is answered flat: building, roomId, timestamp, value, plus every plugin field at the top level. The same names as the live tick, so a page that loads history and then live data handles one shape.

dashboard query:

timeRangeWindowBucket
missing or 1Dlast 24 h1 hour
1Wlast 7 days1 day
1Mlast 30 days1 day
customstart to end (end defaults to now). No start is 4001 day

aggMode: sum, min, max, anything else avg.

Example, connected devices. At 10:00:00 lobby reports 12, hall 5. office last reported at 09:40 (its router was removed). Answer: {"totalDeviceCount": 17, "timestamp": <10:00:00>}. office is left out: only rooms at the newest report count. The sum is done here, never by a client, and only a count adds up: never route an average through it.


What it stores

TableHoldsNotes
readingsbuilding_id, room_id, metric, ts, value, payloadHypertable. Unique on building_id, metric, room_id, ts. Compressed after 7 days, dropped after 14
readings_hourlyHourly summariesContinuous aggregate, refreshed every hour over the last 3 days
thresholdsbuilding_id, room_id, metric, boundsroom_id null = the building row
sensorssensor_id (key), building_id, room_id, name, sensor_type, driver, endpointroom_id null = outdoors
buildings, building_roomsRegistration: names and rooms. device_key_epoch

Kafka topics it creates: building-registration-requested, building-registration-completed, alerts, wifi-observations (10 min).

The unique key building_id, metric, room_id, ts is what makes a repeated batch harmless to the archive: the second insert simply stores nothing.


Configuration

VariableDefaultWhat it is
DATABASE_URLrequiredPostgreSQL with TimescaleDB
TELEMETRY_DEVICE_MASTER_KEYrequired, 32 bytes or moreDerives every device key
TELEMETRY_INGEST_SECRETunsetA shared key that signs for any building. Local and simulators only
REDIS_URLredis://redis:6379Tick fan-out
KAFKA_BROKERSkafka:9092Registration, alerts, Wi-Fi, room counts
DIGITAL_TWIN_URLhttp://digital-twin:3000Ownership lookup
SIMULATORSunsetJSON {name: {url, kinds}}. Compose gives temperature, peopleCount to sensor-simulator, airQuality to aq-simulator, router to ap-simulator
PORT3000HTTP port

Metrics

NameWhat it counts
telemetry_ingest_total{metric, outcome}Readings accepted, batches invalid, unsigned, bad_signature
telemetry_ingest_persist_failures_total, …_persist_duration_secondsInsert failures and time
telemetry_threshold_breaches_total{metric, direction}Breaches found
telemetry_alerts_published_total{topic, outcome}Kafka answers to alerts
telemetry_published_total{outcome}Redis publishes
telemetry_registration_total{outcome}, …_duration_secondsready, failed, unacknowledged
telemetry_occupancy_ticks_totalRoom-count messages
telemetry_query_duration_seconds{query}Read times
telemetry_authz_denials_total{action}read, edit denials
telemetry_http_requests_total, …_error_requests_total, …_duration_secondsEvery route
telemetry_db_pool_connectionsPool use

Tests

just test telemetry               # unit, kernel against fakes, no database
just test telemetry-integration   # tests/*.rs against throwaway TimescaleDB, Redis, Kafka
WhereWhat it checks
src/kernel/*.rsEvery rule above against in-memory fakes: all-or-nothing batch, the 500 cap, every breach its own alert, a failed name lookup still alerts by id, Wi-Fi counting cases, sensors batch errors, simulation plans, actions
src/adapters/ingest_auth.rsGolden HMAC vectors shared with Go. The collector request fixture. A key per building. Rotation revokes. The shared key only when set. Short keys refused
src/adapters/driven/*.rsThreshold cache: one query per tick, invalidation per building and metric, capacity. Twin lookup cache and timeout. Dispatch renames. Simulator bodies match the fixture
tests/api.rsHTTP: /contracts equals fixtures/metric-contract.json. Every ingest-batch.json case and rejection
tests/persistence.rsThe PostgreSQL adapters
tests/fanout.rsA tick reaches telemetry:raw
tests/alerts.rsA breach reaches alerts keyed by building and room, even before anyone subscribes
tests/registration.rsReady answer. A repeat converges. Failed with reason. A bad message does not stop the consumer. A request sent before start is answered. An unpublished answer is read again
tests/cedar_conformance.rsCedar decisions match the shared golden cases
tests/architecture.rsThe layering rules in the table above

Gaps