Communication & Data Flow

CrowdVision uses three communication styles, chosen deliberately per relationship:

• Synchronous, user-driven — HTTP through the gateway, and the handful of service-to-service calls made on a user’s behalf. The caller’s identity travels with the request.
• Synchronous, control-plane — HTTP calls between backend-only services that have no end user attached (signup provisioning). Authenticated by a shared secret instead of a user identity, because there is no user identity to carry.
• Asynchronous, fire-and-forget (publish/subscribe) — over Redis, for the high-frequency telemetry and alert flows, where producers must not be coupled to, or blocked by, consumers, and no message needs to survive a restart.
• Asynchronous, durable (event log) — over Kafka, for the one cross-service outcome that must eventually be known on both sides even if either service restarts mid-flight: a new building’s registration.

This page maps all three, then walks five end-to-end flows.

Identity & Trust: How a Caller’s Identity Travels

There is no central session store and no per-request call back to an identity authority. Two distinct mechanisms carry trust, and which one applies depends on whether a human is behind the request:

Both patterns exist for the same reason: nobody re-implements JWT verification or invents a second signing scheme per pair of services. Every mesh-claims consumer and every HMAC consumer reuses one shared convention, respectively — see Identity & Tenancy Architecture and Service Mesh Architecture for the full verification mechanics.

A third category — no authentication at all — is used only where forwarding an identity would be meaningless or actively wrong: device-facing ingestion (a sensor has no user token to present) and a small number of internal bootstrap calls that write nothing sensitive (flagged individually in the table below).

Synchronous Communication

Browser ↔ Gateway

All external traffic enters through the single gateway (Caddy in compose, Istio Gateway API in k8s), which routes by URL prefix. See the routing table in the Overview — not repeated here.

Service-to-Service Calls

These calls are intentionally few. The design keeps the request path short and avoids synchronous chains on the hot path; anything high-frequency goes over the broker instead.

Device Ingestion

POST /ingest on telemetry — no user token (a sensor gateway has none to present), so it is exempt from the gateway’s auth gate and carries its own credential instead: X-Signature, an HMAC-SHA256 over the raw body keyed by TELEMETRY_INGEST_SECRET — the same X-Signature scheme as the control-plane hops above, but a separate key, since this one leaves the mesh and lives on customer premises. Unsigned or mis-signed requests are 401 before any work. The endpoint validates, persists and evaluates thresholds before answering 202 Accepted, so a 202 means the reading is durable and queryable.

Asynchronous Communication

Redis is the publish/subscribe backbone for everything high-frequency and fire-and-forget; Kafka carries the two flows that must survive a restart. Three channels, one stream and one topic carry the event flow:

Publishers never know who (if anyone) is listening, and the broker absorbs bursts, so a slow or restarting consumer cannot back-pressure a sensor.

Kafka

Unlike Redis, a Kafka topic is a durable log: a consumer that’s down when a message is produced still sees it once it comes back, which is why this is the transport for the one flow where digital-twin and telemetry must eventually agree even if either one restarts mid-registration — not for anything high-frequency.

Both payloads and both topic names are defined once, in schemas/twin-schema, and each service re-exports them from its own adapters/topics.rs — see Twin Schema.

Consumer groups are named after what they consume, not after the service consuming them, so a group name stays right even if the service is renamed.

Publishing is not blocking on either side: digital-twin’s worker returns as soon as the first message reaches the broker, and telemetry’s consumer processes it on its own schedule. Both topics carry no x-gateway-claims — this is internal transport between two services already inside the trust perimeter, not a client-facing hop. See Digital Twin Service for the full sequence and the idempotency argument (every write either side makes is an upsert, so redelivery converges rather than duplicating).

Browser-Facing Real-Time Channels

Socket.IO (socket)

Web Push (notification)

The client fetches the server’s public VAPID key (GET /notification/public-key), calls the browser’s pushManager.subscribe(), and registers the resulting subscription (POST /notification/subscribe). When an alert fires, notification sends a push payload ({title, message, icon}) via the web-push package directly to the browser’s push service — this path does not go through Redis or socket, and works even if the user has no tab open. A service worker (frontend/public/service-worker.js) handles the push event to show an OS-level notification and notificationclick to focus/open the app.

End-to-End Flows

Path 1: Live Telemetry to the Browser

sequenceDiagram
    participant Sensor as Sensor source
    participant SS as telemetry
    participant R as Redis
    participant CS as dashboard
    participant WS as socket
    participant UI as Vue client

    Sensor->>SS: POST /ingest (X-Signature HMAC)
    Note over SS: verify signature, validate, persist, normalize
    SS-->>Sensor: 202 Accepted
    SS->>R: PUBLISH telemetry:raw
    R-->>CS: telemetry:raw event
    Note over CS: read the event's own buildingId → route
    CS->>R: PUBLISH telemetry:filtered:{buildingId}
    R-->>WS: telemetry:filtered:{buildingId} event
    WS->>UI: emit "telemetry" to room building:{id}

dashboard is the routing stage: it republishes each reading to its own building’s channel (an O(1) lookup on the event’s buildingId), so a reading reaches exactly one building’s feed and never leaks into another’s. It does not filter by metric type — every metric a building produces is forwarded, and the dashboard’s column preferences only decide what the table displays, client-side.

Path 2: Threshold Alert to the User

sequenceDiagram
    participant SS as telemetry
    participant R as Redis
    participant K as Kafka
    participant NS as notification
    participant TW as digital-twin
    participant WS as socket
    participant Push as Browser push service
    participant U as User

    Note over SS: background: reading crosses a configured bound
    SS->>K: produce alerts (key buildingId:roomId)
    K-->>NS: alerts record
    Note over NS: throttle repeats (per-room cooldown)
    NS->>TW: GET /domain/{buildingName}
    TW-->>NS: domain name(s)
    Note over NS: select opted-in recipients
    NS->>R: PUBLISH notifications {domainName}
    NS->>Push: web-push send (VAPID)
    R-->>WS: notifications event
    WS->>U: emit "notification" to room domain:{name}
    Push-->>U: service worker "push" → OS notification

Unlike Path 1, this is a two-hop async chain, not one publish: telemetry never calls notification directly — the threshold trigger and the resolved, recipient-scoped notification are two distinct Redis events, with notification sitting in between doing the one synchronous call it needs (resolving domains from digital-twin) and fanning out over both delivery paths — in-app (broker + socket) and Web Push (direct, works with no tab open).

Path 3: Assistant Tool-Calling

sequenceDiagram
    participant U as Browser
    participant Chat as chat
    participant Agent as agent
    participant Twin as digital-twin
    participant Sensor as telemetry

    U->>Chat: POST /chat/... (x-gateway-claims via gateway)
    Chat->>Agent: POST /ask {question}, x-gateway-claims forwarded
    Note over Agent: LLM decides which tool(s) to call
    Agent->>Twin: GET /buildings/{domain}, x-gateway-claims forwarded
    Twin-->>Agent: building data
    Agent->>Sensor: GET /{metric}/latest, x-gateway-claims forwarded
    Sensor-->>Agent: reading
    Agent-->>Chat: answer
    Chat-->>U: chat response

Every tool call agent makes carries the original user’s x-gateway-claims header, forwarded across two hops (chat → agent → digital-twin/telemetry) — so digital-twin and telemetry apply the same Cedar authorization to an assistant-driven read as they would to a direct API call from that user. chat persists the conversation in its own MongoDB independently of this flow.

Path 4: Login & Identity Resolution

sequenceDiagram
    participant U as User
    participant CG as claims-gateway
    participant KC as Keycloak
    participant TS as tenancy

    U->>CG: POST /login (credentials or IdP redirect)
    CG->>KC: verify credentials / verify token
    KC-->>CG: verified identity + Organization claim
    CG->>TS: GET /internal/memberships (HMAC)
    alt no membership yet
        CG->>TS: POST /internal/provision (HMAC)
        TS-->>CG: membership created from the Organization claim
    end
    CG-->>U: signed internal token (StandardClaims)

This is the only point where tenancy is consulted for identity — once per login, never per request. Every subsequent request in a session is authorized locally, from the token/header, with zero further network hops for auth. See Identity & Tenancy Architecture for how the membership itself is derived, and Service Mesh Architecture for how the resulting token becomes the x-gateway-claims header at the mesh edge.

Path 5: Organization Signup & Provisioning

sequenceDiagram
    participant C as New customer
    participant RS as registry
    participant P as provisioner
    participant TS as tenancy

    C->>RS: POST /organizations (no auth — no credentials exist yet)
    RS-->>C: 201, status=provisioning
    loop every 15s
        P->>RS: GET /internal/organizations/pending (HMAC)
    end
    P->>TS: POST /internal/domains (HMAC, idempotent)
    TS-->>P: 201 or 409 (both = success)
    P->>RS: POST /internal/organizations/{id}/status (HMAC)

This is the control-plane path: no end user is authenticated mid-flow, so every internal hop uses the HMAC X-Signature scheme rather than a forwarded identity header. See Overview for what registry/provisioner/tenancy each own.

Why Four Styles