/

Ambitwin · source-available, not open source · © 2026 Nicolò Ghignatti

Kubernetes & Istio

Keywords on this page
WordMeaningExample
ClusterMachines Kubernetes runs containers onambitwin, made by k3d
k3dRuns a small Kubernetes (k3s) inside Docker containersOne server node
NamespaceA named box that groups a cluster’s objectsambitwin
PodOne or more containers that start and stop togethertelemetry-7f9c…
DeploymentKeeps N copies of a stateless pod runningk8s/deployments/telemetry.yml
StatefulSetLike a Deployment, but each pod keeps its own diskk8s/stateful/telemetry-db.yml
Init containerRuns to the end before the main container startswait-for-db
Secret, ConfigMapValues handed to pods as environment variables; a Secret for private onestelemetry-secret, common-config
IstioA service mesh: it carries, encrypts and checks traffic between podsAmbient mode
Ambient modeIstio without a proxy in each pod: one ztunnel per node does the workistio.io/dataplane-mode: ambient
ztunnelIstio’s per-node proxy: mutual TLS and identity checks between podsA DaemonSet
WaypointAn optional per-namespace proxy for HTTP rules inside the meshNot used here
Gateway APIKubernetes’ standard way to describe an entry point and its routesGateway, HTTPRoute
mTLSBoth sides of a connection prove who they are with certificatesPeerAuthentication STRICT

The cluster setup mirrors Docker Compose: same images, same routes, same claims header. Only the edge changes, from Caddy to Istio.


What is in k8s/

PathHolds
namespace.ymlNamespace ambitwin, labelled istio.io/dataplane-mode: ambient
configmaps/common.ymlcommon-config: PORT, REDIS_URL, KAFKA_BROKERS, service URLs, PUBLIC_URL=http://ambitwin.local
stateful/10 StatefulSets: twin-db, notification-db, dashboard-db, chat-db (MongoDB), telemetry-db (TimescaleDB), agent-db (pgvector), registry-db, tenancy-db (Postgres), redis, kafka
deployments/12 Deployments and their Services: the 11 backend services and frontend
monitoring/prometheus (with its ServiceAccount, ClusterRole and scrape config), grafana
istio-peer-authentication.ymlSTRICT mTLS for the whole namespace
istio-gateway.ymlThe Gateway and one HTTPRoute per path
istio-request-authentication.ymlJWT check at the edge, and the policy saying which paths need it
istio-agent-authz.ymlOnly the edge and chat may call agent

Not in the cluster: Keycloak, the Langfuse stack, the simulators, ap-collector, the docs ingester.

A Deployment, piece by piece

k8s/deployments/telemetry.yml:

PartValueWhy
replicas1Every service runs one copy
initContainerswait-for-db: until nc -z telemetry-db 5432The service crashes if its database is not up yet
imageghcr.io/nickghignatti/ambitwin-telemetry:latest, IfNotPresentUses an image imported by just k8s load without asking GHCR
securityContextallowPrivilegeEscalation: false, readOnlyRootFilesystem: trueNothing in the container can be changed; /tmp is an emptyDir
envFromtelemetry-secret, common-configPrivate values, then shared ones
probeshttpGet /health on 3000: ready after 10 s, alive every 20 sNo command inside the image: distroless has no shell
resourcesasks 100m CPU, 128 Mi; capped at 500m, 256 Mi
annotationsprometheus.io/scrape: "true"Prometheus finds the pod

Services that need more than one dependency get one init container per dependency: dashboard waits for dashboard-db, redis, telemetry and digital-twin, in that order.


just k8s setup: the first time

sequenceDiagram
    autonumber
    participant U as Developer
    participant J as just
    participant K as k3d
    participant C as Cluster
    participant I as istioctl
    U->>J: just k8s setup
    J->>K: cluster create ambitwin, ports 80 and 443, api 6550, no traefik
    K-->>C: one k3s node in Docker
    J->>C: kubectl apply Gateway API CRDs v1.2.0
    J->>I: istioctl install profile ambient, k3d CNI paths
    I-->>C: istiod, ztunnel, istio-cni
    J->>C: apply namespace.yml
    J->>J: scripts/k8s/k8s-secrets.sh reads the env file
    J->>C: one Secret per service
    J->>C: just k8s apply, in order
StepDetail
2-p "80:80@loadbalancer" -p "443:443@loadbalancer" --api-port 6550 --k3s-arg "--disable=traefik@server:0". Traefik is off because Istio is the entry point
4The CRDs teach the cluster what a Gateway and an HTTPRoute are
5--set values.cni.cniConfDir=/var/lib/rancher/k3s/agent/etc/cni/net.d --set values.cni.cniBinDir=/bin. istioctl is pinned to 1.31.3 in .mise.toml
7The namespace label enrolls every pod in ambient mode at creation: no restart needed
8–9See the secrets table below

just k8s apply: the order

#FileWhy here
1namespace.ymlEverything lives in it
2configmaps/Deployments read it at start
3stateful/Databases first: the init containers wait for them
4deployments/Creates chat’s ServiceAccount, which step 8 names
5istio-peer-authentication.ymlmTLS on
6istio-gateway.ymlCreates the edge workload the next two select
7istio-request-authentication.yml
8istio-agent-authz.yml
9monitoring/

Every step is kubectl apply: running it again changes only what differs.

Secrets

scripts/k8s/k8s-secrets.sh reads .env, checks every required key before writing anything, then writes each secret with kubectl create secret … --dry-run=client -o yaml | kubectl apply -f -, so it can run again.

SecretHolds
telemetry-secretDATABASE_URL, TELEMETRY_DEVICE_MASTER_KEY, and TELEMETRY_INGEST_SECRET only if set
notification-secretMONGO_URI, VAPID_PUBLIC_KEY, VAPID_PRIVATE_KEY
registry-secret, tenancy-secretDATABASE_URL, INTERNAL_SIGNING_SECRET
provisioner-secretINTERNAL_SIGNING_SECRET
claims-gateway-secretINTERNAL_SIGNING_SECRET, REGISTRATION_CLIENT_SECRET
claims-gateway-keysecrets/gateway-dev-key.pem, from disk
agent-secretGOOGLE_API_KEY, DEEPSEEK_API_KEY, POSTGRES_URL
chat-secret, digital-twin-secret, dashboard-secretMONGO_URI
*-db-secretThe database’s own user and password
grafana-secretAdmin user and password, default admin / ambitwin
ghcr-pull-secretOnly when GHCR_USERNAME and GHCR_TOKEN are set; GHCR images are public

A request through Istio

sequenceDiagram
    autonumber
    participant B as Browser
    participant GW as ambitwin-gateway-istio
    participant G as claims-gateway
    participant ZT as ztunnel
    participant T as digital-twin
    B->>GW: GET /twin/buildings, cookie authentication_token
    GW->>G: fetch the public keys through its own /gateway route, cached
    GW->>GW: RequestAuthentication gateway-jwt: issuer cv-gateway, signature
    GW->>GW: write the payload into x-gateway-claims
    GW->>GW: AuthorizationPolicy require-gateway-jwt: principal cv-gateway/ada?
    GW->>GW: HTTPRoute digital-twin-route: rewrite /twin to /
    GW->>ZT: GET /buildings
    ZT->>T: over mTLS, STRICT
    T->>T: decode x-gateway-claims, run Cedar
    T-->>B: 200
StepWhat happens
1The token is read from Authorization: Bearer … or the authentication_token cookie
2jwksUri points at the gateway’s own /gateway/* route, so the key fetch also goes through the mesh
3–4outputPayloadToHeader: x-gateway-claims; forwardOriginalToken: false, so the JWT itself goes no further
5Gated paths need a request principal from issuer cv-gateway
6ReplacePrefixMatch: /, same as Caddy’s handle_path
7–8ztunnel on each node wraps pod-to-pod traffic in mTLS; plain text is refused

Which paths need a token

flowchart TD
    R["request path"] --> A{"/twin, /telemetry, /notification, /chat,<br/>/socket.io, /tenancy, /dashboard?"}
    A -- no --> P1["allowed: /gateway, /agent, /grafana, /"]
    A -- yes --> E{"exactly /telemetry/ingest, /collector<br/>or /observations?"}
    E -- yes --> P2["allowed, no principal: HMAC checked in telemetry"]
    E -- no --> J{"valid cv-gateway JWT?"}
    J -- yes --> P3["allowed, with x-gateway-claims"]
    J -- no --> D["denied by the policy"]

agent

RuleWhereEffect
Remove x-gateway-claimsagent-route, RequestHeaderModifierThe edge never passes claims to agent
agent-callersistio-agent-authz.ymlOnly ServiceAccounts ambitwin-gateway-istio and chat may open a connection

Why: agent trusts the claims header without a signature. Only chat may hand it one; ztunnel checks the caller’s identity, which a pod cannot fake.


Parity with Compose

ComposeCluster
EdgeCaddy forward_auth to claims-gateway /verifyIstio RequestAuthentication at the gateway
Extra hop per requestYesNo: the gateway has the keys
Header services readx-gateway-claimsx-gateway-claims
Prefix removalhandle_pathURLRewrite ReplacePrefixMatch
Ungated telemetry paths3 exact handle blocks in Caddyfile3 exact paths in require-gateway-jwt
/agent claimsheader_up -X-Gateway-ClaimsRequestHeaderModifier remove
SSE on /chatflush_interval -1timeouts.request: 120s
Between servicesPlain HTTP on the Compose networkmTLS through ztunnel

The three exact telemetry paths are written twice, by hand. backend/acceptance/edge/ tests the Caddy side; nothing tests the Istio side.


Checks

WhereToolChecks
just k8s validatekubectl apply --dry-run=clientEach file parses and is accepted, without touching the cluster
ci-k8skubeconform 0.6.7, -strict, Kubernetes 1.32.0Every manifest against its schema; Istio and Gateway API kinds through the datree CRD catalog
ci-k8spluto 5.24.0No API version that Kubernetes removed
ci-k8skube-score 1.20.0Good practice. Switched off: network policy, disruption budget, user and group id, read-only root, image tag, pull policy, identical probes

Day to day

RecipeDoes
just k8s buildBuilds 12 images tagged ghcr.io/nickghignatti/ambitwin-*:latest
just k8s loadImports them into k3d, no registry
just k8s push, build-pushPushes them to GHCR
just k8s statuskubectl get all -n ambitwin
just k8s logs <svc>Follows logs by label app=<svc>
just k8s restart <svc>, restart-allRolling restart: picks up a new :latest
just k8s rollout <svc>Waits for a rollout to finish
just k8s stop, startPauses and resumes the cluster; data kept
just k8s hostsPrints 127.0.0.1 ambitwin.local for the hosts file
just k8s deleteDeletes the namespace and every volume in it. No undo

Traps

Related: Docker Images, Docker Compose, CI.