Nothing in the container can be changed; /tmp is an emptyDir
envFrom
telemetry-secret, common-config
Private values, then shared ones
probes
httpGet /health on 3000: ready after 10 s, alive every 20 s
No command inside the image: distroless has no shell
resources
asks 100m CPU, 128 Mi; capped at 500m, 256 Mi
annotations
prometheus.io/scrape: "true"
Prometheus finds the pod
Services that need more than one dependency get one init container per dependency: dashboard waits for dashboard-db, redis, telemetry and digital-twin, in that order.
just k8s setup: the first time
sequenceDiagram
autonumber
participant U as Developer
participant J as just
participant K as k3d
participant C as Cluster
participant I as istioctl
U->>J: just k8s setup
J->>K: cluster create ambitwin, ports 80 and 443, api 6550, no traefik
K-->>C: one k3s node in Docker
J->>C: kubectl apply Gateway API CRDs v1.2.0
J->>I: istioctl install profile ambient, k3d CNI paths
I-->>C: istiod, ztunnel, istio-cni
J->>C: apply namespace.yml
J->>J: scripts/k8s/k8s-secrets.sh reads the env file
J->>C: one Secret per service
J->>C: just k8s apply, in order
Step
Detail
2
-p "80:80@loadbalancer" -p "443:443@loadbalancer" --api-port 6550 --k3s-arg "--disable=traefik@server:0". Traefik is off because Istio is the entry point
4
The CRDs teach the cluster what a Gateway and an HTTPRoute are
5
--set values.cni.cniConfDir=/var/lib/rancher/k3s/agent/etc/cni/net.d --set values.cni.cniBinDir=/bin. istioctl is pinned to 1.31.3 in .mise.toml
7
The namespace label enrolls every pod in ambient mode at creation: no restart needed
8–9
See the secrets table below
just k8s apply: the order
#
File
Why here
1
namespace.yml
Everything lives in it
2
configmaps/
Deployments read it at start
3
stateful/
Databases first: the init containers wait for them
4
deployments/
Creates chat’s ServiceAccount, which step 8 names
5
istio-peer-authentication.yml
mTLS on
6
istio-gateway.yml
Creates the edge workload the next two select
7
istio-request-authentication.yml
8
istio-agent-authz.yml
9
monitoring/
Every step is kubectl apply: running it again changes only what differs.
Secrets
scripts/k8s/k8s-secrets.sh reads .env, checks every required key before writing anything, then writes each secret with kubectl create secret … --dry-run=client -o yaml | kubectl apply -f -, so it can run again.
Secret
Holds
telemetry-secret
DATABASE_URL, TELEMETRY_DEVICE_MASTER_KEY, and TELEMETRY_INGEST_SECRET only if set
Only when GHCR_USERNAME and GHCR_TOKEN are set; GHCR images are public
A request through Istio
sequenceDiagram
autonumber
participant B as Browser
participant GW as ambitwin-gateway-istio
participant G as claims-gateway
participant ZT as ztunnel
participant T as digital-twin
B->>GW: GET /twin/buildings, cookie authentication_token
GW->>G: fetch the public keys through its own /gateway route, cached
GW->>GW: RequestAuthentication gateway-jwt: issuer cv-gateway, signature
GW->>GW: write the payload into x-gateway-claims
GW->>GW: AuthorizationPolicy require-gateway-jwt: principal cv-gateway/ada?
GW->>GW: HTTPRoute digital-twin-route: rewrite /twin to /
GW->>ZT: GET /buildings
ZT->>T: over mTLS, STRICT
T->>T: decode x-gateway-claims, run Cedar
T-->>B: 200
Step
What happens
1
The token is read from Authorization: Bearer … or the authentication_token cookie
2
jwksUri points at the gateway’s own/gateway/* route, so the key fetch also goes through the mesh
3–4
outputPayloadToHeader: x-gateway-claims; forwardOriginalToken: false, so the JWT itself goes no further
5
Gated paths need a request principal from issuer cv-gateway
6
ReplacePrefixMatch: /, same as Caddy’s handle_path
7–8
ztunnel on each node wraps pod-to-pod traffic in mTLS; plain text is refused
Which paths need a token
flowchart TD
R["request path"] --> A{"/twin, /telemetry, /notification, /chat,<br/>/socket.io, /tenancy, /dashboard?"}
A -- no --> P1["allowed: /gateway, /agent, /grafana, /"]
A -- yes --> E{"exactly /telemetry/ingest, /collector<br/>or /observations?"}
E -- yes --> P2["allowed, no principal: HMAC checked in telemetry"]
E -- no --> J{"valid cv-gateway JWT?"}
J -- yes --> P3["allowed, with x-gateway-claims"]
J -- no --> D["denied by the policy"]
agent
Rule
Where
Effect
Remove x-gateway-claims
agent-route, RequestHeaderModifier
The edge never passes claims to agent
agent-callers
istio-agent-authz.yml
Only ServiceAccounts ambitwin-gateway-istio and chat may open a connection
Why: agent trusts the claims header without a signature. Only chat may hand it one; ztunnel checks the caller’s identity, which a pod cannot fake.
Parity with Compose
Compose
Cluster
Edge
Caddy forward_auth to claims-gateway /verify
Istio RequestAuthentication at the gateway
Extra hop per request
Yes
No: the gateway has the keys
Header services read
x-gateway-claims
x-gateway-claims
Prefix removal
handle_path
URLRewriteReplacePrefixMatch
Ungated telemetry paths
3 exact handle blocks in Caddyfile
3 exact paths in require-gateway-jwt
/agent claims
header_up -X-Gateway-Claims
RequestHeaderModifier remove
SSE on /chat
flush_interval -1
timeouts.request: 120s
Between services
Plain HTTP on the Compose network
mTLS through ztunnel
The three exact telemetry paths are written twice, by hand. backend/acceptance/edge/ tests the Caddy side; nothing tests the Istio side.
Checks
Where
Tool
Checks
just k8s validate
kubectl apply --dry-run=client
Each file parses and is accepted, without touching the cluster
ci-k8s
kubeconform 0.6.7, -strict, Kubernetes 1.32.0
Every manifest against its schema; Istio and Gateway API kinds through the datree CRD catalog
ci-k8s
pluto 5.24.0
No API version that Kubernetes removed
ci-k8s
kube-score 1.20.0
Good practice. Switched off: network policy, disruption budget, user and group id, read-only root, image tag, pull policy, identical probes
Prints 127.0.0.1 ambitwin.local for the hosts file
just k8s delete
Deletes the namespace and every volume in it. No undo
Traps
Wrong CNI path, broken node. k3d’s containerd looks for CNI plugins in /bin. Another path makes every pod fail with failed to find plugin "istio-cni", and each failed retry leaks an address until no IP addresses available in range set. Only a new cluster fixes that.
Git Bash rewrites /bin. Run straight from Git Bash, istioctl --set values.cni.cniBinDir=/bin gets a Windows path. just runs recipes through PowerShell on Windows, which does not.
No Keycloak in the cluster.claims-gateway looks for it at host.k3d.internal:8090, so Keycloak must run from Compose, or sign-in fails at start.
A missing token gives a different status. Caddy answers 401 from /verify; Istio’s policy denies a gated path with no principal by its own refusal, not by claims-gateway.
An expired cookie may block everything. Istio rejects a request whose token is present but invalid, on any path. The cookie has no Max-Age, so an expired JWT stays in the browser. Not tested here.
The cookie is Secure. Browsers keep it over plain http only for localhost, and the cluster is reached at http://ambitwin.local. Not tested here.
Image tags drift.grafana is 13.2.4 in Compose and 13.2.1 in the cluster; redis 8.10.2 and 8.10.1.
dashboard-secret names contractsdb.dashboard ignores the database in its URI and opens ambitwin anyway.