Docker Compose Orchestration

Locally, the entire platform runs under Docker Compose. There is no single hand-maintained compose file that lists every service; instead, each service owns its own compose fragment, and a small Node orchestrator assembles them into one runtime file on demand. This page explains that mechanism end to end, so a new contributor can predict exactly what just stack dev does.

The shape of the configuration

Compose configuration is split across three kinds of file.

The root file holds the pieces every run needs regardless of which services are included:

• gateway — the Caddy reverse proxy on port 80 (see Architecture Overview. Its depends_on entries are marked required: false, so the gateway still starts when a service is excluded and simply retries the connection when that service appears.
• redis — the publish/subscribe broker.
• keycloak and keycloak-db — the hosted identity provider every login flows through, and its private Postgres. keycloak imports keycloak/realm-export.json on boot and exposes its own admin console directly on host port 8090 (not behind the gateway) — see Identity & Tenancy Architecture.
• prometheus and grafana — metrics and dashboards.
• The langfuse-*, redis-langfuse, clickhouse, and minio stack — the agent’s trace backend (see Setting Up the Environment for the generated Langfuse credentials and UI URL).

The orchestrator

scripts/compose/compose-run.mjs (invoked by the Justfile, never directly) turns the scattered fragments into one runnable configuration.

flowchart TD
    J["just stack dev / start / down / …"] --> R["compose-run.mjs"]
    R --> W["Walk backend/ and simulators/<br/>apply exclude patterns"]
    W --> G["Write compose.runtime.yml<br/>(an include: list)"]
    G --> D["docker compose -f compose.runtime.yml &lt;up|down|build&gt;"]

Why a generated include: file, not multiple -f flags

Passing several files with repeated -f flags makes Compose resolve every relative path (such as build.context: .) against the first file’s directory, which breaks per-service Dockerfile lookups. Compose’s include: directive instead resolves paths relative to each included file, so build.context: . inside backend/twin-service/docker-compose.yml correctly points at that service’s directory. The orchestrator therefore writes a small compose.runtime.yml whose body is an include: list:

# Auto-generated by scripts/compose/compose-run.mjs — do not commit.
include:
  - path: docker-compose.yml
  - path:
      - backend/twin-service/docker-compose.yml
      - backend/twin-service/docker-compose.dev.yml
  - path: backend/sensor-service/docker-compose.yml
  # … one entry per included service …

A single-path entry becomes one sub-project; a multi-path entry merges a service’s base file with its dev override into one sub-project (two separate include entries cannot define the same service — that raises a conflict).

How the include list is built

1. Always include the root docker-compose.yml.
2. Walk backend/ (skipping tests/) and simulators/, adding each folder’s docker-compose.yml and — in development modes — its docker-compose.dev.yml if present.
3. Add frontend unless excluded.
4. For the integration mode, also include backend/tests/docker-compose.integration.yml.

Excluding services

In dev, dev-build, and start, you can omit services by substring of their folder name. This is how you run a lighter stack when you do not need the agent or the simulators:

just stack dev exclude="agent"            # skip agent-service, agent-db, agent-ingester
just stack dev exclude="agent simulator"  # also skip aq-simulator, sensor-simulator

The down, integration, and build modes always operate on the full set and ignore exclude patterns.

The modes

The first argument to the orchestrator selects the docker compose invocation. Each maps to a just recipe.

The orchestrator runs docker compose with COMPOSE_BAKE=true, which delegates builds to BuildKit Bake for faster, parallel image builds.

Environment and the dev loop

Every stack-starting recipe (dev, dev-build, start) depends on the env recipe, which regenerates \.env (VAPID keys, the local-dev-only eval JWT secret, Langfuse secrets) before Compose runs — see Setting Up Your Environment.

Typical local loop:

just stack dev                 # full stack with hot-reload
just stack dev exclude="agent" # lighter stack without the AI agent
just stack logs twin-service   # follow one service's logs
just stack down                # tear it all down

just stack logs [service] follows logs against the generated compose.runtime.yml, so it targets exactly the services the last run started.

Related operations

For the production-grade orchestration (StatefulSets, ingress, secrets), see Kubernetes Configuration; for the runtime topology and gateway routing, see the Architecture Overview.