CI/CD Pipeline

CrowdVision uses a fully automated CI/CD pipeline on GitHub Actions. Every pull request to master is gated by a suite of checks — builds, tests, linting, security audits, Docker and Kubernetes validation, and a guard that refuses to let any unreviewed path merge unchecked. Merging to master keeps a standing release pull request up to date; merging that release PR cuts a versioned release and pushes Docker images to GHCR. This page documents the workflows; the commit format and release mechanics that drive them are covered in Commits & Release Please.

Workflow naming

All workflow files live in .github/workflows/ and use a three-prefix convention that mirrors their purpose.

Prefix	Category	When it runs
`ci-*`	Continuous integration checks	On every pull request to `master`
`cd-*`	Continuous delivery	On every push to `master`
`maint-*`	Maintenance	Weekly schedule or manual dispatch

The CI gate

Every pull request must pass CI Gate / CI Passed — the single aggregated check that covers all the fast CI. The gate (ci-gate.yml) detects what changed, fans out to the relevant checks in parallel, and reports one result. The only other check worth requiring in branch protection is CodeQL (the Static security analysis section below), which runs as a separate workflow because it needs wider permissions than the gate’s read-only token.

graph LR
    PR[Pull Request] --> CHANGES[changes<br/>path filter]
    CHANGES --> SVC[Service legs<br/>chat · agent · frontend<br/>contracts · notification<br/>telemetry · socket · twin<br/>claims-gateway · tenancy<br/>registry · provisioner]
    CHANGES --> DOCKER[docker<br/>hadolint + build]
    CHANGES --> K8S[k8s<br/>kubeconform · pluto · kube-score]
    CHANGES --> LINT[workflows-lint<br/>actionlint]
    CHANGES --> INFRA[infra<br/>scripts + tooling]
    SVC --> PASSED[CI Passed]
    DOCKER --> PASSED
    K8S --> PASSED
    LINT --> PASSED
    INFRA --> PASSED
    PASSED --> MERGE[merge allowed]

How the gate decides

The changes job runs dorny/paths-filter once and emits a boolean per area. Every downstream leg is guarded by an if: on those booleans, so a leg runs only when a path it owns actually changed. Service legs and the cross-cutting checks are called as reusable workflows (uses: ./.github/workflows/ci-*.yml); the gate itself stays a thin orchestrator — detect, fan out, aggregate.

Unaffected legs are skipped, not failed

A leg whose paths did not change resolves as skipped, which the aggregator counts as a pass. This is what keeps the pipeline fast — a one-line change to one service does not rebuild the world.

The final ci-passed job depends on every leg and runs with if: always(), so it reports even when a leg was skipped. Its verdict logic is deliberately strict:

Whitelist, not blacklist. Each leg’s result must be success or skipped. Anything else — failure, cancelled, or any future result value — fails the gate. A blacklist that only checked for failure would let an unknown state slip through green.
The fan-out root must have succeeded. If the changes job itself failed or was cancelled, every leg was skipped without ever being evaluated — a green gate there would be a false pass. So ci-passed hard-fails unless changes ended in success.

Service CI workflows

Each service has a dedicated reusable workflow that the gate calls when that service’s directory (or its own workflow file) changes. Each installs dependencies, lints, builds, runs tests, and audits its dependencies.

Workflow	Service	Language / test runner
`ci-chat.yml`	`backend/chat`	Node.js 24 / Jest
`ci-twin.yml`	`backend/digital-twin`	Rust stable / cargo llvm-cov
`ci-telemetry.yml`	`backend/telemetry`	Rust stable / cargo llvm-cov
`ci-socket.yml`	`backend/socket`	Rust stable / cargo llvm-cov
`ci-notification.yml`	`backend/notification`	Rust stable / cargo llvm-cov
`ci-contracts.yml`	`backend/dashboard`	Rust stable / cargo llvm-cov
`ci-frontend.yml`	`frontend`	Node.js 24 / Vitest
`ci-agent.yml`	`backend/agent`	Python 3.12 / pytest
`ci-claims-gateway.yml`	`backend/claims-gateway`	Go / `go test`
`ci-tenancy.yml`	`backend/tenancy`	Go / `go test`
`ci-registry.yml`	`backend/registry`	Go / `go test`
`ci-provisioner.yml`	`backend/provisioner`	Go / `go test`

The frontend splits into a quality job (lint, type-check, audit), a test job that shards Vitest across three parallel runners, and a coverage job that runs the suite unsharded to produce a single coverage report. The six Rust services (chat, dashboard, socket, digital-twin, notification, telemetry) all run cargo fmt --check and cargo clippy --all-targets -- -D warnings before cargo llvm-cov, then a rustsec/audit-check. Which infrastructure containers a job starts is driven by the needs_mongo / needs_redis / needs_postgres / needs_kafka flags in .github/services.json, not hard-coded per service: chat and digital-twin get a real mongo:8.3.9, socket a real redis:8.10.1-alpine, and so on, since they exercise their infrastructure directly rather than mocking it. The agent runs ruff check, ruff format --check, and pyright in a quality job, with pytest (unit + integration, with --cov) in a separate tests job. The four Go services each run a gofmt -l check and go vet (lint), go build, then go test -tags=integration — setup-go installs the single Go pin from .mise.toml, the same version every go.mod declares and every Go image builds with; none of them run an audit step yet (see the note below). Coverage reporting is detailed in the Code coverage section below.

The dependency audit is split into two steps

Every Node service runs the audit twice. npm audit --audit-level=high --omit=dev is blocking — a fixable high/critical vulnerability in a production dependency fails the build. npm audit --audit-level=high (the full tree, including dev deps) runs straight after with continue-on-error: true, so dev-only advisories are surfaced in the log without blocking a merge. The split keeps the gate honest about what ships to users while still reporting everything.

The four Go services don’t have an audit step yet

Unlike every other language here, there’s no govulncheck-equivalent step in the four Go CI workflows — .moon/tasks/go.yml doesn’t define an audit task either, so just setup audit silently skips them too (see The Toolchain. Build/lint/test coverage is in place; dependency vulnerability scanning for Go is still open.

Code coverage

Every service CI workflow now measures code coverage as part of its test run and reports it on each pull request — collected with each stack’s native tooling, with no external coverage service involved.

Stack	Command	Report format
Node.js services + frontend	`jest --coverage` · `vitest run --coverage`	Istanbul `coverage-summary.json`
Rust services	`cargo llvm-cov --json`	LLVM JSON
`agent`	`pytest --cov`	Cobertura `coverage.xml`
Go services	`go test -tags=integration -coverpkg=./...`	Cobertura `coverage.xml`

The Node and Rust services write a coverage summary into the GitHub job summary (rendered from the report with jq); the agent surfaces its figures directly in the pytest log via --cov-report=term-missing.

cd-coverage.yml aggregates these on every push to master, for changed services only, and force-pushes badge.json, summary.json and coverage-counts.json to the orphan coverage-data branch — the source the Coverage page reads live.

The baseline is pruned against the registry

A partial run merges the previous coverage-counts.json forward so unchanged services keep their numbers. Left unfiltered, that also carries a renamed or deleted service forward indefinitely: its old key never ages out, and the weighted badge counts it alongside whatever replaced it. merge_baseline() drops any key absent from .github/services.json and logs what it pruned. This is not hypothetical — contracts, registry-service, sensor and tenancy-service had accumulated there, understating the badge by four points.

`gofmt` is a gate, `go vet` never was

go vet does not look at formatting, so until the check above existed nothing in CI did — drift sat in the tree unnoticed. gofmt -l prints offenders and still exits 0, so the step loops over its output and sets the exit status itself; a bare run: gofmt -l . would pass forever. The same check runs first in the shared lint task (.moon/tasks/go.yml), so just lint fails on exactly what CI fails on.

Both Go flags exist to stop the number lying

A plain go test -coverprofile ./... understates every Go module with a support package or a tagged suite, and it did so by a lot.

-coverpkg=./... attributes coverage across package boundaries. Without it each package is measured only against its own _test.go files, so internal/storefake — exercised by every service and api test — reported 0%.
-tags=integration compiles the //go:build integration suites (internal/store in registry and tenancy, internal/events in tenancy). They are real testcontainers tests against a real Postgres; without the tag they compile nowhere, run nowhere, and report 0%.

Applying both moved registry from 26.6% to 55.6% and tenancy from 43.2% to 71.5% without adding a single test. The same tag now runs in tpl-go-ci.yml, so a broken integration suite fails the PR rather than silently never running. Locally: just test registry-integration / just test tenancy-integration.

The coverage summary runs always

The step that publishes the coverage table is guarded by if: always(), so it runs even when a test fails — by default a failed step skips every step after it, yet the coverage of a red run is often exactly when you want to read it. This does not weaken the gate: always() governs only that one reporting step, so a failing test still fails the job. Coverage is report-only for now — no minimum threshold is enforced, so it surfaces the numbers on every run but cannot itself block a merge.

Simulator CI

The two device simulators ship as GHCR images but are not application services, so they get a dedicated ci-simulators.yml — one job each, gated on simulators/.

Job	Runtime	Checks
`sensor-simulator`	Node.js 24 / TypeScript	ESLint (shared config) + `tsc` build. No test suite exists yet, so there is no test step — a bare Jest with zero tests would fail.
`aq-simulator`	Python 3.12	`py_compile` parses every module (smoke.py is not executed — it performs network I/O), then `ruff check` runs with default rules, which catch real defects such as undefined names and unused imports.

Both were previously ungated despite shipping images until this workflow gave them a real bucket.

Cross-cutting checks

These legs validate properties that span the codebase rather than a single service. They are gated by their own path filters and join the same ci-passed aggregator.

Leg	What it validates
`ci-docker.yml`	The `docker` leg. For each changed service that ships an image: hadolint on its `Dockerfile`, then a `push: false` Docker build (a PR-time build check — `cd-registry` does the real publish on `master`). The gha layer cache is scoped per image so services reuse their own layers without evicting each other. Its `if:` starts with `!cancelled()` and only stops on a failed `needs` job: without that, any skipped leg in `needs` skips the build silently and `ci-passed` still goes green.
`ci-k8s.yml`	The `k8s` leg. Validates `k8s/` with kubeconform (schema, against Kubernetes 1.32.0, `-strict`, plus the CRDs-catalog schema location so Istio/Gateway API kinds — `Gateway`, `HTTPRoute`, `PeerAuthentication`, `RequestAuthentication`, `AuthorizationPolicy` — are actually schema-checked rather than erroring with “could not find schema”), pluto (deprecated/removed API versions), and kube-score (best-practice scoring, with a curated set of ignored tests).
`workflows-lint`	A job defined inline in `ci-gate.yml` that validates the workflow files themselves whenever anything under the .`github/` tree changes. actionlint (digest-pinned container) catches malformed expressions and invalid action inputs; shellcheck is bundled but disabled (`-shellcheck=`) until its `run:`-block findings are triaged. zizmor (pinned version, run `--offline --min-severity medium`) then audits for security smells actionlint misses: template injection in `run:` blocks, credential persistence (it is why every read-only checkout sets `persist-credentials: false`), over-broad token permissions, and dangerous triggers.
`ci-infra.yml`	The `infra` leg. Cheap, real validation for repo scripts whenever `scripts/` or `tooling/` change: `node --check` parses every tracked .`mjs`/.`js`, and `bash -n` checks every shell script. Keeps routine script edits green automatically.
`ci-docs.yml`	The `docs` leg. Builds the published site with the same `just docs build` recipe `cd-documentation` uses, whenever `documentation/`, `api/`, the docs recipe, the link guard or the Quarkdown pin changes. Without it the first sign of a broken docs change is a failed deploy on `master`, since nothing else compiles the guide.
`ci-acceptance.yml`	The `acceptance` leg. Runs the cross-service suite in `backend/acceptance/` against a real composed stack. The only leg where a request crosses the edge: `backend/acceptance/edge/` sends traffic through a real Caddy container mounting the repo-root `Caddyfile`, so a change to the ingest ungate or to a gated route fails here rather than in production. Its filter covers `backend/acceptance/`, the `Caddyfile`, `k8s/istio-request-authentication.yml`, and `telemetry`/`socket`/`dashboard` — telemetry is in the list because the near-miss this leg exists for was a change to its route table, not to the edge config. The slowest leg in the gate, hence the narrow filter.
`ci-dependency-review.yml`	The `dependency-review` leg. On any PR that changes a dependency manifest or lockfile, it fails if the PR introduces a dependency carrying a known high/critical advisory (`fail-on-severity: high`). It complements the per-service audit steps: those see the merged lockfile state, whereas this diffs exactly what the PR adds, catching a vulnerable package at the moment it is introduced. Requires the repository’s dependency graph (on by default for public repos).

Every third-party action is pinned to a commit SHA

Each uses: across every workflow references a full 40-character commit SHA (with the human-readable version kept as a trailing comment, e.g. # v6), never a moving tag like @v4. A tag can be silently repointed at malicious code by whoever controls the action; a commit SHA cannot. The repository setting Settings → Actions → “Require actions to be pinned to a full-length commit SHA” enforces this, so no future workflow can reintroduce an unpinned action. Local reusable workflows (uses: ./.github/workflows/…) are exempt — they live in this repo. Bumping a version means resolving the new tag to its SHA (git ls-remote) and updating both the SHA and the comment.

CD workflows

cd-release

Triggered on every push to master. Runs googleapis/release-please-action, which reads commits since the last tag and either opens/updates a release pull request (CHANGELOG.md + every version manifest) or, if its own release PR was just merged, tags the repository and publishes a GitHub Release. A concurrency guard prevents duplicate runs. The full mechanics are in Commits & Release Please.

cd-documentation

Triggered on push to master when documentation/, api/, just/docs.just, scripts/check-docs-links.sh or .mise.toml changes. Runs just docs build — the same recipe as the docs PR leg, so the deploy cannot drift from what the pull request validated — and publishes landing-page/ to GitHub Pages.

The Quarkdown version is pinned, and the pin is read, not repeated

setup-quarkdown installs the latest release when given no version. The site’s theme styles Quarkdown’s generated DOM directly, so an upstream release can rename a class and quietly break the layout while the build stays green. QUARKDOWN_VERSION in .mise.toml is the single source: the workflows read it with sed before installing, and just docs build refuses to run when the locally installed CLI disagrees with it. Quarkdown cannot be a mise [tools] entry — its release asset is a zip containing a bundled JRE, and the ubi/github backends keep only the launcher script, which then cannot find its own classes.

cd-registry

Triggered on push to master when source changes under backend/**, simulators/**, frontend/**, or the workflow file itself. It publishes Docker images to GHCR tagged both :latest and :<git-sha>, for all fourteen images — the ten TypeScript/Rust/Python services and simulators, plus claims-gateway, tenancy, registry, and provisioner.

The publish is gated by a vulnerability scan that runs before anything reaches the registry. Each image is built into the runner’s local daemon (push: false, load: true), scanned by Trivy, and pushed only if the scan passes — so a vulnerable image is never published. The push step rebuilds from the still-warm gha cache, so it costs no recompilation.

Build → scan → push, not push → scan

Scanning after publishing would mean a flagged image is already live by the time the job fails. Ordering the scan as a gate before the push is the whole point. aquasecurity/trivy-action is pinned to a commit digest (not the moving @master) to remove that supply-chain risk.

Selective builds: only changed services are pushed

The workflow does not rebuild every image on every push. A changes job detects which services changed and emits a dynamic build matrix; a build-and-push job (needs: changes) then builds only the selected images, and is skipped entirely when nothing matches. Two cases force a full rebuild: a change to cd-registry.yml itself, and a manual workflow_dispatch with service: all (or a comma-separated list to target specific images).

The Rust base image

The six Rust Dockerfiles build FROM rust-base, which exists on no public registry. Without a supplied context buildx looks up docker.io/library/rust-base and fails with insufficient_scope — a missing image, not a token permission.
A rust-base job builds docker/rust-base/ and pushes ghcr.io/nickghignatti/crowdvision-rust-base:<rust pin>-<Dockerfile sha256, 12 chars>. If the tag is already there, it skips the build.
build-and-push passes it as build-contexts: rust-base=docker-image://… on both of its build steps.
The PR docker leg does not need it: Rust images there use .github/docker/Dockerfile.ci with the prebuilt binary.

Scan scope

The Trivy scan is deliberately scoped to OS packages only (vuln-type: os). Application dependencies are not re-scanned here because they are already gated pre-merge by the per-service audits. The division of labour is: the service CI workflows own language dependencies at PR time; cd-registry Trivy owns the base-OS layer at push time — the part the dependency audits cannot see.

To keep the OS scan green, every production image upgrades its base-OS packages at build time (apk upgrade on Alpine images, apt-get upgrade on the Debian-based contracts, agent, and aq-simulator images). The Debian upgrade lines carry # hadolint ignore=DL3005. Any residual fixable CVE that cannot be patched in-build is waived in the repo-root .trivyignore, where each entry must carry a justifying comment.

Static security analysis (CodeQL)

ci-codeql.yml runs GitHub’s SAST for javascript-typescript, python, rust, and go with the security-extended query suite. It runs automatically on every pull request to master, on every push to master (to refresh the Security tab’s default-branch baseline), and on a weekly schedule (Monday 03:00 UTC) to catch CVEs newly disclosed against unchanged code. Manual workflow_dispatch is retained for on-demand re-scans, and a concurrency guard cancels a scan superseded by a newer commit.

All four languages use build-mode: none — CodeQL extracts from source without a compile. For javascript-typescript and python that is the natural mode; for rust (dashboard and digital-twin) and go (claims-gateway, tenancy, registry, provisioner) it means no toolchain setup is needed for CodeQL’s own purposes, and each of the several independent crates/modules is found and extracted directly from source without a root Cargo.toml/go.work tying them together.

It runs as a standalone workflow rather than a leg of the gate, on purpose: CodeQL needs security-events: write to upload results, whereas the gate’s token is deliberately read-only, and a reusable workflow cannot hold more permission than its caller. Keeping CodeQL separate preserves the gate’s least-privilege token. Add CodeQL as a second required status check in branch protection (alongside CI Gate / CI Passed) to block merges on new high-severity findings.

Maintenance

maint-cleanup runs every Sunday at 02:00 UTC (or on manual dispatch). It removes old package versions from GHCR, keeping a minimum of three versions per image and protecting the :latest tag, so the registry does not grow unbounded.

Dependency updates

Dependency upgrades are made manually (there is no Dependabot). Bump the version in the relevant manifest, then verify and regenerate locks locally before opening a pull request:

just setup deps-check   # every lockfile in sync (npm ci / uv sync --locked / cargo check)
just setup audit        # no known vulnerabilities (npm / uv / cargo audit)

Both checks also run inside the per-service CI workflows on every pull request, so a drifted or vulnerable dependency blocks the merge regardless.

Docker image registry

All images are hosted at ghcr.io/nickghignatti/ and tagged both :latest and :<git-sha>, published automatically by cd-registry on every push to master that touches them. The Kubernetes deployment manifests reference :latest.

Service	GHCR image
`chat`	`ghcr.io/nickghignatti/crowdvision-chat`
`digital-twin`	`ghcr.io/nickghignatti/crowdvision-twin`
`telemetry`	`ghcr.io/nickghignatti/crowdvision-telemetry`
`notification`	`ghcr.io/nickghignatti/crowdvision-notification`
`socket`	`ghcr.io/nickghignatti/crowdvision-socket`
`dashboard`	`ghcr.io/nickghignatti/crowdvision-dashboard`
`agent`	`ghcr.io/nickghignatti/crowdvision-agent`
`frontend`	`ghcr.io/nickghignatti/crowdvision-frontend`
`sensor-simulator`	`ghcr.io/nickghignatti/crowdvision-simulator`
`aq-simulator`	`ghcr.io/nickghignatti/crowdvision-aq-simulator`
`claims-gateway`	`ghcr.io/nickghignatti/crowdvision-claims-gateway`
`provisioner`	`ghcr.io/nickghignatti/crowdvision-provisioner`
`registry`	`ghcr.io/nickghignatti/crowdvision-registry`
`tenancy`	`ghcr.io/nickghignatti/crowdvision-tenancy`

Reproducing the gates locally

Run the same checks CI runs before pushing, to avoid a slow round-trip through a failed pipeline:

just test all            # all unit tests (moon-cached); or just test affected
just lint affected   # ESLint / Ruff on changed projects
just setup deps-check      # lockfile sync
just setup audit           # vulnerability audit

Per-service tests are available as just test chat, just test twin, just test telemetry, just test notification, just test socket, just test frontend, and just test agent (no per-service recipe for dashboard or the four Go control-plane services — use just test affected/all, which cover every registered project via moon run :test). Full backend integration tests run via just test integration (it composes the full stack — see Docker Compose Orchestration.

On this page