Agent Service

The Python agent-service is CrowdVision’s tool-calling assistant. It answers questions by combining a permission-filtered documentation knowledge base with live building and room data from twin-service.

It is built on FastAPI — an async Python web framework chosen because the agent spends most of its time waiting on network I/O (the language-model provider, the database, the twin-service), and an async stack lets one worker handle many in-flight requests instead of blocking a thread per call. For local commands and troubleshooting, see the service README.md. For writing a new tool, see Adding an Agent Tool.

Module Map

graph TD
    ASK["routes/ask.py"] --> AUTH["auth.py\nrequire_user"]
    ASK --> LOOP["agent/loop.py\nAgent.answer, tool-calling loop"]
    LOOP --> LLM["agent/llm/base.py\nLLMClient port"]
    LOOP --> REG["agent/tools/registry.py\nToolRegistry"]
    REG --> BASETOOL["agent/tools/base.py\nTool[ArgsT] port"]
    BASETOOL --> SEARCH["tools/search_docs.py"]
    BASETOOL --> TWIN["tools/twin.py"]
    BASETOOL --> SENSOR["tools/sensor.py"]
    BASETOOL --> DOWN["tools/downstream.py"]
    TWIN --> ACCESS["access.py\ncan_access_domain, accessible_domains"]
    ACCESS --> CEDAR["cedar_authz.py"]
    DOWN --> AUTHHDR["downstream.py\nauth_headers — forwards x-gateway-claims"]
    LOOP --> CITE["citations.py\nextract_citations, strip_hallucinated"]

How It Works

A request to POST /ask is not a single model call. The agent runs a tool-calling loop: the language model is given the question plus the JSON schemas of the available tools, and on each turn it may either answer or ask to call one or more tools. We run the requested tools, feed their results back, and let the model decide again. This “agentic” pattern is preferred over stuffing everything into one prompt because the model itself decides what data it needs and can chain lookups (for example, resolve a building name to an ID, then fetch that building’s rooms).

flowchart TD
    A[question + AuthUser] --> B[bootstrap system + user messages]
    B --> C[LLM chat with tool schemas]
    C -->|tool calls| D[validate args, run tools, append results]
    D --> C
    C -->|final text| E[validate citations, strip invented markers]
    E --> F[response]

The loop is bounded by MAX_TOOL_HOPS so a confused model cannot spin forever; hitting the limit returns the IDK marker with decision=tool_loop_exhausted. Tool errors are handed back to the model as ordinary tool-result messages rather than raised, so the model gets a chance to recover (retry with different arguments, or give up cleanly).

stream=true runs the same loop and then emits the finished answer as a single Server-Sent Events token event followed by done. It is not per-token streaming yet — the event shape exists so the client contract is stable when true streaming is added later.

Scope

The assistant is deliberately not a general-purpose chatbot. Its system prompt restricts it to CrowdVision topics — buildings and rooms, live occupancy and sensor state, and how the platform works — and instructs it to decline anything else (general knowledge, math, coding, translation, writing, other products, personal advice) in one short sentence, without calling a tool.

The motivation is product trust and cost: a building-occupancy assistant that also writes poems or answers trivia invites misuse, leaks the fact that it is “just an LLM”, and spends the shared provider budget on off-topic work. Two refusal behaviors are intentionally kept distinct:

• Out of scope → a brief scoped refusal (“I can only help with CrowdVision…”), no tool call.
• In scope but unanswerable → the prompt instructs the model to return the exact IDK marker after tools come up empty. The service also returns that marker when the tool-loop limit is exhausted; it does not enforce that every model-emitted IDK was preceded by a tool call.

Greetings are treated as in scope and answered directly.

Capabilities

The model can choose from eight read-only tools:

The agent answers with validated documentation citations, a tool-call trace, token usage, estimated/provider-reported cost, an idk flag, and a final decision.

list_sensors: The Device Inventory

The reading tools tell you the values; list_sensors tells you what hardware exists. It answers “which sensors does room X have — does it even measure temperature?” and lets the agent distinguish “no reading right now” from “no sensor installed”, instead of guessing from an empty readings list.

Given a building_id and an optional room_id, the tool:

1. Fetches the building from twin-service (forwarding the caller’s x-gateway-claims header) and rejects inaccessible or unknown buildings without revealing whether they exist; an unknown room_id is rejected before any sensor request is made.
2. Calls sensor-service GET /sensors/buildings/{buildingId} — or …/rooms/{roomId} when a room is given — again as the asking user. Sensor-service reads its Mongo-backed device registry (buildingId, roomId, sensorId, sensorType).
3. Filters the response to rooms that actually belong to the authorized building (downstream data is untrusted) and projects each device to sensor_id, sensor_type, room_id, and the room name resolved from the twin.

Two services are involved: twin-service for authorization and room names, sensor-service for the device registry itself. The tool is read-only and returns an inventory, never measurements — the system prompt steers the model to get_latest_sensor_data / get_sensor_history for those.

Knowledge Base And Retrieval

In Docker Compose, the one-shot agent-ingester waits for the service to become healthy, then ingests \.qd, \.md, and \.markdown files from documentation/user and documentation/developer.

just agent ingest

Ingestion is idempotent by global content hash: identical content is skipped. It does not replace or delete an older document when a source file changes, so a clean rebuild requires clearing the agent tables before re-ingesting. Each source is split by a Markdown-aware chunker that keeps headings, fenced code, and tables intact, because a chunk that splits a code block or a table mid-row retrieves badly.

search_docs runs hybrid retrieval:

1. Embed the query into a vector.
2. Run a pgvector cosine-similarity search (semantic match).
3. Run a PostgreSQL full-text search via tsvector and websearch_to_tsquery (exact-term match).
4. Merge the two ranked lists with Reciprocal Rank Fusion.
5. Pass the merged candidates through the configured reranker (noop is the only current implementation).
6. Return the top TOP_K_FINAL chunks.

The reason for running both searches is that they fail in opposite ways: vector search understands meaning but can miss a rare exact token (an error code, a config key), while full-text search nails exact tokens but is blind to paraphrase. Reciprocal Rank Fusion combines them without the usual problem of incompatible score scales — instead of trying to add a cosine similarity to a ts_rank, it scores each document by 1 / (k + rank) summed across the lists, so only the position in each list matters and documents that both methods rank highly rise to the top.

Both searches filter chunks against the caller’s JWT roles and domains at the SQL boundary, so a user never even retrieves a chunk they are not allowed to see:

chunk permissions are empty            (visible to every authenticated caller)
OR
chunk permissions overlap the caller's JWT roles/domains

Citations

The system prompt asks the model to mark any claim drawn from search_docs with a [^<chunk-id>] footnote. After the model answers, the service validates those markers against the chunks actually returned:

1. extract_citations resolves each marker to a real retrieved chunk.
2. Valid markers become structured citation objects in the response.
3. Unknown markers are reported in retrieval.hallucinated_citations.
4. strip_hallucinated removes the invented markers from the displayed answer.

This guards against a model citing a source it never saw. Note the limit: it proves a marker points at a real retrieved chunk, not that the chunk actually supports the sentence.

API

Docker Compose exposes the service through Caddy at http://localhost/agent. Kubernetes uses the same /agent ingress prefix. Inside the service network it listens on port 3000.

/ingest requires the admin role (Cedar’s IngestDocuments action — a global, non-domain-scoped gate, since re-ingesting the knowledge base isn’t scoped to any one domain), enforced in app/routes/ingest.py via cedar_authz.can_ingest_documents. A caller below that tier gets 403.

The request schema currently exposes top_k, but it is not wired into retrieval. Retrieval depth is controlled by search_docs and the service’s TOP_K_* configuration.

Authentication And Permissions

require_user (app/auth.py) checks the mesh-injected x-gateway-claims header first: Istio’s RequestAuthentication (or, in docker-compose dev, Caddy’s forward_auth against claims-gateway’s /verify) verifies the gateway JWT once at the edge and injects the validated payload as this base64 header — agent-service trusts it rather than re-verifying a JWT itself, the same trust model as every other migrated service (see Node Auth & Error Middleware. There is no JWKS material anywhere in this service’s configuration.

Only when the header is absent does a second, narrower path run: a local-dev bypass for evals/run_evals.py, which sends its own self-minted token as the authentication_token cookie (or a Bearer header) and is verified with HS256 against EVAL_JWT_SECRET — a separate secret, opt-in, never set in any deployed environment, that lets dataset runs work without a full Keycloak + claims-gateway stack. This path never reaches the mesh, so it still needs its own signature check; the mesh-header path never does.

Both token shapes are normalized to the same roles/domains lists: claims-gateway’s real memberships:[{domain,role}] is flattened, and the eval tool’s already-flat roles/domains arrays pass through directly. Roles and domains are combined into AuthUser.permissions and used both for retrieval filtering and role gates. Role-sensitive operations use the same weighted hierarchy as the rest of the fleet (auth-contracts/roles.json): standard_customer (10), business_staff (60), business_admin (80), admin (100). The one role-gated operation today is the per-request answer-model override.

REQUIRE_AUTH=false creates an anonymous user and is intended only for isolated local use.

Provider And Models

One OpenAI-compatible endpoint drives both chat/tool-calling and embeddings. Using a single OpenAI-compatible contract (rather than a vendor SDK) means the model can be swapped — between OpenRouter, a local server, or another gateway — by changing configuration, not code. OpenRouter is the default:

OPENROUTER_API_KEY=<your-openrouter-key>
LLM_BASE_URL=https://openrouter.ai/api/v1
ANSWER_MODEL=openai/gpt-4o-mini
EMBEDDING_MODEL=openai/text-embedding-3-small
EMBEDDING_DIM=768
MAX_OUTPUT_TOKENS=2048

LLM_API_KEY, DEEPSEEK_API_KEY, and GOOGLE_API_KEY are accepted as legacy key aliases, but the supplied key must authenticate against LLM_BASE_URL.

Privileged callers may override the answer model per /ask request:

MODEL_OVERRIDE_MIN_ROLE=business_admin
ALLOWED_MODELS=openai/gpt-4o-mini,google/gemini-2.5-flash

The override spends the server’s shared provider balance, which is why it is gated: it is rejected with 403 below the configured role and with 400 when a non-empty allowlist excludes the model. An empty ALLOWED_MODELS permits any model for callers that pass the role gate. The embedding model is never changed by an answer-model override, because changing it would invalidate the indexed vectors.

Database

Alembic (the schema-migration tool for SQLAlchemy) creates two tables:

• documents: source, content hash, metadata, permissions.
• chunks: document relation, text, section, kind, embedding, generated tsvector, permissions, and metadata.

plus pgvector, full-text GIN, permission GIN, and relation indexes. pgvector is the Postgres extension that adds a vector column type and similarity operators, which is what lets semantic search live in the same database as everything else instead of a separate vector store.

The initial migration defines embedding vector(768). The vector dimension is fixed in the schema, so changing EMBEDDING_DIM requires a matching migration and re-ingestion — an environment-variable change alone is not enough. Runtime queries use async SQLAlchemy with asyncpg; Alembic uses a synchronous driver for migrations.

Evaluation

Behavior is checked with a golden dataset: hand-written questions, each paired with the behavior we expect rather than one exact answer string. A row can assert which tool should fire, which documentation source should be cited, which keywords the answer must contain, whether the response should be the IDK marker, and — for scope — that an off-topic question is declined. Pinning behavior instead of wording keeps the checks robust to harmless rephrasing while still catching real regressions.

These evaluations are deliberately separate from the unit tests. They drive the real running stack against a real provider and depend on an already-ingested corpus, so they are slow, cost money, and are not hermetic — the opposite trade-off from the fast unit tests, and the reason they are run on demand rather than in the normal test run.

Most assertions are deterministic: a tool name appears in the trace, a citation source matches, a substring is present. But some behavior is semantic and substring matching cannot capture it — the clearest case is “did the assistant decline an off-topic question, or did it actually answer it?” Those rows opt into an LLM-as-a-judge: a second model grades the answer against a rubric. The judge defaults to a different model family than the answer model, because a model asked to grade output in its own style tends to favour it, and an independent family reduces that self-preference bias.

Not every failure is a regression. Some rows test behavior the agent is known not to handle yet — an accepted, tracked gap. Marking such a row xfail (“expected to fail”, a convention borrowed from pytest) keeps it in the suite without turning the run red: when it fails as predicted it is reported as XFAIL and the build stays green. The point is that the gap stays visible and counted on every run instead of being deleted and forgotten — and every xfail must carry a written reason, so an accepted gap is never silent debt. If an xfail row ever starts passing — because the gap was actually fixed — it is reported as XPASS, the signal that its marker is now stale and should be removed so the row goes back to guarding the behavior normally.

For the dataset schema, the scoring and exit-code contract, model sweeps, and how to configure the judge, see the agent-service README.md.

Langfuse Observability

The agent is instrumented with OpenTelemetry (OTel), a vendor-neutral tracing standard, so each request produces a trace tree of spans. The motivation is that an agent’s behavior is otherwise opaque — you cannot tell from the final answer alone which tools ran, what retrieval returned, or where the time and tokens went. Langfuse is the backend that ingests those traces and renders them with LLM-specific views (token counts, cost, generations):

agent.answer
  agent.hop.N
    gen_ai.chat <model>
  tool.<name>
    retrieve.embed / retrieve.vector / retrieve.keyword / retrieve.rerank

Spans are typed so Langfuse renders them meaningfully: generation spans carry model, tokens, and cost; retriever spans carry the query and the ranked documents; tool spans carry arguments and results, and failed tools get an error status plus the exception.

Capturing the actual query, document text, and tool I/O is gated by OBSERVE_PAYLOADS, which defaults to off because that content can be sensitive; the development Compose override turns it on for useful local traces. Span types, counts, and error statuses are still emitted when payload capture is disabled.

With OTEL_EXPORTER_OTLP_ENDPOINT set, spans are exported over OTLP. Without it, the agent prints compact one-line span summaries to its logs. Langfuse and the OTLP variables are part of the Compose stack but are not generated by just stack env — see Setting Up Your Environment. Langfuse is not deployed by the current Kubernetes manifests.

Current Deployment Differences

The source of truth for runtime settings is backend/agent-service/app/config.py. LLM_TEMPERATURE controls the default generation temperature, while TWIN_TIMEOUT_SECONDS controls live-data request timeouts. Invalid numeric bounds, unsupported model-override roles or log formats, and malformed service URL schemes fail during settings loading. Startup also fails when the provider key is missing. CORS_ORIGINS is currently defined there but is not consumed by FastAPI middleware.