The Problem
An anomaly score says something is off, not which field moved, how far, or whether it has happened before. The LLM service in EventHorizon closes that gap: it explains a detection on request, answers questions in a chat, drafts Flink SQL, and groups related anomalies into incidents. Two constraints shaped it: develop entirely offline, with no API keys or per-call bill, while keeping a managed cloud model as the production option; and never let an explanation sound authoritative about causes the data did not show. They became two ADRs: a provider abstraction, and retrieval on pgvector.
One Interface, Chosen Once
Every LLM feature talks to one LlmProvider interface (generation with or without history, callback streaming, a name, an availability check), and every blocking generation call returns the same LlmResponse record (content, provider, model, latency). Behind it sit two LangChain4j-backed implementations, Ollama locally and Gemini for the cloud, each a Micronaut bean guarded by its own configuration; without an API key, Gemini never reports itself available. A factory picks the primary provider once, at startup (condensed; the real one logs each fallback reason):
LlmProvider provider = switch (providerName.toLowerCase()) {
case "ollama" -> ollamaProvider;
case "gemini" -> geminiProvider;
default -> null;
};
if (provider == null || !provider.isAvailable()) {
return new MockLlmProvider();
}
return provider;What it buys. Switching providers takes one environment variable, LLM_PROVIDER, plus GEMINI_API_KEY when the target is Gemini. Locally, qwen3:8b runs natively on the host through Ollama, with no network and no key. The service boots with no chat model at all, because the fallback is a mock, and the mock is loud: its provider name and content say so, and the factory’s tests pin that down. Cost control is structural rather than metered: the default is local and free, the paid path needs a deliberately supplied key, and neither provider fills the response’s token counts yet.
What it does not buy. Streaming is simulated in both providers: each generates the full response, then replays it word by word, so the first “streamed” token arrives only after the whole answer exists. Prompts are hand-assembled System:/User:/Assistant: strings, and the library is pinned to a 1.0 alpha. Failure handling has already diverged: the ADR describes a provider that disables itself permanently on its first runtime error, and I replaced that for Ollama the same day with a three-failure cutoff and a 30-second health check that re-enables it. Gemini still has the permanent flag. And because the factory decides once, a provider unavailable at startup is replaced by the mock until restart.
Retrieval on the Database Already There
Retrieval lives in the PostgreSQL the services already run, with pgvector: a document_embeddings table holding 384-dimension vectors next to the tenant, document type (anomaly, schema, rule, or incident), source ID, and content, upserted on re-index, with an IVFFlat cosine index. Embeddings come from all-MiniLM-L6-v2 running in-process through LangChain4j’s ONNX build, so they cost nothing per call and work offline. If that model cannot load, requests that need retrieval fail rather than answer ungrounded; the ADR says the service refuses to start, but the embedding bean is created lazily, another place the record and the code have drifted. Micronaut Data cannot express pgvector’s distance operator, so the query is native JDBC with bound parameters:
SELECT id, document_type, source_id, content, metadata,
1 - (embedding <=> ?::vector) AS similarity
FROM document_embeddings
WHERE tenant_id = ?
AND document_type IN (?, ?)
AND 1 - (embedding <=> ?::vector) > ?
ORDER BY embedding <=> ?::vector LIMIT ?Tenant isolation is a WHERE clause, and the defaults are a 0.7 similarity floor and five documents. Pinecone is a managed service with per-request pricing and lock-in, Weaviate needs its own cluster, Elasticsearch would split embeddings from the rows they describe, and hosted embeddings add a per-token bill and end offline development.
How it grounds an explanation. Retrieval is the smaller half. An explanation prompt carries similar past anomalies and incidents from pgvector, the schema when supplied, and the detection’s own facts: the triggering field, the observed value against its expected baseline, and the component scores. The instruction tells the model to cite the field and the actual numbers, not to speculate about causes such as attacks, breaches, or data corruption unless the data supports them, and to say what is missing instead of guessing. The detection’s numbers do most of the grounding; retrieval adds precedent.
Where It Leaks
Intent routing skipped retrieval silently (found while writing this, now fixed). The chat path classifies intent with regular expressions that wrapped word stems in whole-word boundaries, so the anomal alternative could never match anything: “anomal” is not a complete word. “Explain this anomaly” and “any anomalies today?” fell through to general chat and retrieved nothing. The same shape made the other five patterns miss plurals and inflections such as “schemas”, “queries”, “triggered”, and “incidents”, though they still matched base words. Each stem now ends in \w* inside the boundaries. The classifier had no direct tests (the orchestrator’s spec mocked it); it now has a 26-case Spock spec covering every intent type, the reported phrases, a negative case, and null or blank input. The explain endpoint never used that router, which is why dashboard explanations stayed grounded throughout.
The corpus is mostly seeded. Documents enter through a seeding script and incident grouping. Nothing indexes new anomalies as they are detected, so “similar past anomalies” means whatever was seeded.
Embeddings carry the same skew risk as the ML features. The seeding script embeds documents in Python with sentence-transformers; queries are embedded in Java by LangChain4j. Same model name, two runtimes, and nothing checks that the vectors match: the failure shape from keeping features identical between training and serving, one layer over.
The grounding facts come from the client. The dashboard assembles the detection context in the browser and sends it with the request. The tenant check governs who may ask, not what they claim.
The index and the confidence are unmeasured. The migration builds the IVFFlat index (100 lists) on an empty table, against pgvector’s guidance to build after loading data; with few rows and a single probe, a query can miss close documents or return fewer than it asked for. I have not checked whether the planner uses the index or measured recall; the fix is to build it after seeding, or switch to HNSW. The confidence each explanation reports is the model’s self-assessment, parsed from its text, defaulting to MEDIUM when absent.
Maturity
This is working software on one machine, not a validated product. The factory’s routing, Ollama’s recovery, intent classification, retrieval formatting, chat orchestration, and the SQL validator’s blocklist have unit tests. Nothing tests retrieval against a real pgvector instance or exercises the Gemini provider itself, and there is no golden set for explanation quality; the project plan defers that to the production phase. Generated Flink SQL passes a blocklist check and nothing executes it yet. The design has earned a shape worth keeping: a swappable model, a loud fallback, and explanations anchored to numbers the pipeline actually computed. It has not earned a claim that the explanations are good.