The Problem
I have built the same system three times at two companies. At Banjo, I built it twice: first in Ruby, on platforms spanning corporate security, breaking news detection, and election monitoring, and then again as the platform moved to Java on Flink. At PHC, a health-tech startup, I built the third: a real-time risk anomaly detection and notification platform on Flink, Elasticsearch, Protobuf, and Micronaut, including a custom Flink StateFun framework that ran processors in both Java and Python.
The shape never changed: ingest high-volume event streams, apply rules and models to find the few signals that matter, and put them in front of someone who can act. Each time, the domain was baked into the code. EventHorizon is my attempt to build the shape once with no employer’s domain attached: users define their own schemas and detection rules, and the platform handles validation, scoring, storage, and explanation. It is a personal side project, built in two bursts between January and August 2026 per the commit history (after a 2024 design phase that produced documents, not code), and it has never been deployed anywhere but my own machine.
The Approach
The pipeline is five Flink 1.18 jobs chained over Kafka topics rather than one large job, so each stage can be redeployed, tested, and reasoned about on its own.
Validation comes first: every raw event is checked against its registered schema and failures go to a dead-letter topic. Two detectors then read the validated stream in parallel. The statistical detector keeps keyed rolling state in Flink (per entity when events carry an entity field, pooled per schema when they do not) and applies user-configured rules (z-score, IQR, and deviation from simple or exponential moving averages) from a rule service. The feature extractor writes six windowed features per numeric field to Redis, and the ML scorer runs an Isolation Forest exported to ONNX inside the Flink operator, with no Python in the serving path. The ensemble scorer joins the two scores on event ID with a ten-second timeout, fuses them (0.6 statistical and 0.4 ML, as the submit script sets them), and emits anomalies to a query service backed by Elasticsearch.
Around the pipeline sit six Micronaut 4 services on Java 21 (schema, rules, anomaly query, an ML model registry, the LLM service, and a GraphQL gateway that streams anomalies over SSE), a Next.js dashboard, and Keycloak, with tenant isolation enforced in every service. The LLM service is on demand, not in the stream: it explains anomalies, answers questions, and drafts Flink SQL behind a provider interface (Ollama locally, Gemini as the cloud option) with retrieval from pgvector, covered in one interface, two models, and RAG on pgvector. The stack runs on Minikube, deployed by Skaffold and Kustomize in layers with health checks between them, because starting everything at once crash-looped on a single machine.
The part I care most about is the move off fabricated data. Until July 2026, every anomaly on the dashboard came from generated demo events. Phase 4.7 replaced them with the Numenta Anomaly Benchmark’s machine-temperature series: 22,695 real five-minute sensor readings with four engineer-labeled failure windows, replayed time-compressed through the live pipeline. A second source, an hourly collector for a public retail catalog, is built and verified against recorded snapshots. Real data immediately exposed two ML bugs hiding each other, and keeping features identical between Python training and Flink serving became its own discipline: one feature contract, two languages.
Results
The full chain runs end to end on one local cluster, from raw event to explained anomaly. Per the commit records of the benchmark work, the first live NAB replay, with the ML score stuck at 0.0, still put statistical detections in three of the four labeled windows. After the key fix, the model-output fix, and windowed features, a later run at an ensemble threshold of 0.65 put at least one detection in all four, with 96 detections at a point precision of 0.57 (0.37 at the default 0.5). I am claiming a recorded smoke test, not a benchmark score: it is one run on one series, the window-level measure is lenient (one detection anywhere in a two-day window counts), the output survives only in the commit message and the runbook, and NAB’s own scorer has not been run. That work is still on the working branch, with its merge to main open.
What is not built matters as much. There is no production deployment and no scale test; one machine caps throughput far below what this architecture exists for. There is no alerting yet, so detection ends at the dashboard. The rule API accepts threshold and SQL rule types that no job evaluates, and the UI marks them “coming soon” rather than letting them silently never fire. The Strimzi-managed Kafka and ClickHouse in the base manifests belong to a cloud target that does not exist yet; the running stack uses a single-node Kafka and no ClickHouse.
The record is 88 commits on the working branch as of August 2026, all mine, 74 of them with an AI co-author trailer: I build AI-paired and say so. Six ADRs capture the decisions; the failure-handling section of the provider-abstraction ADR was superseded the same day, when I added recovery to the Ollama provider. By a static count the JVM side carries roughly 9,600 lines of test code across 64 files, plus pytest suites for the Python code, and CI runs both on every pull request. Next is a properly scored benchmark run and a cloud deployment with a real load test. The fourth time, the architecture was the easy part; proving the numbers were real was the work.