One Feature Contract, Two Languages

The Problem

Training/serving skew is a model learning from one computation of its features and being served another. In EventHorizon the gap is wide: the Isolation Forest is trained in Python with pandas and scikit-learn, exported to ONNX, and scored in Java inside a Flink operator, where it receives a bare array of floats. If the two sides disagree about which number goes in which slot, or how it is computed, the model answers with its usual confidence.

Here a broken contract is quieter than a wrong answer: the scoring function catches inference errors and returns 0.0, so a mismatched vector scores every event as normal. Per the commit records, the ML score on the first NAB machine-temperature replay sat at 0.0 for every event, so that run’s detections came from the statistical side alone. The feature extractor keyed its Redis writes on a hardcoded entity_id field, but the benchmark payloads carry device_id, so it wrote one key per event and the scorer, reading by device, never found them. The scorer fell back to a one-feature vector, the two-feature model rejected it, and the catch block returned 0.0. Fixing the key exposed a second defect the first had masked: the scorer was reading the ONNX label output, where scikit-learn’s +1 means normal, as an anomaly score, and once features flowed it flagged 10,710 anomalies on one replay. Two defects, each hiding the other, and neither one threw.

The Contract

The fix was to write the features down as a contract, under a heading in the benchmark runbook that says it must not drift. Six features per numeric field, over a per-device rolling window of the last N readings:

F        = v                              # raw value
F_delta  = v.diff(1).fillna(0.0)          # full series, 0 on first reading
F_log    = ln(v)                          # only when v > 0
F_mean   = rolling(N, min_periods=1).mean()
F_std    = rolling(N, min_periods=1).std(ddof=0).fillna(0.0)
F_zscore = (v - mean) / std               # inf and NaN mapped to 0

Three rules make it enforceable.

Order is derived, never positioned. Java’s FeatureVector.toFloatArray() sorts feature names lexicographically before building the ONNX input, and the trainer’s column list is sorted([...]). Neither side hand-places a column, so the order is always temperature, temperature_delta, temperature_log, temperature_mean, temperature_std, temperature_zscore, and the input dimension is six.

Semantics are pinned where the libraries disagree. The standard deviation is the population form (ddof=0), not pandas’ default sample form; min_periods=1 gives every event a full vector from its first reading; the delta is a full-series difference, not a within-window one. The Flink side states each choice in code:

// POPULATION variance (ddof=0): E[x^2] - E[x]^2, clamped to avoid tiny negatives.
double variance = Math.max(0.0, sumSq / size - mean * mean);
double std = Math.sqrt(variance);
double zscore = std > 0.0 ? (v - mean) / std : 0.0;
double delta = prev != null ? v - prev : 0.0;

The trainer also computes its rolling windows over the full series before dropping the labeled anomaly windows from the training set, because the stream never skips readings and the training window should not either.

Configuration is shared by name. The Flink job and the trainer both read FEATURE_WINDOW_SIZE=12 (one hour at five-minute cadence), and the submit script sets ENTITY_ID_FIELD for the extractor that writes the Redis key and the scorer that reads it from the same variable.

How It Is Enforced

Until recently, enforcement ran one way, and I found that while writing this. A Java parity test drove a fixed sequence through a real keyed Flink pipeline and asserted every feature to 1e-6, but its expected values came from a reference script that was never committed, and the Python side had no test at all: change ddof in the trainer and every build would have stayed green. Regenerating the values from the trainer’s own feature code reproduced the Java constants exactly, so nothing had drifted; the gap was in verification, not in the numbers. Now a generator script computes a golden fixture with the trainer’s build_feature_frame, a pytest test asserts that the trainer reproduces it, and the Java test asserts the Flink output and the feature-name order against the same values, both within 1e-6. Edge tests cover cold start, per-device isolation, device_id keying, and ignoring the benchmark’s string timestamp. The export step checks the ONNX model’s input dimension, and on an inference failure the scorer now logs the vector length against it.

What It Costs

Every feature is written twice, in two languages, with two numerical methods, so parity is a tolerance, not an identity. Model and extractor must ship together: retrain and register the model before rolling the Flink jobs, or a stale model scores 0.0.

The ensemble makes a break worse than neutral. As the submit script configures it, the fused score is 0.6 times the statistical score plus 0.4 times the ML score, both in [0, 1], and the tuned benchmark threshold is 0.65. An ML score stuck at 0.0 caps the fused score at 0.6, so a broken contract does not merely weaken detection on the fused path; it makes it arithmetically impossible. That is why the scorer carries an exclusion list: schemas configured as not matching the loaded model are dropped before scoring, and the ensemble falls back to a statistical-only result instead of fusing a zero.

What Is Still Open

Half of the parity check runs only on my machine. CI runs the Java side on every pull request, but not the new Python test yet. The fixture is written as two identical copies, one per side, kept in step only by the generator, and the window-3 buffer-trim case is still pinned on the Java side alone.

The feature set can change shape. For a non-positive value the extractor omits the log feature, and because the Redis write merges fields into the existing hash, the scorer then reads the previous reading’s log, or a five-feature vector if there was none. The trainer never sees the case: it drops non-positive rows. The golden fixture is strictly positive, so it does not exercise this either.

The contract covers values, not time. The scorer reads whatever features are newest in Redis for the device and ignores the timestamp the extractor stores, so under a time-compressed replay an event can be scored with a neighbor’s features.

Identity sits outside it. The scorer loads one model file per job, copied into the Flink task manager, and never reads the feature names the exporter embeds in it. The extractor keys only on string entity IDs while the scorer stringifies any value, so a numeric device ID would split them silently.

As for results, this is groundwork for scoring, not a score. Per the commit records, the first live replay, with the ML score stuck at 0.0, still put statistical detections in three of the four labeled windows; after the key fix, the output fix, and windowed features, a run at the 0.65 threshold hit all four, with 96 detections at a point precision of 0.57. That is one run on one series, measured by my own window-level script, with output that was not kept, and NAB’s official scorer has not been run. A contract nothing checks from both ends is only a comment.