The Problem
Document processing had been built one source at a time (a source being the external system a notification arrives from). Years of source-by-source growth left 535 concrete processor files (roughly 397 per-source) by the April 2026 audit, with the count still growing, and every new source meant another Java class, another deployment, and another set of regression risks. Observability lived in log lines, authentication was federated through a sprawling legacy system, and the entire pipeline rode inside the original monolith with no clear seam to extract.
I wanted a platform with seams instead of a monolith with conventions: a single engine that reads its behavior from configuration, a worker tier that owns the source-specific quirks, an observability surface for the operators who actually live in this system, and an authentication boundary that does not require the legacy system to be online.
The Approach
The platform is four core services with a deliberately small contract between them, plus an MCP bridge taking shape alongside them.
Orchestrator owns the DAG: it reads the template for the source, dispatches eligible steps, and merges sub-processor results into a single JSONB document per processing context. Terminal evaluation, optimistic-lock retry, and a strict publish-before-commit boundary make it safe under concurrent step completion, and a retried dispatch is field-for-field the original message, source-config pin and replay lineage included. Workers host ten generic pipeline stages (content extraction, source matching, record lookup, deduplication, authentication, document fetch, restricted-document handling, fallback capture, package assembly, and storage) that consume from RabbitMQ queues and publish a completion message back when a job finishes. The workers themselves are essentially stateless: they hold no direct database access, and during a stage run they read and write pipeline data over gRPC through the orchestrator and legacy-model data through a separate data gRPC service. Dashboard is a Vue 3 SPA (Vue 3.5 on Vite 6, with a vitest suite) that gives operators a view of every processing context in flight, plus versioned config editors and a patch-and-replay surface whose mutations all belong to the orchestrator’s API. Auth API is a standalone Micronaut JWT issuer with a JWKS endpoint, designed so the platform’s authentication boundary does not depend on the legacy stack. MCP Bridge, still taking shape, exposes a curated subset of platform operations to MCP-aware tooling.
The contract between services is shallow on purpose. The orchestrator is the only writer to its own JSONB. The workers are the only callers of the source-specific HTTP and email surfaces. The dashboard owns no data and no endpoints; every mutation it offers is a call into the service that owns the record. The auth API issues and rotates signing keys; nobody else owns key material. Each service has its own deployment cadence, its own database role, and its own ingress rule.
Results
The platform is four focused services built to retire the 535-processor legacy pipeline, each with a single responsibility and its own deployment seam. The first source pattern runs end-to-end on the dev environment, and in September 2026 the first real end-to-end dry run against a live source on that environment returned a mismatch receipt that caught two genuine parity defects before any production traffic. The promotion rule for staging, production, and the cutover is that every one of the roughly dozen in-scope sources proves end-to-end on local and the dev environment first: one working source promotes nothing. All 411 commits in the first four months had a single human author, working AI-paired throughout, and that is still true of the roughly 430 commits on the four main branches today, about two-thirds of which carry an AI co-author trailer. Onboarding a source is still configuration rather than code, and the path to that configuration is now a discovery session that walks one real notification the way a human would; the result is curated into version-controlled fixtures alongside the versioned config rows, and a zero-side-effect dry run proves the outcome against the live source before anything is promoted. The legacy processors are an oracle to sample where a legacy record exists, never the specification to reproduce. All three Java services run Micronaut 5 on Java 25 with Gradle 9.5, adopted mid-build without pausing feature work. Four gated stages of test-driven development with formal architectural review produced roughly 580 tests at the last of those gates by the gate records, a suite since grown to roughly 1,200 by a static count across the four repositories, and surfaced three architecture-level defects (a multi-pod DDL race, half-logic in DAG terminal evaluation, missing consumer wrappers) before any of them could ship. The live dry run has since added two more to that list.
Each component is documented in its own case study: the orchestrator and its DAG model; the workers and the stage design that coexists safely with the legacy system; the dashboard and the API-perimeter lesson it taught us; and the auth API and the dual-controller alias pattern that made a zero-downtime route migration possible.