The Problem
The legacy document-processing system had grown to 535 concrete processor files by the April 2026 audit, roughly 397 of them per-source (a source being the external system a notification arrives from), and the count is still growing. Each new source onboarding meant new code, a new deployment, and a fresh round of regression testing. The inheritance had also become unmaintainable: the processors extend and extend again, so a small change to a shared step meant tracing a handful to a dozen files across base classes, extended classes, and specific processor implementations, and the deep chains obscured what any given processor actually did.
I wanted a single engine that read its behavior from configuration instead of code. Sources would be onboarded by adding rows to a database table, not by writing and shipping Java.
The Approach
I designed Pipeline Orchestrator as a config-driven DAG engine on Micronaut and Postgres. It started on Micronaut 4 and today runs Micronaut 5 on Java 25, migrated mid-build without pausing feature work. Each source maps onto one of three reusable DAG patterns (email-based intake, cookie-session authentication, and full browser automation; the founding bet was seven patterns covering the entire source estate), stored as versioned template rows: the earliest were Flyway-seeded, and today they are managed through a JSON-schema-validated write API, with per-source fixtures from discovery walks version-controlled alongside them. At runtime the orchestrator instantiates a processing context, dispatches the eligible steps to RabbitMQ for sub-processors to execute, then merges their results back into a JSONB column on the context as completion messages arrive.
Three invariants kept the engine honest. First, terminal evaluation is not just a check for “no more steps.” Some DAGs halt early (duplicate detected, record restricted), so the orchestrator evaluates terminal state both before and after dispatch, and a step’s halt signal must short-circuit the dispatch loop, not race it. Second, optimistic-lock retry: when two sub-processors complete at almost the same instant, both want to merge into the same JSONB document. A @Version field on the context plus a bounded retry loop with a fresh transactional read handles the collision without losing work. Third, publish before commit is forbidden: the RabbitMQ dispatch lives outside the @Transactional boundary, so a rollback never leaks a phantom message into the queue.
Two more safeguards round out the engine. Every dispatch attempt is deduplicated by a unique constraint on (context, step, attempt), so a redelivered message can never double-execute a step. And failed steps back off through a tiered ladder of RabbitMQ delay queues (seconds at first, hours at the top) instead of hot-looping against an upstream system that is already struggling. A later fix closed a gap in that ladder: a retried dispatch is now field-for-field the original message apart from its attempt number and timestamp, carrying the same source-config pin and, for a replay, the same lineage, where it had been silently rebuilding both as empty.
The DAG vocabulary is intentionally small. A step has a list of dependencies, an optional fallback step, and a terminal flag. There are no per-source hooks, no scripting language inside the config. Anything source-specific lives in the sub-processor, not the orchestrator.
The newest surface is a dry-run endpoint. An administrator submits a real notification and a candidate source config, not necessarily the current version, and the engine runs the full DAG on its normal message path with every durable side effect suppressed end to end: no record, document, or package rows, no propagation to the legacy job table, uploads quarantined under a scratch prefix. The lookup, deduplication, and fetch stages still execute against the live source, which is the point. What comes back is a receipt: a seven-field outcome tuple diffed field by field against the expected outcome, an overall MATCH or MISMATCH, and per-step status. A kill switch, a concurrency cap, a size cap, and a retention sweeper keep it from becoming a load generator.
Results
A single deployable service runs the first source pattern end-to-end on the dev environment, and its dry-run endpoint produced the platform’s first real receipt against a live source in September 2026: a MISMATCH that caught two genuine parity defects before any production traffic. Nothing promotes to staging or production until every in-scope source proves end-to-end; one working source promotes nothing. Onboarding a source is still configuration through a versioned write API rather than a code change, but that configuration now comes out of a discovery walk of one real notification and is proven by a dry run before it is promoted. The orchestrator’s review gate passed on the third attempt: the first two iterations missed the half-logic in terminal evaluation and the publish-before-commit invariant. Those bugs would have produced phantom in-flight steps and rollback-silenced messages under real traffic, and the gate caught them before any traffic flowed.