The Problem
By late summer 2026 the pipeline platform could run its first source pattern end-to-end on the dev environment, and it was tempting to call that a milestone. I called it something else: one working source promotes nothing. Roughly a dozen sources were in scope, each an external system with its own notification format, link structure, and login quirks, and proving one said very little about the other eleven.
The second problem was onboarding itself. Porting each legacy processor (read the old Java, reproduce it in configuration, test against the old code’s values) treats a decade of accreted code as the specification, bugs included. The legacy processor is an oracle to sample where a record exists, never the spec. And the only way to know a candidate configuration is right is to run it against the real source, which on a dev database shared with the legacy system means every experiment risks a real write.
By August 2026 both answers had changed: staging, production, the traffic cutover, and scale-out work all gate on every in-scope source proving end-to-end on local and the dev environment, onboarding became discovery-driven, and proof became a dry run.
The Approach
A discovery walk replaces the port. A session walks one real notification the way a person at a keyboard would: open the email, follow every link, capture the request chain in the browser, then pick the cheapest machine path that reproduces it (plain HTTP over cookie-backed HTTP over a JSON hop over browser automation) and prove each hop browser-free before it counts. What the walk learns becomes config-as-data (a routing rule, a typed source config, a DAG template choice) plus a version-controlled fixture set with a synthetic notification, the expected outcome, and a transcript. Credentials never enter any of it: the transcript records cookie and header names only, the agent never types or screenshots a login, and a redaction check runs before every commit. Anything the platform cannot express yet is recorded as a named capability gap (twelve classes so far) against the closest expressible config, so the dry run goes red on exactly that field and nothing else.
A dry run replaces guessing. An administrator submits a real notification and a candidate source config, not necessarily the current version, and the orchestrator runs the full DAG on its normal message path with every durable side effect suppressed end to end. The dry-run flag rides in every dispatch, retries and replays included: the store stage quarantines uploads under a scratch prefix and writes no document row, the package stage reports the package it would have created, and record lookup never auto-creates a record. The lookup, deduplication, and fetch stages still execute against the live source, which is the point: production’s exact code path, minus the writes. What comes back is a receipt: a seven-field outcome tuple diffed field by field against the expectation, an overall MATCH or MISMATCH, and per-step status. A kill switch, concurrency and size caps, a retention sweeper, and a gate on templates with a real authentication step keep the endpoint from becoming a load generator.
A matrix replaces “it works on my machine.” A build task runs every source’s fixtures through a dry run on a local cluster with a fresh database, where the full tuple must match, and on the dev environment, where a notification the legacy system has already processed must halt as a duplicate with a package hash equal to the legacy one: the hash-parity proof. Any cell that is not equal needs a written, approved rationale naming the gap or the ticket, or the run fails.
Results
In early September 2026 the first real end-to-end dry run against a live source completed on the dev environment with zero side effects, row counts identical before and after, and returned a MISMATCH naming two genuine parity defects. The package-hash computation joined its parts with a different delimiter than the legacy processor, so the deduplication key differed for an identical notification. And the record-lookup stage keyed on the record identifier from the subject line rather than the one in the notification body. Three configuration fields were missing as well. Every one of those would have produced silent duplicates or wrong-record matches in production, and none of the roughly 1,200 tests across the four repositories could have asked about them, because the answers lived in the upstream system, not in the code.
The honest status is early: no in-scope source has a committed matrix row yet, and the curation dashboard and self-onboarding loop that will close the cycle (park a failing notification, discover, curate, replay) are designed and ticketed, not built. The retention sweeper turned out to be silent on the dev environment, its own bug, and the upload quarantine path is unproven until a notification with live links arrives. Nothing promotes to staging or production until the exit criterion is met, and that line changes on sign-off, never on optimism. The cheapest production incident is the one you can run on purpose with the side effects turned off.