The Problem
Every tenant’s documents on the platform are protected by a three-level AWS KMS hierarchy: one customer master key per environment, one data key per tenant generated under that CMK, and one key per document or credential, envelope-encrypted with the tenant’s key and stored in Postgres. When the platform moved onto new hosting infrastructure, every tenant key had to be re-encrypted under a new CMK, in production, without a single tenant losing access to its own documents. The gate I built for that work is the whole story, and it closed with one database query: group every tenant key row by its KMS key ARN and count them. On production day, that query returned a single row: 207 keys, all on the new CMK, zero on the legacy alias.
Getting to that one row took roughly six and a half months of staged, gated work, because the failure modes are not forgiving. An encryption-context mismatch fails loudly, an explicit KMS exception, not a silent corruption. A key that simply never gets migrated fails differently: a document that looks recoverable until the moment someone actually needs it back. I needed a migration that could not leave a tenant key in an ambiguous state, and a way to prove, after the fact and without trusting my own notes, that nothing had been left behind.
The Approach
I built a single migration task: for each tenant key, decrypt with the legacy CMK, re-encrypt with the new one, and write the result inside one database transaction, so a failure partway through never leaves a key half-migrated. It is roughly 430 lines of Java, dry-run mode built in from the start rather than bolted on, and an idempotency check that skips any key already migrated, so re-running the tool after a partial failure is safe rather than dangerous.
The safety net came first. Before I touched the migration logic, I rebuilt the service’s test suite from five tests to 162, organized by layer: service, security, controller, repository, and the migration and mapping code itself. Six known production bugs, among them a dropped content type on one code path and a null pointer on an unknown key lookup, went into the suite as characterization tests, pinned rather than fixed, so that later refactors could not silently change behavior nobody had signed off on changing. I built and directed that safety net through an internal TDD agent workflow I designed for exactly this kind of change, pairing with it layer by layer rather than writing every test by hand, and the commit history carries its co-authorship openly.
Every stage was a gate, not a milestone. The dev environment ran a dry run against 29 tenant keys: 100 percent success, zero database writes, because dry-run mode never commits. The staging environment then ran the migration live twice, 47 keys and then 49 more, per the runbook records, each preceded by a checklist and followed by a findings document before anyone signed off on moving to the next stage. Production was different in kind, not just scale: the migration ran as one gated step inside a larger database cutover for the whole platform, at the point in that cutover where a mistake would have been expensive to unwind. It was designed to be abortable at every step, and the migration tool itself never shipped inside the production application jar; it lives behind an environment-gated test spec so it cannot run by accident.
Results
Every tenant key on the production CMK checks out: a single row, 207 keys, zero on the legacy alias. Roughly 190 of those were moved by the migration tool; the other 17 belong to tenants onboarded after the cutover and were created under the new key directly, so they were never a migration risk in the first place. That is not a self-report: it is the direct output of the same grouped query anyone with database access can run again today.
The dev and production numbers are the ones I stand behind without qualification, one because it wrote nothing to the database and the other because it is independently reproducible right now. The staging figures are true per the runbook records, but both staging documents were committed to the repository well after the dates they describe, so I hold them at arm’s length rather than citing them the way I cite the production count. The standalone production runbook I spent months writing also never ran as its own independent procedure. By the time production was ready, the key migration had been absorbed as one stage inside the platform’s larger cutover, executed under that cutover’s own checklist instead of mine. The design held up; the document I wrote did not end up being the operative script.
A migration is done when the database says so, not when the runbook does.