The Problem
Two live Grafana environments, a shared non-production instance backing the dev and staging environments and a separate production instance, ran on pure click-ops. Every dashboard, every alert rule, and every notification route had been created by hand through the UI. There was no version history, no diff to review before a change went live, and no way to tell what had changed between one week and the next except by comparing screenshots. Twenty-one dashboards and a scattered set of alert rules already existed and worked, which made the project harder, not easier: a rewrite from scratch was not an option, because the operators depending on those dashboards could not tolerate losing them while I rebuilt what they already had.
I wanted the opposite of a rewrite. I wanted to bring the existing state under version control without breaking it, then extend it in a form the team could safely touch. That meant designing an adoption path for content that already existed, not just a deployment path for content I was about to create.
The Approach
The design is a Terraform root applied twice, once per Grafana instance, discovering dashboard JSON from a folder tree that mirrors Grafana’s own folder structure and enforcing that each file’s name equals the dashboard’s UID.
Adoption without import. Terraform’s normal path for claiming an existing resource is an import block, one per resource, painful at any real scale. Instead I exported the 21 existing dashboards from the UI, normalized them into the repository’s naming convention, and applied them with an overwrite flag, letting Terraform claim ownership of a dashboard that already existed under the same UID rather than recreating it. Twenty-six Terraform resources landed on each instance, with parity verified across non-production and production before I touched a single alert rule.
Rename-safety for a risk the design itself introduced. Keying every dashboard resource by filename makes the system readable, the file tree is the folder tree, but it also means a bare rename plans as a destroy and a recreate, which for a dashboard means losing its history and breaking anything that pointed at its UID. I added a plan-time precondition that fails the run if the filename and the UID inside the JSON ever disagree, documented a moved-block procedure for the rare intentional rename, and adopted a reviewer rule that treats any plan showing a dashboard or folder destroy as an automatic stop. The guardrail exists because the same convention that makes the system simple is the one that makes a careless rename dangerous.
Alerting ported, not reinvented. Fourteen alert rules across three rule groups came over from whatever had been alerting before: scheduler rules, queue rules, and a poller watchdog, each checked against a live metric before it was written down as Terraform. One rule referenced a metric that no longer existed, and the port caught it instead of carrying it forward broken. Thresholds live in per-environment Terraform variables, never interpolated directly into a PromQL expression, so non-production and production can diverge on a number without diverging on a query.
Tenant isolation without a rewrite. Once the baseline was live, every alert routed to one Slack channel regardless of tenant, and the twenty backend dashboards had no way to scope a view to a single tenant’s traffic. I moved routing to per-tenant contact points generated with a Terraform for_each, and wrote a deterministic transformer that threads a namespace template variable into roughly 280 PromQL expressions across those twenty dashboards, wherever the underlying metric was actually tenant-separable. A new tenant lands with its alerts paused until its metrics are verified live, then gets unpaused in a follow-up change, never armed by default.
Results
The full set of twenty-four dashboard JSON files now lives in git and deploys through the same Terraform apply on every change, with drift caught by a nightly job instead of discovered by an operator. Sixty of the repository’s sixty-one commits are mine; the remaining one is a colleague merging a request I had explicitly handed off. That split is worth being honest about in the other direction too: of the twenty-eight merge requests that landed, twenty-seven were my own, reviewed by no one but me. The platform is built for team contribution that has not fully arrived yet, and I would rather say that plainly than let it look like a team built it. One more honesty note belongs here: the alert-rule folder move during adoption caused a brief alert-state reset, so “no downtime” is not a claim I make without qualification, even though the migration overall shipped without operators losing visibility into a running system.
If you introduce a risk, ship its safety rail in the same change.