The Problem
Our team was eight engineers, nine including our engineering lead, and I was the only one with DevOps and infrastructure experience. After the contractors who had originally built our infrastructure left, I became the default DevOps engineer, despite it being my least favorite part of the stack.
The immediate problem was our EKS clusters. They were running Kubernetes 1.30, which was approaching end of standard support, and AWS charges extended support fees for any cluster that stays on a version past that window. With three clusters running roughly 25 production services, those charges would add up fast. A minimal bump to 1.31 would have kept the fees at bay, but only for another three to six months before the next version aged out behind it, so the target became 1.33, the current release at the time: one upgrade effort for the longest runway available. On top of that, the Terraform codebase the contractors had left behind was in rough shape. Plans would not validate cleanly, and there were issues in both the plan and apply stages that had to be resolved before any upgrade work could begin.
The biggest technical challenge was Karpenter. Our clusters were running Karpenter v0.37, and the upgrade to Karpenter v1 (v1.0.2) was a significant breaking change that touched node provisioning, configuration schemas, and cluster autoscaling behavior. Getting that migration right was critical, because Karpenter manages the compute that all our services run on.
To complicate matters further, midway through the upgrade project we signed a deal with an outside managed-operations partner to eventually migrate our environments to their platform. But we could not stop the EKS upgrade; the extended support charges were already looming. I had to finish the work knowing the infrastructure would eventually be migrated anyway. It was a pragmatic call: pay the charges or finish the upgrade. We finished the upgrade.
The Approach
I started by stabilizing the existing Terraform. The entire infrastructure lived in a single Terraform repository, and I had to fix validation errors and broken plan/apply pipelines before I could safely make changes. Once the Terraform was healthy, I planned the EKS upgrade rollout.
Each upgrade was planned carefully with multiple dry runs to verify correctness before touching production. VPC peering provided VPN connectivity for engineer access during the process. GitLab CI/CD pipelines ran Terraform plan and apply stages, and I tested each change against a fresh environment to confirm the Terraform was complete and valid, effectively a disaster recovery validation at the same time.
The Karpenter v1 migration was the hardest part. The upgrade introduced breaking changes to node pool configuration and provisioner behavior. After each upgrade, there were leftover Karpenter nodes from previous versions that would not drain cleanly and required manual cleanup. That was where most of the hands-on intervention happened.
Results
All three clusters, running roughly 25 services, were upgraded over a four-week period with roughly five minutes of total logged cluster downtime. The bulk of the process was automated through Terraform and GitLab CI/CD pipelines, though Karpenter node cleanup and certain edge cases required manual intervention. The upgraded clusters now run EKS 1.33 with Karpenter v1, avoiding extended support charges and leaving the infrastructure in a clean state, whether for continued use or the eventual migration to the partner’s managed platform. That migration has since completed: the platform now runs in the new partner-managed AWS accounts, and the legacy infrastructure and its Terraform have been fully torn down.
Disaster recovery was validated as a side effect of the process: because I tested every Terraform plan in a clean environment, we confirmed that our infrastructure could be fully recreated from code.