Transmission 035 ended with the node group still provisioning after 30 minutes. It finished overnight. The cluster came up clean on Kubernetes 1.31.

Today the goal was to upgrade it to 1.32.

The constraint

The upgrade required four changes:

1. variables.tf — change eks_cluster_version default from "1.31" to "1.32"
2. phase1_addons.tf — pin all four addon versions to their 1.32-compatible releases

# variables.tf
- default = "1.31"
+ default = "1.32"

# phase1_addons.tf — pinned versions queried from AWS
addon_version = "v1.23.1-eksbuild.1"       # vpc-cni
addon_version = "v1.32.13-eksbuild.28"     # kube-proxy
addon_version = "v1.11.4-eksbuild.53"      # coredns
addon_version = "v1.65.0-eksbuild.2"       # aws-ebs-csi-driver

tofu validate passed. The plan saved to phase1.tfplan.

Plan: 77 to add, 1 to change, 12 to destroy.

The count was higher than expected for a version bump. The reason was state drift from the interrupted apply in 035: many resources existed in AWS but were missing from state, so the plan planned to create them again. I flagged this, understood the drift, and ran the apply anyway.

The proof

The first failure: terraform.tfstate.backup access denied

The apply ran for about 45 seconds — IAM roles, KMS key, VPC, subnets, security groups all created — and stopped:

Error: failed to create local state backup file: open terraform.tfstate.backup: Access is denied.

Error: creating ECR Repository (nexusflow-gateway): RepositoryAlreadyExistsException
Error: creating ECR Repository (nexusflow-worker): RepositoryAlreadyExistsException
Error: creating CloudWatch Logs Log Group (/aws/eks/nexusflow/cluster): ResourceAlreadyExistsException

OpenTofu had created a second VPC, new IAM roles, and new security groups in AWS but could not write updated state to disk. terraform.tfstate.backup was owned by a different Windows user. The three AlreadyExistsException errors confirmed that the ECR repositories and CloudWatch log group already existed from the 035 apply — the plan had not known that because they were missing from state.

State was not saved. The new AWS resources existed in AWS and nowhere else.

The second failure: stale plan

I ran attrib -R terraform.tfstate.backup and re-ran the apply. It failed again:

Error: Saved plan is stale

The given plan file can no longer be applied because the state was changed by
another operation after the plan was created.

The first partial apply had changed state enough that phase1.tfplan was invalid. I re-planned.

The third failure: NAT gateway creates, state still cannot save

The new plan was 29 resources. I applied it. The NAT gateway provisioned after two minutes:

module.vpc.aws_nat_gateway.this[0]: Creation complete after 2m1s [id=nat-0387633789471caab]

Then the backup error came back and stopped the apply again:

Error: Failed to persist state to backend

The error shown above has prevented OpenTofu from writing the updated state to the
configured backend. To allow for recovery, the state has been written to the file
"errored.tfstate" in the current working directory.

    tofu state push errored.tfstate

OpenTofu wrote an errored.tfstate containing the NAT gateway. The main terraform.tfstate was not updated. The state had forked.

Moving to WSL

takeown failed from PowerShell — “does not have ownership privileges.” The backup file was owned by a user the current session could not override.

I switched to WSL Ubuntu 24.04, where the D: drive mounts as /mnt/d with rwxrwxrwx:

cp errored.tfstate terraform.tfstate
rm -f errored.tfstate terraform.tfstate.backup
touch terraform.tfstate.backup && chmod 666 terraform.tfstate.backup

I promoted errored.tfstate over terraform.tfstate. That was safe because errored.tfstate cleanly captured the newly provisioned NAT gateway that failed to flush to the main file, without dropping any prior state.

OpenTofu is not installed in WSL by default. The deb install succeeded, but tofu init timed out reaching registry.opentofu.org over IPv6. Disabling IPv6 fixed provider downloads:

curl --proto '=https' --tlsv1.2 -fsSL https://get.opentofu.org/install-opentofu.sh | sudo bash -s -- --install-method deb
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
tofu init

OpenTofu has been successfully initialized!

Importing the missing resources

The recovered state was missing the three resources that caused AlreadyExistsException. Importing them one at a time failed because outputs.tf directly indexed both ECR repos by key — when only one was in state, the output for the other crashed evaluation and OpenTofu rolled back the import.

I wrapped the offending outputs in try() before importing. That is the correct behaviour regardless: if a repo is not in state yet, the output should return empty rather than crash:

# before
value = aws_ecr_repository.this["nexusflow-gateway"].repository_url

# after
value = try(aws_ecr_repository.this["nexusflow-gateway"].repository_url, "")

All three imports then succeeded and held:

tofu import 'aws_ecr_repository.this["nexusflow-gateway"]' nexusflow-gateway
tofu import 'aws_ecr_repository.this["nexusflow-worker"]' nexusflow-worker
tofu import 'module.eks.aws_cloudwatch_log_group.this[0]' /aws/eks/nexusflow/cluster

$ tofu state list | grep -E "ecr_repository|cloudwatch"
aws_ecr_repository.app
aws_ecr_repository.this["nexusflow-gateway"]
aws_ecr_repository.this["nexusflow-worker"]
module.eks.aws_cloudwatch_log_group.this[0]

The cluster replace conflict

A fresh plan from WSL ran clean. I applied it. The apply updated the CloudWatch log group tags, created the NAT route, created the ECR lifecycle policies, and then hit the cluster:

Error: creating EKS Cluster (nexusflow): ResourceInUseException:
Cluster already exists with name: nexusflow

The plan had marked the cluster as +/- — create replacement then destroy. The reason: the IAM role ARN stored in the cluster’s state entry was the 035 role (nexusflow-cluster-20260913...), but the IAM role resource now in state was the new role created by today’s partial applies (nexusflow-cluster-20260914...). The subnet IDs had also changed — the cluster was on the old VPC’s subnets, not the new VPC’s subnets now in state.

OpenTofu tried to create the replacement cluster before destroying the old one. EKS does not allow two clusters with the same name in the same account. The create failed. There is no clean way to resolve this conflict without deleting the existing cluster first so the plan can recreate it on the correct VPC.

I stopped here. Five hours on a four-line change was enough.

Where this sits

Next session: aws eks delete-cluster --name nexusflow, wait for deletion, re-plan, apply.