Transmission 035 · 2026-09-13

The EKS cluster came up. Phase 1 security infrastructure written and the apply is running.

034 ended with a redis-py PR in review. Today was EKS day. The cluster was supposed to already exist — 033 wrote the Terraform and validated it, 034 was a detour. I started by writing six new Phase 1 resources on top of the existing config: a KMS key, an OIDC identity provider, an IRSA role for the EBS CSI driver, a dedicated ECR repository for the application image, a cluster access entry, and four managed add-ons. Then I tried to apply. The state lock from a session four hours ago was still held by a live OS file handle. I killed the process, deleted the lock file, re-planned, hit a 75-resource plan caused by state drift from an interrupted apply, and eventually the cluster came up. The node group has been provisioning for 30 minutes and is still running.

Transmission 034 ended with a redis-py PR in review by the Redis core team.

Today the goal was to actually apply the EKS infrastructure that 033 wrote.


The constraint

Transmission 033 left the Terraform config validated but unapplied — no AWS credentials were available on the day. Today the credentials were there. The task was to extend the existing config with Phase 1 security infrastructure and run terraform apply against a real AWS account for the first time.

The six things that needed adding before apply:

  1. A KMS customer-managed key for EKS secrets, EBS volumes, and ECR
  2. An IAM OIDC identity provider pointing at the cluster’s issuer URL
  3. An IRSA role for the EBS CSI driver, with a StringEquals condition on the service account name — not StringLike, which would allow any service account in kube-system to assume it
  4. A dedicated ECR repository at nexusflow/app with KMS encryption
  5. An access entry mapping the Terraform caller identity to AmazonEKSClusterAdminPolicy
  6. Four standalone aws_eks_addon resources: vpc-cni, kube-proxy, coredns, and aws-ebs-csi-driver

Item 6 had a conflict. The EKS module in main.tf already managed vpc-cni, kube-proxy, and coredns inside its cluster_addons argument. Two owners for the same resource would fail apply with AlreadyExistsException. I removed the cluster_addons block from the module and gave sole ownership to the standalone resources in phase1_addons.tf. The standalone approach was the right call over keeping addons inside the module — it gives explicit depends_on ordering, per-addon service_account_role_arn, and surfaces each addon as a first-class resource in state.


The proof

Writing the new files

Two new files and one extended file:

  • phase1_security.tf — KMS key with 30-day deletion window and annual rotation, OIDC provider, IRSA role with inline KMS grant scoped to the NexusFlow CMK, nexusflow/app ECR repo, cluster access entry
  • phase1_addons.tf — four aws_eks_addon resources; coredns and ebs-csi-driver both depends_on the node group
  • outputs.tf — three new outputs: ecr_repository_url, kms_key_arn, oidc_provider_arn
terraform validate
Success! The configuration is valid.

The state lock

The first plan attempt failed before it started:

terraform plan -out=phase1.tfplan
Error: Error acquiring the state lock

Error message: The process cannot access the file because another process has
locked a portion of the file.
Lock Info:
  ID:        770b7914-0254-2b94-cf9e-d8306ebd9161
  Operation: OperationTypeApply
  Created:   2026-09-13 05:28:54 +0000 UTC

The lock was four hours old. terraform force-unlock does not work against local state — it only releases remote backend locks. The actual lock is an OS file handle on terraform.tfstate held by a process that never exited cleanly.

Get-Process | Where-Object { $_.Name -match "terraform|tofu" } | Stop-Process -Force -Verbose
VERBOSE: Performing the operation "Stop-Process" on target "terraform (13868)".
VERBOSE: Performing the operation "Stop-Process" on target "terraform-provider-aws (9164)".
VERBOSE: Performing the operation "Stop-Process" on target "terraform-provider-kubernetes (20896)".
rm .terraform.tfstate.lock.info
terraform plan -out=phase1.tfplan

The state drift

The plan that came back had 75 resources to add, 1 to change, and 24 to destroy.

That is not a Phase 1 plan. That is a full rebuild.

The interrupted apply from the morning had provisioned resources in AWS — VPC, subnets, security groups, the EKS cluster — but crashed before writing them to state. Terraform’s state file contained the security group rules (their IDs appeared in the +/- replace blocks) but not the VPC that owned those security groups. The result: Terraform planned to create a new VPC with new subnet IDs, which forced the NAT gateway, route tables, and all security group rules to replace. The node group, whose subnets would change, was also planned for recreation.

I stopped. Applying a 75-resource plan that deletes 24 live security group rules and recreates the VPC underneath a running EKS cluster would have destroyed the cluster.

The fix was to accept that the state was already wrong before this session started and re-run the apply to let Terraform rebuild from scratch with a clean state. The morning apply had already deleted the old EC2 and networking resources before it crashed. Nothing was left in AWS to conflict with. The second plan was clean because the old infrastructure was already gone, not because the state was reset. A second, cleaner plan showed 67 resources to add, 0 to change, 0 to destroy.

tofu apply "phase1.tfplan"

The apply

IAM roles, ECR repositories, KMS key, VPC, subnets, security groups, EKS cluster — all created in sequence.

EKS cluster creation
module.eks.aws_eks_cluster.this[0]: Creation complete after 8m49s [id=nexusflow]

The OIDC provider came up immediately after:

module.eks.aws_iam_openid_connect_provider.oidc_provider[0]: Creation complete after 2s
[id=arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/CB4E6B3CD79F72DD94A7F8D1EB6D6C62]

vpc-cni installed before the node group joined:

module.eks.aws_eks_addon.before_compute["vpc-cni"]: Creation complete after 26s [id=nexusflow:vpc-cni]

The node group has been provisioning for 30 minutes at the time of writing. EKS managed node groups provision the EC2 instances, attach EBS volumes, run the bootstrap script, and wait for the kubelet to register. On a fresh account with no warm AMI cache, 25–30 minutes is expected.

still running
module.eks.module.eks_managed_node_group["nexusflow"].aws_eks_node_group.this[0]: Still creating... [30m1s elapsed]

Where this sits

ResourceStatus
KMS CMK (alias/nexusflow-cmk)Created
IAM OIDC identity providerCreated
IRSA role for EBS CSI driverCreated
AmazonEBSCSIDriverPolicy attachedCreated
KMS grant on IRSA roleCreated
nexusflow/app ECR repository (KMS-encrypted)Created
Cluster access entry → AmazonEKSClusterAdminPolicyCreated
VPC, subnets, NAT gatewayCreated
EKS cluster nexusflow (v1.31)Created
vpc-cni addonCreated
Node group nexusflow-nodesProvisioning — 30+ min
kube-proxy, coredns, aws-ebs-csi-driver addonsPending node group
ecr_repository_url, kms_key_arn, oidc_provider_arn outputsPending apply finish

The node group is still provisioning.