I am writing this one because not writing it would be a different kind of giving up.

037 ended with the cluster and node group existing in AWS and missing from state. The goal today was simple: import those two resources, run the remaining plan, get nodes running.

Six applies. One destroy. Nothing is running.

The constraint

Each apply waited 30 minutes before reporting failure. Each error was different from the last. Three of them were the same error repeated — the same 32 minutes, the same message, three times in a row — while the cause was still being understood.

This took the entire day.

The proof

Import: clean

The stale DynamoDB lock cleared with tofu force-unlock. The node group import failed once with the wrong separator (/ instead of :), then succeeded:

tofu import 'module.eks.module.eks_managed_node_group["nexusflow"].aws_eks_node_group.this[0]' \
  nexusflow:nexusflow-nodes-20260915082737288500000006

Import successful!

Plan: six adds, twelve destroys. Applied.

Applies 1, 2, and 3: the same error three times

AsgInstanceLaunchFailures: Could not launch On-Demand Instances.
InvalidParameterCombination - The specified instance type is not eligible
for Free Tier.

The instance type is t3.medium. It has nothing to do with Free Tier. The real problem was in main.tf: disk_size was set directly on the node group while the EKS module was simultaneously creating a launch template. Those two disk configurations cannot coexist. AWS wraps the conflict in a Free Tier message because the validation fails at the ASG launch step, not at the plan step.

It took three failed applies — each one 32 minutes long — before this was clear. After the third I grepped the entire codebase for t3.medium references to confirm the instance type was correct, then wrote a terraform.tfvars with t3.micro to rule out the Free Tier message being literal:

echo 'node_instance_type = "t3.micro"' > terraform.tfvars

The fix was to replace disk_size with block_device_mappings:

block_device_mappings = {
  xvda = {
    device_name = "/dev/xvda"
    ebs = {
      volume_size           = var.node_disk_size_gb
      volume_type           = "gp3"
      encrypted             = true
      kms_key_id            = aws_kms_key.nexusflow.arn
      delete_on_termination = true
    }
  }
}

Applied again. Apply 4.

Apply 4: KMS key in incorrect state

Waited 33 minutes.

Client.InvalidKMSKey.InvalidState: The KMS key provided is in an
incorrect state

The key is enabled:

aws kms describe-key --key-id a14873f4-0b4c-4079-8dac-f52a79924ba1 \
  --query "KeyMetadata.[KeyId, KeyState, Enabled]"

["a14873f4-0b4c-4079-8dac-f52a79924ba1", "Enabled", true]

InvalidState here does not mean disabled. It means AWSServiceRoleForAutoScaling — the role that actually creates EBS volume grants when an ASG launches instances — is not in the KMS key policy. The key policy had ec2.amazonaws.com. It was missing the Auto Scaling service-linked role entirely. No grant, no volume, unhealthy instance.

Two statements added to phase1_security.tf:

statement {
  sid    = "AllowAutoScalingSLRUsage"
  principals {
    type        = "AWS"
    identifiers = [
      "arn:aws:iam::${local.account_id}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
    ]
  }
  actions = ["kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*",
             "kms:GenerateDataKey*", "kms:DescribeKey"]
}

statement {
  sid    = "AllowAutoScalingSLRCreateGrant"
  principals {
    type        = "AWS"
    identifiers = [
      "arn:aws:iam::${local.account_id}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
    ]
  }
  actions = ["kms:CreateGrant"]
  condition {
    test     = "Bool"
    variable = "kms:GrantIsForAWSResource"
    values   = ["true"]
  }
}

Applied again. Apply 5.

Apply 5: dependency cycle

Failed immediately — no 30-minute wait.

Error: Cycle: data.aws_iam_policy_document.kms_key_policy,
aws_kms_key.nexusflow,
module.eks.var.eks_managed_node_groups (expand, input),
module.eks.output.eks_managed_node_groups (expand),
data.aws_iam_policy_document.kms_key_policy

I had added the node IAM role ARN — module.eks.eks_managed_node_groups["nexusflow"].iam_role_arn — to the key policy data source. That reference created a loop: the key policy depends on the EKS module output, the EKS module depends on the KMS key ARN passed through block_device_mappings, and the KMS key depends on the key policy. Closed circle.

The node role does not need to be in the key policy. The kms:CreateGrant call made by AWSServiceRoleForAutoScaling creates a per-instance ephemeral grant — that grant is what gives the instance access to decrypt its own root volume. Removing the node role statement breaks the cycle without changing encryption behaviour.

Applied again. Apply 6.

Apply 6: the node group came up

The node group reached ACTIVE. Then the apply moved on to the addons and waited 20 minutes:

Error: waiting for EKS Add-On (nexusflow:coredns) create: timeout while
waiting for state to become 'ACTIVE' (last state: 'CREATING', timeout: 20m0s)

Error: waiting for EKS Add-On (nexusflow:aws-ebs-csi-driver) create: timeout
while waiting for state to become 'ACTIVE' (last state: 'DEGRADED', timeout: 20m0s)

CoreDNS needs healthy nodes to schedule its pods on. The EBS CSI driver went DEGRADED. Both timed out at the 20-minute mark. The node group itself was ACTIVE in AWS — the instances were up — but something was still wrong at the networking or bootstrap layer.

I ran destroy.

Where this sits

The launch template conflict is fixed. The KMS key policy is fixed. The dependency cycle is fixed. The node group reached ACTIVE once. The addons did not.

Why the addons degraded is the open question tomorrow.