The Helm chart exists. The cluster ran out of CPU. The worker is crashing and minikube will not restart.
030 ended with the migration Job proven and the full pipeline running clean. Today the goal was to convert the raw kustomize manifests into a Helm chart, install it, and verify the same stack runs under Helm. The chart came together in one session — Chart.yaml, values.yaml, eleven template files, a values-minikube.yaml override. The first install failed because the nexusflow namespace carried kustomize labels and Helm refused to adopt it. Deleted the namespace. The second install succeeded and immediately timed out waiting for the db-migrate hook — postgres was still initialising when the five-minute default expired. Three more attempts to stabilise the upgrade hit the Kubernetes Job immutability constraint: once a Job exists, its spec.template cannot be patched. Every helm upgrade call failed until the Job was manually deleted before each attempt. The real blocker underneath all of it was CPU: five replicas of simple-model-api and the Prometheus stack were consuming all four allocatable CPUs on the node. postgres-0 sat Pending for 43 minutes. Scaling simple-model-api to zero and deleting postgres-0 manually let the StatefulSet recreate the pod with the updated spec. postgres came up. redis came up. The migration Job completed. Then the worker entered CrashLoopBackOff. The cluster became unresponsive before the logs could be read. minikube stop ran clean. minikube start failed — MINIKUBE_ACTIVE_DOCKERD=minikube was still set from the earlier eval, and minikube could not resolve the driver on restart. Stopped there.
Transmission 030 ended with the full pipeline proven: POST, worker, postgres write, GET with complete JSON.
Today was the Helm conversion. The chart runs. The worker does not.
The constraint
The raw kustomize manifests in k8s/base/ work. The goal was to produce a Helm chart in charts/nexusflow/ that renders the same cluster, parameterised through values.yaml, and installs cleanly with helm install nexusflow ./charts/nexusflow.
Eleven template files: namespace, configmap, secret, gateway deployment and service, worker deployment, postgres StatefulSet with its init-SQL ConfigMap and two services, redis StatefulSet with its config ConfigMap and two services, the db-migrate Job, and NOTES.txt. All values extracted into values.yaml — replica counts, image tags, storage sizes, resource requests, probe timings, service type, storage class.
The install did not go cleanly.
The proof
Blocker 1: the namespace carried kustomize labels
helm install nexusflow ./charts/nexusflow
Error: INSTALLATION FAILED: unable to continue with install: Namespace "nexusflow" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: key "app.kubernetes.io/managed-by" must equal "Helm": current value is "kustomize"
The nexusflow namespace was created by kubectl apply -k k8s/overlays/local in a previous session. It has app.kubernetes.io/managed-by: kustomize. Helm requires that label to read Helm and requires two annotations — meta.helm.sh/release-name and meta.helm.sh/release-namespace — before it will manage a resource it did not create.
Two options: patch the namespace annotations and labels so Helm can adopt it, or delete the namespace and let Helm create it from scratch. Deleted the namespace. A patch would have left every other kustomize-owned resource in the namespace with the same conflict.
kubectl delete namespace nexusflow
helm install nexusflow ./charts/nexusflow
Blocker 2: the db-migrate hook timed out
The chart had the migration Job annotated as a post-install hook. Helm creates all chart resources, then runs the hook, then waits for it to finish. The default wait is five minutes.
Error: INSTALLATION FAILED: failed post-install: resource Job/nexusflow/db-migrate not ready. status: InProgress, message: Job in progress
context deadline exceeded
The hook’s init container polls nc -z postgres 5432 in a loop. postgres-0 was still a StatefulSet pod waiting on PVC provisioning. The Job init container kept looping. Five minutes passed.
The hook annotation was removed. The Job became a plain resource deployed alongside everything else. That fixed the timeout — but introduced the next problem on the first helm upgrade.
Blocker 3: Job spec.template is immutable
helm upgrade nexusflow ./charts/nexusflow -f charts/nexusflow/values-minikube.yaml
Error: UPGRADE FAILED: server-side apply failed for object nexusflow/db-migrate batch/v1, Kind=Job: Job.batch "db-migrate" is invalid: spec.template: Invalid value: field is immutable
Kubernetes does not allow patching a Job’s pod template after creation. helm upgrade tries to apply the new template via server-side apply. The API server rejects it. The only ways around this: delete the Job before every upgrade, or use a Helm hook with before-hook-creation delete policy, which tells Helm to delete the previous Job instance before creating the new one.
The hook annotations went back on the Job:
annotations:
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
before-hook-creation means Helm deletes the existing Job before rendering the new one — no immutability conflict. hook-succeeded means the Job pod is deleted after it completes, alongside the ttlSecondsAfterFinished: 300 on the spec itself.
The wait-for-migration init containers in gateway and worker also had to come out. A hook runs after all chart resources are applied — gateway and worker pods exist at that point, and their init containers would poll for a Job that does not exist yet, deadlocking the deployment.
Blocker 4: the node had no CPU left
With the hook restored and the upgrade running with --timeout 15m, postgres-0 stayed Pending.
kubectl get events -n nexusflow --field-selector=reason=FailedScheduling
Warning FailedScheduling pod/postgres-0 0/1 nodes are available: 1 Insufficient cpu. no new claims to deallocate, preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
The node has four allocatable CPUs. Listing everything running:
default prometheus-grafana-7dc9bbf8cf-lsw9t 3/3 Running 35 79d
default prometheus-prometheus-kube-prometheus-prometheus-0 2/2 Running 22 81d
default simple-model-api-deployment-5bc668d8f8-7f5sw 1/1 Running 4 5d
default simple-model-api-deployment-5bc668d8f8-c5vqz 1/1 Running 4 5d
default simple-model-api-deployment-5bc668d8f8-l69hb 1/1 Running 0 2d
default simple-model-api-deployment-5bc668d8f8-snmsl 1/1 Running 6 14d
default simple-model-api-deployment-5bc668d8f8-tjcgr 1/1 Running 5 6d
...
Resource Requests Limits
cpu 4 (100%) 12200m (305%)
Five replicas of simple-model-api and the Prometheus stack consumed the entire CPU request budget. NexusFlow’s pods — even at 50m each — could not be scheduled.
values-minikube.yaml was updated to remove CPU requests entirely, leaving only memory requests. Pods with no CPU request go into the BestEffort QoS class and bypass the scheduler’s CPU accounting. This is acceptable for a dev node that is already overcommitted at the limit level.
kubectl scale deployment simple-model-api-deployment --replicas=0 -n default
That freed roughly 1000m of request headroom. postgres-0 was still Pending because StatefulSets do not recreate a pod that never became healthy — the old spec with CPU requests was still attached to the existing pod object. Deleting postgres-0 manually forced the StatefulSet controller to recreate it with the updated template:
postgres-0 0/1 ContainerCreating 0 0s
postgres-0 0/1 Running 0 9s
postgres-0 1/1 Running 0 20s
redis-0 1/1 Running 0 11m
db-migrate 0/1 PodInitializing 0 13m
db-migrate 0/1 Completed 0 13m
worker 1/1 Running 0 13m
postgres up. redis up. Migration completed. Worker reached Running.
Blocker 5: the worker entered CrashLoopBackOff
worker-d77698d99-l7tvk 1/1 Running 0 13m
worker-d77698d99-l7tvk 0/1 Error 0 13m
worker-d77698d99-l7tvk 1/1 Running 1 (5s ago) 13m
worker-d77698d99-l7tvk 0/1 Error 1 (35s ago) 14m
worker-d77698d99-l7tvk 0/1 CrashLoopBackOff 2 (8s ago) 15m
The worker started, ran briefly, then crashed. The cycle repeated five times. The cause is not the liveness probe — that was fixed in 029, and the Helm template carries the same kill -0 1 probe. The actual crash reason is unknown. The cluster became unresponsive before kubectl logs could return anything useful — etcdserver: request timed out on a simple pod delete.
minikube would not restart
minikube stop && minikube start
✋ Stopping node "minikube" ...
🛑 1 node stopped.
❌ Exiting due to DRV_UNSUPPORTED_OS: The driver '' is not supported on windows/amd64
eval $(minikube docker-env) from the earlier build session had set MINIKUBE_ACTIVE_DOCKERD=minikube in the shell environment. After the stop, minikube tried to start using the driver string from that variable — which was now empty or malformed. The driver field resolved to an empty string. Windows/amd64 does not support an empty driver.
The fix is to run eval $(minikube docker-env -u) to unset those variables before stopping, or open a fresh shell before running minikube start. That is tomorrow’s first command.
I stopped the cluster and walked away.
Where this sits
| Item | Status |
|---|---|
| Helm chart — all eleven templates | Done |
values.yaml — all parameters extracted | Done |
values-minikube.yaml — no CPU requests, single replicas | Done |
| Namespace ownership conflict — deleted and recreated under Helm | Done |
db-migrate hook — before-hook-creation delete policy | Done |
wait-for-migration init containers removed from gateway and worker | Done |
simple-model-api scaled to 0 to free CPU headroom | Done |
| postgres-0 manually deleted to pick up updated pod spec | Done |
| postgres, redis, migration Job — all reaching healthy state | Done |
| Worker crash cause | Unknown |
minikube start after stop | Broken — MINIKUBE_ACTIVE_DOCKERD env var |
| Gateway pod status at end of session | Init:0/2 — did not reach Running |
Tomorrow: open a fresh shell, start minikube, read the worker logs.