Transmission 031 ended with one sentence: tomorrow, open a fresh shell, start minikube, read the worker logs.

Today: opened a fresh shell. Six things were wrong before a GET request came back clean.

The constraint

031 left the cluster in this state: gateway at Init:0/2, worker in CrashLoopBackOff after nine restarts, minikube broken by a stale MINIKUBE_ACTIVE_DOCKERD environment variable left from an earlier eval $(minikube docker-env) call. The cause of the worker crash was unknown — the cluster became unresponsive before logs could be read.

The goal today was to diagnose both the init container blockage and the worker crash, then get every pod to 1/1 Running.

The proof

Fix 1: minikube and the stale env var

031 documented the fix. A fresh PowerShell terminal has no MINIKUBE_ACTIVE_DOCKERD set. minikube started clean.

minikube start
kubectl get nodes

NAME       STATUS   ROLES           AGE   VERSION
minikube   Ready    control-plane   98d   v1.35.1

Scaled simple-model-api to zero to free CPU headroom, same as 031.

kubectl scale deployment simple-model-api-deployment --replicas=0 -n default
kubectl get pods -n nexusflow

NAME                       READY   STATUS             RESTARTS        AGE
gateway-658f489f68-72nd8   0/1     Init:0/2           0               44h
worker-d77698d99-l7tvk     0/1     CrashLoopBackOff   9 (13s ago)     44h
postgres-0                 1/1     Running            2 (4m57s ago)   43h
redis-0                    1/1     Running            1 (10m ago)     44h

The gateway pods did not move. Still Init:0/2.

Blocker 1: the init containers never pulled

kubectl describe pod -l app.kubernetes.io/name=gateway -n nexusflow

Both busybox init containers showed Container ID: empty. Not a connectivity failure — a connectivity failure shows a running container that cannot reach the endpoint. An empty Container ID means the container never started. The image was not in the minikube Docker daemon.

busybox:1.36 and bitnami/kubectl:latest had never been loaded into the minikube daemon. The node cannot reach Docker Hub from inside the cluster. Both had to be pulled on the host and loaded manually.

docker pull busybox:1.36
minikube image load busybox:1.36
docker pull bitnami/kubectl:latest
minikube image load bitnami/kubectl:latest

docker.io/bitnami/kubectl:latest
docker.io/library/busybox:1.36
docker.io/library/nexusflow-gateway:1.0.0
docker.io/library/nexusflow-worker:1.0.0

Both app images were already present from a previous session. The init container images were not.

Blocker 2: wait-for-migration was missing from both Helm templates

With busybox available, the worker init containers started. The worker entered Init:2/3 — two passed, one stalled. That third container is wait-for-migration. It exists in _helpers.tpl and in both kustomize base manifests. It was never included in either Helm deployment template.

# charts/nexusflow/templates/worker-deployment.yaml — before
initContainers:
  {{- include "nexusflow.initWaitForPostgres" . | nindent 8 }}
  {{- include "nexusflow.initWaitForRedis" . | nindent 8 }}
  # wait-for-migration was missing

The kustomize path and the Helm path had diverged silently. The helper was defined and ready; the include line was just never written.

# after
initContainers:
  {{- include "nexusflow.initWaitForPostgres" . | nindent 8 }}
  {{- include "nexusflow.initWaitForRedis" . | nindent 8 }}
  {{- include "nexusflow.initWaitForMigration" . | nindent 8 }}

Added to both gateway and worker templates.

Blocker 3: the gateway-db-patch ConfigMap did not exist

Adding the migration init container to the gateway template surfaced the next failure. The gateway pod mounts gateway-db-patch as a volume over /app/db.py. That ConfigMap was created by hand in session 030 and was never added to any Helm template. Under Helm management, it did not exist.

Warning  FailedMount  kubelet  MountVolume.SetUp failed for volume "db-patch":
configmap "gateway-db-patch" not found

Created it immediately to unblock the pod:

kubectl create configmap gateway-db-patch --from-file=db.py=gateway/db.py -n nexusflow

Then added it as a permanent Helm template. Helm reads files inside the chart directory via .Files.Get. Copied gateway/db.py to charts/nexusflow/files/gateway-db.py and wrote the template:

# charts/nexusflow/templates/gateway-db-patch.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: gateway-db-patch
  namespace: {{ include "nexusflow.namespace" . }}
data:
  db.py: |
{{ .Files.Get "files/gateway-db.py" | indent 4 }}

The ConfigMap is now created on every helm upgrade and will never be absent again.

Blocker 4: wait-for-migration loops forever when the job is TTL-deleted

The worker reached Init:2/3 and stayed there. The wait-for-migration container was running but not exiting. The original script:

until kubectl get job db-migrate -n nexusflow \
      -o jsonpath='{.status.succeeded}' 2>/dev/null | grep -q "1"; do
  echo "Migration not yet complete, retrying in 3s..."; sleep 3;
done

The db-migrate Job completes in under two minutes. ttlSecondsAfterFinished: 300 deletes it five minutes after completion. By the time bitnami/kubectl pulls, starts, and begins polling, the Job is already gone. kubectl get job db-migrate returns nothing. grep -q "1" never matches. The loop runs forever.

I considered raising the TTL from 300s to 1800s so the container would have more time to poll before the Job disappeared. That just delays the same race — any finite TTL creates a window where this script fails. Treating absence as completion is the correct fix: if the Job no longer exists, it either succeeded and was cleaned up, or was never run. Both cases mean proceeding is safe.

until
  STATUS=$(kubectl get job db-migrate -n nexusflow \
             -o jsonpath='{.status.succeeded}' 2>/dev/null);
  [ "$STATUS" = "1" ] || \
  ! kubectl get job db-migrate -n nexusflow > /dev/null 2>&1;
do
  echo "Migration not yet complete, retrying in 3s..."; sleep 3;
done

Applied to _helpers.tpl, k8s/base/gateway-deployment.yaml, and k8s/base/worker-deployment.yaml.

Blocker 5: Helm could not install — wrong ownership annotations on existing resources

helm upgrade --install nexusflow ./charts/nexusflow \
  -f charts/nexusflow/values-minikube.yaml -n nexusflow

Error: unable to continue with install: Secret "nexusflow-secrets" in namespace "nexusflow"
exists and cannot be imported into the current release: invalid ownership metadata;
annotation validation error: key "meta.helm.sh/release-namespace" must equal "nexusflow":
current value is "default"

The existing resources carried meta.helm.sh/release-namespace: default from an earlier partial Helm attempt. In 031, the fix was to delete the namespace entirely and let Helm recreate it. That is not viable here — postgres and redis PVCs hold real data. Deleting the namespace deletes the PVCs.

Patched all 13 resources with the correct Helm ownership annotations instead:

kubectl annotate secret/nexusflow-secrets deployment/gateway deployment/worker \
  statefulset/postgres statefulset/redis \
  service/gateway service/postgres service/postgres-headless \
  service/redis service/redis-headless \
  configmap/nexusflow-config configmap/postgres-init-sql configmap/redis-config \
  -n nexusflow \
  meta.helm.sh/release-name=nexusflow \
  meta.helm.sh/release-namespace=nexusflow --overwrite

After adoption, helm upgrade --install succeeded. STATUS: deployed, REVISION: 1.

Blocker 6: Helm overwrote the postgres password

The gateway entered CrashLoopBackOff immediately after reaching Running:

asyncpg.exceptions.InvalidPasswordError: password authentication failed for user "nexus"

values.yaml sets secrets.postgresPassword: change-me-in-production. values-minikube.yaml had no secrets block. Helm rendered the secret with the placeholder and applied it, overwriting nexusflow-secrets — which held localpassword123, the password postgres was actually initialised with.

Added the correct credentials to values-minikube.yaml:

secrets:
  createSecret: true
  postgresUser: nexus
  postgresPassword: localpassword123
  databaseUrl: postgresql://nexus:localpassword123@postgres:5432/nexusflow
  redisUrl: redis://redis:6379/0

Upgraded. REVISION: 3.

Fig 1: Swagger UI confirming all three endpoints live at /docs with schemas, Idempotency-Key header, and 202/200 status codes.

The resolution

kubectl rollout status deployment/gateway deployment/worker -n nexusflow

deployment "gateway" successfully rolled out
deployment "worker" successfully rolled out

NAME                       READY   STATUS    RESTARTS   AGE
gateway-557cf49778-cdb87   1/1     Running   0          50s
postgres-0                 1/1     Running   2          44h
redis-0                    1/1     Running   1          45h
worker-6797777459-m6qsx    1/1     Running   0          51s

kubectl port-forward svc/gateway 8000:80 -n nexusflow

{"service": "gateway", "status": "ok", "timestamp": "2026-09-07T08:10:52.143358+00:00"}

{
  "task_id": "422572b6-6a6b-4667-92ab-2c0809fe82e0",
  "name": "smoke-test",
  "status": "accepted",
  "accepted_at": "2026-09-07T08:11:37.619522+00:00",
  "idempotency_key": null
}

{
  "task_id": "422572b6-6a6b-4667-92ab-2c0809fe82e0",
  "name": "smoke-test",
  "status": "completed",
  "payload": {"hello": "world"},
  "result": {"name": "smoke-test", "processed": true},
  "retry_count": 0,
  "max_retries": 3,
  "error_message": null,
  "created_at": "2026-09-07T08:11:37.619522+00:00",
  "updated_at": "2026-09-07T08:11:39.030929+00:00"
}

Worker log at the moment the task moved through:

2026-09-07T08:11:38Z [INFO] nexusflow.worker — Picked up task_id=422572b6-6a6b-4667-92ab-2c0809fe82e0  attempt=1/4
2026-09-07T08:11:38Z [INFO] nexusflow.worker — Processing task_id=422572b6-6a6b-4667-92ab-2c0809fe82e0  name=smoke-test
2026-09-07T08:11:39Z [INFO] nexusflow.worker — Task completed  task_id=422572b6-6a6b-4667-92ab-2c0809fe82e0

Idempotency confirmed: same Idempotency-Key header, first call returned 202, second returned 200.

Where this sits

The Helm chart now manages the full stack cleanly. Every pod reaches Running without manual intervention.