Transmission 028 ended with one sentence: the old PVC needs to go before the next apply.

The apply ran today. Two things broke before the cluster was clean.

The constraint

028 left the Kubernetes manifests in a working state — security context fixed on postgres, probes corrected to use TCP — but ended before the cluster could be brought up clean. The stale PVC from the earlier failed session was still on the node. Running a fresh apply on top of it would give postgres a data directory written by the old crashing pod under the wrong uid.

The first step was cleanup:

kubectl delete -k k8s/overlays/local --ignore-not-found=true
kubectl delete pvc postgres-data-postgres-0 redis-data-redis-0 -n nexusflow --ignore-not-found=true

Then reapply. The namespace, configmaps, secrets, services, and statefulsets all created. Redis came up. Postgres came up. Gateway and worker went into ErrImagePull.

The proof

Problem 1: images built into the wrong daemon

NAME                       READY   STATUS              RESTARTS   AGE
gateway-58767cc5f8-765cg   0/1     Init:0/2            0          18s
postgres-0                 0/1     Running             0          17s
redis-0                    0/1     ContainerCreating   0          17s
worker-5d8df68784-ld66r    0/1     Init:0/2            0          18s
redis-0                    1/1     Running             0          22s
postgres-0                 1/1     Running             0          60s
gateway-58767cc5f8-765cg   0/1     ErrImagePull        0          94s
worker-5d8df68784-ld66r    0/1     ErrImagePull        0          103s
gateway-58767cc5f8-765cg   0/1     ImagePullBackOff    0          2m7s
worker-5d8df68784-ld66r    0/1     ImagePullBackOff    0          2m15s

The init containers passed. Postgres and redis were running. Then gateway and worker hit ErrImagePull once their main containers tried to start.

imagePullPolicy: Always in the base manifests tells Kubernetes to pull from the remote registry on every pod start. nexusflow/gateway:latest and nexusflow/worker:latest do not exist on Docker Hub. They exist in minikube’s internal Docker daemon — but only if you point the Docker CLI at it with eval $(minikube docker-env) before building. The first build had not done that. The images landed in the host Docker daemon, invisible to the cluster.

eval $(minikube docker-env) ran. Both images rebuilt inside minikube. A rollout restart still produced ErrImagePull — imagePullPolicy: Always ignores locally available images and attempts the registry pull regardless. A full delete and reapply was needed. After that, the images were found locally and the pods started.

The correct fix is to stop relying on registry-pull fallback behaviour entirely and tell the cluster to use what is already there. imagePullPolicy: Never was added as a patch in the local overlay — not in the base, because the base also feeds the EKS overlay where images do come from a real registry. A base change would break production. An overlay patch keeps both environments correct:

# k8s/overlays/local/kustomization.yaml
- patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: gateway
      namespace: nexusflow
    spec:
      template:
        spec:
          containers:
            - name: gateway
              imagePullPolicy: Never
- patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: worker
      namespace: nexusflow
    spec:
      template:
        spec:
          containers:
            - name: worker
              imagePullPolicy: Never

Problem 2: the probe was killing a healthy process

After the reapply, all four pods reached Running. Then the worker started cycling.

worker-f457f5c6f-zd47l    1/1     Running             1 (27s ago)   5m54s
worker-f457f5c6f-zd47l    1/1     Running             2 (3s ago)    7m33s
worker-f457f5c6f-zd47l    1/1     Running             3 (6s ago)    9m18s
worker-f457f5c6f-zd47l    1/1     Running             4 (4s ago)    10m
worker-f457f5c6f-zd47l    1/1     Running             5 (4s ago)    12m
worker-f457f5c6f-zd47l    0/1     CrashLoopBackOff    6 (0s ago)    15m

Eight restarts. The container kept coming back as Running between each one, which rules out a startup bug. The logs made it clear the worker was not crashing:

2026-08-31T08:52:15Z [INFO] nexusflow.worker — Connected to Redis at redis:6379
2026-08-31T08:52:15Z [INFO] nexusflow.worker — Consumer group 'workers' already exists. Joining.
2026-08-31T08:52:15Z [INFO] nexusflow.worker — Worker 'worker-f457f5c6f-zd47l' starting — listening on stream 'tasks' (group: workers)
2026-08-31T08:52:16Z [INFO] nexusflow.db — PostgreSQL pool ready  host=postgres db=nexusflow  min=1 max=5
2026-08-31T08:53:51Z [INFO] nexusflow.worker — Received signal 15 — finishing current task then shutting down.
2026-08-31T08:53:52Z [INFO] nexusflow.db — PostgreSQL pool closed.
2026-08-31T08:53:52Z [INFO] nexusflow.worker — Shutdown flag set. Worker 'worker-f457f5c6f-zd47l' exiting cleanly.

Signal 15. The worker caught SIGTERM, ran its shutdown path, and exited with code 0. Kubernetes sent the signal. The liveness probe triggered it.

kubectl describe had the reason:

Warning  Unhealthy  kubelet  Liveness probe failed: OCI runtime exec failed: exec failed:
  unable to start container process: exec: "pgrep": executable file not found in $PATH

The probe in the base manifest:

livenessProbe:
  exec:
    command:
      - pgrep
      - -f
      - worker.py
  initialDelaySeconds: 20
  periodSeconds: 30
  timeoutSeconds: 5
  failureThreshold: 3

pgrep is part of the procps package. python:3.12-slim does not include it. Every probe call failed with executable file not found. After three consecutive failures — 3 × 30 seconds — Kubernetes killed the container. The worker logged the clean shutdown, restarted, ran fine for another 90 seconds, and the cycle repeated.

The fix was replacing the probe with a shell builtin that requires no additional package. The worker’s CMD is exec form — CMD ["python", "-u", "worker.py"] — which means the Python process is PID 1 in the container. kill -0 1 checks whether PID 1 is alive without sending any signal. It is a shell builtin. It cannot fail because of a missing binary:

livenessProbe:
  exec:
    command:
      - sh
      - -c
      - "kill -0 1"
  initialDelaySeconds: 20
  periodSeconds: 30
  timeoutSeconds: 5
  failureThreshold: 3

The alternative was installing procps in the Dockerfile runtime stage — valid, but it requires a Dockerfile change, an image rebuild, and a redeployment cycle. kill -0 1 requires none of that and works in any debian-based image.

The apply that fixed both

kubectl apply -k k8s/overlays/local

NAME                      READY   STATUS        RESTARTS       AGE
gateway-cc4547955-5fb6s   1/1     Running       0              26m
postgres-0                1/1     Running       0              26m
redis-0                   1/1     Running       0              26m
worker-686bbcf858-s88wb   1/1     Running       0              23s
worker-f457f5c6f-zd47l    1/1     Terminating   8 (6m7s ago)   26m

The rolling update replaced the old pod. The new one came up with the corrected probe and imagePullPolicy: Never. No image pull errors. No liveness failures. Restarts stayed at zero.

NAME                      READY   STATUS    RESTARTS   AGE
gateway-cc4547955-5fb6s   1/1     Running   0          33m
postgres-0                1/1     Running   0          33m
redis-0                   1/1     Running   0          33m
worker-686bbcf858-s88wb   1/1     Running   0          7m18s

The kubelet RBAC noise at startup

Minikube started with a wall of kubelet errors on every pod in the namespace:

Failed to get status for pod: no relationship found between node 'minikube' and this object

This is a known artefact of Kubernetes v1.35 and the Node Authorizer. On restart, the kubelet temporarily loses its cached node-pod binding for pods from the previous session. The errors clear once pods are rescheduled. They did not cause either problem and did not need intervention.

Where this sits

The cluster is clean. The next session is the first one where real traffic can go through it.