Transmission 027 ended with one sentence: the next layer is Kubernetes. Today was that layer.

The cluster came up. Postgres did not.

The constraint

027 left the Docker Compose stack fully hardened and a complete Kustomize manifest tree written: base layer with namespace, configmaps, secrets template, postgres StatefulSet, redis StatefulSet, gateway Deployment, worker Deployment, and two overlays — local and eks. The local overlay patched replica counts down to one, set the storageClass to standard, and generated a dev secret inline.

The first kubectl apply -k k8s/overlays/local failed before any resource was created. The secret generator was looking up the base secret with an empty namespace field and could not find anything to replace.

That took two iterations to fix. First: bases: renamed to resources:. Second: behavior: replace added to the secretGenerator block. Third: namespace: nexusflow added inside the generator item itself — without it, Kustomize constructs the ResId with Namespace:"" regardless of the top-level namespace field, and the lookup fails.

After those fixes the apply went through.

The proof

The cluster came up

namespace/nexusflow created
configmap/nexusflow-config created
configmap/postgres-init-sql created
configmap/redis-config created
secret/nexusflow-secrets created
service/gateway created
service/postgres created
service/postgres-headless created
service/redis created
service/redis-headless created
deployment.apps/gateway created
deployment.apps/worker created
statefulset.apps/postgres created
statefulset.apps/redis created

Everything created. Then:

NAME                       READY   STATUS     RESTARTS   AGE
gateway-58767cc5f8-rhrxx   0/1     Init:0/2   0          42s
postgres-0                 0/1     Pending    0          40s
redis-0                    0/1     Pending    0          38s
worker-5d8df68784-kbzjx    0/1     Init:0/2   0          41s

postgres-0 and redis-0 stuck at Pending. kubectl describe pod postgres-0 gave the reason:

Warning  FailedScheduling  default-scheduler  0/1 nodes are available:
  1 Insufficient cpu. no new claims to deallocate

The node was at 98% CPU requested. Five replicas of a model API deployment were sitting idle in the default namespace, each requesting 500m CPU. Scaling that deployment to one replica freed enough headroom.

kubectl scale deployment simple-model-api-deployment --replicas=1 -n default

Both StatefulSets scheduled. Redis came up clean. Postgres did not.

The postgres crash loop

chmod: /var/run/postgresql: Operation not permitted

PostgreSQL Database directory appears to contain a database; Skipping initialization

2026-08-30 08:00:58.923 UTC [1] LOG:  starting PostgreSQL 16.15 ...
2026-08-30 08:00:59.077 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2026-08-30 08:00:59.468 UTC [1] LOG:  database system is ready to accept connections
2026-08-30 08:02:31.160 UTC [1] LOG:  received fast shutdown request

The pod started. Postgres initialised. Then the liveness probe killed it.

The probe failure message:

Warning  Unhealthy  kubelet  Liveness probe failed: /var/run/postgresql:5432 - no attempt

The chmod error appeared on every start and the probe never passed.

What was actually happening

The postgres StatefulSet had this in the container securityContext:

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: false
  runAsUser: 999    # postgres uid
  runAsGroup: 999

And this at the pod level:

securityContext:
  fsGroup: 999
  runAsNonRoot: true

The postgres:16-alpine image entrypoint is a shell script. On every start it calls chmod 755 /var/run/postgresql and chown postgres /var/run/postgresql before handing control to the postgres process. That step runs as root. After it completes, the entrypoint uses gosu to drop from root to uid 999 and exec into postgres. The image handles its own privilege drop.

Setting runAsUser: 999 at the Kubernetes level means the entrypoint script starts as uid 999 from the first instruction. The chmod call fails immediately because /var/run/postgresql is owned by root and uid 999 cannot change its permissions. The postgres process still starts — it does not abort on that error — but the Unix socket never gets created at the expected path with the expected permissions. Every pg_isready call then fails because it tries to connect via that socket and finds nothing.

Setting runAsNonRoot: true at the pod level compounds it: Kubernetes itself rejects any process that tries to start as root, which blocks the entrypoint’s entire design.

The fix was removing both lines:

# pod securityContext — before
securityContext:
  fsGroup: 999
  runAsNonRoot: true  # ← removed

# container securityContext — before
securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: false
  runAsUser: 999    # ← removed
  runAsGroup: 999   # ← removed

fsGroup: 999 stays. It sets the GID on the PVC mount so the postgres process can read and write its data directory after the privilege drop. That is what it is for.

The probe also needed fixing. pg_isready without -h connects via the Unix socket at /var/run/postgresql/.s.PGSQL.5432. That socket path was the broken one. Adding -h localhost forces a TCP connection to port 5432 and bypasses the socket entirely:

livenessProbe:
  exec:
    command:
      - pg_isready
      - -h
      - localhost
      - -p
      - "5432"
      - -U
      - $(POSTGRES_USER)
      - -d
      - $(POSTGRES_DB)

Same change applied to the readinessProbe.

Where the session stopped

The manifest fixes are in. The PVC from the earlier failed apply still exists on the cluster with the old data directory. A fresh apply without deleting that PVC will give postgres a data directory that was written by the old (crashing) pod under different permissions. That needs to go before the next apply.

kubectl delete -k k8s/overlays/local
kubectl delete pvc postgres-data-postgres-0 redis-data-redis-0 -n nexusflow

Then reapply. The session ended before that ran.

Where this sits

The security context was right for the gateway and worker — both use images built to run as a non-root user from the first instruction. It was wrong for postgres, which is built to start as root, set up the socket directory, and drop privileges itself. Tomorrow the PVC gets deleted and the apply runs again.