Transmission 029 ended with one sentence: the next session is the first one where real traffic can go through it.

Today was that session. Five things broke before a GET request returned clean JSON.

The constraint

029 left all four pods running with zero restarts. The cluster was clean. The intent today was simple: POST a task, watch it move through the pipeline, GET the result.

The first POST revealed the schema was never applied. Everything after that was consequence.

The proof

Blocker 1: the tasks table did not exist

curl -X POST http://localhost:8000/tasks \
  -H "Content-Type: application/json" \
  -d '{"name": "e2e_test_task", "task_type": "process_payload", "payload": {"source": "e2e_test", "data": "hello nexusflow"}}'

File "/app/main.py", line 214, in create_task
    await db.create_task(
  File "/app/db.py", line 121, in create_task
    await conn.execute(
asyncpg.exceptions.UndefinedTableError: relation "tasks" does not exist

The postgres/init.sql schema file is mounted at /docker-entrypoint-initdb.d/init.sql in the postgres pod. The official postgres image only runs scripts in that directory on the very first start, when PGDATA is empty. The PVC from 029’s session still had data in it. Postgres skipped the init script entirely on restart. The tasks table was never created.

The fix was to apply the schema manually against the running pod. Three attempts before it went through.

Blocker 2: wrong username

kubectl exec -i -n nexusflow postgres-0 -- psql -U nexusflow -d nexusflow < postgres/init.sql

psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed:
FATAL:  role "nexusflow" does not exist

The actual postgres superuser is stored in nexusflow-secrets. Decoding it:

kubectl get secret nexusflow-secrets -n nexusflow \
  -o jsonpath='{.data.POSTGRES_USER}' | base64 -d

nexus

The user is nexus, not nexusflow.

Blocker 3: Git Bash stdin redirect permission denied

kubectl exec -i -n nexusflow postgres-0 -- psql -U nexus -d nexusflow < postgres/init.sql

bash: postgres/init.sql: Permission denied

Git Bash on Windows cannot redirect local files into kubectl exec -i. The < operator tries to open the file through a path that kubectl cannot access in this context. The workaround is to copy the file into the pod first and run it with -f.

Blocker 4: Git Bash path mangling on kubectl cp

kubectl cp postgres/init.sql nexusflow/postgres-0:/tmp/init.sql

Error from server (NotFound): pods "nexusflow\\postgres-0;C" not found

Git Bash sees nexusflow/postgres-0:/tmp/init.sql and interprets the colon as a Windows drive letter separator, mangling the pod path. MSYS_NO_PATHCONV=1 disables that conversion:

MSYS_NO_PATHCONV=1 kubectl cp postgres/init.sql nexusflow/postgres-0:/tmp/init.sql
MSYS_NO_PATHCONV=1 kubectl exec -n nexusflow postgres-0 -- psql -U nexus -d nexusflow -f /tmp/init.sql

CREATE TABLE
CREATE INDEX
CREATE INDEX

kubectl exec -n nexusflow postgres-0 -- psql -U nexus -d nexusflow -c "\dt"

       List of relations
 Schema | Name  | Type  | Owner
--------+-------+-------+-------
 public | tasks | table | nexus
(1 row)

The table exists. The POST went through:

{
  "task_id": "cc9851c4-de61-4a66-b705-827aaa4419f3",
  "name": "e2e_test_task",
  "status": "accepted",
  "accepted_at": "2026-09-03T07:30:17.669348+00:00",
  "idempotency_key": null
}

Blocker 5: JSONB deserialization — the comment in the code was wrong

curl http://localhost:8000/tasks/cc9851c4-de61-4a66-b705-827aaa4419f3

pydantic_core._pydantic_core.ValidationError: 2 validation errors for TaskStatusResponse
payload
  Input should be a valid dictionary [type=dict_type,
  input_value='{"data": "hello nexusflo..., "source": "e2e_test"}',
  input_type=str]
result
  Input should be a valid dictionary [type=dict_type,
  input_value='{"name": "e2e_test_task", "processed": true}',
  input_type=str]

The fetch_task function in db.py returned payload and result directly from the asyncpg row. The comment on that line read:

"payload": row["payload"],   # already a dict — asyncpg deserialises JSONB

That comment is wrong. asyncpg returns JSONB columns as raw JSON strings, not Python dicts. Pydantic’s TaskStatusResponse declared both fields as dict | None and rejected the strings.

The fix:

"payload": json.loads(row["payload"]) if isinstance(row["payload"], str) else row["payload"],
"result":  json.loads(row["result"])  if isinstance(row["result"],  str) else row["result"],

The isinstance guard handles both cases — asyncpg behaviour could differ across driver versions, and a bare json.loads(None) would raise.

The deployment had the wrong image name the whole time

After the fix, the gateway was rebuilt and loaded into minikube:

docker build -t nexusflow-gateway:latest ./gateway
minikube image load nexusflow-gateway:latest
kubectl rollout restart deployment/gateway -n nexusflow

The GET still returned 500. Checking what was actually running in the pod:

kubectl exec -n nexusflow deployment/gateway -- grep -n "json.loads" /app/db.py

No output. The old unpatched file was in the container. The base manifest had:

image: nexusflow/gateway:latest
imagePullPolicy: Never

The image was built as nexusflow-gateway:latest — dash, not slash. The cluster was running whatever it had cached under nexusflow/gateway:latest, which was the original unpatched version. The fix to db.py never landed in the running pod.

Two changes to the base manifest: image name corrected to nexusflow-gateway:latest, imagePullPolicy changed to IfNotPresent. The patched db.py was also mounted via ConfigMap over /app/db.py to get the fix live without waiting for another full image rebuild cycle. readOnlyRootFilesystem: true blocked direct writes into the container, so the ConfigMap mount was the only path.

After redeployment:

kubectl exec -n nexusflow deployment/gateway -- grep -n "json.loads" /app/db.py

232:  "payload": json.loads(row["payload"]) if isinstance(row["payload"], str) else row["payload"],
233:  "result":  json.loads(row["result"])  if isinstance(row["result"],  str) else row["result"],

{
  "task_id": "cc9851c4-de61-4a66-b705-827aaa4419f3",
  "name": "e2e_test_task",
  "status": "completed",
  "payload": {"data": "hello nexusflow", "source": "e2e_test"},
  "result": {"name": "e2e_test_task", "processed": true},
  "retry_count": 0,
  "max_retries": 3,
  "error_message": null,
  "created_at": "2026-09-03T07:30:17.669348+00:00",
  "updated_at": "2026-09-03T07:30:18.183711+00:00"
}

status: completed. The worker picked up the task from the Redis stream, processed it, and wrote the result back to postgres — all while the five blockers were being worked through.

Making the schema permanent: the migration Job

The manual psql fix works once. It does not survive a minikube delete or any reset that wipes the PVC. A one-shot Kubernetes Job now runs on every kubectl apply — idempotent because init.sql uses CREATE TABLE IF NOT EXISTS and CREATE INDEX IF NOT EXISTS throughout. The alternative was relying on /docker-entrypoint-initdb.d/ and accepting that it silently does nothing if PGDATA already exists. That is what caused today’s session.

# k8s/base/db-migrate-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  namespace: nexusflow
spec:
  backoffLimit: 3
  ttlSecondsAfterFinished: 300
  template:
    spec:
      restartPolicy: OnFailure
      initContainers:
        - name: wait-for-postgres
          image: busybox:1.36
          command: [sh, -c, "until nc -z postgres 5432; do sleep 2; done"]
      containers:
        - name: migrate
          image: postgres:16-alpine
          command: [sh, -c, "psql $DATABASE_URL -f /sql/init.sql"]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: nexusflow-secrets
                  key: DATABASE_URL
          volumeMounts:
            - name: init-sql
              mountPath: /sql/init.sql
              subPath: init.sql
      volumes:
        - name: init-sql
          configMap:
            name: postgres-init-sql

ttlSecondsAfterFinished: 300 cleans up the pod automatically after five minutes. The gateway and worker now have a wait-for-migration init container that polls until job/db-migrate reaches status.succeeded == 1 before the application process starts.

Applied against the live cluster:

NAME         STATUS                COMPLETIONS   DURATION   AGE
db-migrate   Running               0/1           23s        24s
db-migrate   SuccessCriteriaMet    0/1           2m30s      2m31s
db-migrate   Complete              1/1           2m30s      2m32s

The pod log was gone by the time kubectl logs ran — ttlSecondsAfterFinished had already cleaned it up. That is the correct behaviour.

Where this sits

The full pipeline ran clean. The schema problem cannot recur.