Transmissions
redis-py pipeline type hints: a 17-month-old open issue and a direct conversation with the core team
033 ended with the EKS config validated and the Helm chart linted, waiting on AWS credentials. Today was different: no NexusFlow, no Kubernetes. I opened a real open-source issue on redis/redis-py, diagnosed why method chaining on Pipeline fails mypy, hit the same architectural wall the maintainer hit when she researched it a year ago, and started a direct conversation with the Redis core team on GitHub. The PR is not open yet. The maintainer is reviewing the trade-off. The branch exists. The code passes mypy, ruff, and vulture.
Continue reading →The cluster was clean. OpenTofu installed. The Terraform config validated and the Helm chart linted.
032 ended with every pod at 1/1 Running and the full pipeline clean. Today the goal was to build the AWS EKS infrastructure layer: Terraform files for VPC, cluster, and ECR repositories, and a production Helm values override for the move off Minikube. The first task was getting a working IaC tool — HashiCorp Terraform refused to download due to regional trade controls. Went to OpenTofu instead. Got the binary, added it to PATH, aliased it as terraform. Wrote four Terraform files and a values-eks.yaml, then validated both. terraform validate returned success. helm lint returned 0 failed. The cluster is still running on Minikube. The EKS config is ready to apply when AWS credentials are available.
Continue reading →The init containers never started. Six fixes before the gateway returned ok.
031 ended with the worker in CrashLoopBackOff, the gateway stuck at Init:0/2, and minikube broken by a stale MINIKUBE_ACTIVE_DOCKERD variable. Today: fresh shell, minikube started clean, node reached Ready. The gateway pods did not move. kubectl describe showed Container ID empty on both busybox init containers — not a connectivity failure, an image pull failure. busybox:1.36 was never in the minikube daemon cache. Pulled and loaded busybox and bitnami/kubectl. That exposed the next problem: wait-for-migration was defined in _helpers.tpl and in both kustomize base manifests but was never included in either Helm deployment template. The Helm chart had been running with two init containers while the base had three. Adding the third exposed the missing gateway-db-patch ConfigMap — not in any Helm template, previously created by hand. Created it immediately and added it as a permanent Helm template using .Files.Get. The wait-for-migration script had a second bug: it only exits when the job shows succeeded=1. The db-migrate Job has a 300-second TTL. By the time bitnami/kubectl starts polling, the Job is already deleted. Fixed by treating job-not-found as completion. The existing resources carried wrong Helm ownership annotations from an earlier partial attempt. Patched all 13 with the correct release-name and release-namespace. helm upgrade --install succeeded. Then the gateway hit InvalidPasswordError: Helm had overwritten nexusflow-secrets with the default placeholder change-me-in-production, against a postgres instance initialised with localpassword123. Added the correct credentials to values-minikube.yaml. REVISION: 3. Gateway 1/1 Running. Worker 1/1 Running. POST returned 202. GET returned status: completed. Idempotency: first call 202, second 200.
Continue reading →The Helm chart exists. The cluster ran out of CPU. The worker is crashing and minikube will not restart.
030 ended with the migration Job proven and the full pipeline running clean. Today the goal was to convert the raw kustomize manifests into a Helm chart, install it, and verify the same stack runs under Helm. The chart came together in one session — Chart.yaml, values.yaml, eleven template files, a values-minikube.yaml override. The first install failed because the nexusflow namespace carried kustomize labels and Helm refused to adopt it. Deleted the namespace. The second install succeeded and immediately timed out waiting for the db-migrate hook — postgres was still initialising when the five-minute default expired. Three more attempts to stabilise the upgrade hit the Kubernetes Job immutability constraint: once a Job exists, its spec.template cannot be patched. Every helm upgrade call failed until the Job was manually deleted before each attempt. The real blocker underneath all of it was CPU: five replicas of simple-model-api and the Prometheus stack were consuming all four allocatable CPUs on the node. postgres-0 sat Pending for 43 minutes. Scaling simple-model-api to zero and deleting postgres-0 manually let the StatefulSet recreate the pod with the updated spec. postgres came up. redis came up. The migration Job completed. Then the worker entered CrashLoopBackOff. The cluster became unresponsive before the logs could be read. minikube stop ran clean. minikube start failed — MINIKUBE_ACTIVE_DOCKERD=minikube was still set from the earlier eval, and minikube could not resolve the driver on restart. Stopped there.
Continue reading →The tasks table did not exist. Five blockers before a clean GET request.
029 ended with all four pods running and zero restarts. Today was the first session where real traffic was supposed to go through. The first POST hit a validation error — missing name field, easy fix. The second POST hit a 500. The gateway log said asyncpg.exceptions.UndefinedTableError: relation 'tasks' does not exist. The postgres image only runs init scripts on first start when PGDATA is empty. The PVC already had data from the previous session. The schema was never applied. Three attempts to apply it manually each hit a different blocker: wrong username, Git Bash stdin redirect permission denied on Windows, Git Bash path mangling on kubectl cp. MSYS_NO_PATHCONV=1 fixed the copy. The table was created. The POST returned a task ID. Then GET /tasks/{id} hit a 500. asyncpg returns JSONB columns as raw JSON strings, not dicts. The gateway code said 'already a dict — asyncpg deserialises JSONB' in a comment that was wrong. json.loads() fixed it. The deployment had the wrong image name — nexusflow/gateway:latest in the manifest, nexusflow-gateway:latest as built. The pod was running the old unpatched image the whole time. The fix landed after correcting the name, mounting the patched db.py via ConfigMap, and redeploying. The GET returned the full task record with status: completed. To prevent the schema problem from recurring, a one-shot Kubernetes Job now runs psql against init.sql on every apply.
Continue reading →The images built into the wrong daemon. Then a broken probe killed a worker that was running fine.
028 left one step pending: delete the stale PVC and apply the cluster. The delete ran. The apply ran. Gateway and worker hit ErrImagePull immediately. imagePullPolicy: Always in the base manifests told Kubernetes to pull from Docker Hub. The images only existed in the minikube internal Docker daemon — and the first build had not run eval $(minikube docker-env), so they were not even there. eval ran, both images rebuilt inside minikube, rollout restart still failed. After a full delete and reapply with the images solidly in the minikube daemon, the cluster came up. All four pods reached Running. Then the worker started cycling: Running → CrashLoopBackOff, every 90 seconds, eight times. The logs showed clean exits on signal 15. The worker was not crashing. The liveness probe was killing it. The probe ran pgrep -f worker.py. pgrep is not in python:3.12-slim. Two patches: imagePullPolicy: Never added to the local overlay so the cluster never tries Docker Hub again, and the probe replaced with sh -c 'kill -0 1' — a shell builtin that checks whether PID 1 is alive. One kubectl apply. All four pods running. Zero restarts.
Continue reading →The postgres pod kept restarting. Two lines in the security context were the problem.
027 ended with the application layer hardened and the Kubernetes manifests generated. Today was supposed to be the first kubectl apply. It was. The cluster came up — namespace, configmaps, secret, services, statefulsets, deployments — but postgres-0 went into CrashLoopBackOff immediately. The log said chmod: /var/run/postgresql: Operation not permitted on every start. runAsUser: 999 in the container securityContext was blocking the postgres entrypoint from running its own root-level setup before dropping to the postgres user. Two lines removed. The probes got -h localhost to force TCP over the broken socket path. The session ended before a clean apply. The old PVC still holds the stale volume.
Continue reading →Three hardening batches. The worker no longer crashes on startup and the API now deduplicates.
026 closed with two open items: the PostgreSQL startup race and telemetry blind spots in the API. This session closed both, then added two more resilience features to the worker and an idempotency layer to the gateway. Four areas touched: a retry loop inside init_pool so the worker waits for Postgres instead of crashing, retry telemetry exposed through GET /tasks/{task_id}, exponential back-off replacing the flat 1-second sleep between retries, and a background asyncio task that periodically scans the PEL with XCLAIM to reclaim messages orphaned by mid-task worker crashes. Then a fifth: an idempotency_key column on the tasks table and a deduplication check in POST /tasks. The first submission with a given key creates the task. Every subsequent submission with the same key returns the original row.
Continue reading →The worker now retries. The stack proved it.
Transmission 025 ended with one thing unconfirmed: whether a task submitted through the gateway actually lands in the database, gets picked up by the worker, and ends in the tasks table with status = completed. That curl command and that psql query ran first. Both passed. Then the session moved on to the next layer: error handling and a retry mechanism. Three new columns on the tasks table, three new database helpers in worker/db.py, and a rewritten run() loop that implements the full retry state machine. A fail_task submission hit the API, exhausted all three retries, and landed in the database with status = failed, retry_count = 3, and error_message = Simulated failure for testing. The machine does what the spec said.
Continue reading →db.py existed on disk and nowhere else. Three blockers before the stack ran clean.
Added PostgreSQL task state persistence to NexusFlow in a session that produced three separate blockers before the stack ran clean. The gateway and worker each got an async db.py module backed by an asyncpg connection pool. The first docker compose up showed empty ps output — the containers had exited immediately. Both Dockerfiles were written before db.py existed and only copied the original entry-point files. db.py was on disk and absent from every image. The fix was switching from explicit per-file COPY to COPY . . with .dockerignore files in each service directory. A PostgreSQL crash recovery added 90 seconds of fsync time on the next start after an unclean shutdown. The curl health check failed silently because curl is not in python:3.12-slim. All three fixed. The full stack is running. A task submitted via POST has not yet been traced end to end through the worker and into the database.
Continue reading →NexusFlow started running. The full stack came up in one command.
Built NexusFlow from scratch in a single session: a FastAPI gateway, a Redis Streams worker, a PostgreSQL service, two production-grade multi-stage Dockerfiles, and a docker-compose.yml that wires all four together. The first docker compose up --build took 188 seconds to pull and build. The second took 17 seconds — everything cached. A POST /tasks request returned 202 Accepted, published to Redis, and the worker consumed and acknowledged the message. The full event loop closed end to end.
Continue reading →The API left the cluster and landed on Cloud Run.
The Dockerfile had port 8000 hardcoded in four separate places: the ENV block, EXPOSE, the HEALTHCHECK URL, and the gunicorn --bind flag. Cloud Run injects a dynamic PORT at runtime. A hardcoded port means the process binds to the wrong address, the health check fires against the wrong port, and Cloud Run kills the container before the first request. All four were fixed. The image was built and verified locally at PORT=9000, then deployed to Cloud Run europe-west3. The /health endpoint returned 200 from the live URL. A curl against /predict with a real image returned five ResNet-50 predictions.
Continue reading →Four problems in one session. The probe would have killed every deploy.
A review of the model API turned up four problems: HPA comments had the wrong CPU math, the CI pipeline pushed to GHCR with no test gate, the codebase had no automated tests, and a liveness probe configuration would have caused CrashLoopBackOff on every deploy once ResNet-50 takes longer than 75 seconds to load. All four are fixed. 41 tests pass. A live Minikube rollout confirmed the probes.
Continue reading →The image is off the local machine. GitHub builds it now.
The Minikube manual build loop is gone. A GitHub Actions workflow builds the Docker image on ubuntu-latest, tags it with :latest and :sha-<commit>, and pushes both to ghcr.io. The GHA layer cache skips the PyTorch layer on every run after the first. deployment.yaml points to ghcr.io. Two pods confirmed running the remote image.
Continue reading →Drizzle schema in production, three stores seeded, and four dashboards talking to each other.
The old Supabase tables were dropped. A Drizzle schema with 19 tables was pushed to production. Three partner users, three stores, and 13 products were seeded. The customer app, partner dashboard, admin dashboard, and API server all read from the same live database. No mock data remains in the core order flow.
Continue reading →Replaced every mock in the passenger app and wrapped it for iOS and Android.
27 files swept to light mode. The ride request flow now inserts directly into Supabase and subscribes to postgres_changes for real-time driver updates. Eight files had mock arrays stripped and replaced with live queries. Capacitor wrapped the Vite bundle into native iOS and Android projects.
Continue reading →The pipeline is proven. The dashboard still has blank panels.
100 requests confirmed the API is stable. Prometheus Explore proved cAdvisor is scraping. The CPU Usage panel was fixed by removing a cluster label filter that Minikube never populates. The top stat cards and memory graphs are still blank — same fix, not yet applied.
Continue reading →Restructured the codebase, built three new verticals, and closed every security vulnerability.
A Python script migrated 25 files into a feature-based layout. Travel Tickets, Courier & Moving, and Kampala Real Estate Broker were built from scratch. The monorepo came out of a security audit clean.
Continue reading →Pausing Mombasa, adding courier logistics, and routing them to the hub
I removed Mombasa from geofencing configurations and active services, built the Courier and Moving vertical with simulated dispatch, and added direct links for tickets and courier deliveries directly inside the Kampala hub page.
Continue reading →The cluster ran out of room. Two pods broke. Scaling down fixed both.
A Helm upgrade to revision 5 triggered new monitoring pods on a node already out of CPU and memory. Grafana froze during a Docker Hub handshake. kube-state-metrics crashed 11 times. Scaling the API down to 2 replicas gave the node enough room. Both pods recovered.
Continue reading →The quota I set in 012 blocked the rollout I ran today.
Fixed the cAdvisor TLS gap, bumped CPU limits for the 97-second startup problem, and hit three consecutive Helm errors before the upgrade landed. Then the namespace quota from 012 blocked every new pod. Deleted the quota. The rollout completed in 23 seconds. The pipeline is verified.
Continue reading →One codebase, three shop types, and a Zustand subscription loop crash.
Hubal POS split into three adaptive variants — pharmacy, cosmetics, supermarket — from a single codebase. A Zustand subscription pattern crashed the app at the end of a long session.
Continue reading →The quota landed. The utilisation panels still say nothing.
The slow pod from 011 came back late again — 97 seconds between gunicorn and application startup. Resource limits hit the namespace. Grafana's utilisation panels are still blank, and now I know exactly why.
Continue reading →ResNet-50 saw the load. The dashboards learned to watch.
100 waves of traffic through all four pods. Grafana panels built from scratch. The request rate and inference latency are now visible in real time.
Continue reading →The 404 was a ghost. The image was stale.
Prometheus scraped /metrics and got 404. The route existed in main.py on disk. It did not exist inside the running container. The pods were serving an image that predated all the monitoring code.
Continue reading →Prometheus is in. The build keeps dying.
Wiring Prometheus into the cluster hit three walls: a broken package manager, a missing metrics route, and a Docker build that hangs for over an hour inside Minikube every time pip touches the network.
Continue reading →ResNet-50 takes 60 seconds to load. The cluster didn't know that.
A slow PyTorch startup window caused 502s on freshly rolled pods. I added resource ceilings, a dual-metric HPA, and startup probes to close the gap.
Continue reading →Tibibu: automated pipeline for publishing children's books
Building an automated pipeline using docx XML parsing, Gemini API rewriting, Pillow image processing, and a FastAPI/React creator tool.
Continue reading →The heartbeat murder: diagnostic of a 502 bad gateway
Debugging PyTorch CPU-bound inference causing Gunicorn worker SIGKILL timeouts and HTTP 502 Bad Gateway responses.
Continue reading →Auditing Minikube resource allocations to prevent unnecessary rebuilds
Using container inspection to verify cluster memory before deleting the environment.
Continue reading →Simple Model API — Production ML Inference Service
A FastAPI service wrapping ResNet-50 for image classification. Production patterns: health probes, correlation IDs, structured errors.
Continue reading →Hubal — offline-first POS and installment tracking
Two Vite+React apps for shop owners: a point-of-sale with IndexedDB offline sync, and a phone installment system with an immutable payment ledger.
Continue reading →Nasez — financial controls for fuel stations
Shift-based verification engine for petroleum stations. PostgreSQL triggers calculate expected cash from meter readings. Variance triggers supervisor review.
Continue reading →Fikalo — logistics and real estate for East Africa
A monorepo running logistics in Kampala and real estate in Mogadishu. One Supabase project, PostGIS geofencing, PesaPal payments.
Continue reading →