NexusFlow started running. The full stack came up in one command.
Built NexusFlow from scratch in a single session: a FastAPI gateway, a Redis Streams worker, a PostgreSQL service, two production-grade multi-stage Dockerfiles, and a docker-compose.yml that wires all four together. The first docker compose up --build took 188 seconds to pull and build. The second took 17 seconds — everything cached. A POST /tasks request returned 202 Accepted, published to Redis, and the worker consumed and acknowledged the message. The full event loop closed end to end.
NexusFlow is a learning project. The goal is to go from nothing to a running microservices platform, then deploy it to Kubernetes on AWS EKS. This transmission covers the first day.
The constraint
Starting from an empty folder. No services, no images, no compose file. The goal was to have two containerized microservices talking to each other through Redis Streams before the session ended.
Redis Streams was chosen over a simple list queue (BLPOP) because streams support consumer groups. Multiple worker replicas can read from the same stream without duplicating work. Unacknowledged messages stay in the Pending Entries List and get re-delivered if a worker dies. That matters at scale. A list queue does not give you that without extra code.
The proof
What was built
Five files across two services and the project root.
gateway/requirements.txt — FastAPI, Uvicorn with standard extras, Redis client, Pydantic.
gateway/main.py — FastAPI application with two endpoints. /health returns service name, status, and a UTC timestamp. POST /tasks validates the request body, generates a UUID, publishes to the tasks Redis Stream, and returns HTTP 202 Accepted. The Redis client is module-level so the connection pool is shared across requests. payload is serialised to a JSON string before publishing because Redis Stream field values must be scalars — a dict raises a DataError.
worker/requirements.txt — Redis client and Pydantic only. No web framework.
worker/worker.py — Blocking consumer using XREADGROUP. Connects to Redis with exponential back-off up to ten attempts. Creates the consumer group with MKSTREAM if it does not exist. Reads one message at a time, calls process_task(), then XACKs only on success. SIGTERM sets a shutdown flag; the loop finishes its current task before exiting. Unacknowledged messages stay in the PEL and are re-delivered after restart.
docker-compose.yml — Four services on a shared bridge network (nexusflow_net). postgres and redis use named volumes. All depends_on conditions use service_healthy, not service_started — the gateway does not start until pg_isready and redis-cli ping both pass. The worker depends only on Redis.
The Dockerfiles
Both services use the same two-stage pattern: a builder stage installs dependencies into /install/packages, the runtime stage copies only the installed packages into /usr/local. No pip, no build cache, no wheel files in the final image. A non-root user (appuser) runs the process in both containers. Kubernetes Pod Security Standards (restricted) require a non-root UID. Running as root is a direct privilege escalation risk if the process is ever compromised.
requirements.txt is copied before the application source in both Dockerfiles. Editing main.py or worker.py reuses the cached pip install layer.
First build
[+] Building 188.4s (13/13) FINISHED
=> [builder 4/4] RUN pip install --upgrade pip --no-cache-dir ... 90.4s
=> [runtime 4/5] COPY --from=builder /install/packages /usr/local 3.0s
=> [runtime 5/5] COPY main.py . 1.6s
=> exporting to image 22.7s
=> => naming to docker.io/nexusflow/gateway:latest
90 seconds of that was pip downloading FastAPI, Uvicorn, and their dependencies cold. The next build hit cache and finished in under two seconds for those layers.
docker compose up —build — second run
After stopping the cluster and clearing the conflicting container (simple-model-api-api-1 was holding port 8000), the second docker compose up --build completed in 17 seconds. Every layer cached.
[+] up 6/6
✔ Image nexusflow-worker Built 16.8s
✔ Image nexusflow-gateway Built 16.8s
✔ Network nexusflow_nexusflow_net Created
✔ Container nexusflow-redis-1 Running
✔ Container nexusflow-worker-1 Recreated
... 2 more
Services coming up
Redis initialised first and passed its health check. The worker connected and created the consumer group:
worker-1 | 2026-08-24T06:43:34Z [INFO] nexusflow.worker — Connected to Redis at redis:6379
worker-1 | 2026-08-24T06:43:34Z [INFO] nexusflow.worker — Consumer group 'workers' created on stream 'tasks'.
worker-1 | 2026-08-24T06:43:34Z [INFO] nexusflow.worker — Worker 'worker-1' starting — listening on stream 'tasks' (group: workers)
PostgreSQL passed pg_isready and is idling in the stack. No schema exists yet — the service is there, nothing has written to it. The gateway started after both dependencies were healthy:
gateway-1 | INFO: Started server process [1]
gateway-1 | INFO: Waiting for application startup.
gateway-1 | INFO: Application startup complete.
gateway-1 | INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
The first task through the loop
A send-email task was submitted while the gateway was running. The worker consumed it 4 seconds later:
worker-1 | 2026-08-24T07:25:02Z [INFO] nexusflow.worker — Processing task_id=1787556301266-0 name=send-email
worker-1 | 2026-08-24T07:25:02Z [INFO] nexusflow.worker — Task completed task_id=1787556301266-0
Live curl
$ curl -X POST http://localhost:8000/tasks \
-H "Content-Type: application/json" \
-d '{"name": "send-email", "payload": {"to": "user@example.com", "template": "welcome"}}'
{"task_id":"571a0512-8dde-48e0-9e0f-32354cf34552","name":"send-email","status":"accepted","accepted_at":"2026-08-24T07:24:58.793639+00:00"}
202 Accepted. The task_id in the response matches the stream message ID the worker logged.
On a second session docker compose up, the worker reconnected and printed Consumer group 'workers' already exists. Joining. — the group and the stream persisted across restarts because the Redis volume was intact.
Where this sits
| Item | Status |
|---|---|
gateway/main.py — /health and POST /tasks endpoints | Done |
gateway/requirements.txt | Done |
gateway/Dockerfile — multi-stage, non-root user | Done |
worker/worker.py — XREADGROUP consumer with graceful shutdown | Done |
worker/requirements.txt | Done |
worker/Dockerfile — multi-stage, non-root user | Done |
docker-compose.yml — four services, health checks, named volumes | Done |
Redis Streams xadd wired into POST /tasks | Done |
| Gateway published task, worker consumed and ACKed it | Done |
Full stack started with docker compose up --build | Done |
| PostgreSQL schema and task state persistence | Pending |
GET /tasks/{task_id} status endpoint | Pending |
The event loop is closed. Gateway publishes, worker consumes. Postgres is in the stack but nothing has touched it — there is no schema, no writes, no way to ask what happened to a task after the worker ACKed it. That comes next: a table for task state, the worker writing to it on pickup and completion, and a status endpoint so the gateway can answer GET /tasks/{task_id}. That work stays in Docker Compose. The platform does not move to EKS until the full local loop — publish, process, persist, query — is working.