Four problems in one session. The probe would have killed every deploy.
A review of the model API turned up four problems: HPA comments had the wrong CPU math, the CI pipeline pushed to GHCR with no test gate, the codebase had no automated tests, and a liveness probe configuration would have caused CrashLoopBackOff on every deploy once ResNet-50 takes longer than 75 seconds to load. All four are fixed. 41 tests pass. A live Minikube rollout confirmed the probes.
The codebase had been in production-facing territory for a while without a single automated test. That was the first problem. It turned out not to be the most dangerous one.
The constraint
Four things were wrong at the same time.
The HPA in kubernetes/hpa.yaml had inline comments stating the CPU request was 250m and the utilisation target was 175m. The actual CPU request in deployment.yaml was 500m. The comments were stale from before a resource limit change. The HPA was computing the right numbers at runtime — Kubernetes reads the live requests.cpu value, not the comment — but the documentation was lying.
The CI pipeline from transmission 021 had one job: build the Docker image and push it to GHCR on every push to main. No test step. A broken preprocessing function or an off-by-one in the response schema would build, push, and deploy without a checkpoint.
The codebase had no tests at all. pytest, httpx, and python-multipart were not in requirements.txt.
The deployment had a readinessProbe and livenessProbe but no startupProbe. The liveness probe had initialDelaySeconds: 30 and failureThreshold: 3 at a 15-second period. That means Kubernetes runs the first liveness check at 30 seconds, then kills the pod after three consecutive failures at 45, 60, and 75 seconds. ResNet-50 takes 97 seconds to deserialize on a CPU node. Every deploy would have ended in CrashLoopBackOff — the pod killed 22 seconds before it was ever ready.
The proof
Fixing the HPA math
The CPU target comment had not been updated when the resource request changed:
# hpa.yaml — comment said
# target: 175m average (70% of 250m request)
# deployment.yaml — actual request
requests:
cpu: "500m"
The comment was corrected to match:
# target: 350m average (70% of 500m request)
No runtime behaviour changed. This was a documentation problem, not a configuration problem.
Adding the test suite
Two files cover the preprocessing pipeline and the API layer.
tests/test_inference.py has 16 tests for preprocess_image() in inference.py: RGB PNG, JPEG, RGBA alpha-stripping, grayscale channel expansion, small and large images, empty bytes, corrupt bytes, truncated PNG, output dtype, and frozen dataclass immutability.
tests/test_main.py has 25 tests against the FastAPI endpoints: /health with and without a loaded model, /info schema and constraint fields, /metrics Prometheus scrape output, and /predict covering valid uploads, empty files (400), oversized files (413), corrupt images (422), no model (503), top_k boundary violations, and correlation ID header uniqueness.
The tests inject a MagicMock into app.state.model and swap in a no-op lifespan. ResNet-50 never downloads. The test client starts instantly.
Running them locally for the first time revealed python-multipart was missing from requirements.txt. FastAPI raises a runtime error at collection time if it is absent and the endpoint accepts file uploads. It was added alongside pytest and httpx.
Full suite result:
collected 41 items
tests/test_inference.py::TestPreprocessImageSuccess::test_rgb_png_succeeds PASSED [ 2%]
tests/test_inference.py::TestPreprocessImageSuccess::test_jpeg_succeeds PASSED [ 4%]
...
tests/test_main.py::TestCorrelationID::test_each_request_gets_unique_correlation_id PASSED [100%]
======================= 41 passed, 1 warning in 17.31s ========================
Adding the CI test gate
The CI pipeline had one job. I split it into two:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: pip install --index-url https://download.pytorch.org/whl/cpu -r requirements.txt
- run: pytest tests/ -v --tb=short
build-and-push:
needs: test
...
needs: test means GitHub Actions will not start the Docker build if the test job fails. Broken code stays off GHCR.
Adding the startup probe and verifying the rollout
The probe block in deployment.yaml was missing startupProbe. I added it before the other two probes:
startupProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 18 # 18 x 10s = 180s startup budget
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 15
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 30
periodSeconds: 15
timeoutSeconds: 5
failureThreshold: 3
Kubernetes does not run readinessProbe or livenessProbe until startupProbe passes at least once. The 180-second window absorbs the 97-second model load with 83 seconds of headroom. After startup, liveness watches for frozen workers and readiness gates load balancer traffic.
The cluster was stopped when I went to apply it. kubectl rollout restart refused the connection:
$ kubectl rollout restart deployment/simple-model-api-deployment
Unable to connect to the server: dial tcp 127.0.0.1:59109: connectex: No connection could be made because the target machine actively refused it.
After minikube start, the first rollout status timed out at 300 seconds:
$ kubectl rollout status deployment/simple-model-api-deployment --timeout=300s
Waiting for deployment "simple-model-api-deployment" rollout to finish: 1 out of 2 new replicas have been updated...
error: timed out waiting for the condition
The second attempt completed:
$ kubectl rollout status deployment/simple-model-api-deployment --timeout=300s
Waiting for deployment "simple-model-api-deployment" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "simple-model-api-deployment" rollout to finish: 1 old replicas are pending termination...
deployment "simple-model-api-deployment" successfully rolled out
kubectl get pods showed three replicas instead of two:
NAME READY STATUS RESTARTS AGE
simple-model-api-deployment-5b65cbf74d-cx87s 1/1 Running 0 77s
simple-model-api-deployment-5b65cbf74d-n4sz7 1/1 Running 0 56s
simple-model-api-deployment-5b65cbf74d-szw7s 1/1 Running 0 8m27s
The HPA had scaled up:
$ kubectl get hpa simple-model-api-hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
simple-model-api-hpa Deployment/simple-model-api-deployment cpu: 0%/70%, memory: 74%/80% 2 10 3 62d
Memory hit 74% during container initialization — close to the 80% threshold. The HPA added a third replica to absorb the pressure from concurrent weight deserialization across two pods starting at the same time.
I applied the updated manifest and ran kubectl describe to confirm:
$ kubectl apply -f kubernetes/deployment.yaml
deployment.apps/simple-model-api-deployment configured
$ kubectl rollout status deployment/simple-model-api-deployment --timeout=300s
Waiting for deployment "simple-model-api-deployment" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "simple-model-api-deployment" rollout to finish: 1 old replicas are pending termination...
deployment "simple-model-api-deployment" successfully rolled out
$ kubectl describe deployment simple-model-api-deployment | grep -A 10 -E "(Startup|Readiness|Liveness)"
Liveness: http-get http://:8000/health delay=30s timeout=5s period=15s #success=1 #failure=3
Readiness: http-get http://:8000/health delay=15s timeout=5s period=10s #success=1 #failure=3
Startup: http-get http://:8000/health delay=5s timeout=5s period=10s #success=1 #failure=18
Where this sits
| Item | Status |
|---|---|
HPA CPU request math corrected in comments (250m to 500m, 175m to 350m) | Done |
pytest, httpx, python-multipart added to requirements.txt | Done |
16 unit tests for preprocess_image() in test_inference.py | Done |
25 integration tests for API endpoints in test_main.py | Done |
test job added to CI pipeline — gates build-and-push | Done |
startupProbe added (180s budget, port 8000) | Done |
readinessProbe configured (15s initial delay, port 8000) | Done |
livenessProbe configured (30s initial delay, port 8000) | Done |
| Zero-downtime rolling update verified in Minikube | Done |
| HPA memory scaling under startup load observed | Done |
Three pods are running. The probes are active. The next broken commit will not reach the cluster.