The cluster ran out of room. Two pods broke. Scaling down fixed both.

014 verified the pipeline and left one thing open: confirm the Grafana utilisation panels after the cAdvisor TLS fix. I came in today to do that. The node had other plans.

The constraint

The cluster started with kube-state-metrics already at 100 restarts over 8 days:

$ kubectl get pods
prometheus-kube-state-metrics-678b9d8db-9sxlp   1/1   Running   100 (86s ago)   8d

The logs showed the liveness probe at /livez returning 503 repeatedly:

W0618 06:38:38.482931  1 server.go:556] Failed to contact API server for /livez: got 0
W0618 06:38:46.481706  1 server.go:556] Failed to contact API server for /livez: got 0

100 restarts on a metrics agent is not normal. The node was already under pressure before I ran anything.

The proof

The Helm upgrade landed. The node did not have room for what came next.

$ helm upgrade prometheus prometheus-community/kube-prometheus-stack \
  -f kubernetes/values.yaml --namespace default --cleanup-on-fail
Release "prometheus" has been upgraded. Happy Helming!
REVISION: 5
STATUS: deployed

The upgrade succeeded. New pods started spinning up — a new Grafana, a new kube-state-metrics, a new operator. The HPA was still enforcing a minimum of 3 model API replicas. The node was already carrying the full monitoring stack plus five API pods. The scheduler could not fit one more:

$ kubectl describe pod simple-model-api-deployment-6ff5fb88cb-p5gcx
Events:
  Warning  FailedScheduling  default-scheduler  0/1 nodes are available:
  1 Insufficient cpu. preemption: 0/1 nodes available:
  1 No preemption victims found for incoming pod.

Grafana stuck in ContainerCreating for 40 minutes

The new Grafana pod 7dc9bbf8cf-dzvsg sat at 0/3 ContainerCreating for 40 minutes. The only event was a pull that never finished:

Events:
  Normal  Pulling  39m  kubelet  Pulling image "docker.io/grafana/grafana:13.0.2"

I checked whether the image was reachable inside the node:

$ minikube ssh -- sudo crictl pull docker.io/grafana/grafana:13.0.2
FATA[0016] pulling image: Error response from daemon: Head
"https://registry-1.docker.io/v2/grafana/grafana/manifests/13.0.2":
unable to decode token response: context deadline exceeded

crictl failed. The Docker Hub token handshake timed out — the node’s network link to the registry froze during the TLS exchange. I tried the raw Docker daemon:

$ minikube ssh -- docker pull docker.io/grafana/grafana:13.0.2
Digest: sha256:5dad0df181cb644a14e13617b913b261a54f7d4fd4510721dba420929f35bea2
Status: Image is up to date for grafana/grafana:13.0.2

The image was already on the node. crictl was hitting the registry for a manifest check even though nothing needed downloading. Force-deleting the stuck pod alone would not fix it — the new pod would hit the same frozen path. Pulling through the raw Docker daemon first bypassed the token check. Then I deleted the pod:

$ kubectl delete pod prometheus-grafana-7dc9bbf8cf-dzvsg --grace-period=0 --force
pod "prometheus-grafana-7dc9bbf8cf-dzvsg" force deleted

A new pod lsw9t spawned and grabbed the local image immediately.

kube-state-metrics in CrashLoopBackOff

While Grafana was stuck, kube-state-metrics hit CrashLoopBackOff at restart 10:

prometheus-kube-state-metrics-6cc7c56db5-6vdrv   0/1   CrashLoopBackOff   10 (22s ago)   63m

I did not touch it. The node had no CPU left to keep it stable. The fix came from reducing the API replicas.

Scaling down freed the node

I deleted the HPA and scaled manually:

$ kubectl delete hpa simple-model-api-hpa
horizontalpodautoscaler.autoscaling "simple-model-api-hpa" deleted

$ kubectl scale deployment simple-model-api-deployment --replicas=2
deployment.apps/simple-model-api-deployment scaled

Two replicas instead of five gave the monitoring stack room to breathe. kube-state-metrics stopped crashing and stabilised at restart 11 on its own. Grafana lsw9t climbed from 2/3 to 3/3. The Grafana container crashed twice before it held — exit code 2 both times:

    State:      Running
      Started:  Thu, 18 Jun 2026 10:59:09 +0300
    Last State: Terminated
      Reason:   Error
      Exit Code: 2
      Started:  Thu, 18 Jun 2026 10:56:25 +0300
      Finished: Thu, 18 Jun 2026 10:59:08 +0300
    Ready:      True
    Restart Count: 2

Final state:

$ kubectl get pods
alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2   Running   0    73m
prometheus-grafana-7dc9bbf8cf-lsw9t                      3/3   Running   2    17m
prometheus-kube-prometheus-operator-7ff545b6fb-pdrdt     1/1   Running   0    74m
prometheus-kube-state-metrics-6cc7c56db5-6vdrv           1/1   Running   11   74m
prometheus-prometheus-kube-prometheus-prometheus-0       2/2   Running   2    2d1h
simple-model-api-deployment-6ff5fb88cb-sb2f6             1/1   Running   0    95m

HPA back with corrected minimums

$ kubectl apply -f kubernetes/hpa.yaml
horizontalpodautoscaler.autoscaling/simple-model-api-hpa created

$ kubectl get hpa simple-model-api-hpa
NAME                   REFERENCE                                TARGETS                        MINPODS   MAXPODS   REPLICAS
simple-model-api-hpa   Deployment/simple-model-api-deployment   cpu: 0%/70%, memory: 73%/80%   2         10        2

Minimum replicas is now 2, down from 3. The node can hold the monitoring stack and the API together without hitting the ceiling.

Where this sits

Item	Status
Helm upgrade — revision 5	Done
Grafana stuck in ContainerCreating — 40 minutes	Fixed — local image pull then force delete
kube-state-metrics CrashLoopBackOff	Recovered — node had room once API scaled down
API replicas reduced to 2	Done
HPA minimum corrected to 2	Done
Grafana utilisation panels — cAdvisor data flowing	Not verified — next session

Both problems came from the same place. The node ran out of room and the monitoring stack started falling apart. Scaling down one deployment fixed two broken pods.