Pending Pods

KUBERNETES RUNBOOK

Pending Pods

Read the scheduler evidence and systematically check capacity, placement rules, storage and namespace constraints.

What does Pending mean?

A Pending pod has been accepted by Kubernetes but has not completed the steps needed to run. Often it has not been assigned to a node, but storage or other prerequisites can also be involved. Do not assume “Pending = add more nodes.”

60-second diagnosis

kubectl get pod <pod> -n <namespace> -o wide
kubectl describe pod <pod> -n <namespace>

If the NODE column is empty, read Events for the scheduler reason.

Pod Pending↓Is a node assigned?↓
No → read FailedScheduling Events: resources / taints / affinity / selectors
Storage message → inspect PVC / StorageClass / volume Events
Other admission/quota message → inspect quota / admission policy

Step 1 — Check whether a node is assigned

kubectl get pod <pod> -n <namespace> -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP       NODE
api-0   0/1     Pending   0          2m    <none>   <none>

What to check: NODE <none> means scheduling has not completed, so scheduler Events are the next evidence source. If NODE is <none>, the pod has not been scheduled. If NODE is already populated, scheduling succeeded; inspect Events for image pull, volume attach/mount, sandbox/network or init/startup problems.

Step 2 — Read scheduler Events

kubectl describe pod <pod> -n <namespace>

Scroll to Events near the bottom. You may see:

0/3 nodes are available:
2 Insufficient memory,
1 node(s) had untolerated taint

This is a map of why candidate nodes were rejected: two lacked requested memory and one had a taint the pod did not tolerate.

You can also retrieve Events for the pod:

kubectl get events -n <namespace> --field-selector involvedObject.name=<pod> --sort-by=.lastTimestamp

Step 2 — If resources are insufficient

kubectl get pod <pod> -n <namespace> -o jsonpath="{.spec.containers[*].resources}"
kubectl describe node <node>

The scheduler primarily considers requested resources when placing pods. Compare workload requests with node allocatable capacity and existing allocations. kubectl top shows usage, which is useful context but is not the same thing as scheduler requests.

Step 3 — If taints block the pod

kubectl describe node <node>
kubectl get pod <pod> -n <namespace> -o yaml

Find node Taints and pod tolerations. Taints are often intentional isolation. Add a toleration only if this workload belongs on those nodes.

Step 4 — Check nodeSelector and affinity

In the pod/workload YAML, inspect nodeSelector, required nodeAffinity, and required pod affinity/anti-affinity. Then compare required labels with:

kubectl get nodes --show-labels

A constraint can be valid syntactically but impossible for the current node set.

Step 5 — Check PVCs when Events mention storage

kubectl get pvc -n <namespace>
kubectl describe pvc <pvc> -n <namespace>
kubectl get storageclass

Look for unbound claims, provisioning failures, topology conflicts, unavailable volumes or an incorrect/missing StorageClass.

Step 6 — Check quotas

kubectl get resourcequota -n <namespace>
kubectl describe resourcequota -n <namespace>

Quota or admission-policy failures may appear earlier in creation rather than as a long-lived Pending pod, so use the exact API/event message to choose this branch.

Fix the actual constraint in the workload, capacity, labels, taints or storage configuration. Avoid weakening placement rules or resource requests merely to make the pod schedule.

NODE is assigned but the Pod is still Pending

Scheduling is no longer the primary suspect. Run kubectl describe pod and inspect Events for FailedMount, FailedAttachVolume, FailedCreatePodSandBox, image-pull errors or init/startup activity. If you see ImagePullBackOff, follow the ImagePullBackOff runbook.

Verify

kubectl get pod <pod> -n <namespace> -w
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Confirm a suitable node is assigned and the pod then progresses through image pull, container startup and readiness.

Common mistakes

  • Deleting the pod repeatedly while constraints remain unchanged.
  • Using current CPU/memory usage as if it were the scheduler request calculation.
  • Adding broad tolerations without understanding the taint.
  • Ignoring PVC events.