Kubernetes OOMKilled Troubleshooting

What does OOMKilled mean?

OOM means Out Of Memory. When Kubernetes reports a container termination reason of OOMKilled, the process was killed in a memory-exhaustion situation. A common case is the container exceeding its configured memory limit. First confirm the evidence rather than assuming every exit code 137 has the same cause.

60-second diagnosis

kubectl describe pod <pod> -n <namespace>
kubectl top pod <pod> -n <namespace> --containers

Container restarted↓Last State says OOMKilled?↓

Yes → check limit + memory history
No → return to CrashLoopBackOff diagnosis

Step 1 — Confirm the termination reason

kubectl describe pod <pod> -n <namespace>

Look under the affected container for:

Last State:
  Terminated:
    Reason: OOMKilled
    Exit Code: 137

Interpretation: OOMKilled is the strongest clue here. Exit code 137 means the process ended after SIGKILL; do not use 137 alone to prove the exact memory cause.

Step 2 — Find requests and limits

kubectl get pod <pod> -n <namespace> -o jsonpath="{.spec.containers[*].resources}"

Memory request: used by scheduling and resource guarantees.
Memory limit: the container memory boundary enforced by the runtime/kernel mechanisms.
Usage: what the workload is actually consuming.

Step 3 — Inspect current and historical memory

kubectl top pod <pod> -n <namespace> --containers

POD       NAME   CPU(cores)   MEMORY(bytes)
api-7d9   api    220m         486Mi

What to check: compare current memory with the container limit, but remember this is only a current snapshot. If the container already restarted, the peak that caused the kill may be gone. Current usage after restart may be misleading. In Prometheus/Grafana or your monitoring platform, inspect the period before the kill. Ask whether memory grows slowly, spikes with traffic, or jumps during startup.

Step 4 — Identify the memory pattern

A single snapshot cannot establish a leak. For a quick live check:

watch -n 5 kubectl top pod <pod> -n <namespace> --containers

Prefer historical monitoring for the period before the kill.

Starts near the limit — check startup footprint and runtime sizing.

Steadily rises — investigate retained objects, unbounded caches/queues and leaks.

Spikes with traffic — compare concurrency, payload size, buffering/batching and scaling.

Many workloads affected — inspect node memory pressure and eviction evidence.

Correlate memory with traffic, jobs and application events before concluding it is a leak.

kubectl describe node <node>
kubectl top node <node>

Step 5 — Check runtime and process settings

kubectl get pod <pod> -n <namespace> -o yaml

JVM

Look for -Xmx, MaxRAMPercentage and JAVA_TOOL_OPTIONS. If available:

kubectl exec <pod> -n <namespace> -- jcmd 1 VM.flags

Heap is not the whole container footprint; native/runtime memory needs headroom.

Node.js

Inspect NODE_OPTIONS and --max-old-space-size. V8 heap is not necessarily total process memory.

Python / multi-worker servers

Check worker count, caches, queues and batch size. If available:

kubectl exec <pod> -n <namespace> -- ps aux

Multiple workers can multiply memory requirements.

Step 6 — Choose the fix from the evidence

A higher memory limit may be appropriate when the old limit was genuinely undersized. Make resource changes in the workload source of truth, but do not use a larger limit as a substitute for investigating an uncontrolled growth pattern.

Verify

kubectl get pods -n <namespace>
kubectl top pod <pod> -n <namespace> --containers

Confirm restart count remains stable and memory stays within an expected range under representative load for longer than the previous failure interval.

Common mistakes

Assuming every exit 137 is proven OOMKilled without checking Last State/events.
Looking only at memory after the container restarted.
Confusing requests with limits.
Increasing limits repeatedly without investigating growth.