Fast troubleshooting flow

1 — If the process suddenly says “Killed”, prove whether OOM was involved

dmesg | grep -Ei 'out of memory|oom|killed process'

Look for kernel messages explicitly naming an out-of-memory event or a killed process. A bare Killed message by itself is not enough to prove the Linux OOM killer was responsible.

Out of memory: Killed process 18421 (python3) ...

If dmesg access is restricted, use authorized elevated access or the logging mechanism available on that host. On systemd/journald systems, journalctl -k is an optional alternative—not a requirement for this runbook.

2 — Check current memory, but understand what it can and cannot prove

free -h

               total        used        free      shared  buff/cache   available
Mem:            15Gi        12Gi       420Mi       310Mi        3.0Gi        2.4Gi
Swap:          2.0Gi       1.7Gi       300Mi

What to check: available is the first useful capacity signal here; free alone can be small on a healthy Linux host because cache is reclaimable. Swap usage is context—not proof by itself. Focus especially on available memory and swap, not just the free column. Linux can reclaim caches when applications need memory. Also remember: after the offending process has been killed, free -h may look healthy because its memory has already been released. That is why historical/kernel evidence comes first.

3 — Check swap using existing system information

cat /proc/swaps

Filename                Type        Size     Used Priority
/swapfile               file     2097148  1740800      -2

What to check: whether swap exists and how much is currently used. This snapshot does not show swap-in/swap-out rate; pair it with vmstat or monitoring when performance is the concern. No swap is not automatically wrong, and adding swap is not automatically the fix. Heavy swap activity plus poor performance is a sign of pressure; an application that grows without bound still needs its memory behavior fixed.

4 — Find current memory consumers

ps aux --sort=-%mem | head

Use this to identify candidates, then inspect a specific process:

ps -o pid,ppid,%mem,rss,vsz,cmd -p <PID>
grep -E 'VmRSS|VmSize|VmSwap' /proc/<PID>/status

RSS/VmRSS is the process's resident physical-memory footprint at that moment. VSZ/VmSize is virtual address space and can be much larger than actual resident RAM, so do not treat VSZ as “RAM used.”

5 — Observe safely while reproducing

top

For a field-by-field explanation, see Understanding the Linux top Command. If the workload can be reproduced safely, watch whether one process steadily climbs, jumps during a particular phase, or whether many processes compete for memory. Do not repeatedly rerun a job that can destabilize a production host just to observe it.

6 — Interpret the pattern

7 — Memory-heavy scripts: reduce the working set when possible

A script that loads an entire large dataset into memory may fail even when its logic is correct. Prefer bounded processing when the workload allows it: read a chunk → process it → write/release results → continue. Other fixes can include reducing concurrency, batch size or cache size, or correcting a leak. Add RAM only when the workload legitimately requires more capacity.

8 — OOM scoring is diagnostic, not extra memory

cat /proc/<PID>/oom_score
cat /proc/<PID>/oom_score_adj

These values help explain OOM selection. Do not change oom_score_adj merely to protect the failing application: making one process less killable does not create memory and may shift the victim to another process.

Verify

Run the workload under representative conditions and observe it longer than the previous failure point. Confirm memory reaches a stable/bounded pattern, the host retains healthy available memory, and no new OOM event appears.

Running in Kubernetes?

Container limits add another layer. Continue with Kubernetes OOMKilled Troubleshooting to inspect container limits, requests and pod-level evidence.