Linux Process Keeps Getting Killed

LINUX RUNBOOK

Linux Process Keeps Getting Killed or Restarting

Determine whether a disappearing process was OOM-killed, signaled, crashed, restarted by a service manager or constrained by its runtime.

Do not diagnose from the word “Killed” alone. Capture the exit status, kernel evidence and service/runtime evidence before deciding what happened.

Fast troubleshooting flow

Capture exit status / timestamp↓Check OOM evidence↓Check service-manager/runtime logs↓Look for crash/core-dump evidence↓Identify who/what sent the signal or restart↓Fix cause and verify

1 — Capture the exit status when you can reproduce safely

./my-script
echo $?

Shell exit statuses above 128 often indicate termination by a signal using the common 128 + signal convention. For example, 137 commonly corresponds to SIGKILL (9), but that still does not tell you who sent SIGKILL or why.

2 — Check whether the kernel OOM killer was involved

dmesg | grep -Ei 'out of memory|oom|killed process'
Out of memory: Killed process 18421 (python3) ...

If the kernel explicitly records an OOM kill, continue with High Memory / OOM Killed Troubleshooting. If there is no such evidence, keep investigating.

3 — If it is a systemd service, inspect the unit

systemctl status <service>
journalctl -u <service> --since "30 minutes ago"
myworker.service: Main process exited, code=killed, status=9/KILL
myworker.service: Failed with result signal.
myworker.service: Scheduled restart job, restart counter is at 4.

What to check: code=/status= tell you how the main process ended, while a restart-counter message explains why it appears again. This proves a signal/restart path, but status=9/KILL alone still does not prove OOM. Look for the process exit code/signal, restart attempts, watchdog/timeouts and the application’s own error immediately before termination. Restart= policies can make a crash look like an unexplained restart loop.

4 — Check for a crash/core dump where supported

coredumpctl list --since "30 minutes ago"

coredumpctl is systemd-specific and core-dump collection may be disabled. If a matching dump exists, use the application/runtime’s debugging workflow rather than assuming an infrastructure kill.

5 — Consider deliberate signals and external supervisors

Deployment scripts, cron jobs, watchdogs, process supervisors, container runtimes and operators can stop/restart a process. Correlate timestamps across those systems. SIGTERM is commonly used for graceful shutdown; SIGKILL cannot be handled by the process and may come from OOM handling or an external actor.

6 — Containers add another boundary

If the process runs in a container, inspect the container/pod exit reason and configured memory/runtime limits. A host process investigation alone may miss a cgroup/container limit.

Do not “fix” this by making the process harder to kill. Changing OOM preference, disabling watchdogs or removing restart controls can hide the symptom while leaving the resource leak or crash intact.

Verify

Run through the previous failure window and confirm the process stays healthy. Verify both the application behavior and the evidence source that identified the cause: no new OOM event, crash, timeout or supervisor-triggered restart.