Linux Disk I/O Troubleshooting

Start broad, then install/use specialist tools if available. vmstat is commonly present; iostat comes from the sysstat package and may not be installed.

Fast troubleshooting flow

Confirm latency symptom↓vmstat: blocked tasks + iowait↓iostat -xz: device latency + queue↓Identify process I/O↓Map process → files/workload↓Fix workload/storage cause↓Verify latency

1 — Look for blocked work and I/O wait

vmstat 1 5

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  5      0 420000  82000 901000    0    0  9600  2100 3400 5900 14  4 38 44  0

What to check: after the first historical row, persistent b plus elevated wa during the slowdown strengthens the I/O hypothesis. bi/bo show block I/O activity but are not latency measurements. After the first row, watch b (blocked tasks), bi/bo (block input/output) and wa (CPU time idle while outstanding I/O exists). A single spike is not enough; correlate sustained values with the slowdown.

2 — Inspect devices with iostat when available

iostat -xz 1 5

Device   r/s   w/s  r_await w_await aqu-sz  %util
nvme0n1  82.1 310.4    2.3    18.7   6.42   91.3

await includes queue + service time; aqu-sz is average queue length. %util is time during which I/O requests were issued to the device. Near 100% can indicate saturation for serial devices, but modern SSDs/RAID can serve requests in parallel, so do not use %util alone as a performance limit.

3 — Identify processes generating I/O

pidstat -d 1 5

UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
999      2187      12.0    18432.0       0.0  postgres
1001    18421       0.0     8420.0       0.0  java

What to check: sustained read/write rates and whether the top I/O process matches the affected workload. A high rate alone is not bad; correlate it with latency and blocked-work evidence. If pidstat is unavailable, inspect a known candidate through procfs:

cat /proc/<PID>/io

rchar: 1842291
wchar: 928441203
syscr: 8124
syscw: 48291
read_bytes: 4096
write_bytes: 912261120
cancelled_write_bytes: 0

What to check: sample the counters more than once. A rapidly increasing write_bytes identifies storage writes attributed to this process; wchar is bytes passed to write-like syscalls and is not the same thing as physical device writes. Next: map the PID to its files/mount and correlate with device latency.

Compare counters over time. read_bytes and write_bytes reflect storage I/O attributed to that task; do not confuse every character read/written by the process with physical device traffic.

4 — Map the workload to files and mounts

readlink -f /proc/<PID>/cwd
ls -l /proc/<PID>/fd | head

Use this as a lead, not proof of which file is hot. Application logs, database metrics and filesystem/device mapping may be needed for complex workloads.

5 — Interpret common patterns

High write latency + log-heavy process — inspect logging rate, sync behavior, rotation and destination.

Database dominates I/O — inspect query/load changes, cache hit behavior, checkpoints/WAL and database-specific metrics before touching files.

Many blocked tasks, device latency high — investigate storage backend, filesystem, volume/cloud-disk limits and competing workloads.

Filesystem is simply full — switch to Disk Usage / No Space Left.

Avoid destructive “cleanup” or filesystem changes during diagnosis. High I/O does not mean data is disposable. Databases, WAL/journals and container state require application-aware handling.

Verify

Confirm the original request/job latency improves, blocked-task/I/O-wait signals normalize, and device latency/queueing returns to the workload’s established baseline under representative traffic.