Finding I/O Bottlenecks on Linux with top, iostat, sar and pidstat

This article is a localized edition of the Japanese original. Read the Japanese original

When a server is slow but the CPU is mostly idle and work still does not progress, disk reads and writes (I/O) may be the bottleneck. This article narrows it down step by step with standard Linux commands: "Is I/O the cause?", "Which disk?" and "Which process?". The command behavior and output columns were checked on sysstat 12.7.5 in Debian 13 and against the official iostat manual as of November 2026.

Preparation: install sysstat

iostat, sar and pidstat are in the sysstat package.

# Ubuntu/Debian
sudo apt install sysstat

# RHEL-based (AlmaLinux, Rocky Linux, etc.)
sudo dnf install sysstat

Step 1: Check for I/O wait with top and vmstat

In the third line of top (%Cpu(s)), wa is the percentage of time the CPU sat idle waiting for I/O to complete.

%Cpu(s):  2.1 us,  1.0 sy,  0.0 ni, 60.3 id, 36.4 wa,  0.0 hi,  0.2 si,  0.0 st

When wa is high and us and sy (CPU work) are low, as in this example, suspect I/O wait. In the top screen, processes whose S (state) column shows D are blocked waiting for I/O or similar to complete. Note that top shows CPU and memory usage per process, but not the amount of I/O per process (use the commands in Step 4 for that).

To see the numbers side by side at a fixed interval, vmstat is handy.

vmstat 1 5    # 5 samples, 1 second apart

wa only grows while the CPU is otherwise idle, so on a server whose CPU is busy it can stay low even when I/O is saturated. Do not judge by wa alone; check the disk side with iostat, next.

Step 2: Find the saturated disk with iostat -x

iostat -dxyz 1 5

Without -y, the first report is the average since boot, which differs from what is happening now.

The columns to look at are as follows.

How to read them

• When r_await and w_await rise well above their usual values, the disk cannot keep up with requests and they are waiting. What counts as normal varies greatly by disk type, so compare against your usual values.
• When aqu-sz is large, requests are piling up.
• On disks that handle one request at a time, such as HDDs, a %util value pinned near the maximum indicates saturation. However, as the iostat manual says, for devices that serve requests in parallel, such as RAID arrays and modern SSDs, %util does not reflect their performance limits. SSDs and NVMe drives may have headroom even with %util near the maximum, so judge by await and aqu-sz instead.

Older material mentions columns such as svctm, avgrq-sz and avgqu-sz. Current sysstat no longer shows svctm, avgrq-sz has been replaced by areq-sz (in kilobytes), and avgqu-sz by aqu-sz.

To see which file system a device name (sda, nvme0n1, dm-0 and so on) corresponds to, use lsblk.

lsblk -o NAME,SIZE,TYPE,MOUNTPOINTS

Step 3: Look at past activity with sar

iostat only shows what is happening now. For cases like "it was slow only last night", look at the data that sar records periodically.

Enable data collection

# Ubuntu/Debian: enable collection if it is disabled
grep ENABLED /etc/default/sysstat
sudo sed -i 's/^ENABLED="false"/ENABLED="true"/' /etc/default/sysstat
sudo systemctl enable --now sysstat
sudo systemctl restart sysstat

# RHEL-based
sudo systemctl enable --now sysstat

On Debian 13, ENABLED was "false" right after installation. Once enabled, a systemd timer (sysstat-collect.timer) records data every 10 minutes by default, and a file per day (saDD, where DD is the day of the month) is created in /var/log/sysstat/ on Debian/Ubuntu or /var/log/sa/ on RHEL-based systems. Data is only recorded from the moment you enable collection, so turn it on before problems occur.

View the recorded data

# Today's disk data (-p shows device names such as sda)
sar -d -p

# Today's CPU data (look at the %iowait column)
sar -u

# Data for the 10th of the month, from 22:00 to 23:00
sar -d -p -f /var/log/sysstat/sa10 -s 22:00:00 -e 23:00:00

The columns of sar -d are tps (requests per second), rkB/s and wkB/s, areq-sz, aqu-sz, await and %util. Read them the same way as iostat -x and look for time periods where await and aqu-sz rise. Matching those periods against what was running at the time (backups, cron batch jobs, log compression and so on) often reveals the cause.

If you add an interval and count, as in sar -d 1 5, sar measures the current state instead of reading the records.

Step 4: Find I/O-heavy processes with pidstat and iotop

pidstat -d

pidstat -d 1 5

To see other users' processes as well, run it with sudo.

iotop

iotop lists processes ordered by the amount of I/O, like top (install it with sudo apt install iotop or similar).

sudo iotop -o    # show only processes actually doing I/O

Depending on the kernel configuration, iotop prints a warning about CONFIG_TASK_DELAY_ACCT or kernel.task_delayacct at startup, and the IO> column (percentage of time waiting on I/O) is not shown. The read and write amount columns still work in that case.

Examples of what to do once you find the cause

Notes

• A single measurement is only a snapshot. Measure for several minutes to tens of minutes, or use the sar records to see how things change over time.
• On a VPS or in the cloud, slowness can also come from other tenants sharing the same physical disk or from per-plan IOPS limits. If await is high while your own server does little I/O, check information from your provider as well.

For other uses of sar (CPU, memory, network), see Recording system statistics with the Linux sar command.

Summary

1. Use wa and b in top and vmstat to see whether processes are waiting on I/O.
2. Use iostat -dxyz 1 5 to find disks whose r_await, w_await and aqu-sz are rising. On SSDs and NVMe drives, do not judge by %util alone.
3. Look at past periods with sar -d -p. Enable data collection in advance.
4. Identify the processes doing the most reading and writing with pidstat -d or iotop -o, and deal with them.

Questions about WordPress? The publisher of this site is based in Japan and replies in Japanese. Contact (Japanese)