Performance Issues

The first step in performance troubleshooting is identifying the bottleneck: CPU, memory, disk I/O, or network. This article introduces commonly used performance analysis tools and troubleshooting methods.

Quick Overview: Overall System Status

Before diving into analysis, get a snapshot of the overall system status.

$ uptime

Example output: 10:32:05 up 45 days, load average: 2.50, 1.80, 0.95

The load average values represent the 1-minute, 5-minute, and 15-minute averages. As a rule of thumb, load should not consistently exceed the number of CPU cores.

$ nproc

$ free -h          # Memory
$ df -h            # Disk space
$ uptime           # Load
$ sudo ss -s       # Network connection statistics

CPU Performance Troubleshooting

Using top

$ top

Key metrics in the top interface:

%us — User-space CPU usage
%sy — Kernel-space CPU usage
%wa — CPU time waiting for I/O (high values indicate disk is the bottleneck)
%id — Idle CPU time

Common operations:

Press 1 — Show per-core CPU usage
Press P — Sort by CPU usage
Press M — Sort by memory usage
Press c — Show full command line

$ top -bn1 | head -30

Using htop

htop provides a more intuitive interface and needs to be installed from EPEL:

$ sudo dnf install -y epel-release
$ sudo dnf install -y htop

$ htop

In htop, press F6 to select sort criteria and F5 to toggle the tree view.

Using mpstat to View Per-Core CPU Status

$ sudo dnf install -y sysstat

$ mpstat -P ALL 1 5

Identifying High-CPU Processes

$ ps aux --sort=-%cpu | head -11

$ top -H -p <PID>

Common CPU Bottleneck Causes

Application bugs — Infinite loops, inefficient algorithms
Too many processes/threads — High context-switching overhead
Interrupt storms — Check /proc/interrupts
Compilation or compression tasks — CPU-intensive operations

$ vmstat 1 5

The cs column shows context switches per interval. Tens of thousands per second or more may impact performance.

Memory Performance Troubleshooting

Using free to Check Memory

$ free -h

Understanding the output:

total — Total physical memory
used — Memory in use
free — Completely free memory
buff/cache — Memory used as buffers/cache (reclaimable)
available — Actually available memory (free + reclaimable cache)

Important: Linux aggressively uses free memory for disk caching, which is normal behavior. To determine whether memory is insufficient, look at the available column, not the free column.

Monitoring with vmstat

$ vmstat 1 10

Key columns:

si / so — Pages swapped in/out. Consistently greater than 0 indicates memory shortage
free — Free memory (KB)
buff / cache — Buffer and cache memory

Identifying Memory-Hungry Processes

$ ps aux --sort=-%mem | head -11

$ cat /proc/<PID>/status | grep -E '(VmRSS|VmSize|VmSwap)'

Checking Swap Usage

$ swapon --show

$ for pid in /proc/[0-9]*; do
    name=$(cat $pid/comm 2>/dev/null)
    swap=$(grep VmSwap $pid/status 2>/dev/null | awk '{print $2}')
    [ -n "$swap" ] && [ "$swap" -gt 0 ] && echo "$swap kB - $name (PID: $(basename $pid))"
  done | sort -rn | head -10

Common Memory Bottleneck Causes

Memory leaks — An application continuously consuming more memory
Unreleased cache — Certain applications holding large caches
OOM Killer — The system automatically kills processes when memory is exhausted

$ dmesg | grep -i "out of memory"
$ journalctl --since "7 days ago" | grep -i "oom-kill"

Disk I/O Performance Troubleshooting

Using iostat

$ iostat -x 1 5

Key metrics:

%util — Disk utilization. Consistently near 100% means the disk is saturated
await — Average I/O wait time (milliseconds). Normal is 5-20ms for HDD, 0.5-2ms for SSD
r/s, w/s — Read/write operations per second (IOPS)
rkB/s, wkB/s — Read/write throughput per second

$ iostat -x /dev/sda 1 5

Using iotop to Identify I/O-Heavy Processes

$ sudo dnf install -y iotop

$ sudo iotop -o

The -o flag shows only processes with I/O activity.

Checking Disk Space

$ df -h

$ df -i

Note: Even if disk space appears to be available, inode exhaustion can also prevent new files from being created.

$ sudo du -sh /var/log/* | sort -rh | head -10

$ sudo find / -xdev -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh | head -10

Common Disk Bottleneck Causes

Oversized log files — /var/log is full
Database I/O intensive — Unoptimized queries
Frequent swap reads/writes — Actually caused by memory shortage
Degraded RAID — Array performance degradation

Network Performance Troubleshooting

Using sar to Check Network Throughput

$ sar -n DEV 1 5

Key metrics:

rxpck/s, txpck/s — Packets received/sent per second
rxkB/s, txkB/s — Bytes received/sent per second
%ifutil — Network interface utilization

Using ss to Check Connection Status

$ ss -s

$ ss -ant | awk '{print $1}' | sort | uniq -c | sort -rn

A large number of TIME_WAIT connections may indicate excessive short-lived connections. A large number of CLOSE_WAIT may indicate an application bug.

Bandwidth Testing

$ sudo dnf install -y iperf3

$ iperf3 -s

$ iperf3 -c <server-IP>

Common Network Bottleneck Causes

Bandwidth saturation — Traffic exceeds NIC or link capacity
Packet loss — Poor network quality or buffer overflow
Too many connections — Exceeding system or application limits
DNS latency — DNS lookup on every request

Comprehensive Analysis Tool: sar

sar (System Activity Reporter) can review historical performance data.

$ sudo dnf install -y sysstat
$ sudo systemctl enable --now sysstat

$ sar -u

$ sar -r

$ sar -d

$ sar -u -f /var/log/sa/sa20

$ sar -u -s 08:00:00 -e 12:00:00

Quick Performance Issue Identification Flow

Follow this sequence to quickly identify most performance issues:

1. Check System Load

$ echo "Load: $(cat /proc/loadavg | awk '{print $1, $2, $3}'), CPU cores: $(nproc)"

2. Determine the Bottleneck Type

$ vmstat 1 5

High us + sy → CPU bottleneck
High wa → Disk I/O bottleneck
si/so consistently above 0 → Memory shortage
All of the above normal but system is slow → Likely a network or application-layer issue

3. Deep Dive

Based on the results from step 2, use the tools in the corresponding section for in-depth analysis.

Common Quick Fixes

High CPU: Limit Process CPU Usage

$ nice -n 19 <command>

$ sudo dnf install -y cpulimit
$ sudo cpulimit -p <PID> -l 50   # Limit to 50% CPU

Insufficient Memory: Adjust Swap

$ sudo fallocate -l 2G /swapfile
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile

$ sudo sync && sudo sysctl vm.drop_caches=3

Disk Full: Quickly Free Space

$ sudo dnf clean all

$ sudo journalctl --vacuum-size=200M

$ sudo dnf remove --oldinstallonly --setopt installonly_limit=2 kernel

Too Many Network Connections: Tune Kernel Parameters

$ sysctl net.core.somaxconn
$ cat /proc/sys/net/ipv4/tcp_max_tw_buckets

$ sudo sysctl -w net.core.somaxconn=65535
$ sudo sysctl -w net.ipv4.tcp_max_tw_buckets=65535