Performance Tuning

Performance tuning is a systematic process: first identify the bottleneck, then optimize accordingly. Blindly adjusting parameters often does more harm than good. This article covers common tuning techniques and their applicable scenarios, helping you improve server performance step by step from kernel parameters to the application layer.

Pre-Tuning Preparation

Establishing a Baseline

Before tuning, you must record the current performance baseline; otherwise, there is no way to measure the effect of optimizations.

# Record basic system information
uname -a
cat /etc/os-release
lscpu
free -h
lsblk

# Install performance analysis tools
sudo dnf install sysstat perf iotop htop -y

# Enable sysstat data collection
sudo systemctl enable --now sysstat

# View historical performance data
sar -u 1 5      # CPU usage
sar -r 1 5      # Memory usage
sar -d 1 5      # Disk I/O
sar -n DEV 1 5  # Network throughput

Identifying the Bottleneck

# Quickly determine the bottleneck type
# CPU bottleneck: load average consistently higher than the number of CPU cores
uptime

# Memory bottleneck: low available memory with active swap
free -h
vmstat 1 5

# I/O bottleneck: high iowait
iostat -xz 1 5

# Network bottleneck: packet drops or saturated bandwidth
ss -s
sar -n DEV 1 5

Kernel Parameter Tuning (sysctl)

sysctl is used to modify kernel parameters at runtime.

Viewing and Modifying Parameters

# View all parameters
sysctl -a

# View a specific parameter
sysctl net.core.somaxconn
sysctl vm.swappiness

# Temporary change (lost after reboot)
sudo sysctl -w net.core.somaxconn=65535

# Permanent change
sudo tee /etc/sysctl.d/99-tuning.conf > /dev/null <<'EOF'
# Custom kernel parameter tuning
EOF

# Apply the configuration
sudo sysctl --system

Network Parameter Tuning

sudo tee /etc/sysctl.d/99-network.conf > /dev/null <<'EOF'
# ==============================
# Network Performance Tuning
# ==============================

# --- TCP Buffers ---
# Default and maximum TCP send/receive buffers (bytes)
# Format: min  default  max
net.core.rmem_default = 262144
net.core.rmem_max = 16777216
net.core.wmem_default = 262144
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# --- Connection Queues ---
# Maximum listen queue length (affects connection establishment under high concurrency)
net.core.somaxconn = 65535

# Half-open connection queue size
net.ipv4.tcp_max_syn_backlog = 65535

# NIC receive queue length
net.core.netdev_max_backlog = 65535

# --- TCP Connection Optimization ---
# Enable TCP Fast Open (reduces handshake latency)
net.ipv4.tcp_fastopen = 3

# TIME_WAIT related
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_max_tw_buckets = 262144

# Keepalive parameters
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 5

# --- Congestion Control ---
# Use the BBR congestion control algorithm (recommended)
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# --- Local Port Range ---
net.ipv4.ip_local_port_range = 1024 65535

# --- Other ---
# Allow more orphan sockets
net.ipv4.tcp_max_orphans = 262144

# SYN Flood protection
net.ipv4.tcp_syncookies = 1
EOF

sudo sysctl --system

Verify that BBR is active:

sysctl net.ipv4.tcp_congestion_control
# Output should be: net.ipv4.tcp_congestion_control = bbr

lsmod | grep bbr

Memory Parameter Tuning

sudo tee /etc/sysctl.d/99-memory.conf > /dev/null <<'EOF'
# ==============================
# Memory Tuning
# ==============================

# Swappiness: controls the kernel's tendency to swap memory pages to swap
# 0 = avoid swap as much as possible (suitable for memory-rich database servers)
# 10 = low swap tendency (recommended for most server scenarios)
# 60 = default value
vm.swappiness = 10

# Dirty page flush policy
# Background flush starts when dirty pages exceed this percentage of memory
vm.dirty_background_ratio = 5
# Foreground processes must wait for flush when dirty pages exceed this ratio
vm.dirty_ratio = 15
# Dirty page lifetime (centiseconds, 500 = 5 seconds)
vm.dirty_expire_centisecs = 500
# Flush thread wakeup interval (centiseconds)
vm.dirty_writeback_centisecs = 100

# VFS cache pressure (default 100)
# Below 100 = prefer keeping dentry/inode cache
# Above 100 = prefer reclaiming cache
vm.vfs_cache_pressure = 50

# Behavior when memory is exhausted
# 0 = heuristic OOM (default)
# 1 = allow overcommit
# 2 = disallow overcommit
vm.overcommit_memory = 0
EOF

sudo sysctl --system

Huge Pages

Huge pages reduce TLB misses and improve performance for memory-intensive applications (such as databases).

# View current huge page configuration
grep -i huge /proc/meminfo

# Calculate the number of huge pages needed
# For example, to allocate 4GB for the database: 4096MB / 2MB (default huge page size) = 2048 pages
echo "Huge pages needed: $((4096 / 2))"

# Set the number of huge pages
sudo sysctl -w vm.nr_hugepages=2048

# Permanent configuration
echo 'vm.nr_hugepages = 2048' | sudo tee -a /etc/sysctl.d/99-memory.conf
sudo sysctl --system

# Verify
grep -i huge /proc/meminfo
# HugePages_Total:    2048
# HugePages_Free:     2048
# Hugepagesize:       2048 kB

Caution

Once allocated, huge page memory cannot be used for other purposes. Calculate the exact amount based on your application’s actual memory needs and avoid over-allocation.

Transparent Huge Pages (THP)

THP is managed automatically by the kernel. It is beneficial for most server applications, but some databases (such as MongoDB and Redis) recommend disabling THP to avoid latency jitter.

# Check current status
cat /sys/kernel/mm/transparent_hugepage/enabled

# Temporarily disable
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

# Permanently disable (via systemd)
sudo tee /etc/systemd/system/disable-thp.service > /dev/null <<'EOF'
[Unit]
Description=Disable Transparent Huge Pages
DefaultDependencies=no
After=sysinit.target local-fs.target
Before=basic.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'
ExecStart=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/defrag'

[Install]
WantedBy=basic.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable disable-thp

ulimit Resource Limits

ulimit controls the maximum system resources available to user processes. Default values are typically conservative and may need adjusting for high-concurrency scenarios.

Viewing Current Limits

# View all limits for the current user
ulimit -a

# Key parameters:
# -n  Maximum open files (open files)
# -u  Maximum processes (max user processes)
# -l  Maximum locked memory (max locked memory)

Modifying Limits

# Modify system-level limits
sudo tee /etc/security/limits.d/99-tuning.conf > /dev/null <<'EOF'
# Format: <domain> <type> <item> <value>

# All users
*    soft    nofile    65535
*    hard    nofile    131072
*    soft    nproc     65535
*    hard    nproc     131072

# Specific user (e.g., nginx)
nginx    soft    nofile    131072
nginx    hard    nofile    262144

# Root user
root    soft    nofile    131072
root    hard    nofile    262144
EOF

For systemd-managed services, you also need to set limits in the service file:

# Method 1: Modify systemd default limits
sudo mkdir -p /etc/systemd/system.conf.d
sudo tee /etc/systemd/system.conf.d/limits.conf > /dev/null <<'EOF'
[Manager]
DefaultLimitNOFILE=131072
DefaultLimitNPROC=65535
EOF

# Method 2: Set limits for a specific service (recommended)
sudo systemctl edit nginx
# Add in the editor:
# [Service]
# LimitNOFILE=131072
# LimitNPROC=65535

sudo systemctl daemon-reload
sudo systemctl restart nginx

# Verify the actual limits for the service
cat /proc/$(pidof nginx | awk '{print $1}')/limits

tuned Tuning Profiles

tuned is a dynamic tuning daemon provided by RHEL-based distributions, with multiple built-in predefined tuning profiles.

Installation and Usage

sudo dnf install tuned -y
sudo systemctl enable --now tuned

# List all available tuning profiles
tuned-adm list

# Common profiles:
# throughput-performance  - High throughput optimization
# latency-performance     - Low latency optimization
# network-latency         - Network low latency
# network-throughput      - Network high throughput
# virtual-guest           - Virtual machine guest
# virtual-host            - Virtualization host
# postgresql              - PostgreSQL optimization

# View current profile
tuned-adm active

# Switch profiles
sudo tuned-adm profile throughput-performance

# Get the recommended profile
tuned-adm recommend

# View profile details
tuned-adm profile_info throughput-performance

Custom tuned Profile

# Create a custom profile based on an existing one
sudo mkdir -p /etc/tuned/my-web-server

sudo tee /etc/tuned/my-web-server/tuned.conf > /dev/null <<'EOF'
[main]
summary=Custom tuning for web server
include=throughput-performance

[sysctl]
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_tw_reuse = 1
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
vm.swappiness = 10

[vm]
transparent_hugepages = never

[disk]
readahead = 4096
EOF

# Apply the custom profile
sudo tuned-adm profile my-web-server
sudo tuned-adm active

I/O Scheduler

The I/O scheduler determines how disk read/write requests are ordered and merged.

Viewing and Modifying

# View the current I/O scheduler
cat /sys/block/sda/queue/scheduler
# Example output: [mq-deadline] kyber bfq none

# Temporary change
echo "kyber" | sudo tee /sys/block/sda/queue/scheduler

# Permanent change (via udev rules)
sudo tee /etc/udev/rules.d/60-io-scheduler.rules > /dev/null <<'EOF'
# SSD: use none (noop) or mq-deadline
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="none"

# HDD: use bfq
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"
EOF

Scheduler Selection Guide

Scheduler	Suitable Scenario
none (noop)	NVMe SSDs, virtual machines (host already handles scheduling)
mq-deadline	General SSDs, database scenarios (guarantees deadlines)
bfq	HDDs, desktop/interactive scenarios (fair scheduling)
kyber	High-speed SSDs, high throughput scenarios

I/O Read-Ahead Tuning

# View the current read-ahead value (in 512-byte sectors)
cat /sys/block/sda/queue/read_ahead_kb

# For sequential read-intensive scenarios, increase read-ahead
echo 2048 | sudo tee /sys/block/sda/queue/read_ahead_kb

# For random read-intensive scenarios (e.g., databases), reduce read-ahead
echo 128 | sudo tee /sys/block/sda/queue/read_ahead_kb

Advanced Network Performance Tuning

Interrupt Affinity

Bind NIC interrupts to specific CPU cores to reduce context switching:

# View NIC interrupt distribution
cat /proc/interrupts | grep eth0

# Automatic setup (using the irqbalance service)
sudo systemctl enable --now irqbalance

# Or manually bind (for multi-queue NICs)
# View NIC queues
ls /sys/class/net/eth0/queues/

# Installing the tuned network plugin can auto-optimize this

NIC Parameter Tuning

# Install ethtool
sudo dnf install ethtool -y

# View NIC information
ethtool eth0

# View and modify ring buffer size
ethtool -g eth0                          # View
sudo ethtool -G eth0 rx 4096 tx 4096    # Modify

# View and enable NIC features
ethtool -k eth0                          # View
sudo ethtool -K eth0 tso on             # Enable TCP Segmentation Offload
sudo ethtool -K eth0 gro on             # Enable Generic Receive Offload
sudo ethtool -K eth0 gso on             # Enable Generic Segmentation Offload

Connection Tracking Table Optimization

High-concurrency proxy/NAT servers need a larger connection tracking table:

sudo tee /etc/sysctl.d/99-conntrack.conf > /dev/null <<'EOF'
# Maximum connection tracking table entries
net.netfilter.nf_conntrack_max = 1048576

# Connection tracking hash table bucket size
# Typically set to 1/4 of nf_conntrack_max
net.netfilter.nf_conntrack_buckets = 262144

# Shorten timeout for various states
net.netfilter.nf_conntrack_tcp_timeout_established = 3600
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 15
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 30
EOF

sudo sysctl --system

File System Tuning

XFS Tuning

# View XFS mount options
mount | grep xfs

# Optimize mount options (edit /etc/fstab)
# noatime    - Don't update access time, reduces writes
# nodiratime - Don't update directory access time
# logbufs=8  - Increase log buffer count
# Example:
# /dev/sda1  /data  xfs  defaults,noatime,nodiratime,logbufs=8  0  0

# Remount to apply without rebooting
sudo mount -o remount,noatime,nodiratime /data

ext4 Tuning

# Optimize mount options
# noatime       - Don't update access time
# commit=60     - Data commit interval (seconds); increasing improves performance but increases data loss risk
# Example:
# /dev/sdb1  /data  ext4  defaults,noatime,commit=60  0  2

sudo mount -o remount,noatime /data

Performance Tuning Check Script

/usr/local/bin/perf_check.sh

#!/bin/bash
# Check current system performance parameter configuration

echo "============================================"
echo "  Performance Parameter Check - $(date)"
echo "============================================"
echo ""

echo "--- CPU ---"
echo "Cores: $(nproc)"
echo "Current Load: $(uptime | awk -F'load average:' '{print $2}')"
echo "Tuned Profile: $(tuned-adm active 2>/dev/null || echo 'tuned not installed')"
echo ""

echo "--- Memory ---"
free -h
echo "swappiness: $(sysctl -n vm.swappiness)"
echo "dirty_ratio: $(sysctl -n vm.dirty_ratio)"
echo "dirty_background_ratio: $(sysctl -n vm.dirty_background_ratio)"
echo "THP: $(cat /sys/kernel/mm/transparent_hugepage/enabled)"
echo "HugePages: $(grep HugePages_Total /proc/meminfo)"
echo ""

echo "--- Network ---"
echo "somaxconn: $(sysctl -n net.core.somaxconn)"
echo "tcp_max_syn_backlog: $(sysctl -n net.ipv4.tcp_max_syn_backlog)"
echo "tcp_congestion: $(sysctl -n net.ipv4.tcp_congestion_control)"
echo "ip_local_port_range: $(sysctl -n net.ipv4.ip_local_port_range)"
echo "tcp_tw_reuse: $(sysctl -n net.ipv4.tcp_tw_reuse)"
echo ""

echo "--- File Descriptors ---"
echo "System limit: $(sysctl -n fs.file-max)"
echo "Current usage: $(cat /proc/sys/fs/file-nr)"
echo "ulimit -n (current user): $(ulimit -n)"
echo ""

echo "--- I/O ---"
for disk in $(lsblk -dn -o NAME); do
    scheduler=$(cat /sys/block/$disk/queue/scheduler 2>/dev/null)
    readahead=$(cat /sys/block/$disk/queue/read_ahead_kb 2>/dev/null)
    rotational=$(cat /sys/block/$disk/queue/rotational 2>/dev/null)
    type="HDD"
    [ "$rotational" = "0" ] && type="SSD"
    echo "${disk} (${type}): scheduler=${scheduler}, readahead=${readahead}KB"
done
echo ""

echo "--- Connection Tracking ---"
if [ -f /proc/sys/net/netfilter/nf_conntrack_max ]; then
    echo "Max connections: $(sysctl -n net.netfilter.nf_conntrack_max)"
    echo "Current connections: $(cat /proc/sys/net/netfilter/nf_conntrack_count 2>/dev/null || echo N/A)"
else
    echo "nf_conntrack module not loaded"
fi

chmod +x /usr/local/bin/perf_check.sh

Tuning Principles

1. Measure before optimizing — Use sar, vmstat, iostat, perf to identify the real bottleneck
2. Change one parameter at a time — This is the only way to confirm which change produced the effect
3. Record every change — Makes it easy to roll back and review
4. Validate with load testing — Use tools like ab, wrk, fio, sysbench to verify optimization results
5. Watch for side effects — Optimizing one parameter may introduce issues in other areas

# Common benchmarking tools
sudo dnf install httpd-tools -y   # Provides ab
# ab -n 10000 -c 100 http://localhost/

# fio disk performance test
sudo dnf install fio -y
fio --name=randread --ioengine=libaio --rw=randread --bs=4k \
    --size=1G --numjobs=4 --runtime=60 --group_reporting

# sysbench CPU test
sudo dnf install sysbench -y
sysbench cpu --threads=$(nproc) run