Log Management

Logs are the primary data source for troubleshooting issues, auditing security events, and analyzing system behavior. This article explains how to effectively manage and analyze logs on RHEL-based distributions.

Log System Overview

In CentOS/AlmaLinux/Rocky Linux, the logging system consists of two core components:

journald (systemd-journald): The systemd journal daemon that collects all systemd unit, kernel, and boot logs, stored in binary format
rsyslog: The traditional syslog daemon that receives logs forwarded by journald, writes them to text files, and supports remote transmission

The two work together: journald handles collection, while rsyslog handles persistent storage and forwarding.

Applications/Services -> journald (binary log) -> rsyslog -> text logs under /var/log/
                                                     |
                                                Remote log server

journald Configuration and Usage

Basic Queries

# View all logs
journalctl

# View the latest logs (similar to tail -f)
journalctl -f

# View only the last 100 lines
journalctl -n 100

# View logs for a specific service
journalctl -u nginx
journalctl -u sshd -f

# View kernel logs
journalctl -k

# View logs by time range
journalctl --since "2026-03-20 00:00:00" --until "2026-03-20 23:59:59"
journalctl --since "1 hour ago"
journalctl --since today

# Filter by priority (0=emerg, 3=err, 4=warning, 6=info)
journalctl -p err                # Errors and above only
journalctl -p warning -u nginx   # Warning-level nginx logs

# Output in JSON format
journalctl -u nginx -o json-pretty -n 5

# Check disk usage
journalctl --disk-usage

Persistent Storage Configuration

By default, journald logs may only be stored in memory (/run/log/journal/) and are lost after a reboot. To enable persistent storage:

# Create the persistent storage directory
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal

# Edit the configuration
sudo tee /etc/systemd/journald.conf.d/persistent.conf > /dev/null <<'EOF'
[Journal]
Storage=persistent
SystemMaxUse=2G
SystemMaxFileSize=256M
MaxRetentionSec=3month
Compress=yes
EOF

# Restart journald
sudo systemctl restart systemd-journald

Key parameter descriptions:

Parameter	Description
`Storage=persistent`	Persist logs to disk
`SystemMaxUse=2G`	Maximum 2G disk space for logs
`SystemMaxFileSize=256M`	Maximum 256M per log file
`MaxRetentionSec=3month`	Retain logs for up to 3 months
`Compress=yes`	Enable compression

rsyslog Configuration

Default Log Files

# View rsyslog default rules
cat /etc/rsyslog.conf

# Default log layout on RHEL-based systems:
# /var/log/messages    - Most system logs
# /var/log/secure      - Authentication and authorization logs
# /var/log/maillog     - Mail logs
# /var/log/cron        - Cron job logs
# /var/log/boot.log    - Boot logs

Custom Log Rules

# Create a separate log file for a specific program
sudo tee /etc/rsyslog.d/myapp.conf > /dev/null <<'EOF'
# Write logs with the myapp tag to a separate file
if $programname == 'myapp' then /var/log/myapp.log
& stop

# Write all local6 facility logs to a specified file
local6.*    /var/log/custom-app.log
EOF

sudo systemctl restart rsyslog

Log Templates

rsyslog supports custom log formats:

sudo tee /etc/rsyslog.d/custom-template.conf > /dev/null <<'EOF'
# Custom log format template
template(name="CustomFormat" type="string"
    string="%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg%\n"
)

# Apply the custom format to specific logs
local6.*    /var/log/custom-app.log;CustomFormat
EOF

Centralized Logging (Remote Collection)

Configure the receiving end on the log collection server:

sudo tee /etc/rsyslog.d/remote.conf > /dev/null <<'EOF'
# Enable TCP reception (port 514)
module(load="imtcp")
input(type="imtcp" port="514")

# Store remote logs in directories organized by hostname
template(name="RemoteHost" type="string"
    string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
)

# Use custom storage path for remote logs
if $fromhost-ip != '127.0.0.1' then {
    action(type="omfile" dynaFile="RemoteHost")
    stop
}
EOF

sudo systemctl restart rsyslog

# Allow through the firewall
sudo firewall-cmd --permanent --add-port=514/tcp
sudo firewall-cmd --reload

Configure log forwarding on the client:

sudo tee /etc/rsyslog.d/forward.conf > /dev/null <<'EOF'
# Forward all logs to the remote server via TCP
*.* @@log-server.example.com:514

# @  means UDP
# @@ means TCP (recommended for better reliability)

# Enable disk buffer queue if the remote is unavailable
action(
    type="omfwd"
    target="log-server.example.com"
    port="514"
    protocol="tcp"
    queue.type="LinkedList"
    queue.filename="forward_queue"
    queue.maxdiskspace="1g"
    queue.saveonshutdown="on"
    action.resumeRetryCount="-1"
)
EOF

sudo systemctl restart rsyslog

TLS Encrypted Transmission

In production environments, log transmission should be encrypted:

# Install the TLS module
sudo dnf install rsyslog-gnutls -y

# TLS configuration on the log server
sudo tee /etc/rsyslog.d/tls-server.conf > /dev/null <<'EOF'
global(
    defaultNetstreamDriver="gtls"
    defaultNetstreamDriverCAFile="/etc/pki/rsyslog/ca.pem"
    defaultNetstreamDriverCertFile="/etc/pki/rsyslog/server-cert.pem"
    defaultNetstreamDriverKeyFile="/etc/pki/rsyslog/server-key.pem"
)

module(load="imtcp"
    StreamDriver.Name="gtls"
    StreamDriver.Mode="1"
    StreamDriver.AuthMode="x509/name"
)

input(type="imtcp" port="6514")
EOF

# TLS configuration on the client
sudo tee /etc/rsyslog.d/tls-client.conf > /dev/null <<'EOF'
global(
    defaultNetstreamDriver="gtls"
    defaultNetstreamDriverCAFile="/etc/pki/rsyslog/ca.pem"
    defaultNetstreamDriverCertFile="/etc/pki/rsyslog/client-cert.pem"
    defaultNetstreamDriverKeyFile="/etc/pki/rsyslog/client-key.pem"
)

action(
    type="omfwd"
    target="log-server.example.com"
    port="6514"
    protocol="tcp"
    StreamDriver="gtls"
    StreamDriverMode="1"
    StreamDriverAuthMode="x509/name"
)
EOF

logrotate Log Rotation

logrotate prevents log files from growing indefinitely by automatically compressing and cleaning up old logs.

Global Configuration

cat /etc/logrotate.conf

# Default settings:
# - Rotate weekly
# - Keep 4 weeks
# - Create new files
# - Include configurations from /etc/logrotate.d/

Creating Rotation Rules for an Application

sudo tee /etc/logrotate.d/myapp > /dev/null <<'EOF'
/var/log/myapp.log
/var/log/myapp-error.log
{
    daily               # Rotate daily
    missingok           # Don't error if the file is missing
    rotate 30           # Keep 30 rotated copies
    compress            # gzip compression
    delaycompress       # Delay compression by one rotation (allows the program to continue writing)
    notifempty          # Don't rotate empty files
    create 0640 myapp myapp  # Permissions and ownership for the new file
    dateext             # Use date as suffix (instead of numbers)
    dateformat -%Y%m%d  # Date format
    sharedscripts       # Run scripts only once for multiple files
    postrotate
        # Notify the application to reopen log files after rotation
        /bin/kill -USR1 $(cat /var/run/myapp.pid 2>/dev/null) 2>/dev/null || true
    endscript
}
EOF

Nginx Log Rotation Example

sudo tee /etc/logrotate.d/nginx > /dev/null <<'EOF'
/var/log/nginx/*.log {
    daily
    missingok
    rotate 60
    compress
    delaycompress
    notifempty
    create 0640 nginx adm
    dateext
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}
EOF

Manually Testing Rotation

# Debug mode (don't actually execute, just show what would happen)
sudo logrotate -d /etc/logrotate.d/myapp

# Force rotation
sudo logrotate -f /etc/logrotate.d/myapp

# View the logrotate status file
cat /var/lib/logrotate/logrotate.status

Log Analysis Techniques

Filtering with grep

# Search for error messages
grep -i "error" /var/log/messages
grep -i "failed" /var/log/secure

# Search within a specific time period
grep "Mar 20 1[0-5]:" /var/log/messages   # Logs from Mar 20, 10:00-15:59

# Exclude noise
grep -i "error" /var/log/messages | grep -v "No error"

# Recursive search in a directory
grep -r "out of memory" /var/log/

# Show context around matching lines
grep -B 3 -A 5 "kernel panic" /var/log/messages

# Count occurrences
grep -c "Failed password" /var/log/secure

Analyzing with awk

# Count failed SSH login attempts by IP
awk '/Failed password/ {for(i=1;i<=NF;i++) if($i=="from") print $(i+1)}' /var/log/secure \
    | sort | uniq -c | sort -rn | head -20

# Count Nginx requests by hour
awk '{print $4}' /var/log/nginx/access.log \
    | cut -d: -f1-2 | sort | uniq -c

# HTTP status code distribution
awk '{print $9}' /var/log/nginx/access.log \
    | sort | uniq -c | sort -rn

# Find the slowest URLs (assuming response time is in the last column)
awk '{print $NF, $7}' /var/log/nginx/access.log \
    | sort -rn | head -20

# Requests per minute (find traffic peaks)
awk '{print $4}' /var/log/nginx/access.log \
    | cut -d: -f1-3 | sort | uniq -c | sort -rn | head -10

Using sed and Other Tools

# Extract logs for a specific time range (journalctl format)
sed -n '/^Mar 20 14:00/,/^Mar 20 15:00/p' /var/log/messages

# Monitor logs in real time with keyword highlighting
tail -f /var/log/messages | grep --color=auto -i "error\|warning\|fail"

# View login records using last
last -20                    # Last 20 login records
lastb -20                   # Last 20 failed logins
last reboot                 # Reboot history

# Search audit logs using ausearch (requires auditd)
ausearch -m avc             # SELinux denial events
ausearch -k login_events    # Search by audit rule

Practical Log Analysis Script

#!/bin/bash
# Daily log summary report

DATE=$(date -d yesterday '+%b %d')
REPORT="/tmp/log_summary_$(date +%Y%m%d).txt"

{
echo "=========================================="
echo "  Log Summary Report - ${DATE}"
echo "=========================================="
echo ""

echo "--- SSH Failed Login TOP 10 ---"
grep "${DATE}" /var/log/secure 2>/dev/null \
    | grep "Failed password" \
    | awk '{for(i=1;i<=NF;i++) if($i=="from") print $(i+1)}' \
    | sort | uniq -c | sort -rn | head -10
echo ""

echo "--- Error Log Statistics ---"
echo "Errors in messages: $(grep -ci "error" /var/log/messages 2>/dev/null || echo 0)"
echo "Failures in secure: $(grep -ci "fail" /var/log/secure 2>/dev/null || echo 0)"
echo ""

echo "--- Disk Space Alerts ---"
df -h | awk 'NR>1 {gsub(/%/,"",$5); if($5+0>80) print $0}'
echo ""

echo "--- Recent Reboot History ---"
last reboot | head -5
echo ""

echo "--- OOM Killer Events ---"
grep -c "Out of memory" /var/log/messages 2>/dev/null || echo "0 events"

} > "$REPORT"

cat "$REPORT"
# Optional: send via email
# mail -s "Log Summary ${DATE}" [email protected] < "$REPORT"

Advanced: Centralized Log Platforms

When the number of servers grows, a dedicated log platform is needed for unified collection, searching, and analysis.

ELK Stack (Elasticsearch + Logstash + Kibana)

ELK is the most popular log analysis platform:

Elasticsearch: Log storage and search engine
Logstash: Log collection and processing pipeline
Kibana: Web visualization interface

Using Filebeat as a lightweight log shipper:

# Install Filebeat
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

sudo tee /etc/yum.repos.d/elastic.repo > /dev/null <<'EOF'
[elastic-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

sudo dnf install filebeat -y

# Configure Filebeat to collect system logs
sudo tee /etc/filebeat/filebeat.yml > /dev/null <<'EOF'
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/messages
      - /var/log/secure
    fields:
      type: syslog

  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
    fields:
      type: nginx-access

output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]
  username: "elastic"
  password: "your_password"
EOF

sudo systemctl enable --now filebeat

Grafana Loki

Loki is a lightweight log aggregation system from Grafana, positioned as an alternative to ELK with much lower resource consumption:

# Install Promtail (Loki's log collector)
cd /tmp
curl -LO https://github.com/grafana/loki/releases/download/v3.0.0/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip
sudo cp promtail-linux-amd64 /usr/local/bin/promtail
sudo chmod +x /usr/local/bin/promtail

# Basic configuration
sudo mkdir -p /etc/promtail
sudo tee /etc/promtail/config.yml > /dev/null <<'EOF'
server:
  http_listen_port: 9080

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://loki-server:3100/loki/api/v1/push

scrape_configs:
  - job_name: syslog
    static_configs:
      - targets:
          - localhost
        labels:
          job: syslog
          host: __HOSTNAME__
          __path__: /var/log/messages

  - job_name: nginx
    static_configs:
      - targets:
          - localhost
        labels:
          job: nginx
          host: __HOSTNAME__
          __path__: /var/log/nginx/*.log
EOF

# Replace hostname
sudo sed -i "s/__HOSTNAME__/$(hostname)/" /etc/promtail/config.yml

Loki’s advantage is that it does not index the full text of logs. Instead, it organizes log streams using labels, consuming far fewer resources than Elasticsearch, making it ideal for small to medium-scale deployments.

Summary

Task	Tool
View service logs	`journalctl -u service-name`
View system logs	`/var/log/messages` or `journalctl`
Log persistence	journald `Storage=persistent`
Log rotation	logrotate
Remote centralized collection	rsyslog TCP/TLS forwarding
Log search and analysis	grep / awk / journalctl filtering
Large-scale log platform	ELK Stack or Grafana Loki

Develop a habit of regularly reviewing logs and use automated scripts or log platforms to detect anomalies promptly. This is a key practice for keeping servers running smoothly.