Troubleshooting Methodology
When facing issues, blindly searching and trial-and-error often wastes a great deal of time. This article introduces a general troubleshooting framework suitable for EL systems.
Core Principles
Section titled “Core Principles”- Observe first, act later — Gather sufficient information before modifying any configuration
- Start close, then go far — Check the most recent changes first, then expand the investigation scope
- Change one variable at a time — Modify only one thing at a time, verify, then move to the next
- Document the process — Record what you did and what you observed
General Troubleshooting Steps
Section titled “General Troubleshooting Steps”-
Clearly define the symptoms
Can you describe the problem specifically? “The website is down” is not specific enough — “Accessing port 80 returns connection refused” is an effective description.
-
Check recent changes
View recent system logs $ sudo journalctl --since "1 hour ago" --priority errView recently installed/updated packages $ dnf history list --reverse | tail -20 -
Check service status
View the status and logs of the relevant service $ sudo systemctl status <service-name>$ sudo journalctl -u <service-name> -n 50 --no-pager -
Check resources
Disk space $ df -hMemory usage $ free -hCPU and processes $ top -bn1 | head -20 -
Check networking
Port listening status $ sudo ss -tlnpFirewall rules $ sudo firewall-cmd --list-all -
Check SELinux
SELinux denials are a very common source of issues on EL systems:
View SELinux denial logs $ sudo ausearch -m avc -ts recentCheck current SELinux mode $ getenforce -
Review log files
System logs $ sudo journalctl -xeApplication-specific logs (nginx as an example) $ sudo tail -50 /var/log/nginx/error.log
Log Viewing Quick Reference
Section titled “Log Viewing Quick Reference”| Target | Command |
|---|---|
| All system logs | journalctl |
| Current boot logs | journalctl -b |
| Specific service logs | journalctl -u sshd |
| Follow logs in real time | journalctl -f |
| Error level and above | journalctl -p err |
| Specific time range | journalctl --since "2024-01-01" --until "2024-01-02" |
Common Pitfalls
Section titled “Common Pitfalls”- Do not disable SELinux as a first resort — First investigate whether it is an SELinux policy issue
- Do not blindly
chmod 777— This introduces security risks; find the correct permission settings instead - Do not overlook disk space — A full
/var/logis an extremely common cause of failures - Do not forget the firewall — If a service is running but inaccessible externally, it is very likely a firewalld rule issue