Troubleshooting a Linux server in a fixed order: running, listening, reachable, logs, changes

Something on the server is broken, and the instinct is to start changing things: restart the service, then the VM, then edit a config you half remember, then reinstall a package. Sometimes it comes back and you never learn why. Often you have now changed four things and broken a fifth. Good troubleshooting is boring on purpose: a fixed order of questions, each answered by a command, and no changes until you know what is wrong.

The fixed order

Stop at the first unexpected answer and work there.

1. Is it running? systemctl status app, or docker ps for containers.
2. Is it listening? sudo ss -tulpn | grep 8080 shows the port and the address it is bound to.
3. Can you reach it? From the server: curl -v http://127.0.0.1:8080/. From your laptop: the same with the server's address. The difference between those two answers is a bind address, a firewall or the network.
4. What do the logs say? journalctl -u app -n 100, then journalctl -p err -b for everything else that went wrong this boot.
5. What changed? Most outages follow a change. Find it:

tail -n 30 /var/log/apt/history.log             # recent package changes
sudo find /etc -mtime -2 -type f 2>/dev/null    # config changed in 2 days
journalctl --list-boots | tail -3               # did it reboot?
journalctl -b -1 -n 20 --no-pager               # end of the previous boot
uptime

On Debian 13 the last command is gone; the previous boot's journal answers the same question.

The rule that makes the whole order work: write down every change you make while troubleshooting, with the time. A text file in your home directory is enough. If the fix does not work, you can undo exactly what you did. And before changing anything on a VM you care about, take a snapshot, so a bad "fix" is two seconds from undone.

Is the machine itself healthy?

When the problem is "everything is slow" rather than "one thing is broken", look at the whole machine:

uptime          # how long it has run, and the load averages
top             # live: CPU, memory, processes (q to quit)
free -h         # memory and swap
df -h           # disk space per filesystem
sudo iotop -o   # who is using the disk right now (sudo apt install iotop)

Load average, honestly

$ uptime
 10:42:17 up 12 days,  3:05,  2 users,  load average: 3.84, 2.10, 0.95

The three numbers are the average number of processes that were running or waiting to run over the last 1, 5 and 15 minutes. On Linux, processes waiting for disk I/O count too. Read them against the number of cores (nproc): on a 4-core VM, a load around 4 means the CPUs are fully busy, and well above it means work is queueing. The trend matters as much as the value: 3.84, 2.10, 0.95 is climbing, so something started recently. A high load with idle CPUs in top (low us and sy, high wa) points at the disk, not the processor.

Memory and swap

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       2.9Gi       121Mi        12Mi       1.0Gi       880Mi
Swap:          975Mi       410Mi       565Mi

A small free value is not a problem: Linux uses spare memory as disk cache and gives it back when programs need it. The column that matters is available. Swap in use is not a problem by itself; swap that keeps growing while available stays low means the machine needs more memory or a process is leaking. In top, press M to sort by memory.

When the kernel kills something

When memory runs out completely, the kernel's out-of-memory killer picks a process and kills it. The service just "dies", sometimes restarted by systemd, and its own log says nothing useful. The kernel log does:

journalctl -k -b | grep -i -E 'out of memory|oom'
sudo dmesg -T | grep -i -E 'killed process|oom'

The fix is never "turn off the OOM killer". Give the VM more memory, cap the hungry service (MemoryMax= in a systemd drop-in, or a memory limit on the container), or move it somewhere with room.

Disk full

A full root filesystem breaks things in strange ways: services fail to start, logins hang, APT cannot write its lists, databases go read-only. Find where the space went, from the top down:

df -h                                          # which filesystem is full?
df -i                                          # or out of inodes?
sudo du -xh --max-depth=1 / | sort -h | tail   # which top-level directory?
sudo du -xh --max-depth=1 /var | sort -h | tail
sudo find / -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null

The usual suspects:

Do not delete a log file that a running service is still writing to: the space is not freed until the service closes it. Truncate it instead with sudo truncate -s 0 /var/log/app.log. And never delete files in /var/lib/docker by hand; use Docker's own prune commands.

Symptom to first command

Where to start looking for the most common complaints. It is a starting point, not a diagnosis.

Asking for help well

When the answer is in a forum or a bug tracker, the people there can only help with what you show them:

• The exact command you ran and its exact output, copied, not paraphrased.
• The relevant log lines (journalctl -u app -n 50 --no-pager), with secrets removed.
• Versions: cat /etc/debian_version, uname -r, and the app's version.
• What you already tried, and what changed before it broke.

Writing that down often solves the problem before you post it, and it is exactly the note you will want next time.

Checklist

• Follow the order: running, listening, reachable, logs, changes.
• Write down every change you make while troubleshooting.
• Snapshot before trying a fix on anything important.
• Read load average against nproc, and memory by the available column.
• Check the kernel log for OOM kills when a service dies without a reason.
• Truncate live logs instead of deleting them.

Found this useful? Subscribe on YouTube for the daily video, or follow the RSS feed.