The 3am Outage Playbook: Reading Linux Logs Like a Detective

Karani GeoffreyJune 18, 20267 min read

It is 03:12. Your phone is buzzing. The site returns 502. You SSH in and type grep -i error /var/log/* and 40,000 lines scroll past. Twenty minutes later you restart everything, it comes back, and you never find out what happened - which guarantees it happens again next month.

My thesis: the tools are not the problem, the order of operations is. Build a timeline before you build a theory, and hunt the first anomaly rather than the loudest one. Errors cascade; by the time you see the noisy ones you are reading symptoms of symptoms. Here is the method I use, in order.

Step 0: is it still broken, and do not destroy the crime scene

Restarting is a decision to give up on knowing why. Before you restart anything, spend 60 seconds capturing state:

uptime
systemctl --failed
df -h; df -i
free -h
ss -tlnp | head -20
ps -eo pid,user,rss,pcpu,etime,comm --sort=-rss | head -12
sudo dmesg -T | tail -40

Pipe all of that into a file so you can read it after the fire is out:

{ date; uptime; systemctl --failed; df -h; df -i; free -h; ss -tlnp; } \
  > /var/tmp/incident-$(date +%F-%H%M).txt 2>&1

Step 1: fix the clock question

Every log line you are about to read has a timestamp, and you need to know what "now" and "then" mean on this box.

timedatectl
journalctl --list-boots

IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
 -2 4a1e0d9c...  Fri 2026-07-10 09:12:04 EAT  Sat 2026-07-18 22:41:52 EAT
 -1 9f77b3a2...  Sat 2026-07-18 22:42:14 EAT  Tue 2026-07-21 02:47:03 EAT
  0 c0de5511...  Tue 2026-07-21 02:48:31 EAT  Tue 2026-07-21 03:12:44 EAT

That output alone told you something enormous: the machine rebooted at 02:47. This is no longer an application incident. Now go straight to the end of the previous boot:

journalctl -b -1 -p err
journalctl -b -1 -n 200 -o short-precise

If the last entries are ordinary and then simply stop, the machine was killed without a chance to log - host migration, hardware, or the provider. If you see Stopping ... messages, it was a clean shutdown, which means someone or something asked for it.

Step 2: the timeline, narrow and unfiltered

Pick a window that starts about 15 minutes before the first user complaint and read everything. Not filtered by error. Everything.

journalctl --since "2026-07-21 02:30" --until "2026-07-21 03:00" -o short-precise --no-pager

This is the single highest-value command in the article. The first anomaly is often an INFO line - "connection pool exhausted", "reloading configuration", "certificate expires in 0 days" - that a priority filter would have hidden from you. Read it like a story.

Once you have a candidate time, tighten and add priority:

journalctl --since "02:38" --until "02:50" -p warning -o short-precise
journalctl --since "02:38" --until "02:50" -u nginx -u billing -u mariadb

Priorities run 0 (emerg) to 7 (debug), and -p err means "err and worse". Use a range when you want a band: -p 3..4 for errors and warnings only.

Step 3: the journalctl flags worth memorising

Flag	What it does
`-u NAME`	One unit. Repeatable, so `-u nginx -u billing` interleaves both.
`-b`, `-b -1`	This boot, or the previous boot.
`-k`	Kernel messages only. Where OOM kills and disk errors live.
`-g PATTERN`	Grep with PCRE2, matching only the message field. Case-insensitive if all-lowercase.
`-f -n 100`	Follow, showing the last 100 lines first.
`-o short-precise`	Microsecond timestamps. Essential for ordering events under a second apart.
`-o cat`	Message text only, for piping into awk.
`-o json-pretty`	Every field, including `_PID`, `_COMM`, `_SYSTEMD_UNIT`, `_EXE`.
`_PID=`, `_UID=`	Field matches. `journalctl _UID=33` gives you everything www-data logged.
`--since`/`--until`	Accepts "today", "yesterday", "-2h", "09:00", or a full timestamp.

journalctl -u billing -g "timeout|refused|deadlock" --since "-6h"
journalctl -k -g "oom|I/O error|EXT4-fs error" --since "-2d"
journalctl _SYSTEMD_UNIT=billing.service -o json-pretty | head -40

Step 4: correlate with the web server

journald has your services; Nginx has your users. Line them up. Count 5xx per minute straight out of the access log:

awk '$9 ~ /^5/ {print substr($4, 2, 17)}' /var/log/nginx/access.log | uniq -c | tail -20

      3 21/Jul/2026:02:39
     47 21/Jul/2026:02:40
    412 21/Jul/2026:02:41
    398 21/Jul/2026:02:42

02:40 is your onset. Now find who complained loudest and about what:

awk '$9 ~ /^5/ {print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

And the error log will usually name the mechanism outright:

2026/07/21 02:41:07 [error] 1194#1194: *8842 connect() failed (111: Connection refused)
while connecting to upstream, client: 197.232.14.8, server: app.example.co.ke,
request: "GET /api/orders HTTP/2.0", upstream: "http://127.0.0.1:8081/api/orders"

Connection refused means nothing is listening. Not slow, not overloaded - absent. That points you back at the unit, and the story assembles itself:

journalctl -u billing --since "02:35" --until "02:45" -o short-precise

Jul 21 02:40:51.203 web01 systemd[1]: billing.service: A process of this unit has been killed by the OOM killer.
Jul 21 02:40:51.208 web01 systemd[1]: billing.service: Main process exited, code=killed, status=9/KILL
Jul 21 02:40:51.209 web01 systemd[1]: billing.service: Failed with result 'oom-kill'.
Jul 21 02:40:56.401 web01 systemd[1]: billing.service: Scheduled restart job, restart counter is at 5.
Jul 21 02:41:02.887 web01 systemd[1]: billing.service: Start request repeated too quickly.

Five restarts, rate limiter tripped, service left dead, Nginx returns 502 forever. The root cause is memory, not Nginx, and no amount of staring at Nginx would have found it.

The three culprits behind most 3am pages

1. Memory

sudo dmesg -T | grep -i "killed process"
cat /sys/fs/cgroup/system.slice/billing.service/memory.events
cat /proc/pressure/memory

2. Disk, including the one everyone forgets

df -h
df -i          # inodes - a mail queue or session directory can exhaust these at 4% disk use
sudo du -x -d1 /var 2>/dev/null | sort -h | tail
sudo lsof +L1 | head    # files deleted but still held open by a process

lsof +L1 is the one that saves you. You deleted a 30GB log, df still shows the disk full, and the space only returns when the holding process is restarted or the file descriptor is truncated. If du and df disagree, this is why.

3. Certificates and DNS

sudo certbot certificates
echo | openssl s_client -connect app.example.co.ke:443 -servername app.example.co.ke 2>/dev/null \
  | openssl x509 -noout -dates
systemd-analyze verify /etc/systemd/system/*.timer
resolvectl status | head -20
journalctl -u systemd-resolved --since "-1h"

Make sure the logs will still be there tomorrow

On some minimal cloud images, journald storage is volatile, which means everything vanishes on reboot - and reboot is exactly when you need it. Check and fix:

journalctl --disk-usage
ls -ld /var/log/journal 2>/dev/null || echo "VOLATILE - logs die on reboot"

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Then bound the size so logs never fill the disk (which is itself a classic outage):

sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/00-size.conf > /dev/null <<'EOF'
[Journal]
Storage=persistent
SystemMaxUse=800M
SystemMaxFileSize=100M
MaxRetentionSec=30day
EOF
sudo systemctl restart systemd-journald
journalctl --vacuum-size=800M
journalctl --vacuum-time=30d

Make your own logs worth reading

Two rules, and they cost nothing at build time:

Log to stdout and stderr. Let systemd capture it. Your own log-file rotation is another thing to get wrong, and it puts your app's story in a different place from everything else's.
Put a request ID in every line. Generate it in Nginx and pass it through:

proxy_set_header X-Request-ID $request_id;
log_format timed '$remote_addr [$time_local] "$request" $status '
                 'rt=$request_time urt=$upstream_response_time rid=$request_id';

Now one grep joins the user's 502 to the exact stack trace:

RID=$(awk '$0 ~ /" 502 /{print}' /var/log/nginx/billing.access.log | tail -1 | grep -o 'rid=[a-f0-9]*' | cut -d= -f2)
journalctl -u billing -g "$RID" -o short-precise

The habit that ends repeat outages

Before you close the laptop, write four lines into the runbook: when it started, what the first anomalous log line was, what you changed, and what would have detected it sooner. That last one is where your next alert comes from - in the example above it is a PSI threshold on memory, not another dashboard.

Detectives do not start with a suspect. They start with a timeline, and the suspect falls out of it. Logs work the same way.

Practise on a boring day. Pick a random hour last week, read it end to end, and see how much of your server's normal behaviour you did not know about. At 3am you want recognition, not discovery.

#journalctl#linux#logs#troubleshooting#sre