Postgres PANIC 'No space left on device': do not delete pg_wal

The alert fires at 02:14. Disk 100 percent. The application is throwing connection errors, and the Postgres log has stopped being a log and become a suicide note. You look at pg_wal, see nine thousand files with names like 0000000100000A2C000000FE, and every instinct you have says: those are old, delete some, get the database back up.

Do not. That instinct is how a two-hour outage becomes a restore from last night's backup.

WAL files are not logs in the sense you are thinking of. They are the only durable record of writes that have not yet been checkpointed into the data files. Postgres keeps a segment because it still needs it. Delete the wrong one and the cluster will not recover, and the error you get next is far worse than the one you have now.

The exact error: PANIC: could not write to file "pg_wal/xlogtemp..."

This is what it looks like in the server log:

PANIC:  could not write to file "pg_wal/xlogtemp.11399": No space left on device
LOG:  WAL writer process (PID 11399) was terminated by signal 6: Aborted
LOG:  terminating any other active server processes
LOG:  all server processes terminated; reinitializing

Then, on restart, the ugly part:

LOG:  database system was interrupted; last known up at 2026-07-23 22:41:07 UTC
LOG:  database system was not properly shut down; automatic recovery in progress
PANIC:  could not write to file "pg_wal/xlogtemp.11487": No space left on device

That second block is the crash loop people arrive with. Postgres cannot start because recovery itself needs to write, and it cannot write because the disk is still full. It will keep trying, keep panicking, and keep filling your log file, which is often on the same filesystem, which makes it worse.

Why this happens

Postgres is write-ahead logged. Before any change reaches a data file, the change is written and flushed to a WAL segment. That ordering is the entire durability guarantee: if the machine loses power mid-transaction, replaying WAL from the last checkpoint reconstructs a consistent database.

So when the WAL write fails, Postgres has exactly two options. Continue, and silently give up crash safety for every transaction from now on. Or stop hard. It stops hard, with a PANIC, which is the highest severity level in the server and always takes the whole cluster down. This is not a bug or an over-reaction. A database that keeps accepting writes it cannot durably record is worse than a database that is down.

The second question is why pg_wal got big enough to fill the disk in the first place. Under normal operation Postgres recycles segments: at each checkpoint, segments older than the checkpoint's redo point are renamed and reused. That recycling has four separate locks on it, and any one of them held open will make WAL grow without bound.

When you rm a segment that a slot or recovery still needs, you have not freed space, you have removed a piece of the recovery chain. The next startup gives you could not open file "pg_wal/..." or requested WAL segment has already been removed, and at that point your options narrow to a restore from backup or pg_resetwal, which throws away the unreplayed WAL and leaves you with a cluster that starts but is not guaranteed to be consistent. If you have never actually tested that restore, this is the night you find out. That is the whole argument behind building backups you have actually restored.

The fix, in numbered steps

Step 1: Find out what is actually full

df -h
sudo du -xh --max-depth=1 /var/lib/postgresql/16/main | sort -h | tail

You are answering two questions: is pg_wal genuinely the thing that grew, and is it on the same filesystem as everything else. On many installs pg_wal is a symlink to a separate volume, and the full disk is somewhere else entirely. Do not fix the wrong problem.

Step 2: Free space without touching pg_wal

Candidates, in the order I try them:

# 1. Server logs. Usually the biggest easy win during a crash loop.
sudo ls -lhS /var/lib/postgresql/16/main/log | head
sudo truncate -s 0 /var/lib/postgresql/16/main/log/postgresql-2026-07-2*.log

# 2. Orphaned temp files from queries killed by the crash.
sudo find /var/lib/postgresql/16/main/base -type d -name pgsql_tmp -exec du -sh {} \;

# 3. Anything not Postgres at all on the same filesystem.
sudo du -xh --max-depth=2 /var | sort -h | tail -20

Use truncate -s 0 rather than rm on a log the server may still hold open, otherwise the space does not come back until the process closes the descriptor.

Orphaned pgsql_tmp directories are safe to clear only with the server stopped. Postgres removes them itself on startup, so in practice you rarely need to.

Step 3: Grow the volume if you can

If this is a cloud volume, this is the honest fix and it takes minutes:

sudo growpart /dev/nvme0n1 1
sudo resize2fs /dev/nvme0n1p1     # or xfs_growfs /mount/point on XFS
df -h

You need a few hundred megabytes free for Postgres to complete recovery, not gigabytes. Buy yourself headroom now and shrink your bill later, not the other way round.

Step 4: Start Postgres and let it checkpoint

sudo systemctl start postgresql@16-main
sudo tail -f /var/lib/postgresql/16/main/log/postgresql-*.log

Expected output, in this order:

LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  redo starts at A2C/FE000028
LOG:  redo done at A2D/12000000
LOG:  checkpoint starting: end-of-recovery immediate wait
LOG:  checkpoint complete: wrote 41823 buffers ...
LOG:  database system is ready to accept connections

The checkpoint complete line is the one that matters. After it, Postgres is free to recycle segments, and if nothing is holding them, pg_wal will shrink on its own over the next few checkpoints. Watch it:

watch -n 5 'du -sh /var/lib/postgresql/16/main/pg_wal'

Step 5: Find what stopped WAL from being recycled

If the directory does not shrink, something in the diagram above is still holding it. Start with replication slots, because that is the answer most of the time:

SELECT slot_name,
       slot_type,
       active,
       wal_status,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn))
         AS retained_wal
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;

A row with active = false and several gigabytes of retained_wal is your culprit. wal_status tells you how bad it is: reserved is healthy, extended means the slot has gone past wal_keep_size, unreserved means it is at risk, and lost means the required WAL is already gone and that consumer can never catch up.

Then check archiving, if you use it:

SELECT archived_count, last_archived_wal, last_archived_time,
       failed_count, last_failed_wal, last_failed_time
FROM pg_stat_archiver;

A failed_count climbing with a stale last_archived_time means every segment since then is pinned. Fix the archive destination (full disk on the far side, expired credentials, network) and the backlog drains by itself.

Step 6: Drop the dead slot, deliberately

Only after you have confirmed the consumer is genuinely gone and not merely slow:

SELECT pg_drop_replication_slot('old_standby_slot');

This is irreversible for that consumer. If it is a standby, that standby now needs to be rebuilt from a fresh base backup. If it is a change-data-capture pipeline, it will resume from a point that has been discarded and you will have a gap. Know which one you are destroying before you press enter.

Step 7: Put a ceiling on it so this cannot repeat

ALTER SYSTEM SET max_slot_wal_keep_size = '20GB';
SELECT pg_reload_conf();

This caps how much WAL a slot may pin. When a slot exceeds it, Postgres invalidates that slot (wal_status goes to lost) and recycles the WAL instead of filling the disk. You are choosing, in advance, to break one replica rather than take the whole primary down. That is almost always the right trade, and it is the single most valuable line in this article for anyone running a standby they do not monitor closely.

Verification

Three checks, in this order:

# 1. Space is back and stable, not just briefly better
df -h /var/lib/postgresql
du -sh /var/lib/postgresql/16/main/pg_wal

-- 2. The cluster accepts writes and can checkpoint on demand
CHECKPOINT;
CREATE TABLE _wal_smoke_test (t timestamptz);
INSERT INTO _wal_smoke_test VALUES (now());
DROP TABLE _wal_smoke_test;

-- 3. Nothing is pinning WAL any more
SELECT count(*) FROM pg_replication_slots WHERE active = false;
SELECT failed_count FROM pg_stat_archiver;

A manual CHECKPOINT that returns promptly, an insert that commits, zero inactive slots and a failed_count that is not climbing means you are genuinely out of it. The log should also show a clean checkpoint complete with no PANIC lines after your restart timestamp.

What people get wrong

rm the oldest files in pg_wal. The most common and most destructive move. The filenames sort lexically, so "oldest" looks obvious, but the file that has not been checkpointed, archived or consumed by a slot may well be one of those. Deleting it converts a recoverable incident into requested WAL segment has already been removed and a restore from backup. There is no supported way to hand-pick safe segments in a live primary's pg_wal. Postgres already does that at every checkpoint, correctly, using state you cannot see from ls.

pg_resetwal. Sometimes suggested as "just reset the WAL and it starts again". It does start again. It starts by throwing away every change that had not made it into the data files, and the resulting cluster is not guaranteed consistent, including indexes that may now reference rows that do not exist. Its own documentation treats it as a last resort. Treat it as the thing you do after you have already accepted data loss, not as a way to avoid it.

Turning off fsync or full_page_writes to make WAL smaller. full_page_writes = off does reduce WAL volume meaningfully, and it also removes the protection against torn pages on an unclean shutdown. You are on a machine that just had an unclean shutdown. Do not.

Assuming a big pg_wal means Postgres is misconfigured. A busy cluster mid-bulk-load legitimately holds a lot of WAL between checkpoints. Size the volume for the peak. If you are chronically short of room, archiving old rows out of the hot tables beats resizing the volume every quarter.

Restarting repeatedly and hoping. Each attempt writes more log. If the log directory is on the full filesystem, you are making the problem worse while you wait.

Prepare for the next one: the ballast file

Keep a disposable file on the same filesystem so that at 02:14 you have something safe to delete instantly:

sudo fallocate -l 2G /var/lib/postgresql/BALLAST
# during an incident:
sudo rm /var/lib/postgresql/BALLAST
# afterwards, recreate it
sudo fallocate -l 2G /var/lib/postgresql/BALLAST

Two gigabytes of nothing turns a "which of these files can I safely delete" panic into a single command. Pair it with an alert at 80 percent disk rather than 95, because on a busy write workload the window between "noticeable" and "PANIC" can be under an hour. On a small VPS, which is where most of my clients run, it is shorter still.

When it is still broken

1. Postgres still will not start after freeing space. Read the very first PANIC or FATAL in the log, not the last. If it names a missing file rather than a full disk, someone has already deleted from pg_wal and you are now in restore territory.
2. The disk fills again within hours. The slot or archive problem was never fixed, or you have a runaway query writing enormous temp files. Set temp_file_limit per session to make the offending query fail instead of the cluster.
3. pg_wal shrinks but the filesystem stays full. Something is holding deleted files open. sudo lsof +L1 will show them. Usually a log rotated with rm while the server held the handle.
4. Space is fine but writes still fail. Different problem entirely. Check for transaction ID exhaustion using the wraparound runbook, and check that the filesystem has not been remounted read-only after IO errors with dmesg -T | tail.

The rule that survives every version of this incident: Postgres deletes WAL when it is safe to do so, and you cannot compute "safe" from a directory listing. Your job during a disk-full PANIC is to give it room to finish thinking, not to think for it.