'No Space Left on Device' but df Shows Free Space: It's Inodes

It is 1am, a deploy just failed, and the server is refusing to write a single byte. Every command that touches disk dies the same way: No space left on device. So you run df -h, braced for a full disk, and instead you see a partition sitting at 40% used. Gigabytes free. The error must be lying.

The error is not lying. You are checking the wrong number. A filesystem has two independent things it can run out of: space (bytes) and inodes (the little records that describe each file). df -h only shows you the first one. When a directory fills up with millions of tiny files, you exhaust inodes long before you exhaust bytes, and the kernel returns the exact same ENOSPC error for both. Most guides never mention this, so admins spend an hour clearing logs that were never the problem.

The error: "No space left on device" with disk space free

$ echo "test" > /var/www/app/health
-bash: /var/www/app/health: No space left on device

$ df -h /var/www
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        79G   31G   45G  41% /

45G available and the write still fails. Now look at inodes instead:

$ df -i /var/www
Filesystem      Inodes    IUsed IFree IUse% Mounted on
/dev/sda1      5242880  5242880     0  100% /

There it is. IUse% 100%, IFree 0. Every inode is spoken for, so the filesystem cannot create even a zero-byte file. This is the check almost nobody runs first, and it is the whole diagnosis.

Why an "empty" disk runs out of inodes

When you format an ext4 filesystem with mkfs.ext4, it pre-allocates a fixed number of inodes based on the bytes-per-inode ratio, by default one inode for roughly every 16KB of capacity. That ratio is baked in at format time. A 79GB filesystem might get around 5 million inodes. That sounds like a lot until something starts creating files that are far smaller than 16KB on average.

Each file, directory, symlink and socket consumes exactly one inode regardless of size. A 1-byte file and a 1GB file each cost you one inode. So if a process writes ten million 200-byte files, it burns ten million inodes while barely denting the byte count. You end up with a partition that is 40% full and 100% out of inodes at the same time. The kernel has nowhere to record the next file's metadata, so it returns ENOSPC, the same errno as a genuinely full disk.

Step 1: Confirm it is inodes

df -i

Look for any mountpoint at IUse% 100%. If everything is well under 100%, stop here, your problem really is bytes and you should read your database or WAL directory instead. If a mount is at 100% inodes, note which one, because you only need to hunt inside that filesystem.

Step 2: Find the directory hoarding files

You need to find where the millions of files live. This one-liner counts files per directory on a single filesystem (the -xdev flag stops find from crossing into other mounts, which matters):

find / -xdev -type f -printf '%h\n' | sort | uniq -c | sort -rn | head -20

Expected output is a ranked list, largest offender first:

4812233 /var/spool/postfix/maildrop
 218440 /var/lib/php/sessions
  91002 /home/app/.cache/pip/http
   4021 /var/log/journal/9f3c

That top line, nearly five million files in the mail spool, is your inode leak. On a slow disk this find can take a while because it is stat-ing every file; if it crawls, narrow it to the suspect filesystem's mountpoint instead of /.

Step 3: Clear the offending files safely

Once you know the directory, delete or archive its contents. Do not rm -rf the directory itself if it is a spool or session store that a running service expects to exist. Delete the files inside it. For a genuinely huge directory, plain rm * will fail with "argument list too long", so use find with -delete:

# Example: clear a stuck postfix maildrop queue (stop the service first if appropriate)
find /var/spool/postfix/maildrop -type f -delete

# Example: expired PHP sessions older than 2 days
find /var/lib/php/sessions -type f -mtime +2 -delete

Deleting millions of inodes is itself slow, this is metadata work, not bulk byte deletion, so give it time. Watch progress in another terminal with watch df -i.

Step 4: Handle files a process is still holding open

If you deleted files but df -i did not drop, a running process is still holding those inodes open (a deleted-but-open file keeps its inode until the last file descriptor closes). Find and restart the holder:

lsof +L1 | head   # lists open files with a link count of 0 (deleted but held)

Restart the service named in that output and the inodes are released.

Verification

Re-run the inode check and confirm the write works:

df -i /var/www
echo ok > /var/www/app/health && echo "write succeeded"

Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1      5242880  431905 4810975    9% /
write succeeded

IUse% back in single digits and a real file created. That is a real fix, not a hope.

What people get wrong

"df says there is space, so the error is a bug." It is not a bug. ENOSPC covers both bytes and inodes. Trusting df -h alone is the single reason this outage lasts an hour instead of two minutes.

"Just increase the number of inodes." You cannot, not on a live ext4 filesystem. The inode count is fixed at mkfs time. There is no online command to add inodes. The only ways to raise the count are to reformat with a smaller -i bytes-per-inode value (destroys data) or, in narrow cases, use resize2fs while growing the underlying block device, which adds inodes proportionally to the new space. Neither is a 1am fix. If you are permanently generating tiny files by the million, migrate that workload to XFS, which allocates inodes dynamically and does not have a fixed ceiling.

"Delete the biggest files." Size is irrelevant to inodes. Deleting one 40GB log file frees zero inodes and does nothing for this error. You have to delete count, not size.

When it is still broken

• Inodes fill up again within hours. You cleared the symptom, not the source. Something is still writing tiny files continuously: a mail loop, a cron job caching every run, an application logging one file per request. Watch the top directory from Step 2 with watch 'ls /suspect/dir | wc -l' and find the writer.
• The full filesystem is /var/lib/docker. Container image layers and overlay2 diffs are inode-heavy. Run docker system prune and consider a dedicated, larger-inode volume for Docker.
• It is the root filesystem and you cannot even run commands. Free one inode by truncating an existing file in place (: > /var/log/something.log reuses the inode rather than needing a new one), which buys you enough room to run the cleanup.
• You want to never see this again. Add node_filesystem_files_free (or your agent's inode metric) to your monitoring right next to disk-bytes, and alert at 85% inode usage. Most stacks ship the disk-bytes alert and quietly omit inodes. That omission is this whole article.

The lesson that sticks: whenever a disk error makes no sense against df -h, the very next command is df -i. If you also run mail or PHP on the same box, this is the failure that will find you eventually, and knowing the second gauge exists turns a midnight outage into a 90-second fix.