The Disk Is Full. The Files Are Gone.

The alert fires at 2:14 a.m. Disk usage on the app server is at 97%. You SSH in, half awake, and run the command everyone runs:

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1  100G   96G  1.2G  99% /

Ninety-six gigabytes used. So you go find them:

sudo du -sh /* 2>/dev/null

And the total comes to twelve gigabytes.

This is the moment the night gets long. The kernel says the disk is full. The filesystem walk says it isn't. Both are telling the truth, and the gap between them is where the actual problem lives.

Here's the full triage sequence — the order to run things in, what each result means, and the three failure modes that account for most of these pages.

Step one: confirm you're looking at the right filesystem

Before anything else, check that the thing that's full is the thing you think is full. On modern servers / is rarely the only mount, and a container host can have a dozen.

df -h -x tmpfs -x devtmpfs

Drop the pseudo-filesystems and look at what's actually backed by a disk. A very common false alarm: /var/lib/docker sits on its own volume, that volume is full, and / is fine — or the reverse. Fix the mount you're actually out of space on.

Then check the second table, the one people forget exists:

df -i

Filesystem       Inodes   IUsed   IFree IUse% Mounted on
/dev/nvme0n1p1  6553600 6553599       1  100% /

Inode exhaustion looks identical to a full disk from the application's perspective — writes fail with No space left on device — but df -h shows plenty of free space. It happens when something creates millions of tiny files: a session directory that never gets cleaned, a cache with no eviction, a mail queue, a per-request temp file that leaks.

If IUse% is at 100, you don't have a size problem, you have a count problem, and du -sh will actively mislead you because the total bytes are small. Find the directory with the file count instead:

sudo find / -xdev -type f -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head -20

That prints the twenty directories holding the most files. On an inode-exhausted box the answer is usually obvious and usually embarrassing.

Step two: find the big directories, properly

Assuming it is a size problem, walk the tree — but walk it correctly. The naive du -sh /* has two failure modes: it crosses mount points (so a network share or a bind mount inflates your numbers), and it takes forever on a busy server.

sudo du -xh --max-depth=1 / 2>/dev/null | sort -rh | head -20

The -x is the important flag: stay on one filesystem. Then descend into whichever line is large:

sudo du -xh --max-depth=1 /var 2>/dev/null | sort -rh | head -20

Repeat until you find the actual directory. If the machine has it, ncdu does the same thing interactively and is much faster to explore:

sudo ncdu -x /

Arrow keys to descend, d to delete, q to quit. On a server you're debugging at 2 a.m., this is worth installing.

The usual suspects, in rough order of frequency:

Step three: the deleted-but-open file

Now the case from the opening — df says full, du says empty. This is the one that trips people up, and it's the most common cause of that specific discrepancy.

On Linux, deleting a file removes its directory entry. The blocks are only freed when the last open file descriptor to it is closed. If a long-running process has a log file open and something deletes that file out from under it — a cleanup cron job, a careless rm, a logrotate config missing copytruncate or a proper reload signal — the process happily keeps writing to a file that has no name. du walks directory entries, so it can't see it. df asks the filesystem how many blocks are allocated, so it can.

Find them:

sudo lsof -nP +L1

The +L1 filter means "show files with a link count below 1" — exactly the deleted-but-open case.

COMMAND   PID USER   FD   TYPE DEVICE    SIZE/OFF NLINK  NODE NAME
java     4821 app    3w   REG  259,1  84509721600     0 12583 /var/log/app/app.log (deleted)

Eighty-four gigabytes, held open by PID 4821, invisible to du. Two ways out:

The safe one — restart or signal the process so it reopens its log file. For most daemons a systemctl restart or a HUP does it. The space frees instantly.

The no-downtime one — truncate the file descriptor in place through /proc:

sudo truncate -s 0 /proc/4821/fd/3

That empties the file the process is still writing to, without killing it. Use the PID and FD number from the lsof output. Then go fix the rotation config that caused it, because it will happen again next month.

df counts allocated blocks. du counts files it can reach by name. Any time those two disagree, look for something holding a deleted file open.

Step four: the routine reclaims

Once you've handled the emergency, these are the safe, high-yield cleanups. In roughly this order:

systemd journal. Unbounded by default on many distributions, and it will happily eat tens of gigabytes.

journalctl --disk-usage
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=7d

Then make it permanent in /etc/systemd/journald.conf:

SystemMaxUse=500M

Docker. On any host that builds images, this is usually the single biggest win.

docker system df
docker system prune -a --volumes

Read docker system df before you run the prune. The -a flag removes all images not attached to a running container, and --volumes removes unused volumes — which on a database host is exactly the wrong thing to do casually. Know what's on the machine first.

Package manager caches.

sudo apt-get clean            # Debian / Ubuntu
sudo dnf clean all            # Fedora / RHEL

Old kernels, which quietly accumulate in /boot until an upgrade fails because /boot is full:

sudo apt-get autoremove --purge

Log truncation, when you need space right now and can't restart anything. Never rm an active log:

sudo truncate -s 0 /var/log/huge.log

That keeps the inode and the file descriptor valid, so whatever is writing to it keeps working. rm on the same file gets you right back to step three.

Step five: make it not happen again

The 2 a.m. page is a symptom. The fix is boring and takes fifteen minutes:

• Cap the journal. SystemMaxUse in journald.conf. One line, permanent.
• Verify logrotate actually runs. sudo logrotate -d /etc/logrotate.conf does a dry run and shows you what it thinks it's doing. A config that's been silently failing for eight months is extremely common.
• Schedule the Docker prune. A weekly docker system prune -f on build hosts, with the destructive flags chosen deliberately.
• Alert at 80%, not 95%. The difference is whether you're doing maintenance or doing incident response. At 95% on a busy database host you may not have room to run the cleanup itself.
• Watch inodes too. Most default monitoring checks bytes and not inodes, which is why inode exhaustion always arrives as a surprise.
• Give logs and container storage their own volume. Then a runaway log fills a partition instead of taking down the root filesystem and everything with it.

The sequence, condensed

df -h -x tmpfs          # which filesystem, really
df -i                   # inodes, before anything else
du -xh --max-depth=1 /  # walk down, one filesystem only
lsof -nP +L1            # deleted files still held open

Four commands, in that order. The first two tell you which kind of problem you have. The third finds it if it's visible. The fourth finds it when it isn't.

The next time df and du disagree with each other, you won't spend forty minutes wondering which one is lying. Neither of them is — they're just answering different questions, and now you know which is which.