Responding to Disk Pressure on a Linux Host

Purpose

Reclaim space on a filesystem that is full or nearly full, without deleting something that is needed.

Impact

High. A full filesystem causes write failures across everything on the host. Databases may enter read-only mode, log writes fail silently, and under Kubernetes the kubelet will begin evicting pods.

Preconditions

• Root or sudo access on the host
• Knowledge of what the host runs, or someone available who knows

Safety considerations

• Never delete a file you cannot identify. Large and unfamiliar is not the same as unneeded.
• Never blindly clear a directory that an application writes to.
• Prefer truncating an active log to deleting it — deleting a file an application has open does not reclaim the space.
• Take the first reclaim from something certainly safe (package caches, rotated logs) before touching anything else.

Initial checks

$ df -hFilesystem Size Used Avail Use% Mounted on/dev/vda1 200G 200G 0 100% /$ df -iFilesystem Inodes IUsed IFree IUse% Mounted on/dev/vda1 13107200 13101k 5.9k 100% /

Run both. A filesystem can be out of inodes while df -h shows free space — millions of tiny files. The symptom is identical (No space left on device) and the remediation is completely different.

# Where is the space going? Top level first, then descend.
sudo du -xh --max-depth=1 / 2>/dev/null | sort -rh | head -15

The -x matters: it stops du from crossing filesystem boundaries and counting mounted network storage or other volumes.

Decision tree

Filesystem full.

├─ df -i shows inodes exhausted?
│   └─ yes → Diagnosis D: too many small files
│
├─ Does du's total roughly match df's used?
│   ├─ no, du is much smaller → Diagnosis C: deleted files still open
│   └─ yes → continue
│
├─ Is the space in /var/log?          → Diagnosis A: logs
├─ Is it in a container runtime dir?  → Diagnosis B: images and layers
└─ Is it application or user data?    → identify the owner before deleting anything

Diagnosis A — logs

sudo du -xh --max-depth=1 /var/log | sort -rh | head -10

# The systemd journal is frequently the single largest consumer
journalctl --disk-usage

Diagnosis B — container images and layers

docker system df
sudo du -xsh /var/lib/docker/* 2>/dev/null | sort -rh | head

Unused images accumulate indefinitely on nodes that pull frequently and are the most common cause of DiskPressure in a Kubernetes cluster.

Diagnosis C — deleted files still held open

This is the case that confuses people: du reports far less than df. Space is not freed until every open file handle is closed, so a log file deleted while the process still has it open consumes space that nothing can find.

sudo lsof -nP +L1 2>/dev/null | awk '$5 == "REG" {print $1, $2, $7, $NF}' \
  | sort -k3 -n -r | head -20

+L1 lists files with a link count below one — deleted, but still open. The output gives you the process name, PID, size, and path.

Diagnosis D — inode exhaustion

# Find the directories with the most entries
sudo find / -xdev -type d -exec sh -c 'echo "$(ls -A "$1" | wc -l) $1"' _ {} \; 2>/dev/null \
  | sort -rn | head -15

Usually a session directory, a cache, a mail spool, or an application writing one file per event with no cleanup.

Remediation

Work down this list. Stop when you have enough headroom to be out of danger, then address the cause.

Safe reclaim, in order of safety

# 1. Package manager caches -- always safe
sudo apt-get clean                     # Debian/Ubuntu
sudo dnf clean all                     # RHEL/Fedora

# 2. Journal, capped to a size that fits your retention needs
sudo journalctl --vacuum-size=500M

# 3. Rotated logs older than a few days
sudo find /var/log -type f \( -name '*.gz' -o -name '*.[0-9]' \) -mtime +3 -delete

Active logs — truncate, do not delete

# Correct: frees the space immediately, process keeps writing
sudo truncate -s 0 /var/log/large-active.log

Container images

# Unused images, stopped containers, and dangling build cache
docker image prune -a --filter "until=168h"
docker system prune --filter "until=168h"

Deleted-but-open files

Restart the process holding the handle. Identify it from the lsof output first, and restart the service rather than killing the PID so it comes back cleanly:

sudo systemctl restart <service>

If the process cannot be restarted immediately, the space can be reclaimed by truncating through its file descriptor:

# PID and FD number come from the lsof output
sudo truncate -s 0 "/proc/<pid>/fd/<fd>"

Inode exhaustion

Delete the excess files. Do it in batches — a single rm over millions of files will take a long time and may itself fail.

sudo find /path/to/dir -type f -mtime +7 -print0 | xargs -0 -n 1000 rm -f

Validation

# 1. Space and inodes both recovered
df -h /
df -i /

# 2. Writes actually work
sudo touch /var/tmp/disk-check && sudo rm /var/tmp/disk-check && echo "writes OK"

# 3. No deleted-but-open files still holding significant space
sudo lsof -nP +L1 2>/dev/null | awk '$5 == "REG"' | wc -l

# 4. Under Kubernetes: node pressure cleared
kubectl describe node <node> | grep -A5 Conditions

Condition DiskPressure should read False. Until it does, the kubelet will keep evicting pods regardless of what df reports.

Rollback

There is no rollback for a deletion. This is why the safety steps matter more than the speed.

What you can do:

• Restore from backup, if the deleted content was backed up
• For a truncated log, the content is gone — but the file and the process are intact
• For container images, re-pull them; nothing is permanently lost

Before any deletion outside the “safe reclaim” list, record what you are about to remove:

sudo du -xh --max-depth=2 /path > "/root/before-cleanup-$(date +%s).txt"

Escalation

Escalate immediately, before deleting anything, if:

• The space is consumed by a database directory, or by anything you cannot positively identify
• The filesystem refills within minutes of being cleared, which indicates a runaway process rather than accumulation
• Inodes are exhausted by an application’s own data files

Escalate after stabilising if the host has filled repeatedly. Recurrence is a capacity or retention problem, and clearing it again is not the fix.