One Missing Letter Cost 156 GB and a Broken Backup

A nightly backup script failed for 34 days straight, leaking 156 GB of anonymous Docker volumes until the disk hit 100% and the verification step broke itself. The root cause: docker rm -f missing a -v flag in the cleanup trap.

The Setup

Vasyl runs a nightly backup with a verify step. It dumps the database, restores it into a throwaway Postgres container, and runs sanity queries. The logic: a backup you never restored is a hope, not a backup.

The verify step flapped for a week with a cryptic error: postgres did not become ready. The dump itself succeeded. The storage tarball succeeded. Only the gate failed. Then one morning, the box stopped answering SSH.

The Innocent Suspect

First check: disk.

$ df -h /
Filesystem      Size  Used Avail Use%
/dev/sda1        96G   56G   36G  61%

61% used. Plenty of room. Disk is innocent. Move on.

Except df -h / answers a narrower question than the one asked. On this box, Docker's data-root lives on a separate partition — /mnt/data. Everything Docker writes goes there. And /mnt/data looked like this:

$ df -h /mnt/data
Filesystem      Size  Used Avail Use%
/dev/sdb1       196G  183G     0 100%

Zero bytes free. docker system df pointed at the culprit: 156 GB of dangling anonymous volumes. Fifty-seven of them. All identical. All pgdata.

The Mechanism: Three Harmless Facts

Fact one: The official Postgres image declares VOLUME /var/lib/postgresql/data. If you don't mount something there yourself, Docker silently creates an anonymous volume for every container you start.

Fact two: --rm only fires when the container exits on its own. A run that gets killed or times out never reaches that point.

Fact three: The cleanup path for exactly those killed runs did docker rm -f. Without -v. That removes the container but leaves its anonymous volume behind — a full initialized pgdata directory, orphaned, every time.

One leaked volume per bad run. Daily backups. Thirty-four days. 156 GB.

The Loop Closes

This is the part that's beautiful in hindsight. The Docker root fills up. A fresh throwaway Postgres can no longer initdb — nowhere to write its data dir. So the verify step fails. A failed, timed-out run is exactly the kind that skips --rm and goes through the leaky cleanup. Which leaks another volume. Which leaves the disk fuller than before.

The backup broke the very step that verified the backup. A self-reinforcing failure, powered entirely by its own cleanup code.

One thing that mattered a lot at 11pm: real data was never at risk. Prod Postgres and file storage bind-mount to / — the partition sitting comfortably at 61%. The only thing bloating /mnt/data was fifty-seven copies of a database that existed for ninety seconds each, just to prove a dump restores.

The Fix: One Letter, Two Places

# -v removes the container's ANONYMOUS volume too. postgres declares an
# anonymous VOLUME at /var/lib/postgresql/data, so every run that reaches
# `docker rm -f` (a killed/timed-out run where --rm never fired) otherwise
# leaks a full pgdata volume.
cleanup() {
  docker rm -fv "$CONTAINER" >/dev/null 2>&1 || true
}
trap cleanup EXIT

docker rm -f became docker rm -fv — in the trap handler and in the reap of leaked verify containers from previous runs. Plus a catch-all for the runs even a trap can't cover (SIGKILL):

# Belt-and-suspenders: drop any dangling anonymous volumes orphaned before
# this fix (or by an OOM-killed `docker rm`). Named volumes are untouched.
docker volume prune -f >/dev/null 2>&1 || true

And because the original failure hid behind a bare postgres did not become ready, the verify script now diagnoses itself: on readiness failure it dumps the container's docker logs, df -h, and docker system df straight into the CI log, and bails early if the container dies during startup instead of waiting out the full window. The next time this class of bug shows up, the error message will contain its own root cause.

Live remediation was one command. docker volume prune reclaimed 163 GB. /mnt/data went from 100% to 17%.

What I Keep

  • df -h / is not "the disk." If Docker's data-root lives on its own partition, the partition you check by habit can say 61% while the one that matters says 100%.
  • Any image with a VOLUME declaration is a leak waiting for a missing -v. You don't opt into anonymous volumes; they happen to you. Every docker rm without -v on such a container strands one.
  • --rm is happy-path cleanup. The trap handler is the real cleanup — and it must be at least as thorough as --rm would have been, which means it needs -v too.
  • A verify step needs its own observability. Mine guarded the backups for months and then failed with seven words and no evidence. Any gate that can fail should dump the state needed to diagnose the failure, in the failure itself.
  • And one for the road: a full disk still lets you SSH in. When the box stopped answering entirely, that was a clean manual reboot — not the disk. An SSH connect timeout is a host-offline signal. Knowing which symptom belongs to which failure saved me from chasing a second ghost that night.