A disk full at 3am. Everything I should have known about log rotation.

The page came in at 03:12: API returning 500s. SSH took three tries — each login a little slower than the last, which in hindsight was the disk announcing itself. df -h: 100% on /. The culprit: /var/log/app/app.log, forty-one gigabytes of debug logging left on “temporarily” after last week’s investigation, and a logrotate rule that had never fired once in its life.

The archaeology

The rule existed. It was even mostly right. It pointed at /var/log/apps/*.log — and the service logs into /var/log/app/. One letter. /var/lib/logrotate/status confirmed the path had never been considered: the rule ran weekly, matched nothing, and exited green every time for eight months.

The five-second check I did not run: logrotate -d /etc/logrotate.d/app — dry-run mode, prints exactly what would rotate and why. I now run it while writing the rule, not while reading a pager.

Worth knowing before your own 3am: disk-full is worse than the crash you expected. The database starts refusing writes and, on some engines, corrupting them. New SSH sessions fail to allocate. You cannot install the diagnostic tools that would help, because there is no space for the download. Everything compounds. The emergency drill that worked:

df -h                          # which filesystem
du -x --max-depth=1 / | sort -h | tail   # which directory
truncate -s 0 /var/log/app/app.log       # truncate, never delete:
                                         # the process holds the inode,
                                         # deleting the name frees nothing

The four configs, now on every box

1. journald. SystemMaxUse=1G in /etc/systemd/journald.conf. The journal is a log too, and on some distros it defaults to unbounded.
2. A logrotate rule per app, verified. Dry-run it when written, copytruncate where the service cannot reload, and force one rotation in business hours (logrotate -f) so the first real fire is not the first test.
3. Docker. The json-file driver defaults to unbounded container logs — which is a disk-full on every busy host, eventually:

// /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}
// existing containers keep old settings until recreated

4. The app itself. DEBUG left on after an investigation is how forty-one gigabytes happens. The incident that started the debugging is in the log; the off switch never was. Level-down is part of closing the incident, not an aspiration for later.

And the fifth thing, the one that would have made this a Tuesday-afternoon ticket instead of a 3am page: alerting at 80%. Disk pressure is the rare failure that announces itself days in advance, for free.

What changed

• The four configs ship in the cloud-init template now — the Proxmox template that got a plan phase last month got a log policy in the same pass.
• Truncate-before-delete is in the runbook, with the inode explanation, because the delete-the-file move freed exactly zero bytes at 03:20 and cost twenty minutes of confusion.
• Rotation is tested when written and after every log-path change. Rules that have never fired are not rules; they are wishes with a cron entry.

Where I landed

Forty-one gigabytes of debug logs, one-letter typo, one pager. The fix was four lines of config I already knew about. The lesson was never the config — it was that nothing had ever fired it, and nothing would have told me until it was 3am again.