A backup is not a backup until you restore it. A homelab tabletop exercise.

The backup jobs are all green. They have been green for two years. Neither fact means a restore works — green means the archive was written, not that it opens. So: one Saturday morning, a scratch VM on the Proxmox box (the same one that got the OpenTofu treatment), a stopwatch, and one rule — nothing gets fixed mid-drill, everything gets noted.

The rules of the drill

• Restore into a scratch, never in place. A restore that overwrites the only copy is the disaster arriving early, by appointment.
• Time everything. RTO is a number you discover, not one you declare in the runbook.
• Verify, then actually open the thing. Boot the database. Log into the app. Open three files at random and read them.
• Pick the restores you fear, not the easy ones. The easy ones were never in doubt.

The findings

• Postgres dump: restored, twelve minutes. The one restore I had rehearsed before. Rehearsed restores are fast; film at eleven.
• The photo library: restored, nine hours. Four hundred gigabytes over a home uplink is a bandwidth story nobody priced. RTO for the crown jewels is a weekend, and now the weekend is budgeted for.
• Config directory: restored — and three months stale. The source moved to a new path in spring; the job kept backing up the old one, quietly, forever, green the whole time. Stale backups are the finding nobody expects and everybody has.
• The password vault: restorable, and circular. The decryption passphrase lived in the same vault. Twenty uncomfortable minutes and a piece of paper in a drawer broke the loop. Whatever decrypts the backup must exist outside the backup; paper counts.
• Two services I believed were covered were in no job at all. The Home Assistant config. The wiki. “Covered” in my head, absent from the scheduler. This is the finding that justifies the whole exercise.

The runbook writes itself

Every command typed during the drill went into RESTORE.md as I ran it, with the timings. A runbook written from memory after the drill is fiction; the runbook is the log of the drill, and the drill is the only time it gets written honestly.

Cadence that stuck: quarterly for the feared restores, monthly for one random pick — restore roulette, one job, to scratch, timed. The nightly restic check stays, but it validates the archive, not the restore path. Both matter. Only one has ever failed in a way that mattered.

The honest trade-offs

• The drill consumes a Saturday and scratch disk. Cheaper than the alternative; still worth saying out loud.
• Schema drift: an old dump restored into a new app version fails in ways only the drill reveals, and only the drill reveals them early.
• Verification is not restoration. One proves the archive parses; the other proves your Saturday plan works. Do not let the first impersonate the second.
• Findings age. The drill is a snapshot of a moving system; the schedule is the system.

Where I landed

Green checkmarks describe the writer. The drill describes the reader, and only the reader matters at 3am. Two of the five findings were real data-loss risks sitting under two years of green, and the fix for both was an afternoon. The third Saturday is already on the calendar.