Don't Trust a Backup You Haven't Restored

When did anyone last restore your backups, start to finish, and time it? For a lot of teams the true answer is never, and the dashboard has been green the whole way. At one company, backups ran nightly for three years. The infrastructure panel showed green checks. The compliance audit passed every year. The restore had never once been run. Then a production database corrupted at 2 a.m. in November, the restore became mandatory, and the team learned three things about its own system in one very long day.

First, the restore took eleven hours against a confident estimate of two. The bottleneck was how the snapshots were chunked, tuned for backup speed and never for restore speed. Eleven hours of downtime cost four enterprise escalations and an unknown number of small customers who churned without a word.

Second, the restore brought back only part of the data. At that company, roughly 60% came back. The rest lived in a separate database everyone had forgotten sat outside the backup scope. It held order history past ninety days, and that history was gone for good. Support spent the next month rebuilding it by hand.

Third, the runbook lied. The disaster-recovery doc the team leaned on did not match the real restore process. They improvised, which stretched the eleven hours longer and left them unsure the partial restore was even sound. The same rot shows up when the runbook lives in one person's head, or when a Friday deploy breaks by Monday and nobody wrote down why.

The dashboard was telling the truth

Here is the trap. The green checks were accurate. The nightly job did complete. The system worked at the snapshot step every single night. It had never been tested at the restore step, and the restore step is the only one that carries the whole point. Confidence built on the snapshot half of the job is confidence in half a system.

Restore on a schedule

Quarterly restore drills close the gap. Spin up a parallel environment from the newest backup. Time the restore against the wall clock, not the estimate. Validate the data against production, row for row where it matters. Write down what broke. The first drill at most companies turns up at least one gap that would have been a disaster in a live event. The second turns up fewer. By the third or fourth, the team owns a backup system that has met the failure it exists for.

Most teams skip the drills for a simple reason. Drills are infrastructure work nobody requested, and the current setup looks fine. It looks fine right up to the morning it has to work.

A backup you have never restored is a promise you have never read. It might be sound. You find out on the worst possible day, at the worst possible hour, with customers watching. Pick a calmer day this quarter and find out first.