Backup and disaster recovery

The question backups are meant to answer

Nobody buys backups. What people buy is an answer to a question they have started asking at an uncomfortable moment — usually after an outage that went on longer than anyone expected, or after a procurement form from one of your own clients asked for recovery targets and nobody in the business could fill them in.

The question is not "do we have backups". Almost everyone does. It is: if the database were gone in the next hour, how long until we are trading again, and how much work would the team have to do twice? Those two answers have names — RTO and RPO — and until they exist as numbers, a backup is a feeling rather than a plan.

The reason this matters more than it looks is that the two halves of a system fail very differently. A server can be rebuilt. Machines are interchangeable, and a competent rebuild is a bad afternoon. A database cannot be rebuilt, because there is nothing to rebuild it from. The same is true of everything users have uploaded — invoices, photographs, signed documents — which on a lot of small systems is still sitting on the application server's own disk rather than in object storage, and therefore disappears with the machine. Most disaster recovery work is really the discipline of separating those two categories and treating them completely differently.

The failure mode we see most often is not the absence of backups. It is a backup job that has been quietly broken for months. Backups fail silently by nature: a job that never runs produces no error, and success is invisible, so nobody notices anything until the day somebody looks for a file that was never written.

How the work runs

1. Inventory what genuinely has to survive. Database, user uploads, configuration and secrets, DNS records, and the build of the machine itself. This is also where we find out whether uploads live on a server disk, which changes both the backup and the architecture. Half a day to a day.
2. Agree RTO and RPO as written numbers. We put the honest options in front of you with their costs: nightly dumps, hourly, or continuous point-in-time recovery. Tighter targets are not more virtuous, they are more expensive, and the choice belongs to you rather than to a default setting. Half a day.
3. Build the jobs and get a copy out of the account. Snapshots for fast whole-machine recovery, logical dumps for the database because they survive version changes and let you restore one table instead of one server, object storage for files, and a second copy held in a separate account or provider so a single compromised login cannot take both. One to three days.
4. Run a real restore into a clean environment, with a stopwatch. A fresh environment, the backup restored into it, the application started against it, and specific records checked. The time that drill takes is your RTO — measured, not estimated. If it comes out longer than the target agreed in step two, the design changes here rather than during an incident. One to two days.
5. Write the runbook and alert on absence. A short document with the exact commands, in order, for the two or three recovery scenarios that are actually likely — because at three in the morning people should be reading and typing, not reasoning. The monitoring is set to fire when there has been no successful backup within the expected window, which is the only way to catch a job that stopped running rather than one that errored. Half a day to a day.

What moves the price

The range is $500–$8,000 across 1–4 weeks, and where a piece of work lands inside it comes down to three things.

How much data there is, and where it currently sits. A single database on one VPS is a different job from an application whose state is spread across a managed database, an object store and a server disk that nobody documented. Moving a large first copy out of a cloud provider also carries a transfer cost, which we tell you about before it appears on a bill.

How low you need the RPO to go. Nightly is straightforward. Hourly means more moving parts and more to monitor. Continuous point-in-time recovery means write-ahead log shipping and a genuine ongoing commitment. Each step down that ladder adds build time and running cost, and each one is a business decision rather than a technical one.

Whether the environment can be rebuilt or only restored. If the infrastructure is described in code, standing up a clean environment for the restore drill is a command. If the current server was assembled by hand over three years by someone who has moved on, part of this engagement is reconstructing how it was built — because there is no point having a database backup if nothing exists to restore it onto.

What usually goes wrong

The backup has never been restored, so nobody knows it is empty. This is the one that ends businesses. Dumps that exclude a schema, credentials that expired so the job has been writing nothing, archives that are the right size and entirely unreadable, or a dump taken with a client version that the server will no longer accept. Every one of these looks identical to a working backup until the day you need it, which is why the restore drill - not the backup job — is the actual deliverable here.

The only copy lives inside the account it is protecting. Snapshots feel like safety and are stored beside the thing they protect. If the account is suspended over a billing problem, or a root credential is compromised and someone deletes in bulk, the original and the snapshot go together. The fix is unglamorous and cheap: one copy that lives somewhere a single compromised login cannot reach.

Nobody is told when the backup stops. Monitoring is usually configured to alert on errors, and a job that has stopped running never errors. Forty days later somebody asks for last month's data and finds the newest file is from a Tuesday in spring. The alert has to be written the other way round — fire when the freshest successful backup is older than the agreed window — and it needs to reach a person, not a mailbox nobody opens.