Technical Disaster Recovery Plan Template

Start with an inventory of everything you would have to bring back: databases, services, file stores and the outside services you rely on. Then add what nobody thinks of as a system. Amazon Web Services' disaster recovery guidance says to back up code and configuration as well as data¹, because the infrastructure has to be redeployed too.

For each system, write down what must be running before it can start. Order the recovery by those needs first and by importance second. NIST, the US standards agency, asks for recovery procedures in a stepwise, sequential format²: restore and check a server's operating system before its application and data. Your most important service may come back 10th, behind the systems it needs.

Give each system two numbers. NIST's contingency planning guide defines the recovery time objective (RTO)² as the longest a system can stay unavailable before the impact becomes unacceptable. The recovery point objective (RPO) is the point in time you can recover data to, given your latest backup. In short: how long you can be down, and how much recent work you can lose.

The business decides how much downtime and data loss it can take, usually from the deadlines in its business continuity plan. The backup method decides what is possible. Every method has a floor on data loss, and a target below the floor is a wish.

A backup job that reports success tells you a job ran, not that the data can come back. When an engineer accidentally wiped GitLab.com's primary database in January 2017, GitLab found that the process of both finding and using backups failed completely⁴. Its daily database dumps had been failing silently. The site was down for about 18 hours and lost about 6 hours of database changes.

Google's site reliability engineering book quotes a saying: no one really wants to make backups; what people really want are restores⁵. So restore each system for real, into an environment with production's permissions, network and keys. Then check something a person would notice, like the latest order. Look for failures around the data, not just in it:

• The backup was taken from a replica whose replication had quietly stopped
• The encryption key lives in the account you are recovering from
• Old backups aged into an archive tier that takes hours to retrieve
• The restored database came up with default settings and refused connections

Keep at least one copy where an attacker can't reach it. CISA, the US cybersecurity agency, advises offline, encrypted backups of critical data⁶, tested regularly in a disaster recovery scenario. It warns that many ransomware variants hunt for backups they can reach, to delete or encrypt them.

Organize procedures by what is lost, not by what caused it. A flood, a fire and a bad upgrade that take out the same database need the same steps. Four kinds of loss cover most plans: one system, a whole environment or region, the data itself, and a provider you depend on.

Plan hardest for losing the data itself. When records are corrupted, deleted or encrypted, failover won't save you. Amazon's disaster recovery whitepaper warns that replication may not protect against data corruption or destruction¹ unless you also keep versioned or point-in-time backups. Keep backups for longer than it could take you to notice the damage.

Each kind of loss gets the same five parts:

• Trigger: what counts as a disaster, and who can declare one
• Steps in dependency order, each with an owner and a runbook link
• The data you will really lose, from the backup method's floor
• What stays degraded, and for about how long
• Cleanup: replayed transactions, re-sent webhooks, re-run jobs

A chat app that signs in through the identity provider you are recovering will be down when you need it. Name and test a fallback channel, and draft customer status page updates in advance. If an attack caused the loss, your security incident response plan should cover who you must notify and by when.

Tests come in three strengths (tabletop, functional and full interruption), so record which kind you ran. NIST's guide describes a tabletop exercise as discussion-based only, without deploying equipment². A functional exercise has people do their jobs in a simulated environment. For federal systems rated moderate impact, NIST says it should include an element of recovery from backup media.

Start the clock when a monitor would have fired, not at the restore command. Paging people and deciding to declare count against your target too. Amazon's RDS documentation notes that a restored database starts with the default parameter and option groups³ unless you choose your own. It also keeps loading data in the background after it shows as available. Both add minutes that restore estimates miss.

The template on this page has a sheet or document for each step above. Open it with Edit with AI, and River starts by sorting your systems into restored recently, restored long ago, never restored and not backed up at all.