Disaster Recovery That Works: RPO/RTO Without Guessing

Quickstart

If you only have 60–90 minutes, do this. These steps create clarity fast and prevent the two classic failure modes: (1) “we don’t know what we’re protecting” and (2) “we’ve never tried restoring.”

Overview

Disaster recovery (DR) is the set of practices that let you restore service after a serious outage: cloud region failure, ransomware, a broken deploy, corrupted data, a human mistake, or “we deleted the wrong thing.” The goal is not heroics — it’s repeatability.

Core concepts

DR becomes simple when you separate targets (what must be true) from mechanisms (how you achieve it), and you treat recovery as an engineered workflow you can run repeatedly.

Two mental models that prevent bad DR designs

Step-by-step

This is a practical DR build path you can apply to most stacks (VMs, containers, Kubernetes, managed databases, self-hosted databases). The key is to start with a baseline that works, then iterate toward tighter RPO/RTO where it actually matters.

Step 1 — Inventory what you must recover (and in what order)

Step 2 — Choose a recovery strategy per tier

Not everything needs a premium DR strategy. Use tiers to spend effort where it reduces real risk.

Step 3 — Implement backups you can restore (not just store)

Backup design is about three things: frequency (RPO), restore speed (RTO), and survivability (protection from deletion and compromise).

Below is a concrete example for backing up and restoring a Postgres database with a fast, repeatable flow. Adapt the storage backend and secrets handling to your environment (the pattern matters more than the tools).

#!/usr/bin/env bash
set -euo pipefail

# Example: Postgres logical backup to a timestamped file, then integrity check.
# Assumes: PGPASSWORD is set securely (env var, secret manager, injected at runtime).
# Tip: keep backup credentials separate from day-to-day admin credentials.

BACKUP_DIR="${BACKUP_DIR:-/backups}"
DB_HOST="${DB_HOST:-127.0.0.1}"
DB_PORT="${DB_PORT:-5432}"
DB_NAME="${DB_NAME:-appdb}"
DB_USER="${DB_USER:-appuser}"

ts="$(date -u +%Y%m%dT%H%M%SZ)"
file="${BACKUP_DIR}/${DB_NAME}_${ts}.sql.gz"

mkdir -p "${BACKUP_DIR}"

echo "[backup] starting: ${file}"
pg_dump --host "${DB_HOST}" --port "${DB_PORT}" --username "${DB_USER}" --format=p "${DB_NAME}" \
  | gzip -9 > "${file}"

echo "[backup] verifying gzip stream..."
gzip -t "${file}"

echo "[backup] done: $(du -h "${file}" | awk '{print $1}')"

# --- restore example (run in a recovery environment) ---
# gunzip -c "${file}" | psql --host "${DB_HOST}" --port "${DB_PORT}" --username "${DB_USER}" --dbname "${DB_NAME}"
# psql --host "${DB_HOST}" --port "${DB_PORT}" --username "${DB_USER}" --dbname "${DB_NAME}" -c "SELECT 1;"

Step 4 — Automate the repeatable parts (backup jobs + validation)

You don’t need to automate everything on day one. Start with what you run often: backups, restores, and validation checks. Automation reduces RTO because it removes decision points and manual errors.

Here’s a minimal Kubernetes CronJob pattern (conceptually useful even if you don’t run Kubernetes): a scheduled backup, stored externally, with clear separation of config (env) and execution (container).

apiVersion: batch/v1
kind: CronJob
metadata:
  name: pg-backup
spec:
  schedule: "*/15 * * * *" # every 15 minutes (sets your theoretical RPO ceiling)
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 1
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: backup
              image: alpine:3.20
              command: ["/bin/sh", "-lc"]
              args:
                - |
                  set -euo pipefail
                  apk add --no-cache postgresql-client gzip
                  ts="$(date -u +%Y%m%dT%H%M%SZ)"
                  file="/tmp/${DB_NAME}_${ts}.sql.gz"
                  echo "[backup] writing ${file}"
                  pg_dump -h "${DB_HOST}" -U "${DB_USER}" "${DB_NAME}" | gzip -9 > "${file}"
                  gzip -t "${file}"
                  # Upload step omitted (S3/GCS/Azure/NFS) - keep it external + durable.
                  echo "[backup] done"
              env:
                - name: DB_HOST
                  valueFrom: { secretKeyRef: { name: db, key: host } }
                - name: DB_USER
                  valueFrom: { secretKeyRef: { name: db, key: user } }
                - name: DB_NAME
                  valueFrom: { secretKeyRef: { name: db, key: name } }
                - name: PGPASSWORD
                  valueFrom: { secretKeyRef: { name: db, key: password } }

Notice two important details: concurrencyPolicy: Forbid (avoids overlapping backups) and a clear schedule (your theoretical RPO ceiling). Your actual RPO depends on whether jobs succeed and whether uploads are durable — so monitor both.

Step 5 — Write runbooks that reduce decision fatigue

During an incident, people are tired, stressed, and context-switched. A good runbook eliminates ambiguity. Keep it short, command-oriented, and validated in drills.

Step 6 — Test DR like a product: drills, metrics, and iteration

DR testing is where guessing ends. Start small and repeat. A monthly “tiny drill” beats a yearly “big drill” that nobody remembers.

If you store backups as timestamped artifacts, you can compute a simple “backup staleness” indicator and alert when it exceeds your RPO. Here’s a small script pattern that checks the newest backup age in a folder (swap in your storage API if needed).

#!/usr/bin/env python3
import os
import sys
import time
from pathlib import Path

# Compute "backup staleness" (minutes since newest backup file).
# Usage:
#   python backup_staleness.py /path/to/backup_dir 15
# where 15 is your RPO target in minutes.

def newest_mtime_seconds(path: Path) -> float:
    mtimes = []
    for p in path.iterdir():
        if p.is_file():
            mtimes.append(p.stat().st_mtime)
    if not mtimes:
        raise RuntimeError(f"No backup files found in {path}")
    return max(mtimes)

def main() -> int:
    if len(sys.argv) != 3:
        print("Usage: backup_staleness.py <backup_dir> <rpo_minutes>")
        return 2

    backup_dir = Path(sys.argv[1])
    rpo_minutes = float(sys.argv[2])

    if not backup_dir.exists():
        print(f"ERROR: {backup_dir} does not exist")
        return 2

    newest = newest_mtime_seconds(backup_dir)
    age_minutes = (time.time() - newest) / 60.0

    status = "OK" if age_minutes <= rpo_minutes else "ALERT"
    print(f"{status}: newest backup age = {age_minutes:.1f} min (RPO target = {rpo_minutes:.1f} min)")

    return 0 if status == "OK" else 1

if __name__ == "__main__":
    raise SystemExit(main())

Step 7 — Operate DR continuously (so it doesn’t rot)

DR fails silently when it isn’t maintained. Your goal is to make “drift” visible and fix it before an incident.

Common mistakes

Most DR failures are not “we didn’t buy enough infrastructure.” They’re process and assumptions. Here are the pitfalls that create false confidence — and the fixes that restore sanity.

FAQ

What’s the difference between disaster recovery and high availability?

High availability (HA) aims to prevent downtime during common failures (instance crashes, rolling deploys). Disaster recovery (DR) assumes a serious event already happened (region outage, data corruption, compromise) and focuses on restoring service and data within agreed targets. HA reduces the number of incidents; DR limits the damage when the big ones hit.

How do I choose RPO and RTO if stakeholders don’t know?

Start with impact framing: “If we lose 1 hour of data, what breaks?” and “If we’re down for 2 hours, what’s the cost?” If answers are vague, pick a conservative baseline (e.g., RPO 60 min, RTO 4 hours for tier 2), run a drill, measure, and then decide where tighter targets are worth the cost and complexity.

Is “daily backups” enough for most systems?

It depends on how much data loss you can tolerate. Daily backups imply a worst-case RPO close to 24 hours. If your system changes frequently (orders, messages, writes), daily backups often fail the reality test. Many teams move to 15–60 minute backups (or continuous replication) for critical data, and keep daily/weekly snapshots for longer retention.

What’s the fastest way to reduce RTO without major architecture changes?

Reduce manual steps. Pre-provision infrastructure with IaC, automate restore and validation, keep recovery credentials ready (and tested), and practice small drills monthly. RTO is usually dominated by human coordination and “what do we do next?” decisions.

How often should we run DR tests?

A good cadence is monthly small drills (restore into staging + validate) and quarterly deeper drills (failover, access constraints, realistic scenarios). The right cadence is the one that prevents drift and keeps runbooks accurate.

What should we restore first during an incident?

Restore what enables everything else: tier-0 dependencies (identity/secrets, database, DNS/certificates, network access), then the services that provide the minimal viable user experience, then everything else. Your recovery inventory should encode this order.

Cheatsheet

A scan-fast DR checklist you can keep open during planning and drills. If you’re starting from zero, aim to complete the “Baseline” column first.

Wrap-up

Disaster recovery that works is not a document — it’s a practiced capability. The winning pattern is simple: define realistic RPO/RTO per service, pick a recovery mechanism that matches the tier, and test until you have measured results you trust.

If you want to go deeper on adjacent skills that make DR easier — container builds, deployments you can roll back, and visibility during incidents — the related posts below are good follow-ups.

Related posts

• Docker for Developers: Images, Layers, and Caching Explained
• Kubernetes Without Tears: Pods, Services, Ingress in One Story
• CI/CD That Stays Green: Pipeline Patterns That Scale
• GitOps Explained: Deployments You Can Roll Back With Confidence
• Terraform Mistakes: State, Modules, and the ‘One Big Plan’ Trap
• Observability 101: Logs vs Metrics vs Traces (And When You Need Each)