Skip to main content

Backup and disaster recovery

A backup is an unverified claim until it has been restored. This standard covers what is backed up, where the copies live, how they are protected, and how the restore is proven.

Principles​

  1. Back up at the data layer, not the host layer. A snapshot of a corrupt database is a corrupt snapshot.
  2. Copy out of the region. A regional failure should not take the backups with it.
  3. Make backups immutable where the threat model includes ransomware or an insider.
  4. Test the restore, against the RTO, on a schedule.
  5. Encrypt backups with the same rigour as production data.
  6. Know the dependencies. Restoring a database is not restoring a service; DNS, secrets, certificates and configuration have to be present too.

What is protected​

ResourceMechanismFrequencyRetention
RDS / AuroraAWS Backup (snapshot)Daily + PITR35 days PITR; 1 year snapshots
DynamoDBPITR + AWS BackupContinuous + daily35 days PITR; 1 year
EBSAWS BackupDaily30 days
EFSAWS BackupDaily30 days
S3Versioning + replication + AWS BackupContinuousVersioning 90 days; replication to DR region
Secrets ManagerReplication + versionContinuousPer service policy
ConfigurationTerraform in GitEvery changeIndefinite
Container imagesECR replicationEvery buildPer lifecycle policy

Architecture​

                 PRIMARY REGION                 DR REGION
                 ─────────────                  ─────────
  RDS ──┐
  EBS ──┼──► AWS Backup ──► Vault ──copy──► Vault (locked)
  EFS ──┤        │                              │
  Dynamo┘        ▼                              ▼
            Backup plan                  Restore targets
            (schedule + retention)

  S3 ──────────── versioning ────replication──► S3 (replica)

Backup plan​

resource "aws_backup_plan" "production" {
  name = "production-plan"

  rule {
    rule_name         = "daily"
    target_vault_name = aws_backup_vault.primary.name
    schedule          = "cron(0 3 * * ? *)"

    lifecycle {
      delete_after = 35
    }

    copy_action {
      destination_vault_arn = aws_backup_vault.dr.arn
      lifecycle {
        delete_after = 365
      }
    }
  }

  rule {
    rule_name         = "weekly"
    target_vault_name = aws_backup_vault.primary.name
    schedule          = "cron(0 4 ? * SUN *)"

    lifecycle {
      cold_storage_after = 30
      delete_after       = 365
    }
  }
}

Vault lock​

For workloads with a ransomware or insider threat in scope, a vault lock policy prevents deletion of recovery points even by an administrator:

resource "aws_backup_vault_lock_configuration" "dr" {
  backup_vault_name   = aws_backup_vault.dr.name
  changeable_for_days = 3
  min_retention_days  = 90
  max_retention_days  = 3650
}

Restore​

Restore runbook structure​

StepDetail
DeclareWho decides to restore, and how it is communicated
IdentifyWhich recovery point, from which vault, at which timestamp
ProvisionRestore the data store into an isolated network
VerifyRow counts, checksums, application-level smoke test
Cut overDNS or connection string change
ConfirmKPI back within target; incident closed

Measured against RTO​

Every restore drill records:

FieldExample
Recovery point used2026-11-05 03:00 snapshot
Restore start10:02
Data available10:41
Application healthy11:05
Measured RTO63 minutes
Target RTO120 minutes
ResultPass

A drill that is not measured is a rehearsal that proves nothing.

Failure scenarios​

ScenarioResponseDepends on
Instance failureAuto Scaling replacesMulti-AZ, health checks
AZ failureTraffic shifts to remaining AZMulti-AZ design
Database corruptionPoint-in-time restore to a new instancePITR window, restore runbook
Accidental deletionRestore from backup or versioningRetention, vault access
RansomwareRestore from immutable vaultVault lock, isolated recovery account
Regional failureRebuild in DR region, restore data, switch DNSCross-region copies, IaC, runbook

Anti-patterns​

  • Backups in the same account, same region, same credentials as production. A compromised account loses both.
  • Snapshots only, no point-in-time. A logical corruption at 09:00 is in every snapshot taken after it.
  • Untested restores. The backup job's green tick says the backup ran, not that it can be used.
  • No dependency inventory. Restoring the database without the secret, the certificate and the DNS record leaves the service down.
  • Retention by accident. Defaults that keep everything forever cost money; defaults that keep nothing for long enough fail the audit.

Evidence​

  • Backup plan and vault configuration.
  • Cross-region copy confirmation.
  • Retention settings against the policy.
  • Restore drill record with measured RTO.
  • Vault lock configuration where applicable.
  • A dependency inventory for the restore procedure.