Resilience planning
Resilience is designed against two numbers — how long the business can wait, and how much data it can lose. Everything else follows from those.
RTO and RPO
| Term | Definition | Question it answers |
|---|---|---|
| RTO — Recovery Time Objective | Maximum acceptable time to restore service | "How long can we be down?" |
| RPO — Recovery Point Objective | Maximum acceptable data loss, expressed as time | "How much can we lose?" |
Both are business decisions, not technical ones. The delivery team proposes options with cost and complexity attached; the business chooses.
Typical tiers
| Tier | RTO | RPO | Strategy | Relative cost |
|---|---|---|---|---|
| Non-critical | 24 h | 24 h | Backup and restore | Low |
| Standard | 4–8 h | 1–4 h | Backup and restore with warm images | Moderate |
| Important | 1–2 h | 15 min–1 h | Pilot light / warm standby | Higher |
| Critical | < 15 min | Near zero | Multi-site active-active | Highest |
The cost curve steepens sharply below a one-hour RTO. That is the point at which the business case has to be explicit.
DR strategies
| Strategy | Description | RTO | RPO | Cost |
|---|---|---|---|---|
| Backup and restore | Backups in a second region; rebuild infrastructure on demand | Hours to a day | Backup interval | $ |
| Pilot light | Core data replicated; minimal compute always running; scale up on failover | Tens of minutes | Near zero | $$ |
| Warm standby | A full, scaled-down copy running in the second region | Minutes | Near zero | $$$ |
| Multi-site active/active | Both regions serve traffic simultaneously | Near zero | Near zero | $$$$ |
Most workloads are correctly served by backup and restore with a tested restore procedure. Active-active is appropriate only where the business impact of a regional outage justifies the standing cost and the operational complexity of conflict-free multi-region writes.
Designing to the objective
Backup and restore
- AWS Backup plans covering RDS, EBS, EFS, S3, DynamoDB and EFS.
- Cross-region copy to a backup vault in the secondary region.
- Vault Lock where immutability is a compliance requirement.
- Infrastructure reproducible from Terraform in the secondary region.
- A restore runbook that has been executed within the RTO.
The gap that breaks this strategy is almost never the backup — it is the untested restore. A restore that takes nine hours against a two-hour RTO is not a control.
Pilot light
- Data replicated continuously (RDS cross-region read replica, S3 replication, DynamoDB global tables).
- Core network, IAM and security configuration always present in the secondary region.
- Compute defined as code and launched on failover.
- A documented, rehearsed failover sequence.
Warm standby
- A scaled-down duplicate running continuously.
- Health checks and synthetic traffic confirm it works.
- Scale-up runbook and tested promotion.
- DNS or global accelerator switchover.
Active-active
- Data layer that supports multi-region writes, or a strict partitioning scheme.
- Conflict resolution rules that are understood, not emergent.
- Latency-aware routing.
- Continuous verification that both regions serve correct results.
Availability zones
Before reaching for a second region, close the cheaper gap:
- Workload across at least two AZs.
- Load balancer across the same AZs.
- Database Multi-AZ.
- Auto Scaling group spanning AZs, with rebalancing.
- No single NAT gateway serving all AZs.
- Capacity sized so the loss of one AZ does not exceed remaining capacity.
An AZ failure is far more common than a regional one, and a single-AZ design fails the common case.
AWS Resilience Hub
Resilience Hub assesses an application against a policy and reports an estimated RTO/RPO compliance score with recommendations.
Setup
- Define a resiliency policy per tier, with RTO and RPO for software, hardware, AZ and regional disruptions.
- Create an application from the resource group, CloudFormation stack, AppRegistry or Terraform state.
- Attach the policy and an assessment role.
- Run an assessment.
- Implement recommendations; re-assess.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["resiliencehub:List*", "resiliencehub:Describe*"],
"Resource": "*"
}]
}
Using the output
The score is a starting point, not a certificate. The recommendations are the useful part:
- Missing Multi-AZ on a data store.
- No cross-region backup.
- Auto Scaling not spanning AZs.
- Health checks absent or misconfigured.
- No alarm on a critical dependency.
Each recommendation becomes a backlog item with an owner.
Fault injection
An assessment estimates. A test proves.
| Test | Frequency | What it proves |
|---|---|---|
| Instance termination | Monthly | Auto Scaling replaces capacity; requests survive |
| AZ failure simulation | Quarterly | Workload continues on remaining AZs |
| Database failover | Quarterly | Multi-AZ failover meets its target time |
| Restore from backup | Per RTO commitment | Backups are usable and the procedure meets RTO |
| Regional failover | Per DR commitment | The DR plan works, not just exists |
| Dependency failure | Quarterly | Graceful degradation and recovery |
Tests are run in non-production first, and their results are recorded against the RTO/RPO targets.
Evidence
- RTO/RPO agreed with the customer and recorded.
- Resilience Hub assessment and score.
- Backup configuration and retention.
- Restore test record with measured duration.
- Failover test record with measured duration.
- Architecture diagram showing AZ and region topology.