Designing Resilient Architectures

Two services that call each other directly and wait for a response are tightly coupled: if one slows down or fails, the other feels it immediately. Amazon SQS breaks that coupling with a durable message queue sitting between them. A producer writes a message and moves on; a consumer pulls messages at its own pace. When a consumer receives a message, that message becomes invisible to other consumers for the visibility timeout — long enough, ideally, for the consumer to finish processing and delete it. If the consumer never deletes it, the message reappears for someone else to try.

A message that keeps failing shouldn't loop forever. A dead-letter queue (DLQ), configured through a redrive policy, catches any message that has been received more times than a maximum you set, so you can inspect and debug it without it blocking the main queue. Standard queues give the highest throughput with at-least-once delivery and best-effort ordering; FIFO queues trade some throughput for exactly-once processing and strict ordering — the right choice when message order actually matters, like a sequence of account transactions.

aws sqs set-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/orders \
  --attributes '{"RedrivePolicy":"{\"maxReceiveCount\":\"5\",\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:orders-dlq\"}"}'

SQS pairs one producer with one logical set of consumers pulling from the same queue. When several independent systems all need to react to the same event, Amazon SNS is the better fit. SNS is publish/subscribe: a publisher sends one message to a topic, and SNS delivers a copy to every current subscriber — SQS queues, Lambda functions, HTTP endpoints, email, and more. This is the fan-out pattern: the order service publishes "order placed" once, and email, inventory, and analytics each subscribe their own SQS queue to that topic. Adding a fourth subscriber later is just a new subscription — the order service's code never changes.

Handling the resulting request volume is a job for an Application Load Balancer distributing traffic across an EC2 Auto Scaling group. A target tracking scaling policy picks a metric (like average CPU utilization) and a target value, and Auto Scaling adds or removes instances to hold that target — much like a thermostat. Auto Scaling always leans toward availability: it scales out the moment any policy calls for more capacity, but scales in only once every policy agrees it's safe to.

Once traffic is decoupled and load-balanced, you still have to decide what runs the code. Containers, orchestrated by Amazon ECS or Amazon EKS, give you control over the runtime and are a natural fit when you're migrating an existing containerized application or need to run many related services together. Serverless compute — AWS Lambda, or containers run through AWS Fargate without managing servers — scales from zero and bills only while code is actually running, which suits workloads that are idle for long stretches and then burst unpredictably.

For a process with several steps that depend on each other's outcome — charge a card, then reserve inventory, then schedule a shipment, rolling back cleanly if any step fails — AWS Step Functions coordinates the sequence as a state machine, so the coordination logic lives in one place instead of being scattered across each service's error handling.

A single point of failure is any one component whose failure takes the whole workload down with it — and a database sitting in a single Availability Zone is a classic example. Amazon RDS Multi-AZ (with one standby) removes it: RDS automatically provisions a synchronous standby replica in a different AZ, keeps it continuously up to date, and — during planned maintenance, an instance failure, or an AZ disruption — automatically promotes it. The DB instance's endpoint doesn't change, so most applications reconnect without any reconfiguration. AWS states that this failover can complete in as little as 60 seconds.

The standby exists purely for failover. It is not the same feature as a read replica, which is a separate, asynchronously updated copy you create specifically to serve read traffic and reduce load on the primary.

aws rds modify-db-instance \
  --db-instance-identifier orders-db \
  --multi-az \
  --apply-immediately

High availability isn't only about the database surviving a failure — it's also about the database surviving a flood of connections. A workload built on AWS Lambda can scale to hundreds of concurrent invocations in seconds, and if each one opens its own direct database connection, the database can run out of connection slots long before it runs out of compute. Amazon RDS Proxy sits between the application and the database, pooling and reusing a much smaller number of backend connections across many client connections. It also improves failover: because the proxy — not the application — holds the connections, it can reroute traffic to a newly promoted instance without every client having to reconnect from scratch, and it can authenticate using AWS Secrets Manager or IAM rather than a credential baked into the application.

Within one Region, an Application Load Balancer and Multi-AZ cover most availability needs. Across Regions, Amazon Route 53 decides which Region a request even reaches. Simple routing answers with a single resource. Weighted routing splits traffic across resources in proportions you set — useful for a gradual migration or a canary release. Latency routing sends each request to the Region with the best measured latency for that resolver. Failover routing is built for active-passive high availability specifically: Route 53 answers with the primary resource as long as its health check passes, and switches to the secondary the moment that health check fails.

Two numbers frame every disaster recovery decision. Recovery Time Objective (RTO) is how long the business can tolerate being down before service is restored. Recovery Point Objective (RPO) is how much data, measured in time, the business can tolerate losing — the gap since the last point you can recover to. A finance system might need an RTO of hours but an RPO of minutes; a marketing blog might tolerate both being measured in hours.

AWS describes four DR strategies that trade cost and complexity for recovery speed, roughly in this order: backup and restore (cheapest, slowest to recover), pilot light (a small, always-on core with the rest of the stack built up on failover), warm standby (a full-featured but scaled-down duplicate, scaled up on failover), and multi-site active/active (two or more Regions actively serving traffic all the time, most expensive, fastest to recover). None of these is universally "right" — each workload's RTO and RPO, weighed against what that speed costs to maintain, determines which strategy fits.

aws backup start-copy-job \
  --recovery-point-arn arn:aws:backup:us-east-1:123456789012:recovery-point:abc-123 \
  --source-backup-vault-name production-vault \
  --destination-backup-vault-arn arn:aws:backup:us-west-2:123456789012:backup-vault:dr-vault \
  --iam-role-arn arn:aws:iam::123456789012:role/AWSBackupCrossRegionRole

Backup and restore is the simplest strategy: take regular backups (AWS Backup can automate and centralize this) and copy them to a second Region, restoring from them only when disaster strikes. It's the cheapest option specifically because almost nothing runs in the second Region until you need it — which is also why it's the slowest to recover.

Pilot light keeps the smallest core of the workload always on in the second Region: typically the database, kept continuously up to date through replication, while application and web tiers stay switched off until failover, when you deploy or scale them up around that already-live data. Because the data layer is already warm, pilot light recovers faster than backup and restore without the cost of running the full application stack continuously.

Warm standby goes further: a scaled-down but fully functional copy of the entire stack runs continuously in the second Region, ready to be scaled up to full production capacity the moment it's needed. It costs more than pilot light because more of the stack is always running, but it recovers faster because there's less to stand up from scratch during a disaster.

Multi-site active/active goes all the way: two or more Regions run full production capacity and actively serve real traffic all the time, not just standing by. Recovery from a Regional failure can be close to instant, because the other Region or Regions were already handling requests. The cost is real complexity: you have to keep data consistent across Regions and decide how to handle writes that land in two Regions for the same record at nearly the same time.

Whichever strategy you pick, the failover mechanism itself matters. AWS divides its services into a data plane (day-to-day operations like serving a request or reading an object) and a control plane (operations that configure the environment, like creating a resource). Data plane operations are built for higher availability. Building your failover on data plane operations, rather than on a control plane call that might itself be degraded during the very disruption you're recovering from, keeps the failover mechanism from becoming one more thing that can fail.