# AWS hub recovery objective

**Approved objective:** restore the AWS hub application service within **10
minutes** of a detected hub EC2 failure.

**Measured result on 28 July 2026:** **PASS in 244 seconds (4 minutes 4
seconds)**. See the [dated evidence record](../evidence/hub-recovery-2026-07-28.md).

## Scope and boundary

The recovery boundary is the `EG334S-awspublic-server` EC2 application in
`eu-west-1 (Europe - Ireland)`. The VPC, route table, Internet Gateway, Virtual
Private Gateway and both Site-to-Site VPN connections are imported or left
untouched. A VGW or regional failure requires a different design and is not
claimed by this objective.

The recovery controls are deployed as the separate
`eg334s-hub-recovery` CloudFormation stack. This avoids an unsafe in-place
update of the older live hub stack, which predates some resources in the
repository template.

## Recovery sequence

1. EventBridge detects the primary entering `stopped` or `terminated`.
2. A CloudWatch alarm independently detects two consecutive one-minute EC2
   status-check failures.
3. Either signal records the start time and changes the cold recovery Auto
   Scaling group from zero to one `t3.micro`.
4. The replacement starts in `eu-west-1b`, using the existing hub security
   group and route table.
5. After EC2 system and instance checks are both `ok`, Lambda moves the
   30-second private record `hub.eg334s.internal` to the replacement address.
6. A VPC-attached Lambda performs the HTTP service check.

The replacement is stateless. It is suitable for this demo web service, not a
database replication design.

### Why Azure is outside the controlled test

VPN #2 is intentionally parked between demonstrations. Repeating the recovery
test from Azure would first require an approximately 30-minute rebuild of the
paid `VpnGw1AZ` gateway and connection and would temporarily increase Azure
from the approximately US$0.44/day parked rate to the approximately
US$7.85/day fully rebuilt run rate.

That cost is unnecessary for measuring the accepted AWS hub EC2 recovery
boundary. The controlled test therefore verifies HTTP from inside the
associated private AWS network. It does not mark the Azure path as successful;
Azure-to-recovered-hub is **not exercised** until the planned full cross-cloud
rehearsal after VPN #2 is rebuilt for the assessment.

## Cost

At steady state the recovery group has zero instances:

- Route 53 private hosted zone: approximately US$0.50/month.
- Two standard CloudWatch alarms: approximately US$0.20/month.
- Lambda, EventBridge, SSM Parameter Store and private DNS queries are
  negligible at project scale.
- No standby EC2 or public IPv4 charge applies while desired capacity is zero.

During recovery, the replacement `t3.micro`, 8 GiB gp3 disk and public IPv4 are
billed for their actual lifetime. A stopped primary continues to incur its EBS
storage charge.

## Operating commands

Deploy or update the recovery stack with `hub-recovery-eu-west-1.yaml`, using
the current live resource IDs. Validate IDs before every rebuild; the values in
the evidence record are not permanent.

Run the controlled, self-restoring test:

```sh
./scripts/test-hub-recovery.sh
```

Success requires replacement `InService`, DNS moved to its private IP, HTTP
200 through the stable name, elapsed time at or below 600 seconds, and final
restoration of the primary, DNS and zero standby capacity.

Never scale the replacement to zero before primary health and DNS restoration
are confirmed.