# Hub recovery test · 28 July 2026

## Result

**PASS — 244 seconds (4 minutes 4 seconds), against the accepted objective of
600 seconds (10 minutes).**

The test covered loss of the AWS hub EC2 application service in `eu-west-1
(Europe - Ireland)`. It did not test failure of the region, VPC, Virtual
Private Gateway or Site-to-Site VPN service.

## Dated transcript

All times below are UTC on 28 July 2026; add eight hours for SGT.

| Time | Event |
|---|---|
| 07:26:37 | Pre-check started. Primary healthy; recovery ASG desired capacity 0. |
| 07:26:40 | Controlled stop issued for primary `i-074bf0cb910a99e02`. |
| 07:27:19 | EventBridge recovery controller recorded the failure and requested one cold replacement. |
| 07:27:34 | Replacement Lambda workflow started for `i-03d63eb39288c7419`. |
| 07:30:43 | `hub.eg334s.internal` resolved to replacement `10.1.2.225`; VPC probe returned HTTP 200. |
| 07:30:43 | Test declared PASS at 244 seconds from controlled stop. |
| 07:32:57 | Primary status checks healthy, DNS restored to `10.1.1.61`, recovery ASG returned to 0. |

CloudWatch recorded the automation-only interval from controller start to DNS
publication as **187.95 seconds**. The formal 244-second result includes the
earlier operator stop request and the subsequent HTTP probe, so it is the
conservative end-to-end measurement.

## Controls exercised

- Separate CloudFormation stack: `eg334s-hub-recovery`.
- Cold Auto Scaling replacement in a second Availability Zone.
- EventBridge detection of primary `stopped` or `terminated`.
- Two-of-two one-minute `StatusCheckFailed` alarm as an independent trigger.
- EC2 system and instance health gates before private DNS publication.
- Stable 30-second private DNS record: `hub.eg334s.internal`.
- VPC-attached HTTP health probe.
- Measured recovery metric and a 600-second breach alarm.
- Automatic restoration to the original primary and zero standby capacity.

## Post-test verification

- Primary `i-074bf0cb910a99e02`: `running`, system status `ok`, instance status
  `ok`.
- `hub.eg334s.internal`: `10.1.1.61`, TTL 30.
- Recovery Auto Scaling group: desired 0, zero instances.
- HTTP probe through stable DNS: 200.
- VGW `vgw-0bf466d711cc93ac4`: available and attached.
- VPN #1 `vpn-065016cdb8d06a873`: available.
- VPN #2 `vpn-03dcb879c286f5ecd`: available.

## Claim boundary

This proves the accepted **hub EC2 service recovery objective**. The replacement
is stateless and appropriate for the project web service. It does not prove
database recovery, VGW recovery or regional disaster recovery. Those claims
must not be inferred from this result.

Azure was not used as an end-to-end client during this test. VPN #2 was
intentionally parked for cost control; exercising it would have required an
approximately 30-minute rebuild of the Azure `VpnGw1AZ` gateway and connection
and temporarily returned Azure to its approximately US$7.85/day fully rebuilt
run rate instead of the approximately US$0.44/day parked rate. This additional
cost was not required to measure the accepted AWS hub EC2 failure boundary.
The Azure-to-recovered-hub path is therefore **not exercised**, not passed, and
must be included in the planned full cross-cloud rehearsal after VPN #2 is
rebuilt for the assessment.