AWSTemplateFormatVersion: '2010-09-09' Description: > EG334S two-AZ active-active hub-service controls for eu-west-1. This separate stack imports the existing hub network and deliberately creates no VGW, customer gateway or VPN connection. Parameters: HubVpcId: { Type: 'AWS::EC2::VPC::Id' } PrimarySubnetId: { Type: 'AWS::EC2::Subnet::Id' } RecoverySubnetId: Type: 'AWS::EC2::Subnet::Id' Description: Second-AZ subnet output by hub-alb-eu-west-1.yaml. HubSecurityGroupId: { Type: 'AWS::EC2::SecurityGroup::Id' } PrimaryInstanceId: { Type: 'AWS::EC2::Instance::Id' } PrimaryPrivateIp: Type: String AllowedPattern: '^10\.1\.[0-9]{1,3}\.[0-9]{1,3}$' OnPremVpcId: Type: String Description: Simulated on-premises VPC ID in us-east-1. KeyPairName: { Type: 'AWS::EC2::KeyPair::KeyName' } MonitoringTopicArn: { Type: String } AlbTargetGroupArn: Type: String Description: Target group used by the public hub Application Load Balancer. InstanceType: Type: String Default: t3.micro AllowedValues: [t3.micro, t3.small] LatestAmiId: Type: 'AWS::SSM::Parameter::Value' Default: /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 Resources: HubPrivateZone: Type: AWS::Route53::HostedZone Properties: Name: eg334s.internal HostedZoneConfig: Comment: Stable private endpoint for the EG334S hub recovery objective. VPCs: - { VPCId: !Ref HubVpcId, VPCRegion: eu-west-1 } - { VPCId: !Ref OnPremVpcId, VPCRegion: us-east-1 } HostedZoneTags: - { Key: Project, Value: EG334S } - { Key: Purpose, Value: HubRecovery } HubDnsRecord: Type: AWS::Route53::RecordSet Properties: HostedZoneId: !Ref HubPrivateZone Name: hub.eg334s.internal Type: A TTL: 30 ResourceRecords: [!Ref PrimaryPrivateIp] RecoveryStartParameter: Type: AWS::SSM::Parameter Properties: Name: /eg334s/hub-recovery/started-at Type: String Value: '0' Description: Unix epoch recorded when automated hub recovery starts. Tags: { Project: EG334S, Purpose: HubRecovery } RecoveryLaunchTemplate: Type: AWS::EC2::LaunchTemplate Properties: LaunchTemplateName: !Sub '${AWS::StackName}-launch-template' LaunchTemplateData: ImageId: !Ref LatestAmiId InstanceType: !Ref InstanceType KeyName: !Ref KeyPairName MetadataOptions: { HttpTokens: required, HttpEndpoint: enabled } Monitoring: { Enabled: false } NetworkInterfaces: - DeviceIndex: 0 AssociatePublicIpAddress: true Groups: [!Ref HubSecurityGroupId] DeleteOnTermination: true BlockDeviceMappings: - DeviceName: /dev/xvda Ebs: VolumeSize: 8 VolumeType: gp3 Encrypted: true DeleteOnTermination: true UserData: Fn::Base64: !Sub | #!/bin/bash set -euxo pipefail dnf install -y httpd mkdir -p /etc/systemd/system/httpd.service.d cat > /etc/systemd/system/httpd.service.d/recovery.conf <<'EOF' [Service] Restart=always RestartSec=3 EOF TOKEN=$(curl -fsS -X PUT -H 'X-aws-ec2-metadata-token-ttl-seconds: 21600' http://169.254.169.254/latest/api/token) INSTANCE_ID=$(curl -fsS -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id) PRIVATE_IP=$(curl -fsS -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/local-ipv4) cat > /var/www/html/index.html < EG334S · recovered AWS hub

EG334S · AWS public cloud hub

This is the automated recovery instance.

Region: eu-west-1 (Europe - Ireland)
VPC CIDR: 10.1.0.0/16
Instance: $INSTANCE_ID
Private IP: $PRIVATE_IP

EOF printf 'ok\n' > /var/www/html/health systemctl daemon-reload systemctl enable --now httpd TagSpecifications: - ResourceType: instance Tags: - { Key: Name, Value: EG334S-awspublic-server-recovery } - { Key: Project, Value: EG334S } - { Key: Purpose, Value: HubRecovery } - ResourceType: volume Tags: - { Key: Name, Value: EG334S-hub-recovery-volume } - { Key: Project, Value: EG334S } RecoveryAutoScalingGroup: Type: AWS::AutoScaling::AutoScalingGroup DependsOn: RecoveryInstanceRunningRule Properties: AutoScalingGroupName: !Sub '${AWS::StackName}-HubRecoveryAsg' MinSize: '1' DesiredCapacity: '1' MaxSize: '1' HealthCheckType: ELB HealthCheckGracePeriod: 300 VPCZoneIdentifier: [!Ref RecoverySubnetId] TargetGroupARNs: [!Ref AlbTargetGroupArn] LaunchTemplate: LaunchTemplateId: !Ref RecoveryLaunchTemplate Version: !GetAtt RecoveryLaunchTemplate.LatestVersionNumber Tags: - { Key: Project, Value: EG334S, PropagateAtLaunch: true } - { Key: Purpose, Value: HubRecovery, PropagateAtLaunch: true } RecoveryControllerRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: { Service: lambda.amazonaws.com } Action: sts:AssumeRole ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole Policies: - PolicyName: StartColdHubRecovery PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: autoscaling:SetDesiredCapacity Resource: !Sub 'arn:${AWS::Partition}:autoscaling:${AWS::Region}:${AWS::AccountId}:autoScalingGroup:*:autoScalingGroupName/${AWS::StackName}-HubRecoveryAsg' - Effect: Allow Action: ssm:PutParameter Resource: !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/eg334s/hub-recovery/started-at' - Effect: Allow Action: cloudwatch:PutMetricData Resource: '*' Condition: StringEquals: { 'cloudwatch:namespace': EG334S/Recovery } RecoveryController: Type: AWS::Lambda::Function Properties: FunctionName: !Sub '${AWS::StackName}-controller' Runtime: python3.12 Handler: index.handler Role: !GetAtt RecoveryControllerRole.Arn Timeout: 30 Environment: Variables: ASG_NAME: !Sub '${AWS::StackName}-HubRecoveryAsg' PRIMARY_INSTANCE_ID: !Ref PrimaryInstanceId START_PARAMETER: !Ref RecoveryStartParameter Code: ZipFile: | import boto3, json, os, time autoscaling = boto3.client("autoscaling") ssm = boto3.client("ssm") cloudwatch = boto3.client("cloudwatch") def handler(event, context): accepted, reason = False, "unrecognised" if event.get("source") == "aws.ec2": detail = event.get("detail", {}) accepted = detail.get("instance-id") == os.environ["PRIMARY_INSTANCE_ID"] and detail.get("state") in ("stopped", "terminated") reason = "primary-" + str(detail.get("state")) elif event.get("Records"): try: alarm = json.loads(event["Records"][0].get("Sns", {}).get("Message", "{}")) except json.JSONDecodeError: alarm = {} accepted = alarm.get("AlarmName") == "EG334S-Hub-StatusCheckFailed" reason = "status-check-alarm" if not accepted: return {"accepted": False, "reason": reason} started = str(time.time()) ssm.put_parameter(Name=os.environ["START_PARAMETER"], Value=started, Type="String", Overwrite=True) autoscaling.set_desired_capacity(AutoScalingGroupName=os.environ["ASG_NAME"], DesiredCapacity=1, HonorCooldown=False) cloudwatch.put_metric_data(Namespace="EG334S/Recovery", MetricData=[{"MetricName":"RecoveryInitiated","Value":1,"Unit":"Count"}]) return {"accepted": True, "reason": reason, "started": started} PrimaryStateRule: Type: AWS::Events::Rule Properties: Description: Start the cold replacement when the primary stops or terminates. EventPattern: source: [aws.ec2] detail-type: [EC2 Instance State-change Notification] detail: instance-id: [!Ref PrimaryInstanceId] state: [stopped, terminated] State: ENABLED Targets: - { Arn: !GetAtt RecoveryController.Arn, Id: RecoveryController } PrimaryStateRulePermission: Type: AWS::Lambda::Permission Properties: Action: lambda:InvokeFunction FunctionName: !Ref RecoveryController Principal: events.amazonaws.com SourceArn: !GetAtt PrimaryStateRule.Arn PrimaryStatusAlarm: Type: AWS::CloudWatch::Alarm Properties: AlarmName: EG334S-Hub-StatusCheckFailed AlarmDescription: Start cold hub replacement after two failed one-minute checks. Namespace: AWS/EC2 MetricName: StatusCheckFailed Dimensions: [{ Name: InstanceId, Value: !Ref PrimaryInstanceId }] Statistic: Maximum Period: 60 EvaluationPeriods: 2 DatapointsToAlarm: 2 Threshold: 1 ComparisonOperator: GreaterThanOrEqualToThreshold TreatMissingData: missing AlarmActions: [!Ref MonitoringTopicArn] OKActions: [!Ref MonitoringTopicArn] RecoveryControllerSnsPermission: Type: AWS::Lambda::Permission Properties: Action: lambda:InvokeFunction FunctionName: !Ref RecoveryController Principal: sns.amazonaws.com SourceArn: !Ref MonitoringTopicArn RecoveryControllerSubscription: Type: AWS::SNS::Subscription DependsOn: RecoveryControllerSnsPermission Properties: Protocol: lambda TopicArn: !Ref MonitoringTopicArn Endpoint: !GetAtt RecoveryController.Arn DnsUpdaterRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: { Service: lambda.amazonaws.com } Action: sts:AssumeRole ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole Policies: - PolicyName: PublishRecoveredHub PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - autoscaling:DescribeAutoScalingInstances - ec2:DescribeInstances - ec2:DescribeInstanceStatus Resource: '*' - Effect: Allow Action: route53:ChangeResourceRecordSets Resource: !Sub 'arn:${AWS::Partition}:route53:::hostedzone/${HubPrivateZone}' - Effect: Allow Action: ssm:GetParameter Resource: !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/eg334s/hub-recovery/started-at' - Effect: Allow Action: cloudwatch:PutMetricData Resource: '*' Condition: StringEquals: { 'cloudwatch:namespace': EG334S/Recovery } DnsUpdater: Type: AWS::Lambda::Function Properties: FunctionName: !Sub '${AWS::StackName}-dns-updater' Runtime: python3.12 Handler: index.handler Role: !GetAtt DnsUpdaterRole.Arn Timeout: 300 Environment: Variables: ASG_NAME: !Sub '${AWS::StackName}-HubRecoveryAsg' HOSTED_ZONE_ID: !Ref HubPrivateZone RECORD_NAME: hub.eg334s.internal START_PARAMETER: !Ref RecoveryStartParameter Code: ZipFile: | import boto3, os, time autoscaling=boto3.client("autoscaling"); ec2=boto3.client("ec2") route53=boto3.client("route53"); ssm=boto3.client("ssm") cloudwatch=boto3.client("cloudwatch") def handler(event, context): instance_id=event.get("detail",{}).get("instance-id") if not instance_id: return {"accepted":False,"reason":"missing-instance-id"} member=None for _ in range(12): result=autoscaling.describe_auto_scaling_instances(InstanceIds=[instance_id]) if result.get("AutoScalingInstances"): member=result["AutoScalingInstances"][0]; break time.sleep(5) if not member or member.get("AutoScalingGroupName") != os.environ["ASG_NAME"]: return {"accepted":False,"reason":"not-recovery-asg"} healthy=False for _ in range(24): result=ec2.describe_instance_status(InstanceIds=[instance_id],IncludeAllInstances=True).get("InstanceStatuses",[]) if result and result[0].get("SystemStatus",{}).get("Status")=="ok" and result[0].get("InstanceStatus",{}).get("Status")=="ok": healthy=True; break time.sleep(10) if not healthy: raise RuntimeError("replacement status checks timed out") instance=ec2.describe_instances(InstanceIds=[instance_id]) private_ip=instance["Reservations"][0]["Instances"][0]["PrivateIpAddress"] route53.change_resource_record_sets(HostedZoneId=os.environ["HOSTED_ZONE_ID"],ChangeBatch={"Comment":"Automated EG334S hub recovery","Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":os.environ["RECORD_NAME"],"Type":"A","TTL":30,"ResourceRecords":[{"Value":private_ip}]}}]}) started=float(ssm.get_parameter(Name=os.environ["START_PARAMETER"])["Parameter"]["Value"]) elapsed=max(0,time.time()-started) cloudwatch.put_metric_data(Namespace="EG334S/Recovery",MetricData=[{"MetricName":"HubRecoverySeconds","Value":elapsed,"Unit":"Seconds","Dimensions":[{"Name":"InstanceId","Value":instance_id}]}]) return {"accepted":True,"instance_id":instance_id,"private_ip":private_ip,"recovery_seconds":round(elapsed,1)} RecoveryInstanceRunningRule: Type: AWS::Events::Rule Properties: Description: Publish replacement DNS after EC2 status checks pass. EventPattern: source: [aws.ec2] detail-type: [EC2 Instance State-change Notification] detail: { state: [running] } State: ENABLED Targets: - { Arn: !GetAtt DnsUpdater.Arn, Id: DnsUpdater } RecoveryInstanceRunningRulePermission: Type: AWS::Lambda::Permission Properties: Action: lambda:InvokeFunction FunctionName: !Ref DnsUpdater Principal: events.amazonaws.com SourceArn: !GetAtt RecoveryInstanceRunningRule.Arn HealthProbeRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: { Service: lambda.amazonaws.com } Action: sts:AssumeRole ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole - arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole HubHealthProbe: Type: AWS::Lambda::Function DependsOn: HubDnsRecord Properties: FunctionName: !Sub '${AWS::StackName}-health-probe' Runtime: python3.12 Handler: index.handler Role: !GetAtt HealthProbeRole.Arn Timeout: 15 VpcConfig: SecurityGroupIds: [!Ref HubSecurityGroupId] SubnetIds: [!Ref PrimarySubnetId, !Ref RecoverySubnetId] Code: ZipFile: | import urllib.request def handler(event, context): url="http://hub.eg334s.internal/" try: with urllib.request.urlopen(url,timeout=5) as response: body=response.read().decode("utf-8").strip() return {"healthy":response.status==200,"status":response.status,"body":body,"url":url} except Exception as error: return {"healthy":False,"error":str(error),"url":url} RecoveryDurationAlarm: Type: AWS::CloudWatch::Alarm Properties: AlarmName: EG334S-Hub-Recovery-Over-10-Minutes AlarmDescription: Measured hub recovery exceeded the accepted objective. Namespace: EG334S/Recovery MetricName: HubRecoverySeconds Statistic: Maximum Period: 60 EvaluationPeriods: 1 Threshold: 600 ComparisonOperator: GreaterThanThreshold TreatMissingData: notBreaching AlarmActions: [!Ref MonitoringTopicArn] Outputs: RecoveryAutoScalingGroupName: { Value: !Ref RecoveryAutoScalingGroup } RecoverySubnetId: { Value: !Ref RecoverySubnetId } PrivateHostedZoneId: { Value: !Ref HubPrivateZone } StableHubEndpoint: { Value: hub.eg334s.internal } HealthProbeFunctionName: { Value: !Ref HubHealthProbe } RecoveryControllerFunctionName: { Value: !Ref RecoveryController } DnsUpdaterFunctionName: { Value: !Ref DnsUpdater } RecoveryObjectiveSeconds: { Value: '600' }