This blog will show you how to use EC2Rescue Linux to fix unreachable Linux cases. This method can be used to rescue even EC2 Instances not associated with AWS Manager.
This blog will provide more information about AWS System Manager: Step by Step Guide to Centralized Multi-Account OS Patching Using AWS Systems Manager
This will allow AWS to leave behind one CloudFormation stack and create a new VPC. AWS will also launch an EC2RescueInstance to rescue our Unreachable Instance. Once it is available, it will stop the unreachable instance. It will then stop the Instance and create a backup of unreachable Instances. Once the Instance is completed, it will disconnect the root volume from the Unreachable instance and attach it to the Rescue instance. It will then locate the rescue device and mount the rescue volume. It will then run the following commands:
‘/mnt/mount/etc/resolv.conf’ -> ‘/mnt/mount/etc/resolv.conf.back”/etc/resolv.conf’ -> ‘/mnt/mount/etc/resolv.conf”/mnt/mount/usr/bin/ec2rl’ -> ‘/usr/local/ec2rl-1.1.5/ec2rl’123’/mnt/mount/etc/resolv.conf’ -> ‘/mnt/mount/etc/resolv.conf.back”/etc/resolv.conf’ -> ‘/mnt/mount/etc/resolv.conf”/mnt/mount/usr/bin/ec2rl’ -> ‘/usr/local/ec2rl-1.1.5/ec2rl’It will Start chroot and Run EC2 Rescue for Linux. It will then stop the Rescue Instance, and remove the Instance Root Volume from EC2RescueInstance. Once this is done, it will attach the Instance Root Volume to Instance and restore Instance back to its Initial State. Cloudformation will then delete the stack that was created for the rescue operation.
Demonstration: I have changed My Ec2 instance /home directory permissions from 777 to 777. Before running this Automation, it is a good practice to create an AMI for the Instance that is not reachable.
I tried to log back in to my instance again, but I couldn’t SSH to get a “Permission denied” error.
Here, we will use the “AWSSupport-ExecuteEC2Rescue” Automation Document to fix this issue:
Open the Systems Manager by going to AWS Console
Select Automation from the left menu and then click Execute Automation
Select “Self-service support workflows” in the Automation section.
Then choose “AWSSupport-ExecuteEC2Rescue” and click “Next”
Next, take the Instance ID for our Unreachable Instance. Then, enter it in the parameter section.
To automate, click the Execute button
After you click “Execute”, the Automation will begin. You can also see the Status “In Progress”.
You can expand the steps to see more details. Linux Instances will always have a “Failed” status for the first step.
It will take several steps to recover our Unreachable Incident. It creates the Rescue Instance first, then it will stop our Unreachable Instance.
It will now detach the Root Volume from the Unreachable Instance, and attach it to Rescue Instance
Once the file is attached, the procedure runs to fix the issue using EC2Rescue Linuxover rescue Instance
You can monitor the overall status of the procedure using the Execution Status tab in Automation Executions. Wait for it to get “Success”, marking it as complete.
We can now try to connect with our Original Instance.
This method allows you to log in to your EC2 Instance with ease. AWSSupport-ExecuteEC2Rescue is a new method that automates every step required to fix common issues on our unreachable Linux instance utilizing respective EC2Rescue for Linux.
4. About CloudThat
CloudThat is an authorized AWS Well-Architected partner, helping other businesses to build secure, high-performing and resilient infrastructures for their applications and workloads.
CloudThat is also an official AWS (Amazon Web Services), Advanced Consulting Partner and Training Partner, and Microsoft Gold Partner. This partnership helps people learn about the cloud and help businesses achieve higher goals using the best in cloud computing.