I could not ssh in to one of my AWS instances last evening and it wasn’t serving any pages either. AWS management console said it was up, though. Rebooting did not help. The second reboot did not help either. Shutdown and start did not help. I was running out of tricks here!
For some reason, the instance had been running on 100% CPU utilization for days:
(I better do some monitoring in future!)
Even though the CPU usage had dropped after the restarting, the instance would not accept any connections. The only thing I could think of was to either ping the AWS forum, or to get the running volume on some new instance as the instance was an EBS based one. I decided to go with the new volume if the database would not mind too much. Steps I needed to do were:
- Snapshot the running volume
- Create a new volume out of the snapshot on the same availability zone
- Start a new instance with the Launch more like this
- Shutdown the new instance
- Detach the volume on the new instance
- Attach the volume which was created from the snapshot to new instance (need to have the correct attachment information, like /dev/sda1)
- Start the new instance
- Disassociate the Elastic IP from the old instance
- Associate the correct Elastic IP on the new instance
- Test and wish for the best
This actually worked and did not even take too much time. Actually, really cool when thinking about this and imagining I would have had a physical server instead…
Tags: Amazon AWS, EBS, Recovery
