Posts Tagged: EBS

Amazon AWS / Recovery — No Comments
30
Jan 12

Recovering a non responsive AWS instance

I could not ssh in to one of my AWS instances last evening and it wasn’t serving any pages either. AWS management console said it was up, though. Rebooting did not help. The second reboot did not help either. Shutdown and start did not help. I was running out of tricks here!

For some reason, the instance had been running on 100% CPU utilization for days:

(I better do some monitoring in future!)

Even though the CPU usage had dropped after the restarting, the instance would not accept any connections. The only thing I could think of was to either ping the AWS forum, or to get the running volume on some new instance as the instance was an EBS based one. I decided to go with the new volume if the database would not mind too much. Steps I needed to do were:

  1. Snapshot the running volume
  2. Create a new volume out of the snapshot on the same availability zone
  3. Start a new instance with the Launch more like this
  4. Shutdown the new instance
  5. Detach the volume on the new instance
  6. Attach the volume which was created from the snapshot to new instance (need to have the correct  attachment information, like /dev/sda1)
  7. Start the new instance
  8. Disassociate the Elastic IP from the old instance
  9. Associate the correct Elastic IP on the new instance
  10. Test and wish for the best

This actually worked and did not even take too much time. Actually, really cool when thinking about this and imagining I would have had a physical server instead…

Amazon AWS / Recovery — No Comments
20
May 10

EBS-based instance problems

The instance I run this blog was slightly impacted a few days ago. All of a sudden I could not ssh into the instance and the Apache was running really painfully slow. It did not really work at all. While I was already fantasizing that my really super awesome new web-2.0-youtube-facebook-twitter crossbreed vKaiser.com had gotten some traction and was overloaded by the publicity, I ended up in the AWS site to see the service status. The service status was fine and my hopes were still high. Then the truth hit me, there were others as well in the forums who had similar issues, EBS based image becomes unresponsive and reboot does not help. Can’t either take a spapshot of the EBS volume, but stopping and starting might help. Just have to prepare for the instance to go down very, very slowly.

So, as I could not take a snapshot and was not particulary interested in using a few days old snapshots, I decided to just shut down the instance and give it the time it needs. Eventually, the server went down and I could restart it just fine. Situation back to normal. This incident could of course have been avoided easily by having a backup system ready or even a load balanced setup if I would have the money to run it.

No luck in getting traction.

Amazon AWS — No Comments
15
Dec 09

Unexpected Outage

The site went down today for a few hours and the worst thing is, it was sort of my own fault. The last time I was playing with booting from Amazon EBS I must have made a mistake when detaching volumes from the (wrong) instances. Thus, the incident was caused by the EBS volume not being attached. When I was trying out that EBS booting the one and only EBS volume which is attached to this EC2 instance had the “Attachment Information” as “busy” and not as “attached” which seems to be the standard status of a well working volume. I probably detached the volume and the status changed to “busy” state.

I remember wondering what that “busy” meant at that time. Now I know.

It should go without saying that this status information of “busy” is really, really uninformative. How about “detaching” instead when a user wants to detach a volume? And why did it take about one and a half weeks to detach? Is there a log somewhere I really did detach a volume? The lesson learned from this incident is to act if your EBS volume goes to “busy” state. All might work fine for a while but be warned, it will detach at some point. Also, it would be really nice if there would be some abstraction layer in between the real names of the volumes and instances and the ones available to customers. With this layer a user could add more descriptive names to instances or what ever objects there are. Or then really start using different accounts for development and production stuff… Really.

Amazon AWS / Cloud Computing — 1 Comment
6
Dec 09

Booting from Amazon EBS

Amazon has announced a new feature of booting instances from EBS volume. This feature changes radically the way how AWS instances can be preserved if compared to the traditional volume bundling and uploading to S3.

Though this all sounds nice, it isn’t really too easy to convert existing instaces to boot from EBS. All previous instances boot from the local instance disk. Amazon AWS management console indicates the location where the instance boots with the Root Device Type column. Previous instances have the root device type as instance-store while EBS images have the type as ebs.

To get started with the EBS images, there are a few images from Amazon which are useful as a base image. It was really easy to just boot one of them and mount one EBS volume which contained a snapshot of the database and the www root. Installing basic LAMP stuff, changes in httpd.conf and my.cnf to point in the EBS volume and the AWS instance which boots from EBS was ready. I could now create snapshots of the system in minutes and also shut down the system when I don’t need it and thus not get billed for the instance. Awesome! The snapshot also had the EBS volume snapshotted which was mounted to the instance.

The EBS image feature is likely to open a wide range of new applications and really change the way how an elastic service is been constructed. Basically, a member of a pool of web servers can now be created in advance and just turned on when there is a demand to use it. Of course, it first must update itself to be on par with the other pool members.

I am not really sure if it was my old lap top which I used to work with the EBS images or what, but the AWS management console was painfully slow in responding, especially when using Firefox. And when using IE, I did not get anything else in the pop-up window than the button to create the snapshot:

createImage

Firefox, though really slow in responding, gave the option of typing the name in the required field. Also, if you create EBS image and then decide to get rid of the EBS image, you have to delete the AMI first, otherwise the management console will complain that it’s in use.

I have yet to decide should I go with the instance-store or EBS with my instance. It will add something to my costs of running my site in AWS, but that shouldn’t be too much. I find a lot more benefits with EBS than running in instance-store, but then again I fear getting lazy in responding to possible threats of instances going down and disaster recovery.

Pauli Haikonen