Amazon has announced a new feature of booting instances from EBS volume. This feature changes radically the way how AWS instances can be preserved if compared to the traditional volume bundling and uploading to S3.

Though this all sounds nice, it isn’t really too easy to convert existing instaces to boot from EBS. All previous instances boot from the local instance disk. Amazon AWS management console indicates the location where the instance boots with the Root Device Type column. Previous instances have the root device type as instance-store while EBS images have the type as ebs.

To get started with the EBS images, there are a few images from Amazon which are useful as a base image. It was really easy to just boot one of them and mount one EBS volume which contained a snapshot of the database and the www root. Installing basic LAMP stuff, changes in httpd.conf and my.cnf to point in the EBS volume and the AWS instance which boots from EBS was ready. I could now create snapshots of the system in minutes and also shut down the system when I don’t need it and thus not get billed for the instance. Awesome! The snapshot also had the EBS volume snapshotted which was mounted to the instance.

The EBS image feature is likely to open a wide range of new applications and really change the way how an elastic service is been constructed. Basically, a member of a pool of web servers can now be created in advance and just turned on when there is a demand to use it. Of course, it first must update itself to be on par with the other pool members.

I am not really sure if it was my old lap top which I used to work with the EBS images or what, but the AWS management console was painfully slow in responding, especially when using Firefox. And when using IE, I did not get anything else in the pop-up window than the button to create the snapshot:

Firefox, though really slow in responding, gave the option of typing the name in the required field. Also, if you create EBS image and then decide to get rid of the EBS image, you have to delete the AMI first, otherwise the management console will complain that it’s in use.

I have yet to decide should I go with the instance-store or EBS with my instance. It will add something to my costs of running my site in AWS, but that shouldn’t be too much. I find a lot more benefits with EBS than running in instance-store, but then again I fear getting lazy in responding to possible threats of instances going down and disaster recovery.

Pauli Haikonen

Amazon ELB was announced in May and is currently in public beta phase. Previously, HAproxy was the way to go if a customer wanted load balancing within EC2 environment. Amazon ELB is an automatic load balancing solution which detects AWS instance health and distributes traffic accordingly, even across availability zones, but not across regions. The costs of using the service are really not that high, $0.025 per hour and $0.008 per GB transferred.

I wanted to test the ELB service with my simple WordPress installation. I have all the service in one Amazon AWS instance, since, well… I don’t have any sponsors to help with the testing. Thus, I had to setup another instance for the test. Of course, I was then running a duplicate database server as well, but for testing purposes this is all fine. I wanted to test the ELB across availability zones, so I started the instance in eu-west-1a while I had the old instance in eu-west-1b.

Tools:

To get started with Amazon Elastic Load Balancing, ELB API tools must be installed. The tools are installed on your own computer, so you must somehow tell Amazon who you are. There are two ways of describing this to EC2. The tools come with desription in the readme.txt of how to use either one of them. I decided to use the one where each command is appended with descriptions of where the credentials are. You can test the ELB service and your credentials by first querying the currently created load balancers with a command:

C:\Program Files\Support Tools>%AWS_ELB_HOME%\bin\elb-describe-lbs –headers –ec2-cert-filepath=E:\backup_amazon_certs\cert.pem –ec2-private-key-file-path=E:\backup_amazon_certs\pk.pem

The command should return “No LoadBalancers found” if you have not created any and if the command successfully completes.

Installing ELB:

First thing to do is to create a load balancer. The command is:

C:\>%AWS_ELB_HOME%\bin\elb-create-lb wpLoadBalancer –headers –listener “lb-port=80,instance port=80,protocol=HTTP” –availability-zones eu-west-1a –region eu-west-1 –ec2-cert-file path=E:\backup_amazon_certs\cert.pem –ec2-private-key-file-path=E:\backup_amazon_certs\pk.pem

In the above command, I create a load balancer with a name wpLoadBalancer and isntruct the balancer to listen port 80 to incoming requests and also connecting to port 80 in my instances which are to be load balanced. I set the availability zone to be eu-west-1a and the region to be eu-west-1 and the rest is to give information to EC2 of who I am. EC2 will respond with a public DNS name, for example wpLoadBalancer-26728261.eu-west-1.elb.amazonaws.com which doesn’t look too nice, but not to worry, the idea is to create a CNAME record using this information. Of course, this is problematic as you can’t put a CNAME record to the root of the domain.

Next step is to create the health check which the load balancer will use in deciding if the instance is available of not. This is really easy if you know what to do! The load balancer will make make a HTTP GET to the web server for a particular file. If the file is available, the status is OK and the instance is added in the pool of servers. The command is:

C:\>%AWS_ELB_HOME%\bin\elb-configure-healthcheck wpLoadBalancer –headers –region eu-west-1 –target “HTTP:80/ping” –interval 30 –timeout 3 –unhealthy-threshold 2 –healthy-threshold 2 –ec2-cert-file-path=E:\backup_amazon_certs\cert.pem –ec2-private-key-file-path=E:\backup_amazon_certs\pk.pem

EC2 will answer with an ack:
HEALTH-CHECK TARGET INTERVAL TIMEOUT HEALTHY-THRESHOLD UNHEALTHY-THRESHOLD
HEALTH-CHECK HTTP:80/ping 30 3 2 2

This means that the ELB will try to get a file called “ping” from the root of the web server every 30 seconds. The timeout for each request is three seconds and if two pings are missing, the server is removed from the pool until two successful pings are received. I had no previous experience with this, so it took a while to figure out how the ping actually works, but actually all you need is an empty file with that name (or what ever you define in the health check). Important thing is to really keep that empty, because that is just unnecessary traffic if the file would have a size. If the file can’t be found, Apache will respond with a 404 code, while the load balancer wants a 200 code.

Instances are added in the load balancer with the following command:

C:\>%AWS_ELB_HOME%\bin\elb-register-instances-with-lb wpLoadBalancer –headers –region eu-west-1 –instances i-55555555,i-44444444 –ec2-cert-file-path=E:\backup_amazon_certs\cert.pem –ec2-private-key-file-path=E:\backup_amazon_certs\pk.pem

And EC2 responds with:

INSTANCE-ID INSTANCE-ID
INSTANCE-ID i-55555555
INSTANCE-ID i-44444444

One thing to notice here is that you can add instances in the load balancer which are not in the availability zone where the load balancer is. I made a mistake here and were wondering why my other instance’s access log was not getting any hits from the load balancer. The command

C:\>%AWS_ELB_HOME%\bin\elb-describe-instance-health wpLoadBalancer –headers –region eu-west-1 –ec2-cert-file-path=E:\backup_amazon_certs\cert.pem –ec2-private-key-file-path=E:\backup_amazon_certs\pk.pem

And gave a response:

INSTANCE-ID INSTANCE-ID STATE
INSTANCE-ID i-55555555 OutOfService
INSTANCE-ID i-44444444 InService

The problem was fixed by extending the load balancer to cover the eu-west-1b availability zone. How cool is that! Just one command and the balancer covers a new zone! The command was:

C:\>%AWS_ELB_HOME%\bin\elb-enable-zones-for-lb wpLoadBalancer –availability-zones eu-west-1b –headers –region eu-west-1 –ec2-cert-file-path=E:\backup_amazon_certs\cert.pem –ec2-private-key-filepath=E:\backup_amazon_certs\pk.pem

And finally, I had both of my instances with InService state. I could now start the actual balancing of requests for two different sites. I did not do any proper load test by generating load on the instances, I just wanted to test the availability by stopping Apache on either one of the intances. I also edited the www.dkaiser.com CNAME record which was previously pointing to dkaiser.com HOST A record (with elastic IP) and now to the public DNS name of the load balancer.

The first tests were unsuccesful. All was fine when the Apache was running on the instance which had the public Elastic IP configured, suggesting the load balancer did not really distribute traffic to the other node when the first one failed. It just did not work, though the instance was in “InService” state. I then started to google a bit and it became apparent, that the ELB uses a kind of fancy round-robin in distributing requests to a zone. More details here. The point is, there should always be a properly functioning instance per availability zone. This means, having one instance per zone doesn’t really cut it. After I realized my mistake, I had to disable one of the zones from the load balancer, terminate the instance, boot a new instance in eu-west-1b attach volumes and add the new instance to the wpLoadBalancer (once again, how cool is that!). I could now shut down either one of the Apache processes and the site would be up. All as expected.

Summary:

Amazon ELB seems like a great way to load balance traffic. It is also quite cheap. One draw back is the fact that you have to make a CNAME record for the load balancer. This makes it impossible to load balance traffic directed to the root of the site. Fixing this issue is on Amazon’s tasks of future improvements.

Pauli Haikonen

I have put together a task list of what I would like to test with the Amazon AWS infrastructure and so far I have gotten my web server running with EBS. Also the volume bundling and instance creation has been tested a few times. The system has been running quite ok for the past two weeks. It has been interesting to view the error log on Apache, people searching for example the page for phpmyadmin…

Anyway, the next thing I would want to test is to get some kind of monitoring in place. I have some experience with Nagios so I took that route and installed it on the basic m1.small instance using these instructions which got me a clean installation of Nagios. I could then add a host definition of this site and the service which to monitor (http).

I did use the public interface (elastic IP) since it is the only static ip I own. This is, though, the first implication of the problems related to running monitoring system in cloud. With Amazon AWS, you can get by default, five elastic IPs. That will not get you too far, but of course 20 instances is the maximum amount of instances by default anyway, but I have understood more can be purchased if there is a need. How do you deal with the instances that don’t have an elastic IP? You could get around this problem by creating all your instances with a VPN connection and then registering those ips, but well… does not sound too easy.

And then there is the actual alerting when something goes wrong. It’s kind of difficult to have the monitoring server to send SMS messages since it’s impossible to connect a physical device to a virtual machine. I will try installing Skype on the monitoring server and then use Skype to actually send the SMS onwards, but it will still use Internet while on its way to Skype SMS gateway. If there is a connection problem somewhere, the message will not reach me. I should also consider the reliability of the VM running Nagios, which is best effort by most. The system should be clustered using some method, but have to see how Nagios supports this. Oh, and by the way, there has been a few cases when the elastic IP address block (the whole /17) has been blacklisted for spam which in effect stops you from receiving the alerts with email.

To summarize, if I would have the option, I would not run monitoring in Amazon or any other Cloud Computing facility. I would have it the old way – physical – and enjoying the pleasure of firmware upgrades and power failures and all the good stuff.

Pauli Haikonen

We have a lottery draw every Saturday. It’s quite traditional in Finland and people are really active in playing it. Last week we had about 6.9 million Euros for the lucky person getting all seven numbers right. Finns played the game for a total of over 18 million Euros which was the new record. So it came Sunday and the clock was approaching 8.45PM. The draw was on. I was one of the suckers with my own numbers in as well. I was playing the online version of the game and did not have my numbers anywhere else than in veikkaus.fi which is the online game portal of the gaming monopoly in Finland.

The draw was over and clock was about 9 PM. I was trying to log in to the portal but did not even get the front page open. The site was down. A few minutes later the site was down still. It eventually took an hour until the site was functional again. This is a prime example of a site with highly variable traffic load and I can’t help but wonder if cloud computing could help in accommodating with the variable load. This is just a thought, since I don’t have any idea of the application architecture of veikkaus.fi or if it would even be legally possible to burst the excess traffic to, let’s say, Amazon. There are connectors to online banking facilities in veikkaus.fi for example which might make cloud bursting difficult. It would be interesting, though, if this would be possible. This actually is not the first time veikkaus.fi does not work right after the draw is done and I bet there are plenty of people eager and annoyed to not being able to check the results. Come on, I might be a millionaire and have to wait for this site to load!

I would imagine cloud bursting to be difficult, but by no means impossible if there is a will to do it.

Pauli Haikonen

There are probably as many definitions for cloud computing as there are people talking about it. Simon Wardley is one of my favourite speakers and he has given some excellent talks about cloud computing. Few focus on the definitions of what is cloud computing. If I would choose some of Mr. Wardley’s definitions it would be the definition of how a sysadmin would define cloud computing. Of course much of this depends on what type of cloud computing we are talking about. Is it like Infrastructure as a Service (IaaS), Platform as a Service (PaaS) or Software as a Service (SaaS)? For example, Amazon is the market leader in IaaS, Microsoft’s Azure is a cloud platform (PaaS) for running generic applications in their own cloud and Salesforce.com is an example of a SaaS vendor for running a their own specific CRM application in Salesforce’s cloud. All are very much different, and someone might argue Salesforce.com actually not being a cloud offering since they have been in the market long before term cloud computing was even publicly mentioned.

There are a few common denominators, though, with the offerings which I would rate as a cloud offering. First of all, they should not have significant upfront costs, but should be based on usage based billing. The offering should also be elastic which allows the client to add and decrease resource used how they see fit in a self-service manner. This should be possible in nearly real time. This is like Mr. Wardley states, commoditization of IT. The analogue of seeing cloud computing as an electricity grid is brilliant. We are not quite there yet, but are not falling too far away. We are now kind of in state that if a company would go with the IaaS offering, you are basically buying the turbine of a water electricity plant, but you still need a lot of expertise in getting the electricity to your factory, which is building, for example, rubber ducks. If you go with the PaaS offering, you can do with fewer people and need only the people creating the application to, for example, designing the 3D model of your next… superduper rubber duck. If you then again go with the SaaS offering, you can – maybe – get the application almost or completely ready for your specific needs and can do only with the person designing the next superduper rubber duck hit.

So the question of how to make money in cloud markets? Of course it all depends about the resources and where to put the most focus on. All in all, a company building rubber ducks is not interested in running the electricity plant, nor do they want to develop applications, they just want to create the ducks as cheap as possible and sell as good profit as possible. So they would like the SaaS option to be the best if it would be available. They would need someone to create the offering for 3D rubber duck design application. Let’s say I would start offering the application… I then have to find the cheapest way to deliver the application with all the underlying levels, infrastructure, platform and service which I could then sell for the best available price. I am dreaming of an application delivery which would use different clouds for different levels (IaaS, PaaS, SaaS), mixing those to build the most “insert your adjective here” service. Surely, we are not there yet.

SaaS though, is nothing new. There are and have been companies selling SaaS offerings for years and something like Google with their Apps and Gmail and many others is a great example of this. Though, having Google running all the companies applications would make the company totally dependent on Google. If Google were to have a problem, all services would be down. On the other hand (I guess you were waiting for this), if the company would have gone with two different turbines, for example buying the infrastructure level from Amazon and say, GoGrid, and built the application on top of those vendors, redundancy could be achieved. Probably not high-availability, but there would be some ways of creating a disaster recovery plan. You would not be locked in to Google. Or then you just assume Google will not go down, or prepare for this in your SLA.

I rather like the US apps.gov site, which allows government customers to add services in a shopping basket and then roll with those. I would imagine there is a lot of a resource backing this up, but this looks great from the customer viewpoint. I would like to see a similar service for consumers and companies, though I really don’t know how the apps.gov goes after you checkout your SharePoint for 65k (did I mention something about upfront costs?). If the implementation is something involving HA and not vendor lock-in, this starts to sound good. Anyway, I like this, though it does sound like a one more definition of cloud computing.

Pauli Haikonen

Today I realized you have to be really careful on how the Amazon instances are assigned if you are using reserved instances. I did the math and figured out that buying a reserved instance will probably be ideal for me since I rarely need to scale, other than the few times I might be testing how to scale.

So I bought the m1.small instance and had it placed in the best available place automatically. It went to eu-west-1b. Then I created my first instance and started enjoying my 0.04 per hour charge. The account activity was though something else, running at the normal 0.11 per hour rate. Then I had a look at the exact location where I had my instance placed. It was eu-west-1a, not 1b. Though that was chosen automatically as well. Sigh. I also had my EBS volume in 1a as well. I am not sure if you can actually even mix those. I would guess not.

So I pretty much had to do the same thing again which I was doing yesterday. I wanted to give another shot at that fstab issue if it could be solved, so I decided to bundle the image again. I also snapshotted the EBS volume and created a volume out from the snapshot and placed the volume in eu-west-1b. I got my bundle uploaded to S3, so I proceeded to initiate it. Once again, the AWS management console was lagging a bit and did not give me the small instance and actually no RAM ID nor Kernel ID either. Waiting a bit helped here and I got my IDs and small instance as well. I don’t even know what those IDs are, but I guess they are important somewhat…

I was then able to boot up my instance at the glorious eu-west-1b availability zone. I attached the EBS volume created from the snapshot, but the fstab was once again missing the mountpoint, so I had to add that and reboot. MySQL and Apache started fine this time and the site was up. The last thing was to change the Elastic IP to point to the new host and shut down the old server and clean up the old snapshots and EBS volumes. The whole thing took around an hour while sitting on sofa and watching the evening news at the same time.

I had a look at my Account Activity, but so far there has not been any new information about the 0.04 charge… I really hope this was not all for nothing! 
 

Pauli Haikonen

On Sunday I got my virtual server running in Amazon EC2 and it has been happily runnning there since. I have done some homework and know not to rely on Amazon keeping the instance running forever. I should expect it to fail. At the beginning this felt suspicious… I should expect the server to fail? What kind of a service is that? After giving this some thought I eventually realized this is how we should of course expect all IT systems to behave. It just happens that mostly IT people tend to rely on good luck instead of having a tested backup scheme, not to mention a tested disaster recovery plan.

Amazon forces you to think in a different way. The virtual machines run on not-so-high-end server hardware, which means that at any given moment, the server can fail and the poor sysadmin has to figure his/her way out of the situation. Good thing is, Amazon provides some tools to do the task as well, for example the Amazon Simple Storage Service and the Amazon EBS, which provide persistent storage for files used by the instances. The characteristic difference between these two is, that the instance can mount an EBS volume, but S3 works with REST and SOAP interfaces, thus making EBS fast but expensive and S3 slow but cheap.

In my previous post, I setup my first instance and application running in the cloud. I was a bit lucky, since the system was up untill today because I did not do the basic steps of bundling my instance and shapshotting the volume. So, to do this a few things have to be done.

1. Test if you have ec2-ami-tools installed on the instance you are bundling. Install them if they are missing.
2. Move your private key (pk-something.pem) and certificate (cer-something.pem) file to the instance’s /mnt directory (this is fine since /mnt will not be bundled).
3. Use ec2-bundle-vol command to build the bundle (for example: ec2-bundle-vol -d /mnt/image -k pk.pem -c cert.pem -u “AWS account id” -r i386). You might get an error like this, if you are using the standard ami, but this is no reason for concern, as it will most likely complete the bundling:

   NOTE: rsync with preservation of extended file attributes failed. Retrying rsync
   without attempting to preserve extended file attributes…
   NOTE: rsync seemed successful but exited with error code 23. This probably means
   that your version of rsync was built against a kernel with HAVE_LUTIMES defined,
   although the current kernel was not built with this option enabled. The bundling
   process will thus ignore the error and continue bundling. If bundling completes
   successfully, your image should be perfectly usable. We, however, recommend that
   you install a version of rsync that handles this situation more elegantly.

4. Use ec2-upload-bundle command to upload the bundle to S3 (ec2-upload-bundle -b myimages -m image/image.manifest.xml -a “Access Key ID” -s “Secret Access Key”).
5. Register your private AMI. I did this through the AWS management console.

If all went well, you now should have your private AMI created and ready to be provisioned. I proceeded to test my setup with first doing a snapshot of the EBS volume. I did this with xfs_freeze -f /mountpoint command and then snapshotting the volume through AWS console. Of course, I should have done sync and database lock too, but decided to live dangerously since this is just a test setup. After the snapshot was completed I unfreezed the partition and terminated the running instance. I started provisioning the replacement server, but to my surprise I did not have the option of a small instance anymore. They started from large. I was puzzled and did not really want to start provisioning of a large instance. It could have been something with the web GUI so I changed my zone from EU to US and back and suddenly I did get the small one in the list as well. Great! It’s fantastic though to realize how easy it is to scale vertically with Amazon AWS, though it’s not too different than with VMware ESX which goes along the same lines: shut down, change the VM properties and boot up with more RAM and/or CPU.

The server was initiated and I could log on. Apache and MySQL did not start though, because the EBS volume was not attached. I proceed to add that and gave it another reboot but still the services did not start. I then went to look for the mountpoint which had disappeared. Adding this and another reboot later the services were running happily! I also had to manually give the Elastic IP to the new instance. I suppose the mount point information is not included within the bundle, but I have to investigate this further.

I now have my system running again. I also have a little more confidence for the Amazon AWS as well. This effort though required a good part of manual work. The next step would be to automate some of this and to create a recovery plan in case the server fails.

Pauli Haikonen

I haven’t had a blog before - be prepared. 

I’ve been kind of drawn to this new hype term ”Cloud Computing” a while now, I would guess from say, July this year. I wanted to test this stuff myself and actually did try Amazon AWS and  GoGrid, but then decided to go with Amazon. I have been working with IT a few years and have experience with virtualization as well as IT outsourcing. 

Looking at cloud computing from a sysadmin’s view point, I could even buy cloud computing to be a disruptive technology. It has the potential to change the way how IT works. Though, it could very well be that the fad faints as fast as it arrived. A disaster with the bigger cloud providers would probably start that process… let’s hope that will not happen. And, Sidekick/Danger/T-Mobile/Microsoft/Hitachi disaster was not a cloud offering, though at the moment every service delivered through Internet seems to be referred as a cloud application.

I chose Amazon as the IaaS provider because they are the market leader and had two availability zones in EU (I’m from Finland and sadly, rarely travelling). Having a low latency is kind of imperative with any command line tools and this was really annoying with GoGrid due to their geographical placement.

I also did buy a few books about the subject, though there are not too many published yet. I did like George Reese’s book Cloud Application Architectures and can recommend that as an introduction to Cloud Computing with Amazon. I also did buy a book about Google App Engine and one about Microsoft Azure, but since I am more interested about the infrastructure side, those are still to be read.

The first test I wanted to do was to install some form of a linux and then proceed to install some meaninful application. I chose WordPress to be the application which also implied installing Apache, PHP and MySQL as well. I also did use the Amazon Elastic Block Storage (EBS) for the Apache and MySQL database. XFS was used for the EBS partition. The system seems to run quite ok and is relatively responsive, at least when compared with guests on VMware Server running on my PC. Of course, you are testing it right at this moment yourself. This site and blog is that system: the cheapest server Amazon offers.

I should have added all the commands here, but did not write those down this time. In future tests, those will be added as well.

Pauli Haikonen