I could not ssh in to one of my AWS instances last evening and it wasn’t serving any pages either. AWS management console said it was up, though. Rebooting did not help. The second reboot did not help either. Shutdown and start did not help. I was running out of tricks here!

For some reason, the instance had been running on 100% CPU utilization for days:

(I better do some monitoring in future!)

Even though the CPU usage had dropped after the restarting, the instance would not accept any connections. The only thing I could think of was to either ping the AWS forum, or to get the running volume on some new instance as the instance was an EBS based one. I decided to go with the new volume if the database would not mind too much. Steps I needed to do were:

1. Snapshot the running volume
2. Create a new volume out of the snapshot on the same availability zone
3. Start a new instance with the Launch more like this
4. Shutdown the new instance
5. Detach the volume on the new instance
6. Attach the volume which was created from the snapshot to new instance (need to have the correct  attachment information, like /dev/sda1)
7. Start the new instance
8. Disassociate the Elastic IP from the old instance
9. Associate the correct Elastic IP on the new instance
10. Test and wish for the best

This actually worked and did not even take too much time. Actually, really cool when thinking about this and imagining I would have had a physical server instead…

I just realized AWS has a feature called the CloudFormation which allows users to script their technology stack in a convenient and easily understood JSON formatted text files which can then be used to deploy the stack over and over again, always the same way. Fantastic! This eases a the burden of managing a bunch of customized AMIs or other ways of having some custom features introduced to the AMIs. I wonder how I did not notice this feature before. It even has a tab in the AWS Management Console. There are also some sample templates which for example install Drupal or a basic Ruby Hello World example.

As a test, I ran the Drupal installation script and I have to say this was by far the easiest Drupal installation I have ever done. From start to finish in 5 minutes where most of it was just waiting for the deploy to finish. Absolutely great! Minor thing might be to remember that the security keys are not available in all the Regions, at least not in US East (Virginia) my keys were not available which caused the stack deployment to fail without any good reason except key was not found… I was of course first thinking of a typo in the key name. The other thing is that the user must know the instance type name, such as t1.micro while a drop down menu would be great.

There is also a possibility to modify an existing stack which is actually a relatively new feature. This makes it even more usable. It would be interesting to see if I could do a stack for a simple Aegir installation as lately that’s the platform I have been installing the most and doing the manual installation has become kind of boring. CloudFormation would help lot with that!

Today’s breaking news have been Bloomberg’s story about the Sony PSN attack been conducted by using Amazon Web Services. I read the story and feel confused, like how on earth can the source of the servers be any kind of relevancy if they’ve been using a public cloud provider? Come on, Amazon can’t and really should not, follow what their customers do with their servers. This whole thing Bloomberg is writing about is like saying the bank was robbed by a Smith&Wesson and it was Smith&Wesson’s fault.

Of course, there will be a subpoena for getting all the information of the account used in managing the account and I guess they had to use some stolen credit card as well which is interesting. Also, the statement in the Bloomberg’s article about anyone anonymously going and getting an account in AWS is kind of not totally true. Maybe it can be managed somehow if using a stolen credit card, but it’s not an anonymous service as such. And how are you going to prevent that “flaw” in the system of the possibility using stolen cards and false identities? Scan your id and send that as well or visit them at AWS personally? Huh?

In the end of the article, there is a thought-provoking paragraph of “Rethinking the Cloud” because a cloud can be used also for malicious purposes. Yep. I’ll do think about this for a while…

Thinking…

Thinking…

…and it should not matter for the most parts. Say, the whole AWS would be used only for attacks and the service level would degrade and my IPs would be black listed, then I probably would switch to some other provider, but, right now, I am not worried the least bit. I have my application and the service level I need in a good and healthy balance.

I wanted to get a fi-domain as I am building a site for our housing company. It’s very much a pro bono work, but interesting nevertheless. To be honest, this is the first time I have to register a fi-domain and man, it’s not as easy as getting a com or similar domain with DynDNS etc. You need to be a Finnish citizen to be allowed to get one for starters and made sure you are not violating any possible trademarks or even more, some real people with your domain name.

I would perhaps been ok if a DynDNS type of service would exists (well, now as I write this it probably does) in Finland, but the ones I came across were mostly just taking orders and not like dynamically updating their resources… but can’t of course be totally sure. Anyway, I decided to give Amazon Route 53 a go as it is new and I do appreciate the possibility to update the records on command line. Or well, I perhaps did not investigate really too much before signing up.

First I had to though register the fi-domain with Ficora and that took around a day to get the credentials on paper. Yes. On paper. The next step was to register the name and give them two (at this point fictious) name servers. Then I was on my way to Route 53. The first look at the Getting Started Guide is not very encouraging. Need to create some files which contain the access keys and the actual requests. Need to run a perl script to actually create the records. Good thing I bought my first Mac just a few months ago as with Windows this would have sucked.

So the first thing was to create the .aws-secret file which contains your AWS Secret Access Keys it looks something like this:

%awsSecretAccessKeys = (
“my-keys” => {
id => “JISEGIOJDFGSLSDKFG”,
key => “KSLDFSDFGSDFGSasdfsdASFDSDF”,
},
);

And it really needs to be named .aws-secret and have only read permissions as the dnscurl.pl checks this.

Then create the zone you have registered:

<CreateHostedZoneRequest xmlns=”https://route53.amazonaws.com/doc/2010-10-01/”>
<Name>YOURDOMAIN.fi.</Name>
<CallerReference>SOMETHINGRANDOMHERE</CallerReference>
<HostedZoneConfig>
<Comment>Creating first zone</Comment>
</HostedZoneConfig>
</CreateHostedZoneRequest>

Then download dnscurl.pl from the AWS developer tools and run it with these parameters:

dnscurl.pl –keyname my-keys — -X POST -H “Content-Type: text/xml; charset=UTF-8″ –upload-file MyCreateRequest.xml https://route53.amazonaws.com/2010-10-01/hostedzone

You should get something like this in return:

<CreateHostedZoneResponse xmlns=”https://route53.amazonaws.com/doc/2010-10-01/”><HostedZone><Id>/hostedzone/34LJSKFSJGSDFKJ</Id><Name>YOURDOMAIN.fi.</Name><CallerReference>JIjasdmfasfw4af3233</CallerReference><Config><Comment>Creating first zone</Comment></Config></HostedZone><ChangeInfo><Id>/change/23ILKSFJDLSK</Id><Status>PENDING</Status><SubmittedAt>2011-01-24T20:48:47.715Z</SubmittedAt></ChangeInfo><DelegationSet><NameServers><NameServer>ns-1778.awsdns-30.co.uk</NameServer><NameServer>ns-372.awsdns-44.com</NameServer><NameServer>ns-1621.awsdns-38.org</NameServer><NameServer>ns-534.awsdns-04.net</NameServer></NameServers></DelegationSet></CreateHostedZoneResponse>

Here are the real name servers which I had to give to Ficora and it happily said them being ok, so fi-domain is well supported by AWS! Yey!

Then you can start adding records to your zone. First need to create the MyRecordsRequest.xml for the records which could look like this:

<?xml version=”1.0″ encoding=”UTF-8″?>
<ChangeResourceRecordSetsRequest xmlns=”https://route53.amazonaws.com/doc/2010-10-01/”>
<ChangeBatch>
<Comment>
Create A-record
</Comment>
<Changes>
<Change>
<Action>CREATE</Action>
<ResourceRecordSet>
<Name>www.yourdomain.fi.</Name>
<Type>A</Type>
<TTL>14400</TTL>
<ResourceRecords>
<ResourceRecord>
<Value>192.0.0.111</Value>
</ResourceRecord>
</ResourceRecords>
</ResourceRecordSet>
</Change>
</Changes>
</ChangeBatch>
</ChangeResourceRecordSetsRequest>

dnscurl.pl –keyname my-keys — -H “Content-Type: text/xml; charset=UTF-8″ -X POST –upload-file ./MyRecordsRequest.xml https://route53.amazonaws.com/2010-10-01/hostedzone/34LJSKFSJGSDFKJ/rrset

And you should get a response like this:
0.0%
<?xml version=”1.0″?>
<ChangeResourceRecordSetsResponse xmlns=”https://route53.amazonaws.com/doc/2010-10-01/”><ChangeInfo><Id>/change/C3FMNWCVL1YW40</Id><Status>PENDING</Status><SubmittedAt>2011-01-25T19:16:24.181Z</SubmittedAt></ChangeInfo></ChangeResourceRecordSetsResponse>

I got a few problems with “root is not authorized to perform: route53:ChangeResourceRecordSets on resource” because I did not have ./ in front of the MyRecordsRequest.xml, so remember to have it there.

Well hello there! It’s been a while, but I finally found some time to work with the sites and the latest of Amazon Web Services. Lately, AWS has introduced the tiny micro instances with a tempting price tag for small businesses with not too much of a need for high performance. For me, those do sound fantastic for testing purposes as I have been wanting to try running the two sites, this and vkaiser.com on a bit more robust architecture than the current one with just an EBS based AMI and the videos in S3.

I run the typical LAMP stack on one AMI, thus the idea was first to boot up one micro instance and have a look. Well, I chose to go with some old image I had created way long time ago. It also had a LAMP stack installed, but of course it was kind of outdated and the vkaiser.com did not look too good (well, does it now either…), so I figured I could rsync the html folder of the Drupal installation and I did eventually get the rsync with public key working. Then I realized that the db wasn’t really up to date either and the drupal modules would not of course work, so how about connecting to the database on the current “production” which would kind of resemble the hopefully future setup too as running a separate db server (and slave) would just be the way to go at least with Drupal.

Settings in Drupal for remote database connections are really simple. First edit the MySQL configuration (/etc/my.cnf) to have

Bind-address=database_ip

And if you have skip networking defined, comment that out.

Then add remote access permissions to the database for a db user

GRANT ALL ON *.* TO ‘dbuser’@'remote_ip’ IDENTIFIED BY ‘password’;

And modify the settings.php on the remote application server to point to the database server:

$db_url = ‘mysql://dbuser:password@database_ip/database_name’;

Then you can test the connection to the database. At least I got that working, though I was first editing the wrong settings.php file which of course did not prove to be very useful in getting the db connection working.

The real deal was though to see how the t1.micro performs under stress. I browsed a while for some tools with how to do the test, but then I found loadimpact.com which simulates really well concurrent users from 0-50 for free! With some euros, you can get up to 5000 users and customized tests and what not. I like the service, though it went down just as I got my t1.micro tested. The average response time was around 1.5 seconds for the vkaiser.com frontpage and it did not show any real implications of getting slower, thus I should put more load on the micro if I coughed up some cash. I next went on and tested my good old small instance and got about the same results

This wasn’t too scientific, though the results are encouraging. I mean, 50 concurrent users is about 49 more than this site usually has and the micro worked well, so I am planning to make a switch soon… More about that later!

I’ve been neglecting the blog for a while and feel sorry about that. The spring has been busy and will most likely stay like that, some bachelor parties and weddings and I am also going to be a dad in the beginning of June! The boy is already kicking strong!

But I also have some new cloud related things to tell you about. Since the blog isn’t exactly driving traffic too much and I had some free CPU resources, I started a new project, vKaiser.com, which is a more Web 2.0 oriented site. Well, an imitation of YouTube but with heavy connections to social media sites like Facebook and Twitter. The site is by no means ready, but you are welcome to check it out – with Firefox. IE7 is ok too if you are not on compatibility mode. Interesting things to mention is the storage of the videos and thumbnails in S3 and the possibility to use CloudFront too.

And just to make this post a bit more cloud related and not just pitching my new site, a short story of what happened during the development at one point. As said, I had the Facebook Connect module as well as the Drupal for Facebook (yes, I ended up running Drupal as the CMS system) module installed but I had not enabled the Facebook Connect module since the Drupal for Facebook does essentially the same thing of connecting with your Facebook credentials. Or should do. I had and still have problems with the module as it forwards to a page which can’t be found but still after a few refreshes actually logs in. Anyway, I did go and enabled the Facebook Connect module while Drupal for Facebook had the same functionality enabled if another module would work a bit better.

Sure enough, after enabling the module I was watching a white browser screen with an Internal Server Error 500 with no access to the admin interface at all. What to do then? Should I mess with the database? Remove some modules and run update.php? Well, could not even access the update page. Luckily, I was running the site on an EBS based image! I had a week old (yeah, a bit old, but I did not mind) snapshot of the volume so all I had to do was to get the static files out from the bad volume, create a volume of the snapshot, shutdown the instance, detach the bad volume and attach the new volume. Boot up. Reboot had to be done too for some reason before I could see the log from AWS EC2 console. Reattach the elastic ip, copy the static files and I was back in business. Restore time below 10 minutes.

I love EC2.

So you have your great new application utilizing all the awesomeness of Amazon AWS? How you gonna sell it?

There are vague definitions of what is a cloud service and one of the prerequisites was that you could buy the service with your credit card and to pay only for the resources you use. Amazon DevPay allows developers to sell their AMIs (with application installed), all you need is a business in the US.

There are two ways how you can use DevPay: through AMIs you have built with the service installed and also by selling an application which uses S3 as the storage location. It’s a really good start, but still lacks a few things, like if you actually would like to provide high availability for the client who bought your AMI, the client will have to roll their own solution to achieve this. I bet in principle not many clients are willing to do that and would just like the application to be available.

When I think about this dilemma, a solution might be to have some kind of a root AMI which a customer would buy (and pay by the hour like crazy). This would then take care of the availability of the service by seeding new servers which are members of the application through some very wicked autoconfiguration. Actually, Elastra does almost this as their product allows the user to define architectures and then deploy them to Amazon. In principle, it would be possible to have an Elastra AMI with some configuration inside which the client could then deploy, but it does sound like a hack and not really something you could sell as a product. By the way, the Elastra’s product looks great for defining and deploying architectures at least internally within an organization.

The seeding of an application from a root AMI might make clients able to buy a redundant application and not pieces of it. So far, though, this is just a beautiful dream since the DevPay is not available in Europe so it isn’t possible to start with even the simplest model of serving an AMI of your superduperapp for the public. That is, if you don’t implement billing yourself. If I’m not totally wrong, the billing API is not public which makes rolling out your own solution impossible (you can’t either put limits based on usage charges in your account due to this reason).

I would be willing to adjust the definition of a cloud service regarding the pay by the hour and order by credit card before DevPay is available in Europe.

The site went down today for a few hours and the worst thing is, it was sort of my own fault. The last time I was playing with booting from Amazon EBS I must have made a mistake when detaching volumes from the (wrong) instances. Thus, the incident was caused by the EBS volume not being attached. When I was trying out that EBS booting the one and only EBS volume which is attached to this EC2 instance had the “Attachment Information” as “busy” and not as “attached” which seems to be the standard status of a well working volume. I probably detached the volume and the status changed to “busy” state.

I remember wondering what that “busy” meant at that time. Now I know.

It should go without saying that this status information of “busy” is really, really uninformative. How about “detaching” instead when a user wants to detach a volume? And why did it take about one and a half weeks to detach? Is there a log somewhere I really did detach a volume? The lesson learned from this incident is to act if your EBS volume goes to “busy” state. All might work fine for a while but be warned, it will detach at some point. Also, it would be really nice if there would be some abstraction layer in between the real names of the volumes and instances and the ones available to customers. With this layer a user could add more descriptive names to instances or what ever objects there are. Or then really start using different accounts for development and production stuff… Really.

I’ve been thinking the relationship between operating systems and the reason why they exist – the applications. How does PaaS fit into the future of computing or is IaaS just a stepping stone to a world without the traditional one server, one application approach?

Having a background with enterprise IT, I know something (but not much) about deploying applications. Many of them are multi-tiered, the usual being front-end application servers with back-end database servers. Some may have a load balancer in front of the application servers. Thus, there are a few different roles a server must fulfill to deliver the service to a customer using the application these servers in whole produce.

How do you set up an application? It depends about the requirements of course, but basically there are some usual things, at least if you narrow the selectable services. VMware got an idea to bundle a few virtual machines into a vApp which can then be deployed. I have not personally used those, but they seem like an interesting concept. They do work in a bit different way than how an AWS instances would work since with VMware you have the luxury of for example vMotion taking care of VM migration in case the host dies… Giving there is a VMware HA cluster in place. Hiding the complexity this way sounds fantastic! I love it. I do want the same in Amazon AWS!

There has to be a way to group things in Amazon AWS. There are a few tools such as RightScript and Puppet which provide a way to move from a specific AMIs to specific scripts which produce a certain kind of servers. Using these scripts it should be possible to deploy a full application with various components. Puppet also makes it possible to update a class (with Puppet the servers may belong to a class, maybe clasess) of servers to have for example the latest resolv.conf file. Sounds nice! Essentially, this sounds a lot like Microsoft SCCM but for UNIX like operating systems.

How about a future where a business owner could just browse to an IT webstore, select a CMS installation with a rough estimate of usage and the system would just produce it by running the scripts in the backgroud? Or upgrade an existing system? How many IT admins would lose their jobs?

Amazon has announced a new feature of booting instances from EBS volume. This feature changes radically the way how AWS instances can be preserved if compared to the traditional volume bundling and uploading to S3.

Though this all sounds nice, it isn’t really too easy to convert existing instaces to boot from EBS. All previous instances boot from the local instance disk. Amazon AWS management console indicates the location where the instance boots with the Root Device Type column. Previous instances have the root device type as instance-store while EBS images have the type as ebs.

To get started with the EBS images, there are a few images from Amazon which are useful as a base image. It was really easy to just boot one of them and mount one EBS volume which contained a snapshot of the database and the www root. Installing basic LAMP stuff, changes in httpd.conf and my.cnf to point in the EBS volume and the AWS instance which boots from EBS was ready. I could now create snapshots of the system in minutes and also shut down the system when I don’t need it and thus not get billed for the instance. Awesome! The snapshot also had the EBS volume snapshotted which was mounted to the instance.

The EBS image feature is likely to open a wide range of new applications and really change the way how an elastic service is been constructed. Basically, a member of a pool of web servers can now be created in advance and just turned on when there is a demand to use it. Of course, it first must update itself to be on par with the other pool members.

I am not really sure if it was my old lap top which I used to work with the EBS images or what, but the AWS management console was painfully slow in responding, especially when using Firefox. And when using IE, I did not get anything else in the pop-up window than the button to create the snapshot:

Firefox, though really slow in responding, gave the option of typing the name in the required field. Also, if you create EBS image and then decide to get rid of the EBS image, you have to delete the AMI first, otherwise the management console will complain that it’s in use.

I have yet to decide should I go with the instance-store or EBS with my instance. It will add something to my costs of running my site in AWS, but that shouldn’t be too much. I find a lot more benefits with EBS than running in instance-store, but then again I fear getting lazy in responding to possible threats of instances going down and disaster recovery.

Pauli Haikonen