I could not ssh in to one of my AWS instances last evening and it wasn’t serving any pages either. AWS management console said it was up, though. Rebooting did not help. The second reboot did not help either. Shutdown and start did not help. I was running out of tricks here!

For some reason, the instance had been running on 100% CPU utilization for days:

(I better do some monitoring in future!)

Even though the CPU usage had dropped after the restarting, the instance would not accept any connections. The only thing I could think of was to either ping the AWS forum, or to get the running volume on some new instance as the instance was an EBS based one. I decided to go with the new volume if the database would not mind too much. Steps I needed to do were:

1. Snapshot the running volume
2. Create a new volume out of the snapshot on the same availability zone
3. Start a new instance with the Launch more like this
4. Shutdown the new instance
5. Detach the volume on the new instance
6. Attach the volume which was created from the snapshot to new instance (need to have the correct  attachment information, like /dev/sda1)
7. Start the new instance
8. Disassociate the Elastic IP from the old instance
9. Associate the correct Elastic IP on the new instance
10. Test and wish for the best

This actually worked and did not even take too much time. Actually, really cool when thinking about this and imagining I would have had a physical server instead…

I, as well as many others, received today an email from Amazon about the need to reboot one of my instances. Actually, Twitter was already aware of this and was a bit upset of the need. For me, this was the second time since 2009 when Amazon has asked to reboot one of my instances. Once the HW was degraded and now this. I would say it’s quite a decent score since I have averaged something like five instances running all the time.

I am not upset, on the contrary I am happy AWS keeping the infrastructure up to date, be the reason for the reboot what ever. Besides, the systems should be designed so, that rebooting an instance should not take the service down, if you don’t accept it (like I do).

The actual process how AWS did inform the customers did feel ok. At first it was of course just rumours, but then I received an email stating the need which gave an acceptable time to react. When I logged in to the AWS Dashboard, I saw this kind of a message:

Which had a link to further information:

And even more information:

There was an option to do the reboot right now if I wanted, so I did it. At first after the reboot, I was looking at the instance in the dashboard, but the notification icon was still there. I would have thought it would disappear. Then I had a look of the details of the event and it actually had [Completed] written infront of the event:

Which now probably means it’s ok and I am done with this.

I just realized AWS has a feature called the CloudFormation which allows users to script their technology stack in a convenient and easily understood JSON formatted text files which can then be used to deploy the stack over and over again, always the same way. Fantastic! This eases a the burden of managing a bunch of customized AMIs or other ways of having some custom features introduced to the AMIs. I wonder how I did not notice this feature before. It even has a tab in the AWS Management Console. There are also some sample templates which for example install Drupal or a basic Ruby Hello World example.

As a test, I ran the Drupal installation script and I have to say this was by far the easiest Drupal installation I have ever done. From start to finish in 5 minutes where most of it was just waiting for the deploy to finish. Absolutely great! Minor thing might be to remember that the security keys are not available in all the Regions, at least not in US East (Virginia) my keys were not available which caused the stack deployment to fail without any good reason except key was not found… I was of course first thinking of a typo in the key name. The other thing is that the user must know the instance type name, such as t1.micro while a drop down menu would be great.

There is also a possibility to modify an existing stack which is actually a relatively new feature. This makes it even more usable. It would be interesting to see if I could do a stack for a simple Aegir installation as lately that’s the platform I have been installing the most and doing the manual installation has become kind of boring. CloudFormation would help lot with that!

Today’s breaking news have been Bloomberg’s story about the Sony PSN attack been conducted by using Amazon Web Services. I read the story and feel confused, like how on earth can the source of the servers be any kind of relevancy if they’ve been using a public cloud provider? Come on, Amazon can’t and really should not, follow what their customers do with their servers. This whole thing Bloomberg is writing about is like saying the bank was robbed by a Smith&Wesson and it was Smith&Wesson’s fault.

Of course, there will be a subpoena for getting all the information of the account used in managing the account and I guess they had to use some stolen credit card as well which is interesting. Also, the statement in the Bloomberg’s article about anyone anonymously going and getting an account in AWS is kind of not totally true. Maybe it can be managed somehow if using a stolen credit card, but it’s not an anonymous service as such. And how are you going to prevent that “flaw” in the system of the possibility using stolen cards and false identities? Scan your id and send that as well or visit them at AWS personally? Huh?

In the end of the article, there is a thought-provoking paragraph of “Rethinking the Cloud” because a cloud can be used also for malicious purposes. Yep. I’ll do think about this for a while…

Thinking…

Thinking…

…and it should not matter for the most parts. Say, the whole AWS would be used only for attacks and the service level would degrade and my IPs would be black listed, then I probably would switch to some other provider, but, right now, I am not worried the least bit. I have my application and the service level I need in a good and healthy balance.

Quora is down, Reddit is in emergency read only mode. Quite severe this is then!

According to the first investigation (from the AWS health dashboard) the reason for outage was a networking event which caused a large number of EBS volumes being re-mirrored. This caused capacity problems in the affected region. Also there were problems with one control plane which made it difficult to create new EBS volumes and instances. Control plane is a piece of router architecture which is responsible of drawing the network map, if you did not know it… I certainly did not know before.

Of course, there are plenty of other services impacted by the outage and I guess this is a great time to see how different services have been designed to sustain a degradation of some underlying components. Quora is totally dead (well, there is the notification to users) and Reddit is in read only mode. I give my points to Reddit as they have managed to fail gracefully to a cached read only mode.

Funny thing, just today I was reading a text by James Hamilton which is spot on this situation. I need to say I am surprised Quora did not have a fail over to a different location as the other location in US seems to be ok.

I wanted to get a fi-domain as I am building a site for our housing company. It’s very much a pro bono work, but interesting nevertheless. To be honest, this is the first time I have to register a fi-domain and man, it’s not as easy as getting a com or similar domain with DynDNS etc. You need to be a Finnish citizen to be allowed to get one for starters and made sure you are not violating any possible trademarks or even more, some real people with your domain name.

I would perhaps been ok if a DynDNS type of service would exists (well, now as I write this it probably does) in Finland, but the ones I came across were mostly just taking orders and not like dynamically updating their resources… but can’t of course be totally sure. Anyway, I decided to give Amazon Route 53 a go as it is new and I do appreciate the possibility to update the records on command line. Or well, I perhaps did not investigate really too much before signing up.

First I had to though register the fi-domain with Ficora and that took around a day to get the credentials on paper. Yes. On paper. The next step was to register the name and give them two (at this point fictious) name servers. Then I was on my way to Route 53. The first look at the Getting Started Guide is not very encouraging. Need to create some files which contain the access keys and the actual requests. Need to run a perl script to actually create the records. Good thing I bought my first Mac just a few months ago as with Windows this would have sucked.

So the first thing was to create the .aws-secret file which contains your AWS Secret Access Keys it looks something like this:

%awsSecretAccessKeys = (
“my-keys” => {
id => “JISEGIOJDFGSLSDKFG”,
key => “KSLDFSDFGSDFGSasdfsdASFDSDF”,
},
);

And it really needs to be named .aws-secret and have only read permissions as the dnscurl.pl checks this.

Then create the zone you have registered:

<CreateHostedZoneRequest xmlns=”https://route53.amazonaws.com/doc/2010-10-01/”>
<Name>YOURDOMAIN.fi.</Name>
<CallerReference>SOMETHINGRANDOMHERE</CallerReference>
<HostedZoneConfig>
<Comment>Creating first zone</Comment>
</HostedZoneConfig>
</CreateHostedZoneRequest>

Then download dnscurl.pl from the AWS developer tools and run it with these parameters:

dnscurl.pl –keyname my-keys — -X POST -H “Content-Type: text/xml; charset=UTF-8″ –upload-file MyCreateRequest.xml https://route53.amazonaws.com/2010-10-01/hostedzone

You should get something like this in return:

<CreateHostedZoneResponse xmlns=”https://route53.amazonaws.com/doc/2010-10-01/”><HostedZone><Id>/hostedzone/34LJSKFSJGSDFKJ</Id><Name>YOURDOMAIN.fi.</Name><CallerReference>JIjasdmfasfw4af3233</CallerReference><Config><Comment>Creating first zone</Comment></Config></HostedZone><ChangeInfo><Id>/change/23ILKSFJDLSK</Id><Status>PENDING</Status><SubmittedAt>2011-01-24T20:48:47.715Z</SubmittedAt></ChangeInfo><DelegationSet><NameServers><NameServer>ns-1778.awsdns-30.co.uk</NameServer><NameServer>ns-372.awsdns-44.com</NameServer><NameServer>ns-1621.awsdns-38.org</NameServer><NameServer>ns-534.awsdns-04.net</NameServer></NameServers></DelegationSet></CreateHostedZoneResponse>

Here are the real name servers which I had to give to Ficora and it happily said them being ok, so fi-domain is well supported by AWS! Yey!

Then you can start adding records to your zone. First need to create the MyRecordsRequest.xml for the records which could look like this:

<?xml version=”1.0″ encoding=”UTF-8″?>
<ChangeResourceRecordSetsRequest xmlns=”https://route53.amazonaws.com/doc/2010-10-01/”>
<ChangeBatch>
<Comment>
Create A-record
</Comment>
<Changes>
<Change>
<Action>CREATE</Action>
<ResourceRecordSet>
<Name>www.yourdomain.fi.</Name>
<Type>A</Type>
<TTL>14400</TTL>
<ResourceRecords>
<ResourceRecord>
<Value>192.0.0.111</Value>
</ResourceRecord>
</ResourceRecords>
</ResourceRecordSet>
</Change>
</Changes>
</ChangeBatch>
</ChangeResourceRecordSetsRequest>

dnscurl.pl –keyname my-keys — -H “Content-Type: text/xml; charset=UTF-8″ -X POST –upload-file ./MyRecordsRequest.xml https://route53.amazonaws.com/2010-10-01/hostedzone/34LJSKFSJGSDFKJ/rrset

And you should get a response like this:
0.0%
<?xml version=”1.0″?>
<ChangeResourceRecordSetsResponse xmlns=”https://route53.amazonaws.com/doc/2010-10-01/”><ChangeInfo><Id>/change/C3FMNWCVL1YW40</Id><Status>PENDING</Status><SubmittedAt>2011-01-25T19:16:24.181Z</SubmittedAt></ChangeInfo></ChangeResourceRecordSetsResponse>

I got a few problems with “root is not authorized to perform: route53:ChangeResourceRecordSets on resource” because I did not have ./ in front of the MyRecordsRequest.xml, so remember to have it there.

Hello!

I wrote a while back about my new architecture for the vKaiser.com site. I have been relatively happy with the setup until I realized one thing about the system. If I have three frontend servers with Drupal installed in all of them, all of them should be capable of sending email. I had only configured the first one which I have been happy with as it powers the dkaiser.com mail traffic (which is not much to be honest) as well.

So I now had to try to remember what I had done about a year ago with Postfix to get it running the first place. I used a rather old image to build the two other servers which did not even have it installed, only FFMPEG which by the way works pretty well in this kind of distributed system. I had done something with the /etc/postfix/main.cf as I had configured AuthSMTP to work as the relay agent. I found some decent instructions from here which I pretty much followed, while also copying the main.cf from the working server. Those instructions are really spot on, except I had to do postmap /etc/postfix/canonical to actually build the canonical.db, but after that mail started working. By default the sender is apache@example.com which if you haven’t allowed in AuthSMTP won’t get too far.

Now all is good!

Amazon announced free monitoring for EC2 instances. A cool new feature, I have to say. It looks good too. Just check your instances and the monitoring tab. As I had already discussed about those load tests with the t1.micro instances, I have to say it sucks a bit for not having this kind of monitoring available when I was doing it… but maybe I will test again.

I’ve been talking about load balancing already a bit, but that was about the Amazon Elastic Load Balancing. It’s a super easy way to do load balancing, with management now also through the EC2 management console, I believe. Then again, you have to use a CNAME to point to the load balancer which is a restriction as most of the cool guys have their site as http://mysite.com and this is why I did a HAproxy installation too.

So I have this http://vkaiser.com site which is a social site with video upload capabilites and connections to Twitter and Facebook but mostly it’s just my hobby and a test site on how to run Drupal. It’s a basic LAMP installation with EBS based image with the thumbnails and videos in S3 bucket. I’ve been wanting to add a load balancer, multiple web servers and a separate database server for a long time, but now as the t1.micro instances have become available, I have the financial possibilities to add them.

I first started with the load balancer. There are multiple good tutorials out there on how to do this, such as this. That tutorial even has instructions on how to install a high availability load balancing with heartbeat. I did not do that as I could not figure out how assign the virtual ip which the load balancers should share. One other thing which did not seem to work, was the web farm listening ip. For some reason it did not work with the elastic ip I had given for the HAproxy. I had to use a wild card to get connection to web servers working through the HAproxy. It might have something to do with virtual hosts, but I have not tested that.

It might be good to mention, that the connections after the HAproxy are done through the private address space as this does not consume the bandwidth. It might be interesting to see how the system can work with multiple availability zones, given there is a way a round the virtual ip problem. Well, one thing which might work is to have a hot stand-by HAproxy which would check the running HAproxy for availability and then start doing tricks with the AWS api if the other zone would not be available.

Then the file uploads. As it is a video site with the possibility to upload videos, I need to have some way to get the same uploaded files to all of the web servers. A scalable way would have been to install yet two more file servers with high availability, but at this stage I did not do that. I only did rsync with public key authentication between the servers. A good tutorial on how to do the public key stuff can be found here.

I actually have three web servers, which one of them is the database server because I did not add the wordpress installation (this blog) to the web farm yet at least. Thus, the vkaiser web farm has three nodes where the db server is kind of the root. All theme updates are done there and synced forward to the other two nodes. File uploads are synced from the other nodes to the root and from the root to the other two nodes. The slave nodes don’t sync directly between each other because there is no real need as they hop through the db server. In case the db server would be down, the site is gone anyway.

Oh yeah, one thing was the video conversion to flash. I have ffmpeg on the web servers, which is bad bad bad, but now as there are three nodes it should be a slightly better situation.

Next up would then be the master-slave replication for the database, investigating if there is a way to do that HAproxy virtual ip or elastic ip reassignment, move to a file share instead of rsync and get Puppet to take care of the configuration management. Possibly a separate cluster for ffmpeg would be so cool as well. A lot more to do!

Well hello there! It’s been a while, but I finally found some time to work with the sites and the latest of Amazon Web Services. Lately, AWS has introduced the tiny micro instances with a tempting price tag for small businesses with not too much of a need for high performance. For me, those do sound fantastic for testing purposes as I have been wanting to try running the two sites, this and vkaiser.com on a bit more robust architecture than the current one with just an EBS based AMI and the videos in S3.

I run the typical LAMP stack on one AMI, thus the idea was first to boot up one micro instance and have a look. Well, I chose to go with some old image I had created way long time ago. It also had a LAMP stack installed, but of course it was kind of outdated and the vkaiser.com did not look too good (well, does it now either…), so I figured I could rsync the html folder of the Drupal installation and I did eventually get the rsync with public key working. Then I realized that the db wasn’t really up to date either and the drupal modules would not of course work, so how about connecting to the database on the current “production” which would kind of resemble the hopefully future setup too as running a separate db server (and slave) would just be the way to go at least with Drupal.

Settings in Drupal for remote database connections are really simple. First edit the MySQL configuration (/etc/my.cnf) to have

Bind-address=database_ip

And if you have skip networking defined, comment that out.

Then add remote access permissions to the database for a db user

GRANT ALL ON *.* TO ‘dbuser’@'remote_ip’ IDENTIFIED BY ‘password’;

And modify the settings.php on the remote application server to point to the database server:

$db_url = ‘mysql://dbuser:password@database_ip/database_name’;

Then you can test the connection to the database. At least I got that working, though I was first editing the wrong settings.php file which of course did not prove to be very useful in getting the db connection working.

The real deal was though to see how the t1.micro performs under stress. I browsed a while for some tools with how to do the test, but then I found loadimpact.com which simulates really well concurrent users from 0-50 for free! With some euros, you can get up to 5000 users and customized tests and what not. I like the service, though it went down just as I got my t1.micro tested. The average response time was around 1.5 seconds for the vkaiser.com frontpage and it did not show any real implications of getting slower, thus I should put more load on the micro if I coughed up some cash. I next went on and tested my good old small instance and got about the same results

This wasn’t too scientific, though the results are encouraging. I mean, 50 concurrent users is about 49 more than this site usually has and the micro worked well, so I am planning to make a switch soon… More about that later!