Posts Tagged: vkaiser.com

Amazon AWS — No Comments
30
Oct 10

Load Balancing with HAproxy

I’ve been talking about load balancing already a bit, but that was about the Amazon Elastic Load Balancing. It’s a super easy way to do load balancing, with management now also through the EC2 management console, I believe. Then again, you have to use a CNAME to point to the load balancer which is a restriction as most of the cool guys have their site as http://mysite.com and this is why I did a HAproxy installation too.

So I have this http://vkaiser.com site which is a social site with video upload capabilites and connections to Twitter and Facebook but mostly it’s just my hobby and a test site on how to run Drupal. It’s a basic LAMP installation with EBS based image with the thumbnails and videos in S3 bucket. I’ve been wanting to add a load balancer, multiple web servers and a separate database server for a long time, but now as the t1.micro instances have become available, I have the financial possibilities to add them.

I first started with the load balancer. There are multiple good tutorials out there on how to do this, such as this. That tutorial even has instructions on how to install a high availability load balancing with heartbeat. I did not do that as I could not figure out how assign the virtual ip which the load balancers should share. One other thing which did not seem to work, was the web farm listening ip. For some reason it did not work with the elastic ip I had given for the HAproxy. I had to use a wild card to get connection to web servers working through the HAproxy. It might have something to do with virtual hosts, but I have not tested that.

It might be good to mention, that the connections after the HAproxy are done through the private address space as this does not consume the bandwidth. It might be interesting to see how the system can work with multiple availability zones, given there is a way a round the virtual ip problem. Well, one thing which might work is to have a hot stand-by HAproxy which would check the running HAproxy for availability and then start doing tricks with the AWS api if the other zone would not be available.

Then the file uploads. As it is a video site with the possibility to upload videos, I need to have some way to get the same uploaded files to all of the web servers. A scalable way would have been to install yet two more file servers with high availability, but at this stage I did not do that. I only did rsync with public key authentication between the servers. A good tutorial on how to do the public key stuff can be found here.

I actually have three web servers, which one of them is the database server because I did not add the wordpress installation (this blog) to the web farm yet at least. Thus, the vkaiser web farm has three nodes where the db server is kind of the root. All theme updates are done there and synced forward to the other two nodes. File uploads are synced from the other nodes to the root and from the root to the other two nodes. The slave nodes don’t sync directly between each other because there is no real need as they hop through the db server. In case the db server would be down, the site is gone anyway.

Oh yeah, one thing was the video conversion to flash. I have ffmpeg on the web servers, which is bad bad bad, but now as there are three nodes it should be a slightly better situation.

Next up would then be the master-slave replication for the database, investigating if there is a way to do that HAproxy virtual ip or elastic ip reassignment, move to a file share instead of rsync and get Puppet to take care of the configuration management. Possibly a separate cluster for ffmpeg would be so cool as well. A lot more to do!

Amazon AWS / Cloud Computing — 1 Comment
7
Oct 10

Testing t1.micro with loadimpact.com

Well hello there! It’s been a while, but I finally found some time to work with the sites and the latest of Amazon Web Services. Lately, AWS has introduced the tiny micro instances with a tempting price tag for small businesses with not too much of a need for high performance. For me, those do sound fantastic for testing purposes as I have been wanting to try running the two sites, this and vkaiser.com on a bit more robust architecture than the current one with just an EBS based AMI and the videos in S3.

I run the typical LAMP stack on one AMI, thus the idea was first to boot up one micro instance and have a look. Well, I chose to go with some old image I had created way long time ago. It also had a LAMP stack installed, but of course it was kind of outdated and the vkaiser.com did not look too good (well, does it now either…), so I figured I could rsync the html folder of the Drupal installation and I did eventually get the rsync with public key working. Then I realized that the db wasn’t really up to date either and the drupal modules would not of course work, so how about connecting to the database on the current “production” which would kind of resemble the hopefully future setup too as running a separate db server (and slave) would just be the way to go at least with Drupal.

Settings in Drupal for remote database connections are really simple. First edit the MySQL configuration (/etc/my.cnf) to have

Bind-address=database_ip

And if you have skip networking defined, comment that out.

Then add remote access permissions to the database for a db user

GRANT ALL ON *.* TO ‘dbuser’@'remote_ip’ IDENTIFIED BY ‘password’;

And modify the settings.php on the remote application server to point to the database server:

$db_url = ‘mysql://dbuser:password@database_ip/database_name’;

Then you can test the connection to the database. At least I got that working, though I was first editing the wrong settings.php file which of course did not prove to be very useful in getting the db connection working.

The real deal was though to see how the t1.micro performs under stress. I browsed a while for some tools with how to do the test, but then I found loadimpact.com which simulates really well concurrent users from 0-50 for free! With some euros, you can get up to 5000 users and customized tests and what not. I like the service, though it went down just as I got my t1.micro tested. The average response time was around 1.5 seconds for the vkaiser.com frontpage and it did not show any real implications of getting slower, thus I should put more load on the micro if I coughed up some cash. I next went on and tested my good old small instance and got about the same results

This wasn’t too scientific, though the results are encouraging. I mean, 50 concurrent users is about 49 more than this site usually has and the micro worked well, so I am planning to make a switch soon… More about that later!

Amazon AWS / Recovery — No Comments
20
May 10

EBS-based instance problems

The instance I run this blog was slightly impacted a few days ago. All of a sudden I could not ssh into the instance and the Apache was running really painfully slow. It did not really work at all. While I was already fantasizing that my really super awesome new web-2.0-youtube-facebook-twitter crossbreed vKaiser.com had gotten some traction and was overloaded by the publicity, I ended up in the AWS site to see the service status. The service status was fine and my hopes were still high. Then the truth hit me, there were others as well in the forums who had similar issues, EBS based image becomes unresponsive and reboot does not help. Can’t either take a spapshot of the EBS volume, but stopping and starting might help. Just have to prepare for the instance to go down very, very slowly.

So, as I could not take a snapshot and was not particulary interested in using a few days old snapshots, I decided to just shut down the instance and give it the time it needs. Eventually, the server went down and I could restart it just fine. Situation back to normal. This incident could of course have been avoided easily by having a backup system ready or even a load balanced setup if I would have the money to run it.

No luck in getting traction.

Applications / Recovery — 1 Comment
8
Apr 10

vKaiser.com

I’ve been neglecting the blog for a while and feel sorry about that. The spring has been busy and will most likely stay like that, some bachelor parties and weddings and I am also going to be a dad in the beginning of June! The boy is already kicking strong!

But I also have some new cloud related things to tell you about. Since the blog isn’t exactly driving traffic too much and I had some free CPU resources, I started a new project, vKaiser.com, which is a more Web 2.0 oriented site. Well, an imitation of YouTube but with heavy connections to social media sites like Facebook and Twitter. The site is by no means ready, but you are welcome to check it out – with Firefox. IE7 is ok too if you are not on compatibility mode. Interesting things to mention is the storage of the videos and thumbnails in S3 and the possibility to use CloudFront too.

And just to make this post a bit more cloud related and not just pitching my new site, a short story of what happened during the development at one point. As said, I had the Facebook Connect module as well as the Drupal for Facebook (yes, I ended up running Drupal as the CMS system) module installed but I had not enabled the Facebook Connect module since the Drupal for Facebook does essentially the same thing of connecting with your Facebook credentials. Or should do. I had and still have problems with the module as it forwards to a page which can’t be found but still after a few refreshes actually logs in. Anyway, I did go and enabled the Facebook Connect module while Drupal for Facebook had the same functionality enabled if another module would work a bit better.

Sure enough, after enabling the module I was watching a white browser screen with an Internal Server Error 500 with no access to the admin interface at all. What to do then? Should I mess with the database? Remove some modules and run update.php? Well, could not even access the update page. Luckily, I was running the site on an EBS based image! I had a week old (yeah, a bit old, but I did not mind) snapshot of the volume so all I had to do was to get the static files out from the bad volume, create a volume of the snapshot, shutdown the instance, detach the bad volume and attach the new volume. Boot up. Reboot had to be done too for some reason before I could see the log from AWS EC2 console. Reattach the elastic ip, copy the static files and I was back in business. Restore time below 10 minutes.

I love EC2.