Summer is almost here and it was really the time to clean up my blog a bit. Now there is a much more readable theme and all the CityDeal hamburger campaigns are gone. Hopefully nobody will be missing those?

I’ve been writing already about deploying applications instead of virtual machines and how I would like to move away from the machine instance and to a some kind of a service where you could select a system and the service would deploy and manage the needed computing resources. It now seems like this is starting to become a reality as there are providers like DotCloud and PHP Fog who take care of the infra while the customer just deploys the code.

I requested a beta invite to DotCloud which in contrary to PHP Fog is somewhat more multipurpose with other services than the PHP only. Actually, currently they have java, mysql, nodejs, perl, php, php-worker, postgresql, python, python-worker, rabbitmq, redis, ruby, ruby-worker, smtp, static as the deployment options. They also have some nice tutorials about how to use the system. The system is operated mostly locally from CLI and the code is pushed to dotcloud which seems like a nice way to do it as it is analogous to for example editing files locally and then using FTP to transfer the files to a web server.

As a test run, I went with installing WordPress as there is a good tutorial for that and only a few minutes later I had a running blog in DotCloud. I’m really liking this!

I’ve been working with Aegir recently to get some ideas how to host multiple Drupal sites in a somewhat automated and scalable way. To my surprise the performance was actually a lot better with the t1.micro instance which runs the Aegir platform than what I get from the small instance. A video below has a demo when running the identical sites on both servers:

The pageload was around 2.6 seconds with the standard Drupal installation and around a second faster with the Aegir platform. The site on Aegir was dump from the other site taken a few days ago with no significant differences. Have to say I really really like Aegir!

These are more like notes for myself, but I am happy to share if someone else would be interested. No warranties though, if you can’t get Aegir running with these notes :)

Get the packages:

yum install httpd php php-mysql php-gd mysql-server postfix sudo rsync git-core unzip alpine screen

Set the PHP.ini memory limits, 192M works ok

Fix your DNS and how your host resolves:

edit /etc/hosts

x.x.x.x FQDN hostname

hostname FQDN

Install MTA (I run Postfix, so just copypaste from other server)

upload posfix configuration to /etc/postfix

Edit the /etc/postfix/canonical

This will rewrite the name which Drupal uses to send emails to what you need

create canonical db

postmap /etc/postfix/canonical
service postfix start

Create a group for the Aegir user and which Apache uses as well:

groupadd www-data

edit /etc/httpd/conf/httpd.conf to run as www-data

The Aegir user home folder needs to have 755 as the permissions (at least this works)

adduser -r –groups www-data –home-dir /var/home/aegiruser aegiruser

Edit the httpd.conf to have the Aegir confs included

Include /var/home/providus/config/apache.conf

Edit sudoers file

aegiruser ALL=NOPASSWD: /usr/sbin/apachectl

start mysqld

/usr/bin/mysqladmin -u root password ‘password’

wget -O /tmp/install.sh ‘http://git.aegirproject.org/?p=provision.git;a=blob_plain;f=install.sh.txt;hb=provision-0.4-beta2′

./drush/drush.php dl drupal

Well hello there! It’s been a while, but I finally found some time to work with the sites and the latest of Amazon Web Services. Lately, AWS has introduced the tiny micro instances with a tempting price tag for small businesses with not too much of a need for high performance. For me, those do sound fantastic for testing purposes as I have been wanting to try running the two sites, this and vkaiser.com on a bit more robust architecture than the current one with just an EBS based AMI and the videos in S3.

I run the typical LAMP stack on one AMI, thus the idea was first to boot up one micro instance and have a look. Well, I chose to go with some old image I had created way long time ago. It also had a LAMP stack installed, but of course it was kind of outdated and the vkaiser.com did not look too good (well, does it now either…), so I figured I could rsync the html folder of the Drupal installation and I did eventually get the rsync with public key working. Then I realized that the db wasn’t really up to date either and the drupal modules would not of course work, so how about connecting to the database on the current “production” which would kind of resemble the hopefully future setup too as running a separate db server (and slave) would just be the way to go at least with Drupal.

Settings in Drupal for remote database connections are really simple. First edit the MySQL configuration (/etc/my.cnf) to have

Bind-address=database_ip

And if you have skip networking defined, comment that out.

Then add remote access permissions to the database for a db user

GRANT ALL ON *.* TO ‘dbuser’@'remote_ip’ IDENTIFIED BY ‘password’;

And modify the settings.php on the remote application server to point to the database server:

$db_url = ‘mysql://dbuser:password@database_ip/database_name’;

Then you can test the connection to the database. At least I got that working, though I was first editing the wrong settings.php file which of course did not prove to be very useful in getting the db connection working.

The real deal was though to see how the t1.micro performs under stress. I browsed a while for some tools with how to do the test, but then I found loadimpact.com which simulates really well concurrent users from 0-50 for free! With some euros, you can get up to 5000 users and customized tests and what not. I like the service, though it went down just as I got my t1.micro tested. The average response time was around 1.5 seconds for the vkaiser.com frontpage and it did not show any real implications of getting slower, thus I should put more load on the micro if I coughed up some cash. I next went on and tested my good old small instance and got about the same results

This wasn’t too scientific, though the results are encouraging. I mean, 50 concurrent users is about 49 more than this site usually has and the micro worked well, so I am planning to make a switch soon… More about that later!

It’s been a hectic past month to say the least! First of all, I’ve been a proud father for about a month now and everything is fine on that front. Then again, we had a water damage (a forty year old weld let go) in our place and are currently trying to get permits from the city officials if we could actually fix and renovate the appartment a bit. Have a look here of what it looks like:

Sucks…

I have not been doing much of anything cloud related as I also moved to a new job from HP to Fortum and I guess somewhat also because I’m getting fed up with the hype. In my mind, cloud is just a way of sourcing the necessary resources, albeit having some characteristics not found elsewhere.

But, I’ll continue with clouds in fall when I hopefully have some more time.

Have a nice summer!

So you have your great new application utilizing all the awesomeness of Amazon AWS? How you gonna sell it?

There are vague definitions of what is a cloud service and one of the prerequisites was that you could buy the service with your credit card and to pay only for the resources you use. Amazon DevPay allows developers to sell their AMIs (with application installed), all you need is a business in the US.

There are two ways how you can use DevPay: through AMIs you have built with the service installed and also by selling an application which uses S3 as the storage location. It’s a really good start, but still lacks a few things, like if you actually would like to provide high availability for the client who bought your AMI, the client will have to roll their own solution to achieve this. I bet in principle not many clients are willing to do that and would just like the application to be available.

When I think about this dilemma, a solution might be to have some kind of a root AMI which a customer would buy (and pay by the hour like crazy). This would then take care of the availability of the service by seeding new servers which are members of the application through some very wicked autoconfiguration. Actually, Elastra does almost this as their product allows the user to define architectures and then deploy them to Amazon. In principle, it would be possible to have an Elastra AMI with some configuration inside which the client could then deploy, but it does sound like a hack and not really something you could sell as a product. By the way, the Elastra’s product looks great for defining and deploying architectures at least internally within an organization.

The seeding of an application from a root AMI might make clients able to buy a redundant application and not pieces of it. So far, though, this is just a beautiful dream since the DevPay is not available in Europe so it isn’t possible to start with even the simplest model of serving an AMI of your superduperapp for the public. That is, if you don’t implement billing yourself. If I’m not totally wrong, the billing API is not public which makes rolling out your own solution impossible (you can’t either put limits based on usage charges in your account due to this reason).

I would be willing to adjust the definition of a cloud service regarding the pay by the hour and order by credit card before DevPay is available in Europe.

It is 2010 now. This should be the year when cloud computing starts to live up to its expectations and to the hype generated all last year. What are the obstacles cloud vendors might face and will this year really be the year of Cloud?

Virtualization has made an impact in IT and though the ROI figures have not always been too good, virtualization (the servers at least) has become rather mainstream. Why is it so? Could it be that the reason for it was the relative ease of doing a P2V migration? You know, doing the migration on VMware by using the VMware converter is quite straightforward. We have not seen a tool which would make a similar thing with for example Amazon AWS. It just isn’t easy to migrate servers in cloud environments as the application architecture will most likely need some modifications. AWS is really great for greenfield type of application deployment but hardly ready for automated migration of existing apps.

I guess the rapid success of the cloud boils down to a few things. First, there has to be a way of migrating existing applications with ease into the cloud. With AWS, this would probably mean that the life of an instance would be quaranteed, that is, there would be a similar thing as VMware vMotion (migrates virtual machines if the underlying ESX host dies) in AWS. Though, I don’t like how this would enable the building of badly deployed applications once again and make AWS consults job opportunities kinda scarce, maybe. My personal view is that you probably can first migrate the stuff by machines to IaaS providers and maybe later by applications, to for example in Microsoft Azure and then scrap the OS altogether. How I would like that!

I would not expect too much of actual savings when running the applications on IaaS provider, actually. I feel a bit sorry for saying this, though. I hope to be wrong. First of, the costs of the hardware are really nothing if it is managed by tiers of engineers during the application life time and you will need those people in cloud too. Sure, you get some possibility of elasticity, but actually, with legacy apps the elasticity is just like with VMware – vertical. I feel the real benefits of cloud are within the PaaS where the OS is abstracted out from the client. Maybe 2011 or 2012 will then be the real year of the cloud.

I’ve been keeping a close eye on what’s happening in the cloud computing scene in Finland ever since I jumped on the band wagon this summer. If there were some misunderstandings of whether the Sidekick (you remember, last fall the service went tits up and customers could not access their data) was a cloud offering, currently there is a big trend of calling a lot of things a cloud service while in reality they might not be.

Today, Logica announced their new cloud service (pilvipalvelu in Finnish) which they call as Logica Live Connections. It is based on IBM’s Lotus Connections product family. Something just does not sound right. What exactly makes this a cloud offering? It is said to be billed in user/month fashion which, I guess, should make it a cloud service. I would like to see a bit more than that as with the given information, it resembles more of a SaaS offering. And where can I sign up with my credit card? Either it’s not possible or then it is really well hidden. Anyhow, this contradicts the idea of a cloud service being available with only a sign up. I bet you will need a consultant to start using the service too. I really wonder why it has been branded as a cloud service… is it a valid selling point nowadays? A marketing term and nothing else? Shouldn’t you really concentrate on the good aspects of a cloud service like the elasticity and on-demand possibility of resource allocation if it really is a cloud offering?

There has been a few other cases as well, like Cloud Computing Oy sales director’s pitch about being able to point out where the actual cloud server physically sits. I’m sorry for flaming, but oh man, how wrong can it go! Using terms in a wrong way is harmful for the entire business. Well, they are a startup (I believe) and the business is still in its infancy, but I feel sad (like a panda) if there isn’t anything better going on. The bigger consultancy firms in Finland have layed low with their cloud offerings though HP has shown some muscle with the late HP and Microsoft agreement about delivering infrastructure to application which sounds really promising. One other (smaller) firm I have seen active in the cloud field is Codento Oy which has held a few presentations about the subject and seem to really understand the concept. Good stuff.

But. We Finns could be so much more! We have the correct type of environment to build great data centers as it is relatively cold during most of the year, the electricity is not too expensive, we have the skills, we have the political stability, we don’t even have earth quakes or hurricanes. Google has spotted this and is actually ramping up a new data center in Hamina, Finland. Sounds good, but this is only a data center which needs just a few (50) persons replacing faulty servers. How about a real cloud service center? A one where you get computing power, platform or an application by just browsing a catalog and deploying?

I’ve been thinking the relationship between operating systems and the reason why they exist – the applications. How does PaaS fit into the future of computing or is IaaS just a stepping stone to a world without the traditional one server, one application approach?

Having a background with enterprise IT, I know something (but not much) about deploying applications. Many of them are multi-tiered, the usual being front-end application servers with back-end database servers. Some may have a load balancer in front of the application servers. Thus, there are a few different roles a server must fulfill to deliver the service to a customer using the application these servers in whole produce.

How do you set up an application? It depends about the requirements of course, but basically there are some usual things, at least if you narrow the selectable services. VMware got an idea to bundle a few virtual machines into a vApp which can then be deployed. I have not personally used those, but they seem like an interesting concept. They do work in a bit different way than how an AWS instances would work since with VMware you have the luxury of for example vMotion taking care of VM migration in case the host dies… Giving there is a VMware HA cluster in place. Hiding the complexity this way sounds fantastic! I love it. I do want the same in Amazon AWS!

There has to be a way to group things in Amazon AWS. There are a few tools such as RightScript and Puppet which provide a way to move from a specific AMIs to specific scripts which produce a certain kind of servers. Using these scripts it should be possible to deploy a full application with various components. Puppet also makes it possible to update a class (with Puppet the servers may belong to a class, maybe clasess) of servers to have for example the latest resolv.conf file. Sounds nice! Essentially, this sounds a lot like Microsoft SCCM but for UNIX like operating systems.

How about a future where a business owner could just browse to an IT webstore, select a CMS installation with a rough estimate of usage and the system would just produce it by running the scripts in the backgroud? Or upgrade an existing system? How many IT admins would lose their jobs?

Amazon has announced a new feature of booting instances from EBS volume. This feature changes radically the way how AWS instances can be preserved if compared to the traditional volume bundling and uploading to S3.

Though this all sounds nice, it isn’t really too easy to convert existing instaces to boot from EBS. All previous instances boot from the local instance disk. Amazon AWS management console indicates the location where the instance boots with the Root Device Type column. Previous instances have the root device type as instance-store while EBS images have the type as ebs.

To get started with the EBS images, there are a few images from Amazon which are useful as a base image. It was really easy to just boot one of them and mount one EBS volume which contained a snapshot of the database and the www root. Installing basic LAMP stuff, changes in httpd.conf and my.cnf to point in the EBS volume and the AWS instance which boots from EBS was ready. I could now create snapshots of the system in minutes and also shut down the system when I don’t need it and thus not get billed for the instance. Awesome! The snapshot also had the EBS volume snapshotted which was mounted to the instance.

The EBS image feature is likely to open a wide range of new applications and really change the way how an elastic service is been constructed. Basically, a member of a pool of web servers can now be created in advance and just turned on when there is a demand to use it. Of course, it first must update itself to be on par with the other pool members.

I am not really sure if it was my old lap top which I used to work with the EBS images or what, but the AWS management console was painfully slow in responding, especially when using Firefox. And when using IE, I did not get anything else in the pop-up window than the button to create the snapshot:

Firefox, though really slow in responding, gave the option of typing the name in the required field. Also, if you create EBS image and then decide to get rid of the EBS image, you have to delete the AMI first, otherwise the management console will complain that it’s in use.

I have yet to decide should I go with the instance-store or EBS with my instance. It will add something to my costs of running my site in AWS, but that shouldn’t be too much. I find a lot more benefits with EBS than running in instance-store, but then again I fear getting lazy in responding to possible threats of instances going down and disaster recovery.

Pauli Haikonen