Sucks…

I have not been doing much of anything cloud related as I also moved to a new job from HP to Fortum and I guess somewhat also because I’m getting fed up with the hype. In my mind, cloud is just a way of sourcing the necessary resources, albeit having some characteristics not found elsewhere.

But, I’ll continue with clouds in fall when I hopefully have some more time.

Have a nice summer!

So, as I could not take a snapshot and was not particulary interested in using a few days old snapshots, I decided to just shut down the instance and give it the time it needs. Eventually, the server went down and I could restart it just fine. Situation back to normal. This incident could of course have been avoided easily by having a backup system ready or even a load balanced setup if I would have the money to run it.

No luck in getting traction.

Devops is a concatenation of development and operations and essentially is the same what I was looking for in my thesis. We did even try the approach in practice and it did work really well! using Agile methods brought the necessary structure in IT projects. In future, this type of new way of doing IT is probably even more valuable, as customers of IT don’t anymore allow a server deployment to take weeks or even months or ever lasting IT projects without clear visibility.

Besides using Agile methods, what else must the IT do in order to fullfil the needs of a modern enterprise? I have already talked about Puppet which is an open source server management automation tool. Datacenter automation is bound to make a big impact in allowing datacenters (private cloud?) to compete with public clouds in ease and agility of deployment. The role of a sysadmin and the developer could start to blur ans sysadmin has to know about how to deploy servers using a scripting language. Interesting times!

But I also have some new cloud related things to tell you about. Since the blog isn’t exactly driving traffic too much and I had some free CPU resources, I started a new project, vKaiser.com, which is a more Web 2.0 oriented site. Well, an imitation of YouTube but with heavy connections to social media sites like Facebook and Twitter. The site is by no means ready, but you are welcome to check it out – with Firefox. IE7 is ok too if you are not on compatibility mode. Interesting things to mention is the storage of the videos and thumbnails in S3 and the possibility to use CloudFront too.

And just to make this post a bit more cloud related and not just pitching my new site, a short story of what happened during the development at one point. As said, I had the Facebook Connect module as well as the Drupal for Facebook (yes, I ended up running Drupal as the CMS system) module installed but I had not enabled the Facebook Connect module since the Drupal for Facebook does essentially the same thing of connecting with your Facebook credentials. Or should do. I had and still have problems with the module as it forwards to a page which can’t be found but still after a few refreshes actually logs in. Anyway, I did go and enabled the Facebook Connect module while Drupal for Facebook had the same functionality enabled if another module would work a bit better.

Sure enough, after enabling the module I was watching a white browser screen with an Internal Server Error 500 with no access to the admin interface at all. What to do then? Should I mess with the database? Remove some modules and run update.php? Well, could not even access the update page. Luckily, I was running the site on an EBS based image! I had a week old (yeah, a bit old, but I did not mind) snapshot of the volume so all I had to do was to get the static files out from the bad volume, create a volume of the snapshot, shutdown the instance, detach the bad volume and attach the new volume. Boot up. Reboot had to be done too for some reason before I could see the log from AWS EC2 console. Reattach the elastic ip, copy the static files and I was back in business. Restore time below 10 minutes.

I love EC2.

Guest speaker Mr. Martin Buhr from Amazon gave a presentation about Amazon Web Services with some very interesting figures about the exponential growth of AWS. The second guest speaker was a former fellow of mine from F-Secure Corporation, Mr. Pirkka Palomäki, who gave a presentation about the cloud strategy F-Secure Corporation is implementing. It is clearly visible that without F-Secure’s cloud strategy, the rapid detection of new threats would be nearly impossible.

I wish to thank Codento Oy and Mr. Martin Buhr for inviting me to join this event. It was great! Especially great was Codento’s innovative version of an EBS, the Elastic Beer Storage :)

There are vague definitions of what is a cloud service and one of the prerequisites was that you could buy the service with your credit card and to pay only for the resources you use. Amazon DevPay allows developers to sell their AMIs (with application installed), all you need is a business in the US.

There are two ways how you can use DevPay: through AMIs you have built with the service installed and also by selling an application which uses S3 as the storage location. It’s a really good start, but still lacks a few things, like if you actually would like to provide high availability for the client who bought your AMI, the client will have to roll their own solution to achieve this. I bet in principle not many clients are willing to do that and would just like the application to be available.

When I think about this dilemma, a solution might be to have some kind of a root AMI which a customer would buy (and pay by the hour like crazy). This would then take care of the availability of the service by seeding new servers which are members of the application through some very wicked autoconfiguration. Actually, Elastra does almost this as their product allows the user to define architectures and then deploy them to Amazon. In principle, it would be possible to have an Elastra AMI with some configuration inside which the client could then deploy, but it does sound like a hack and not really something you could sell as a product. By the way, the Elastra’s product looks great for defining and deploying architectures at least internally within an organization.

The seeding of an application from a root AMI might make clients able to buy a redundant application and not pieces of it. So far, though, this is just a beautiful dream since the DevPay is not available in Europe so it isn’t possible to start with even the simplest model of serving an AMI of your superduperapp for the public. That is, if you don’t implement billing yourself. If I’m not totally wrong, the billing API is not public which makes rolling out your own solution impossible (you can’t either put limits based on usage charges in your account due to this reason).

I would be willing to adjust the definition of a cloud service regarding the pay by the hour and order by credit card before DevPay is available in Europe.

Virtualization has made an impact in IT and though the ROI figures have not always been too good, virtualization (the servers at least) has become rather mainstream. Why is it so? Could it be that the reason for it was the relative ease of doing a P2V migration? You know, doing the migration on VMware by using the VMware converter is quite straightforward. We have not seen a tool which would make a similar thing with for example Amazon AWS. It just isn’t easy to migrate servers in cloud environments as the application architecture will most likely need some modifications. AWS is really great for greenfield type of application deployment but hardly ready for automated migration of existing apps.

I guess the rapid success of the cloud boils down to a few things. First, there has to be a way of migrating existing applications with ease into the cloud. With AWS, this would probably mean that the life of an instance would be quaranteed, that is, there would be a similar thing as VMware vMotion (migrates virtual machines if the underlying ESX host dies) in AWS. Though, I don’t like how this would enable the building of badly deployed applications once again and make AWS consults job opportunities kinda scarce, maybe. My personal view is that you probably can first migrate the stuff by machines to IaaS providers and maybe later by applications, to for example in Microsoft Azure and then scrap the OS altogether. How I would like that!

I would not expect too much of actual savings when running the applications on IaaS provider, actually. I feel a bit sorry for saying this, though. I hope to be wrong. First of, the costs of the hardware are really nothing if it is managed by tiers of engineers during the application life time and you will need those people in cloud too. Sure, you get some possibility of elasticity, but actually, with legacy apps the elasticity is just like with VMware – vertical. I feel the real benefits of cloud are within the PaaS where the OS is abstracted out from the client. Maybe 2011 or 2012 will then be the real year of the cloud.

Today, Logica announced their new cloud service (pilvipalvelu in Finnish) which they call as Logica Live Connections. It is based on IBM’s Lotus Connections product family. Something just does not sound right. What exactly makes this a cloud offering? It is said to be billed in user/month fashion which, I guess, should make it a cloud service. I would like to see a bit more than that as with the given information, it resembles more of a SaaS offering. And where can I sign up with my credit card? Either it’s not possible or then it is really well hidden. Anyhow, this contradicts the idea of a cloud service being available with only a sign up. I bet you will need a consultant to start using the service too. I really wonder why it has been branded as a cloud service… is it a valid selling point nowadays? A marketing term and nothing else? Shouldn’t you really concentrate on the good aspects of a cloud service like the elasticity and on-demand possibility of resource allocation if it really is a cloud offering?

There has been a few other cases as well, like Cloud Computing Oy sales director’s pitch about being able to point out where the actual cloud server physically sits. I’m sorry for flaming, but oh man, how wrong can it go! Using terms in a wrong way is harmful for the entire business. Well, they are a startup (I believe) and the business is still in its infancy, but I feel sad (like a panda) if there isn’t anything better going on. The bigger consultancy firms in Finland have layed low with their cloud offerings though HP has shown some muscle with the late HP and Microsoft agreement about delivering infrastructure to application which sounds really promising. One other (smaller) firm I have seen active in the cloud field is Codento Oy which has held a few presentations about the subject and seem to really understand the concept. Good stuff.

But. We Finns could be so much more! We have the correct type of environment to build great data centers as it is relatively cold during most of the year, the electricity is not too expensive, we have the skills, we have the political stability, we don’t even have earth quakes or hurricanes. Google has spotted this and is actually ramping up a new data center in Hamina, Finland. Sounds good, but this is only a data center which needs just a few (50) persons replacing faulty servers. How about a real cloud service center? A one where you get computing power, platform or an application by just browsing a catalog and deploying?

I remember wondering what that “busy” meant at that time. Now I know.

It should go without saying that this status information of “busy” is really, really uninformative. How about “detaching” instead when a user wants to detach a volume? And why did it take about one and a half weeks to detach? Is there a log somewhere I really did detach a volume? The lesson learned from this incident is to act if your EBS volume goes to “busy” state. All might work fine for a while but be warned, it will detach at some point. Also, it would be really nice if there would be some abstraction layer in between the real names of the volumes and instances and the ones available to customers. With this layer a user could add more descriptive names to instances or what ever objects there are. Or then really start using different accounts for development and production stuff… Really.

Having a background with enterprise IT, I know something (but not much) about deploying applications. Many of them are multi-tiered, the usual being front-end application servers with back-end database servers. Some may have a load balancer in front of the application servers. Thus, there are a few different roles a server must fulfill to deliver the service to a customer using the application these servers in whole produce.

How do you set up an application? It depends about the requirements of course, but basically there are some usual things, at least if you narrow the selectable services. VMware got an idea to bundle a few virtual machines into a vApp which can then be deployed. I have not personally used those, but they seem like an interesting concept. They do work in a bit different way than how an AWS instances would work since with VMware you have the luxury of for example vMotion taking care of VM migration in case the host dies… Giving there is a VMware HA cluster in place. Hiding the complexity this way sounds fantastic! I love it. I do want the same in Amazon AWS!

There has to be a way to group things in Amazon AWS. There are a few tools such as RightScript and Puppet which provide a way to move from a specific AMIs to specific scripts which produce a certain kind of servers. Using these scripts it should be possible to deploy a full application with various components. Puppet also makes it possible to update a class (with Puppet the servers may belong to a class, maybe clasess) of servers to have for example the latest resolv.conf file. Sounds nice! Essentially, this sounds a lot like Microsoft SCCM but for UNIX like operating systems.

How about a future where a business owner could just browse to an IT webstore, select a CMS installation with a rough estimate of usage and the system would just produce it by running the scripts in the backgroud? Or upgrade an existing system? How many IT admins would lose their jobs?