It doesnt happen often that i am some sort of a fanboy of some company or service, it is especially rare when it is a hosting company.
However for VPS.NET this happened to me 4 months ago when i started to move all my stuff to this provider and instead of using dedicated servers and co-location to XEN instances. In total i paid now over 1400 $ in 4 months to them, hell i even said on twitter that it was worth every cent... was it really?
I was suprised how well that works and how easy it was to scale installations, i also enjoyed it to setup nodes just to test some stuff instead of installing a new server somewhere, hell i even posted a "they are so cool try it out" post here in my blog. But mainly i moved because they promised me a self-healing and scalable infrastructure that even if something fails or burns down will stil work. SAN Storage instead of normal harddisk was the other movement key for me.
Til today i supported this company like some apple fanboys that even would buy steve jobs piss in yoghurt cups. But now they failed me miserable and im not sure if it was a intelligent move from me to get all my stuff to this provider.
But back to what happened...
Yesterday on 6th July 2010 around 3 or 4pm the London Cloud B and C went down. A power-outage killed all servers in the UK2 Datacenter.
Well thats no big trouble, i guess even if it should not happen (UPS, Redundant Power Supply and so on) if you had a dedicated server now or a co-lo you would have your servers back in notime and maybe if at all 30-60 minutes downtime.
But what happens if you run virtual servers like tons of other users in this datacenter and depend on SAN Storage exactly like VPS.NET ?
Well makes around 10-20 Hours downtime. That was the total time that my machines were down or unusable. And now over 24 hours later i stil have one machine in a unusable state and that one and my experience with the support is what makes me really angry and where i lost my fate in this company.
It is not even the fact that vps.net promises self-healing and stable storage facilitys and all this stuff that makes me this angry, it is that they tried to talk me into spending extra money to make up for there failures... but continue reading...
The story of the crashed disk
After most of my vps came back online in the night, one remained unusable. The filesystem was broken and it was read-only but running besides the fact that it claimed to have used 64 ZetaByte of storage ;)
I didnt think of a big deal here but i was unable to fsck the disk, and vps.net said on twitter people with this problem should open a ticket.
Well i did this, and i could bite me in the ass that i didnt download the data when the vps was read-only but stil running. Because now it was lost...
The Support guys later then told me that they tried to repair the disk and said that it was probably the "bootloader"
Hi
I ran a disk check on your VPS but it looks like the bootloader may be corrupt. I would suggest restoring from a backup.
No Problem i thought, that would mean the partitions may be intact and just need to reinstall grub or whatever. But what got my attention here was this part: "I would suggest restoring from a backup."
If you do not know vps.net, it is not like in many dedicated server environments where you get like X gb of ftp backup space that you have to pay for it per month. Pay for rsync backups, pay for disk images.
I have no problem with paying for backups, some of my vps have this feature enabled on vps.net , the really important ones. The Problem here is... the disk did not crash because it was my fault, the disk crashed because they miserable failed in there business.
I did not install backups of this server, whats running on that machine is in my source repository and can be restored, but what bothers me is that i need to reinstall all the stuff on the machine and im used todo highly customized installs from source instead of packages from the distribution. And they suggest me i should pay for backups because they are unstable... thats awesome and that was exactly the point where i lost my fate... I did go to vps.net because they make the impression on there website and from the architecture that a failure like this cannot happen.
Once i asked why storage is so expensive and there counter was because of the expensive SAN systems, obivously this expensive uber systems failed yesterday and are responsible that im left with a corrupt disk that i cannot repair myself.
I asked for a disk image and they offered me to transfer it to a server, i was happy and thought "whooho cool i can restore the fs myself and just reinstall the machine and copy back my stuff" but then suddenly another technican said thats not possible, even if the former one said he will upload that to my server...
But wait... thats not all
I gave now up on this and just decided to re-install the server and copy back my files from the source repository and to redo the work on some stuff that i was missing and that i hopped was in the disk image.
But no... i cannot reinstall the server, just nothing happens. Only ways todo this now? open a ticket and then maybe get this answer or do it right away: delete the vps, create a new vps.
Awesome but i cannot do that, that means i would lose the IP adress on this vps, but i do not want to change this again, means also extra work.
Conclusion?
I think i should now add 5 $ a month to all of my vps todo backups in the vps.net way so that i can restore them easily. But not to protect me from my own faults but to protect me from incompetent service at the hosting company itself. Because there is no such thing as self-healing and this SAN storage they utilize are far away from being save and stable. At least in this london clouds as it seems.
Trust? There is no such thing from me at this moment, they lost it terrible, VPS.NET have to regain that trust and need to make me a fanboy again. I dont care about free hostbill shit, or free ssl certs or whatever if they cannot provide a stable basic service that i expect.
Do it better...
The part is i think i was doing something better when moving to a cloud based hosting service for my machines.
I got my first back in this time very expensive Dedicated Server in the year 2000 when most users were still surfing with modems and ISDN in germany and 100mbit connection with 5gb traffic included and prices of up to 16 DM per additional GB traffic was just awesome and all you needed to serve then-thousands of this users.
This evolved later to more and more dedicated machines, then to cheap tower housing up to my own cages of expensive hardware in datacenters in frankfurt also including virtual enviroments based on xen and vmware.
For my new company i wanted to just didnt want to do that again myself and just off-load this to vps.net, instead of getting some nice reward i was punched in my own face by nothing else then myself for doing this adventure and faced a major downtime of my company store and demonstration websites for products that we sell.
I stil belive that nick, ditlev and all the other involed guys at uk2 do a great job, but i also hope they understand that they now have to regain trust to many customers.
I will stay at vps.net but i do not think i will grow my environment there as i planned todo and just go back to good old colocation or dedicated servers for larger projects, i can buy that at uk2 too, and i will. But that cloud experiment failed. Not because the cloud failed, because my disk in the cloud failed and i had much work todo to restore because i refused to stuck more money in the cloud to protect me from itself.
Thanks for reading and sorry for the spelling mistakes.
