Tuesday, July 08, 2008

Dead lastminute.com - the full(ish) story

So last Friday we revealed how lastminute.com and a string of sister sites had seemingly disappeared off the face of the earth.

It actually turned out to be a pretty serious problem, as our story today indicates.

The sites - including Medhotels, Holiday Autos and Travelocity.co.uk, as well as the Tesco.com white label - remained offline until 2am on Saturday morning, over ten hours after the "power outage" at IT supplier Colt took place.

We have only limited information about the circumstances surrounding the problem but questions will certainly be asked of Colt, no doubt about it.

Some might translate "we're working with our provider Colt to ensure this doesn't happen again", to actually be code for "we're going tear strips off these guys after our sites went dark for over 10 hours on one of the busiest days of the year!" - but probably not.

Anyway, it's a tricky area to analyse as it is completely impossible to monitor every travel site on the web to gauge how often something like this happens.

It was only good fortune that we saw the dead LM last Friday afternoon and stuck around until the wee hours of the following morning to see when it returned. We actually gave up at 1am and went to bed!

So, how common is the problem? What do travel companies do in a situation like this? Apart from panic.

Kevin May, editor, Travolution


Darren Cronian said...

I know when the Travel rants blog goes out of action as I receive an email telling me so – it’s a cheap service which checks the server, and pings it regularly to make sure it’s alive. I am sure people behind Lastminute would have something much more technology advanced.

I’ve heard of services which sends text messages as well..

I am surprised to hear that the sites were down for up to 10 hours, that’s not good for a company that large.

How often this type of thing happens depends on if the site is hosted on a shared or dedicated server, a company the size of Lastminute will most certainly have their own dedicated servers, so there’s less chance of this happening, regularly. If like me your on a shared service with hundreds of other website, then the site can down more often.

It depends on the hosting company in this situation.

Matt said...

It's amazing how often these things happen at the weekend. And in truth if you are hosted outside of your own four walls there is very little you can do except wait for the provider to sort it out - what sort of contingency plan is there for sites of this size and with this much content? None I would have thought. Incredibly frustrating.

Darren Cronian said...

Matt, a contingency plan should be in place.

I am no server expert, but I have heard of some companies who host their site on two servers, one goes down, they flick the switch make changes to the DNS settings, which are quickly picked up across the internet. Whilst it’s not perfect and you’d only do it if the server is going to be down a while, but it means that there’s still a large number of visitors who can access the site.

Richard Hartigan said...

Darren, that is correct. I am surprised sites of this size don't have something like this already in place. Regardless of server numbers, there will always be a single point of failure and a contingency plan should exist.

I have previously written about a simple three point strategy that online business should look to adopt.

Darren Cronian said...

Some great points in that post Richard, especially about PPC revenue which it could have cost them.

I love the MySpace idea too.

I always say 'I am no expert on..' but I always seem to shock myself that I know more than I think I do!

Mark said...

No a lot you can do if the power goes no matter how many backup servers you have.

If the power goes you need a full disaster recovery plan to switch into place.

This would only work if they have a second site constantly updating/running, ready to be swapped over at the flick of a switch.

Costly to setup and maintain but maybe even more so if you don't

david havard said...

Given the size of these players you would expect that full disaster recovery to be in place. But before that I would be asking what happened to the hosts backup generators - they should be standard fare for any modern web host?