Aspiring Evil Genius @ Work

Last week we saw one of the dozens of hotels that we support suddenly go 'red' in our monitoring system. It only took a few minutes to confirm that their Internet service provider had cut their service somewhere outside the hotel. A couple of phone calls got an explanation - there was a scheduled 'disconnect' order (not for non-payment), but it had been executed a week early. They turned service back on within a few hours and we notified the General Manager at that location of what had happened.

Though our customer made it clear to the carrier that they HAD NOT approved a disconnect and service should continue, the circuit went down a week later at noon on the original termination date. As of now, the hotel has had no Internet access for 48 hours and, this being a Friday we have to assume that if it isn't fixed in the next few hours there will be no service all weekend. And they are completely full this weekend.

No Internet at the Front Desk, no Internet for the Guests, no convenient temporary answer like Clearwire, nothing. There's nothing our customer can do and nothing we can do to speed up the carrier. In 48 hours they have not been able to undo the keystrokes that turned the service off. Brining in a new service provider will take at least a week.

And by now everyone is thinking through the monetary damage to the hotel's business - room credits, loss of future revenue and any future business that they cannot book right now because the Property Management System is cut off from the outside world.

And this is not some small-town ISP with duct tape and antennas running a wireless service - it's a T1 from a sizable carrier, which should be the single most reliable service you can get. Our customer is certainly not guilty of taking risks.

So what's the reasonable answer to avoid the possibility of a 3rd party having such a dire negative impact on your operations? We've installed a second ISP circuit in many of our hotels in the last two years. In most cases it has been to increase web browsing speed for the guests as more of them are on-line at peak times. But the secondary benefit is redundancy.

As long as you can add a second carrier that shares as few weak links as possible (don't get two of the same circuit and try to get one that does not use the phone company's wired connection into the building as T1s and DSL do), you can typically add a 2nd circuit for $50/month or less. If you never need, you've spent up to $600 per year for insurance. If you do need it, you may pay for a year of service in just one night.

If you have questions about the benefits and requirement for dual-ISPs, drop me a note at matt-at-tnoc-dot-us.