billso.com

One of the country’s largest web hosting companies, The Planet, is still down after a fire knocked out their main data center’s transformer. To their credit, Planet managers have been providing regular updates on a web forum. Here’s update #2, which is a good example:

Today at approximately 5:45 p.m. [CDT], a transformer in our H1 data center in Houston caught fire, thus requiring us to take down all generators as instructed by the fire department. All servers are down.

The Houston Fire Department ordered Planet staff to shut down the data center’s electric generators. Approximately 7500 9000 web servers and 9000 7500 customers are affected.

One reason that I use DreamHost.com to host billso.com is the incredible level of transparency that DreamHost provides. DreamHost is employee-owned, which helps explains their reporting policy. I’ve never had a major problem with DreamHost, but I know that I can check the status reports for the data center and most of their servers at any time, on the Web or with RSS

See Center Networks for more comments about the Planet fire. I agree with Allen Stern - given the number of servers and clients affected, I expected to hear much more on the blogosphere this morning. The outage affects Planet’s Server Command, ResellOne, and legacy EV1 customers.

The Hosting News posted an excellent article on 29 May 2008 about several recently completed projects at The Planet. It’s tempting to think that this project and the fire are connected, but there’s no indication of that yet. The Planet used to be known as EV1. I remember EV1 from from my years in Austin, when that company offered cut-rate dial-up internet service and web hosting. Their radio commercials were just awful.

The Register and Broadband Reports have posted very brief reports, and here’s the Wikipedia page for EV1.

HPU has rolled out a campus emergency notification system that will notify students and employees by email and text message. The system was announced on Pipeline on 25 September. I discussed similar systems on 17 April, and USA Today discussed Rave in an article on 25 October 2006.

HPU students and staff can enroll online with Rave at https://www.getrave.com/login/hpu/

The service is free, and the enrollment process requires a Pipeline ID and password.

I did notice that the telephone number fields will not support international calling codes. Most students on Oahu would use a North American phone number, but there might be distance education students who would want emergency notification messages. The Rave system could be used to announce unscheduled WebCT and Pipeline outages.

Subscribers can get their Rave notifications by email, which is a good alternative for users of push email systems like the iPhone, Blackberrys, Sidekicks and some Windows Mobile devices.

Also, the enrollment form appears to validate entries as they are typed. Many web forms include programming code to catch data entry errors in the web browser, before the submit button is pressed. Client-side validation can reduce network traffic and server load, if the validation scripts are well-written and carefully tested.

To test the form, I entered some incorrect data. Then I tried to edit the data. I had to click and hold with my trackpad while retyping the numeric fields.

It’s not the most obvious solution – I tried deleting and backspacing over the bad data, but the cursor kept jumping to another field. The validation script assumed that I’d entered the data correctly the first time.

The Rave system does send a confirmation message to the mobile phone number that was entered on the enrollment screen. The user get on a web browser and enter a 4-digit code from the text message to activate their account.

The email side of the system does not send an activation or conformation code, which I found curious. I entered my Gmail email address.

There’s been a surge in notification system deployments since April, as universities and colleges look for better ways to notify stakeholders of important events.

Some universities have found these systems useful when the weather gets bad. At other schools, these systems are also used to notify students when classes are canceled or events are rescheduled.

An emergency notification system is also an important part of disaster recovery planning (DRP) for any large organization. CIOs and their staff must have a reasonable, well-tested action plan to resolve issues that threaten the organization’s employees, staff and information systems. Rave’s system can improve a DRP implementation by asking people to staff off-campus during an emergency.

Of course, the Rave system works best when many users are registered with their correct information. Students tend to change email addresses and phone numbers often, and there’s nothing that compels students to use the system.

Another issue involves marketing. An emergency notification system should not be used to send advertising and marketing messages to users. There are some services that allow these messages. However, the last thing I expect from an emergency notification service is spam and ads.

From the Honolulu Advertiser and the Honolulu Star-Bulletin: We did have a tsunami yesterday. City officials asked people to stay out of the water at the beach parks, but the siren system was not activated.

The effects were dramatic, especially in Haunama Bay, but there was very little damage.

Still, it was the largest tsunami in 40 years, and one wonders if any local businesses set a disaster recovery plan (DRP) in motion. The wave was generated by an 8+ magnitude eartquake in Japan, as shown in this graphic from the Advertiser.

This article first appeared on my old blog at http://www.bloglines.com/blog/wsodeman?id=204

http://www.honoluluadvertiser.com/apps/pbcs.dll/article?AID=/20061019/NEWS01/610190341

In this morning’s Honolulu Advertiser, Rick Daysog reports that HECO could have added redundant systems to prevent a blackout, but the project might cost US$1B in current dollars.

That works out to US$3400 per electric customer, using an estimate of 291,000 residential and commercial electrical customers on Oahu. The number of customers is less than the number of people on the island, as there are many households with more than a single resident. A household usually counts as a single customer as far as the electric company is concerned.

Oahu residents are asking questions that Californians have asked over and over after much larger earthquakes. In information systems, vendors, departments and managers sometimes create service level agreements that define service standards, service availability, the kinds of interruptions and problems that can occur, and specify procedures to escalate and remediate the problems when they do happen.

SLAs also define the responsibilities of the multiple parties involved in the agreement. If management decides to add more desktop computers, or implement additional server-based applications, then the agreement should be modified to account for the additional resources and time required.

SLAs often accompany disaster readiness plans (DRPs) and contingency plans, as companies become more reliant on telecommunications and information systems to conduct their most basic operations.

Sunday provided several examples of contingency planning. Some Oahu supermarkets had backup generators ready to power the cash registers and credit card readers. In an emergency, it’s difficult to run out to the bank and get more cash. Hospitals, police and fire departments called in extra staff to assist. The state considered calling out the Hawaii National Guard, which was actually on duty last weekend, but decided against that move.