Scott McPherson's Web Presence

Computerworld magazine does a fine job of keeping pandemic preparedness on the minds of Chief Information Officers (that's Head Geek of corporate and government IT-dom), as well as decision-makers and IT personnel.

Anyway, in their latest issue appears this gem of a story:

Eight-day IT outage would cripple most companies

Gartner survey finds business continuity plans lack ability to withstand longer outages

January 10, 2008 (Computerworld) -- A Gartner Inc. poll of information security and risk management professionals released today shows that most business continuity plans could not withstand a regional disaster because they are built to overcome severe outages lasting only up to seven days.

Gartner analyst Roberta Witty said that the results of the poll show that organizations must "mature" their business continuity and disaster recovery strategies to enable IT operations and staffers to endure outages of at least 30 days. Such efforts would require additional IT budget spending and collaboration across enterprise business units at most corporations, she noted.

Gartner surveyed 359 IT professionals from the U.S., U.K. and Canada during 2007 on their business continuity efforts, and nearly 60% said that their business continuity plans are limited to outages of seven days or less.

Further, results showed most companies focus on rebounding from internal IT disruptions, not from regional disasters that could also damage facilities. A very shortsighted tactic, remarked Witty, considering damage caused by Hurricane Katrina in 2005, as well as potential harm from outages, terrorist attacks, pandemics, service provider outages, civil unrest or other unpredictable event.

"If you start looking at some of the events we've [experienced] over the last few years, companies must plan for events that actually take much longer to recover from," Witty said. "This is an issue [businesses] have to deal with -- it's in front of everyone's face right now." 

The survey found that 77% of companies have come up with a business continuity plan covering power outages caused by fire, while 72% have a plan to get up and running after a natural disaster. Only 50% of companies are prepared to rebound from terrorism-related IT outages.

Witty did say that companies are starting to take pandemic concerns more seriously than in the past. The survey showed that 29% of organizations now have pandemic recovery measures in place, up from just 8% in 2005.

To withstand an outage of up to 30 days, companies must improve cross-training efforts and streamline emergency management, notification and incident management techniques for quicker response, she added. "That's what [business continuity] is about. If you don't have people to manage it, a data center is useless," Witty remarked.

I lectured on pandemic preparedness at Gartner's international conference in Orlando in 2006, and I know Ken McGee of Gartner, who (along with yours truly, of course) is one of the few recognized bona fide IT pandemic experts on the planet.  So Gartner is extremely well-focused on this topic.  Their research on this, and other topics, is first-rate.  It's "take it to the bank"-type material.

So when Gartner says sixty percent of American corporate and government organizations cannot sustain disaster recovery services beyond seven days, believe it.  And that is extremely bad news for any calamity, be it caused by a virus or a match.

Let me take you through the world of telecommuting plans.  They all originate in large data centers -- operations centers with floor tiles raised over twelve inches from the floor to accept conduits full of cables and cooling pipes and to keep equipment high and dry if those pipes burst and water seeps in.  They are also very chilly, so they can keep the multitudes of computers cool and, thus, more efficient.  (Heat is the enemy of computers, which is why you should blow out all computers thoroughly with canned air at least once a year.)  These data centers are also stuffed with what we call remote-access servers.  These servers are powered by UNIX, or Linux, or Microsoft Server products.  They run Windows Terminal Services, or Citrix, or some other emulation software.  And they need lots and lots and lots of bandwidth and processor power and energy.

And the stuff in data centers breaks sometimes.  Computers are machines, too, like the washer. Anything from a poorly-seated accessory card to a botched software patch can render a million dollars' worth of remote access equipment unusable.  That, in turn, requires hands-on work to fix.  You can't fix a physically broken appliance remotely, even if it is a computer.  You occasionally, sometimes frequently, need to take the machine physically down and get into it up to your elbows.  That takes people, people.  And if you are down 30% to 40% on your server team staff, you are in deep trouble.

Now the remote-access packets of data pass through the network; banks of appliances called routers and switches, any of which can break, for the same reasons as above.  Then after this leaves your organization's firewall (another appliance), you have to rely upon the Internet, and the same dynamics apply to all the equipment that runs the Internet.  Finally, you get to your PC or laptop, maybe in a hotel in Burbank, or your home in Tuscaloosa.  So if your cable/DSL modem is working properly, AND if you can get to the Internet, AND you can get back to your hosting data center, AND that remote equipment is running, AND you can log in to your validation server:  Well, that's a huge bunch of "ifs", even on a good day.  The fact this stuff even works most of the time is a huge testament to IT everywhere.  So hug a geek today!

This applies to all the participants in the work-at-home food chain:  The organization, the IT service provider, the telecommunications provider, all the way to the electric company.  If any of these links fails (and it will in a pandemic), the entire chain is worthless.  That is why corporations and governments alike must prepare to make the calls to bring people back into their offices when the Internet becomes unreliable.

That is also why these same organizations must undertake measures NOW to acquire masks, gloves, hand sanitizer and disinfectant wipes.  But that is a lesson left for another day.

The biggest concern after staff shortages and broken stuff is the issue of supply chain failures.  The Just-in-time supply chain, as we all know and preach, is lethally exposed during a pandemic.  During the runup to Y2K, we drilled incessantly in Florida for supply chain failures.  We even went so far as to have the National Guard ready to escort convoys of Winn-Dixie food from warehouses in Alabama to their distribution points within Florida's Panhandle. 

In a pandemic, everything will be constrained and in short supply.  This especially means spare parts and replacement equipment for IT, since so much of it comes from overseas (Asia).  It is difficult to get some networking equipment delivered quickly on a good day, let alone in the middle of an influenza pandemic.  In fact, Michael Dell told me personally in 2006 that the SARS experience has fueled Dell's initiative to try and develop a Singapore-to-Ireland revolving door of manufacturing during a pandemic.  The theory is that while one area is savaged, the other might be on the path to recovery.  The company is making the best assumption it can; namely, that it must find a way to continue operations, or perish.  Dell will also try and maintain larger inventories of certain parts, although those components change so quickly that it is an egregious violation of Dell's own business model to store anything in too much quantity for too long.

It might surprise some to know that Dell has taken such a proactive approach to pandemic planning.  But I know Dell to be a forward-thinking and forward-leaning corporation, so it is not surprising to see them adopt such an approach.  The problem is that Dell is so alone when it comes to such planning.  And this is reinforced by Gartner's latest study, which again reinforces the limitless, ignorant arrogance of people -- including IT people and their superiors, regrettably -- to think a calamity will never happen to them.

Lower Manhattan and New Orleans professionals know the tremendous impact an extended calamity can cause.  That is why companies such as Merrill Lynch are global Best Practices at disaster recovery.  There's nothing like experience to help shape attitudes.