24 May 2006

By GWEN FLORIO
Tribune Capitol Bureau

HELENA, MT — The state’s computer system was back online Tuesday after being down for nearly 24 hours and, in the process, providing a good practice run for a true emergency.

“Monday was certainly an event of concern,” said David Ewer, budget director for Gov. Brian Schweitzer.

Hardware failure caused the problem, which began at about 2 a.m. Monday and was resolved at about the same time Tuesday morning, said Dawn Pizzini, spokeswoman for the state’s Information Technology Services Division. Systems actually began going back online at about 10 p.m., she said.

“It involved hardware that people had never seen fail before,” Ewer said.

“I’ve been in state government 20 years, and I don’t remember (the system) having been down more than four to six hours before,” he said.

The state’s most vital functions were preserved during the outage, officials said. People on food stamps were able to use their special debit cards and law enforcement officers had at least partial access to a system used for background checks.

But, said Ewer, “frankly, we’re not satisfied that it worked well enough.”

He did, however, praise the state’s technology staff for working hard and quickly to restore service.

While the computers were down, people couldn’t apply for hunting and fishing licenses online. The state couldn’t print tax refund or pension checks — or any checks for that matter. Social service agencies had to process applications for aid by hand. And every single state worker accustomed to conducting most of his or her business via e-mail had to resort to faxes and phones.

“We take the Internet and cell phones and e-mail for granted, but it hasn’t been around that long,” Ewer said.

The outage illustrated two things, he said.

“It shows how important it is to have a good, secure, state-of-the-art data center, and a secure backup system. Second, we need to do a better job of having a plan of action that gets us through, say, 48 hours with a minimal disruption of services.”

He said that improvements to the system, whose servers are located in the basement of the Sam W. Mitchell building, will be “a very high priority” in Schweitzer’s next budget.

“We’ve got five floors of toilets above our computers,” he said. “It’s not a good system.” He estimated that improvements will cost about $20 million to $25 million.

For starters, the state would like to house the servers and the Information Technology Services Division in their own secure building, and then have a backup center elsewhere, as in far from Helena.

“You’d want to try and not have it in the same earthquake zone, not the same telephone zone, not the same electricity grid, not in the same floodplain. You’d want to have as many things different so that if something natural happens, hopefully you wouldn’t get hit that badly,” Ewer said.

The best thing about Monday’s equipment failure, he said, was that it provided a wake-up call.

“This has always been on my to-do list,” he said. “But now it’s taken on a different kind of priority.”

The state last saw a large computer outage when a virus infected the system approximately four years ago, according to the Associated Press.

Contact Gwen Florio at 406-442-9493, or gfloriogreatfal.gannett.com