Storm Warning

Document Sample
Storm Warning Powered By Docstoc
					                                                                         How Jefferson Parish planned for and
                                                                         responded to the worst disaster in
                                                                         U.S. History, and the lessons IT
                                                                         professionals can learn from it.




IT Influencer Series

Storm Warning


Contents
            Katrina
                  1.
                  2.
                        Storm Warning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1
                        Cascading Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2
                  3.    Communications Collapse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3
                  4.    Tulane’s Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3
                        Planes, Trains & Automobiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4
                        Home Away from Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4
                  5.    ‘We couldn’t find them’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6
                  6.    Pain at the Pumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6
                  7.    Data Center Two-Step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
                  8.    The 3-Day Assumption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
                  9.    Lessons in the Wind . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9
                  10.   A Personal Toll . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
                        ‘I Had a Pretty Good Idea’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
                        Out of House . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11
                  11.   IT Lessons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11
                  12.   Work to Do . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11
                  13.   Respect the Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12
                  14.   Missions Critical . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12
                        Behind the Storm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12
                        The Best Laid Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13
                  15.   Conclusion: The Gathering Storm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13
                  16.   An Inside Job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14
                        Night at the Improv . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14
                        ‘The Luckiest Guy in the World’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14
                  17.   Credit Where Credit’s Due . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15
                        Stay at Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
                        ‘The Bayou Is What You Get’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
 St rm
Warning
How Jefferson Parish planned for and responded to
the worst disaster in U.S. history, and the lessons IT
professionals can learn from it. STORY AND PHOTOS BY MICHAEL DESMOND



T       he Jefferson Parish Emergency Operat-
        ing Center (EOC) squats over the low,
        grassy plain south of the Mississippi River.
It’s a brutish, four-story block of a building that
sits in the shadow of a massive radio tower. A
brick smokestack stabs at the Louisiana sky.
                                                       great crescent, from the shores of Lake
                                                       Pontchartrain in the northwest to the Mississip-
                                                       pi River-bounding communities of Marrero and
                                                       Gretna to the south and southeast. Inside the
                                                       EOC building, emergency personnel worked to
                                                       evacuate, assist and rescue citizens across this
  If the EOC building looks like a bomb shelter,       expansive swath of earth.
it’s because it once was. On Aug. 29, 2005, that         The Jefferson Parish EOC was well equipped
structure survived the greatest bomb ever dropped      for the task. There were hardened landlines,
on the United States: Hurricane Katrina.               800MHz radios, and cell and satellite phones
  Nearly 100 souls had crammed into the forti-         tapping multiple networks. There was a fully
fied structure on the morning of Katrina’s land-       redundant computer network, a fleet of remote
fall. Most came from across Jefferson Parish.          sensors and cameras, and a detailed disaster plan
With a population of more than 450,000, the            promising a multi-layered government
parish cradles the city of New Orleans in a            response. And, of course, there was a system of




                                           1 | Redmondmag.com
St rm Warning
 canals, levees and pumping stations designed to channel            shelters with supplies and backup power. His team was in
 flood waters out of the area.                                      place at the EOC. Key personnel stood by in neighboring
   Within hours of Katrina coming ashore, every one of              parishes to provide day one communications and recovery.
 those systems would fail catastrophically.                          “Up to the breaching of the levees, this was a perfect
   This is the story of the Jefferson Parish EOC and the            model,” Maestri says from his office on the fourth floor.
 people who struggled to run it as Katrina laid waste to the        “We knew what we thought could happen. We knew what
 region. Despite the meticulous planning, robust systems
 redundancy and heroic efforts of parish personnel, the Jef-
 ferson Parish EOC team—indeed, the entire region—
 struggled to accomplish the most basic tasks. Nine months
 later, it is not at all clear that the parish and its businesses
 are any more prepared for another major event than they
 were on that last Sunday of August 2005.

 Cascading Failure
 Dr. Walter S. Maestri had been director of the Jefferson
 Parish Emergency Management Department since 1996.
 The Sunday before landfall, he and nearly 100 parish
 staffers packed into the bunker-like EOC building, with its
 thick concrete walls and windowless rooms. Katrina boiled
 to a frightening crescendo just 200 miles to the south.
 Now a Category 5 storm, among the most powerful ever
 recorded, it steamed directly toward New Orleans.
   But when the storm weakened and turned unexpectedly to           Jason Phillips monitors water-level and pump-station status
 the east just hours before landfall, Maestri was confident.        from his workstations at the Jefferson Parish EOC in Marrero.
 More than 90 percent of parish residents had evacuated.            we had to deal with. We had prepared the plans, spread the
 Those unable to evacuate were housed at hospitals and              message. Everything was according to plan. No surprises,
                                                                    no surprises.”
                                                                      Maestri’s voice trails off. He sounds almost reluctant to
                                                                    go on. “Then, of course, the storm comes in—and things
                                                                    happen.”
                                                                      The power failed first, victim of the thrashing winds that
                                                                    toppled poles, tangled lines and threw down transformers
                                                                    and other gear. The EOC, like the parish’s two data cen-
                                                                    ters, is supported by diesel generators that kicked in as
                                                                    soon as service was disrupted. In the EOC, as well as at
                                                                    local hospitals and other critical facilities, the lights stayed
                                                                    on and the computers kept humming.
                                                                      Out in the field, the parish’s network of nearly 200
                                                                    radio-based remote sensor stations switched to battery
                                                                    backup power. Jason Phillips, supervisory control and data
                                                                    acquisition manager for the parish, watched as the real-
                                                                    time feeds displayed on his twin monitors began to drop
                                                                    off one by one. Most batteries lasted from 45 to 120 min-
                                                                    utes before going dark. Wind meters, shorn of their direc-
                                                                    tional vanes, stopped transmitting telemetry on wind
                                                                    speeds, while many lakeside sensors kept working until
                                                                    the waters swallowed them. The last sensor quit with a
                                                                    reading of 29.6 feet.
                                                                      “The ones we lost went underwater. Where we had them
                                                                    mounted we never would think the water on the lake
                                  Nearly 100 people crammed
                            into the bunker-like EOC building
                                                                    would get that high,” Phillips says. “But it did.”
                                     during Hurricane Katrina.        Voice and data communications were next to fail. When
                                                                    the power quit, general Internet access went with it.



                                                     2 | Redmondmag.com
St rm Warning
 Phone service throughout the area faltered or failed
 entirely within hours of landfall. At the EOC, staff mem-
 bers could receive calls throughout the storm, but were
 unable to dial out. It would take five days for BellSouth to
 restore service to the EOC, one of the highest-priority
 service points in the parish.

 Communications Collapse
 Of course, the parish disaster plan anticipated these out-
 ages. Staffers had been issued cell phones from a diverse
 set of service providers to help ensure voice communica-
 tions. “Verizon, Sprint, Nextel, Cingular, T-Mobile,”               The gutted interior of Casual Home Furniture, which
 Maestri counts off in rapid succession.                             took on five feet of water when New Orleans flood
                                                                     waters flowed down Airline Highway. The store today
   In fact, most service failed as cellular towers sustained         does business from a large tent in the parking lot.
 damage, switching stations lost power and call volume
 swamped the networks. Even the dozen or so EOC-oper-              na on the 21-story Galleria building blew away. That sev-
 ated Motorola satellite phones faltered during the storm,         ered links to police, fire, hospital and key emergency per-
 their signals blocked by thick cloud cover and heavy              sonnel situated on the north side of the Huey P. Long
 rain—though they would prove invaluable in the days               Bridge. When crews rushed to fix the damage after the
 that followed.                                                    storm, they discovered that T1 service between the Galle-
   The one system that worked—the parish’s nearly 2,000            ria and the EOC was dead as well.
 radios—suffered a crippling blow when the repeater anten-           Like an army slicing through the weakened flank of an



  Tulane’s Education
  Six months after a glancing blow from Hurricane Katrina
  flooded New Orleans, the city’s downtown is a work in
  progress. Street lights remain dark and power in office
  buildings fails regularly. A half block from the Superdome,
  its savaged roof now repaired, stands the dark monolith of
  the Hyatt Hotel. The building’s façade is an ugly patchwork
  of blown out windows and white plastic sheeting, a caustic
  visual reminder of just how much work remains to be done.
    Paul Barron, interim CIO of Tulane University, has a birds’
  eye view of it all from his 14th floor office on Poydras
  Street, right across from the Superdome. Like New Orleans
  itself, Hurricane Katrina nearly finished off the 172-year-
  old university, Barron says. “During the first 60 days, it was
  unclear if Tulane was going to survive or not.”
    Calamity struck quickly. “That Friday night I was at a pre-
  season Saints game, and people were talking a little bit
  about the hurricane. But nobody was talking about evacu-
  ating,” Barron says.
    Certainly not Tulane’s IT department. Tim Deeves, director
  of network services, recalls kicking off full system backups
  Friday evening. It was a fortunate thing, in hindsight, that
  the storm struck after the weekend—it gave the IT staff          More than half a year after the storm, battered facades,
  time to fully back up their systems. That work went on           intermittent power failures and non-working traffic
  until Sunday at 2 p.m., when Deeves himself shut down the        lights plague downtown New Orleans.
  school e-mail system.
    “I was a little bit worried,” Deeves says. “At one point I                                         Continued on page 4




                                                     3 | Redmondmag.com
St rm Warning
 Continued from page 3                                             They stopped at the Jefferson Parish operations center set
 was the only person in the building, I was worried about        up by the West Bank Expressway. There, McGinity and his
 the elevator losing power. I said, ‘Chris, if you don’t hear    team climbed into a deuce-and-a-half troop truck, along
 about me by 1 o’clock, call public safety and tell them to      with a contingent of heavily armed police. The transport
 come get me out of the elevator.’”                              rumbled past the nearly-empty Superdome and circled the
   The plan called for a courier from the Tulane security cen-   building on Poydras Street, looking for an approach where
 ter to accept the tapes from Deeves and transfer them to a      the water was shallow. National guardsmen, patrolling the
 nearby security center site. But when the courier failed to     streets on Sea-Doos, looked on as McGinity dismounted,
 show up, Deeves was told to leave the tapes out in the          waded to the entrance and reached into the murky waters
 14th floor hallway for pickup. When the courier finally         for the lock at the bottom of the doors.
 arrived, he found the doors locked—the building staff had         The walk up to the 14th floor was uneventful, but the
 fled. Those tapes would spend more than a week stewing in       team found the entrance to the data center locked tight—
 the late-summer Louisiana heat.                                 the security protocol that should have opened the door
   As it turned out, it’s a good thing the tapes never got to    when power went out had failed. One group searched near-
 the security center, which ended up largely underwater.         by offices for a key, while the other began hauling down
   “The interesting thing is the security center is four and     the backup tapes found in the outer hall. A few minutes,
 one-half blocks from here. That was not very smart,” Barron     McGinity and his guys were in the data center room, tear-
 says. “We lucked out in terms of getting our tapes, because     ing open disk arrays and servers in the sweltering heat.
 the security center is much lower than we are, and there          “We pulled disk drives and the tapes, up until the point
 was water. We got lucky.”                                       where we just couldn’t pull anymore stuff because time had
                                                                 run out,” McGinity says, estimating that the air in the
 Planes, Trains & Automobiles                                    outer hallways was about 110 degrees.
 With the region devastated, Tulane President Scott Cowen          The tapes in the boxes were a priority. After that, the drives
 made a decision—delivering payroll to school employees          from the systems were gathered in case they needed them for
 would be the first mission. With credit card and other          hardware being set up remotely. “We left one machine half
 electronic transactions unavailable for weeks after the         open with a board almost out of it,” McGinity says.
 storm, people needed immediate access to cash. To provide         By nine that night, the team was on a plane out of Baton
 that, the university needed to recover the tapes Deeves         Rouge on its way to Houston. “Well, everything is going
 had left on the 14th floor of the Poydras Street building.      according to plan, right?” McGinity jokes, drawing a laugh.
 It fell to Rick McGinity,                                       “We get our tapes and we take it to the hardware. We just
 director of operating                                           didn’t know where the hardware was.”
 system and database
 services, to lead a                                             Home Away from Home
 group into New Orleans                                          The hardware, it turned out, was in Vorhees, New Jersey.
 a week later to recover                                         Barron had arranged for mainframe space and resources
 the data.                                                       with Sungard’s offices in Vorhees, and McGinity and several
   What follows reads                                            others flew up with the tapes in a chartered jet Wednesday
 like something from                                             afternoon. The group worked frantically to get its
 the script of the movie                                         applications set up on a logical partition (LPAR) of an IBM
 “Planes, Trains & Auto-                                         mainframe operated by availability services provider
 mobiles.” McGinity and                                          Sungard. Twenty-four hours later, at 8 p.m. Thursday,
 his team couldn’t fly                                           Tulane’s payroll system was up and running. The university
 into New Orleans Air-     Tulane’s interim CIO Paul             was back in business.
 port—that was under       Barron says the school erred            “They had an environment, DASD (direct attached storage
 military control—so       when it moved its data center         device) and things, running. They were able to carve up the
 they landed in Baton      downtown.                             resources and things we would need running up there,”
 Rouge, picked up two                                            McGinity recalls. “We just did some tweaks that morning to
 staff members and drove to one of their homes. There they       get everything in. There are always differences, but they were
 picked up a second car. Once on the interstate, they            negligible enough that Mary’s team was able to take over.”
 dropped in behind what McGinity describes as “a whole             As head of the software application services team, Mary
 stream of ambulances going about 100 miles per hour.” The       Walsh configures and manages the payroll system. In the
 group talked its way past several roadblocks.                                                           Continued on page 5




                                                   4 | Redmondmag.com
St rm Warning
  Continued from page 4                                             • Ensure that recovery plans are implemented uniformly
  days following the storm, President Cowen had made              across all departments, not just the university in general.
  restoring payroll operations a top priority. University           • Create a team of first responders to handle assigned
  employees had to get paid, he reasoned, if the university       tasks, such as preservation of irreplaceable research assets
  was going to survive.                                           like lab specimens.
    With the software quickly in order, Walsh turned to maxi-       • Require submission of cellular phone numbers and mul-
  mizing the cash income of Tulane’s employees. She turned        tiple e-mail addresses for all staff.
  off income deductions and stopped monthly payments for            • Create listserves using Tulane and personal e-mail
  things like campus parking. Direct deposit payments went        addresses in order to broadcast to the community.
  out to employees, but the complete suspension of mail             • Create a remote call center outside of the 504
  service in New Orleans and the prolonged evacuation of          area code.
  staffers to points unknown often meant that paper checks          Challenges will remain. For instance, Tulane had moved its
  failed to find a home.                                          data center into the 22-story Poydras Building in July, less
    “The hard copy checks—that took a lot of perseverance.        than two months before the storm struck. Prior to that, the
  We had to track people down,” Walsh recalls. “It was an         data center had been located in the university’s uptown
  ongoing effort.”                                                campus, which has recovered more quickly than the area
    The effort paid off. In January, the campus reopened to       around the Superdome.
  students. Barron says more than 80 percent of students            “This had been the corporate offices for some oil company
  returned, despite many spending the previous semester at        and they had a data center here,” says Barron, “so we
  other schools. While many problems and challenges remain,       decided to use the data center [facility]. Was it a good
  the school that very nearly died in the rising waters of Kat-   decision? Probably not. Will we keep it here? I think so.”
  rina is back on track. In fact, the IT staff is planning to     Barron says if the university could do it all over, the data
  migrate the aging mail server infrastructure to Microsoft       center would probably have ended up in Covington,
  Exchange 2003 once classes let out for the summer.              Louisiana, on the north shore of Lake Pontchartrain. “It is
    In the meantime, school administrators are working to         generally viewed as safe from any storm surge,” he says.
  beef up Tulane’s readiness for another storm. Among               So will Tulane move its data center again?
  the changes:                                                      “Sunk costs have a big impact on people, even though it
    • Immediately review operations with an eye toward            shouldn’t,” Barron answers. “I don’t see us in the short
  short- and long-term improvements in the school’s hurri-        term doing that, but we may in the longer term.”
  cane response.                                                                                                       — M.D.


 enemy, Katrina had cut off the EOC from the East Bank            when he learned that the freshly mounted antenna had
 communities pinned between Lake Pontchartrain and the            been disconnected by FEMA personnel.
 Mississippi River.                                                 “We had an instance where we got our antenna back up
   These were some of the hardest hit areas in Jefferson          and someone at FEMA came in and put their communica-
 Parish. Flooding devastated portions of Metairie, particu-       tions up, because of the operation that was going on [near
 larly in the south, where floodwaters from New Orleans
 followed Airline Highway into the parish. Powerful north
 winds rotating off the retreating storm battered lakeside
 communities. Vital resources like the New Orleans airport
 and the I-10 highway linking the area to Baton Rouge
 were a top priority.
   Technicians rushed to replace the antenna and set up a
 point-to-point microwave link between the EOC in
 Marrero and the Galleria building—about an 8-mile
 shot. That enabled 800MHz radio communication
 parish-wide for first responders, but less than two days
 later the link failed again.
   As emergency coordinator for Jefferson Parish, Tom
 Rodrigue is responsible for keeping communications up
 during a storm. When Katrina came ashore, he was ready           Katrina tore open the Yenni Building roof, disabling the data
 to battle wind, water and line damage. But he was stunned        center and damaging the upper floors.



                                                    5 | Redmondmag.com
St rm Warning
 the Galleria site]. At which time the sheriff went back up      “It was the pony express—people going from place to
 there and put [on] armed guards, took their antenna down,       place carrying messages.”
 put ours back up, and says, ‘Anybody tries to take it down
 ...’” Rodrigue pauses as a co-worker finishes the thought,      ‘We couldn’t find them’
 “… they are going to shoot you.”                                Actually, Jefferson Parish was fortunate. Unlike surround-
    Even with its radio service restored to the East Bank, the   ing communities, it could radio state agencies.
 EOC struggled to communicate with state, federal and              “St. Bernard Parish, for whatever reason, couldn’t reach
 other parish agencies. Jammed frequencies and incompati-        the state,” Maestri says of the neighboring parish to the
 ble radio networks stymied cooperation—Rodrigue                 east, which was utterly devastated by flooding. “So what
 describes the system at the time as “inoperable.” Those dif-    happened was St. Bernard Parish could reach us and we
 ficulties enhanced the isolation created by rising floodwa-     could relay messages to the state. We were relaying mes-
 ters and broken roads.                                          sages for multiple parishes.”
    “We had to resort to runners,” says Carolyn Capdeville,        The city of New Orleans, however, simply dropped off
 Jefferson Parish coordinator for emergency management.          the map. “They abandoned their EOC and set up their



  Pain at the Pumps
  In New Orleans, studies have shown that structural flaws         Tom Rodrigue says similar issues plague the design of the
  and poor maintenance most likely caused the breeches that      canals that stretch miles inland from Lake Pontchartrain.
  flooded 80 percent of the city. Perhaps no single element      When barriers like the 17th Street Canal fail upstream from
  was more devastating to the area than the Mississippi River    the pumping station, there’s nothing to prevent the lake
  Gulf Outlet (MR-GO), a shipping shortcut designed to cut 37    from rushing freely through the gap.
  twisting miles out of the trip from the Port of New Orleans’     “When you think about it, they allowed the lake to intro-
  Inner Harbor Navigation Channel to the Gulf of Mexico.         duce itself all the way into the city, by having the pumping
                                                                 station not at the lake, but back here,” says Rodrigue, stab-
                                                                 bing a finger at a map of New Orleans, at a point about two
                                                                 miles south of the lake. “The lake goes all the way into here,
                                                                 then you pump the water through that canal to the lake.”
                                                                   In Jefferson Parish, the lakeside pumps are situated right
                                                                 on the shore. The problem was, they weren’t manned.
                                                                   When parish President Aaron Broussard invoked the so-
                                                                 called Doomsday Plan on August 28, the day before Katrina
                                                                 struck, it required that all 236 pump operators evacuate to
                                                                 Washington Parish, about 100 miles north. Lacking a safe
                                                                 place to house the workers, the parish ordered its person-
                                                                 nel out and shut down the pumps. The result: Flooding
                                                                 developed into major events, as waters pooled in Metairie,
                                                                 Kenner and elsewhere.
                                                                   The parish learned a hard lesson that day. Broussard has
                                                                 since announced that pump operators will stay local, and
                                                                 that safe houses will be built at pump stations to house
  Rodrigue says a flawed canal system helped create              the workers during storms. In addition, cities in the parish,
  disastrous scenes, like this one two blocks from the           such as Kenner, are cross-training personnel to take over at
  17th Street Canal breech.                                      the pumps should parish staff be unavailable.
                                                                   The result is a more robust—and more realistic—emergency
    Opinions differ, but many engineers believe the MR-GO        plan. The original plan simply abandoned one of the most
  channeled gulf waters into a fast-moving surge that under-     vital defenses of the parish—its pumping stations—and put
  mined levees and led directly to the catastrophic flooding     thousands of residents and emergency personnel at risk. The
  in New Orleans’ 9th Ward and St. Bernard Parish. In short:     lesson: Make sure your emergency procedures, however well
  MR-GO may have been a gun-barrel thrust into the ribs of       intentioned, don’t do more damage than they avoid.
  the metropolitan area.                                                                                             — M.D.



                                                   6 | Redmondmag.com
St rm Warning
                                                                                                net. You’ve got to take it
                                                                                                this way or you are going to
                                                                                                lose us. There was no way
                                                                                                to get to E Team.”
                                                                                                  The EOC team finally
                                                                                                convinced the state to
                                                                                                accept requests by phone.
                                                                                                But with cell and landline
                                                                                                service both out, even that
                                                                                                was dicey. “Eventually the
                                                                                                only communications sys-
                                                                                                tem to stay up during the
                                                                                                entire ordeal was satellite
                                                                                                telephones,” says
                                                                                                Rodrigue. “Had it not
                                                                                                been for that we wouldn’t
                                                                                                have been able to make
                                                                                                any requests to the state.”
                                                                                                  Parish workers soon
                                                                                                learned that cell phones
                                                                                                with out-of-state area codes
                                                                                                actually worked—even in
   Canal breaches in the Lower 9th
   Ward and at the 17th Street                                                                  the stricken region. In the
   Canal (shown) devastated large                                                               weeks after Katrina, team
   sections of New Orleans. Many                                                                members began using cell
   areas remain uninhabitable.
                                                                                                phones with numbers based
                                                                                                in Texas and other states. It
                                                                                                was just one of many
                                                                                                improvisations that helped
                                                                                                keep critical systems and
 temporary operating facilities in different buildings down-    communications functioning during the crisis.
 town, and we couldn’t find them. Their towers went down,        “It’s not the technology that saves anybody, it’s the peo-
 their radio systems went down, or whatever, and we could-      ple who use the technology,” Capdeville says. “That’s
 n’t find them.                                                 missed a lot of times, the human part of this.”
   “Communication interoperability is a tremendous prob-
 lem, and nobody learned the lesson from 9/11. That             Data Center Two-Step
 problem, through Katrina, still exists,” Maestri warns.        Jefferson Parish maintains two discreet data centers, locat-
 “Our challenge is not interoperability, it’s operability.      ed more than 10 miles apart. Identically configured, each
 The commitment has been made that everybody will               includes a single IBM AS/400 iSeries midframe server,
 share the same system.”                                        which hosts financial, payroll, utility billing and other
   The EOC staff did get one pleasant surprise: Cellular        applications. The servers connect to a 2TB SAN and are
 voice service collapsed within hours of landfall, but text-    linked to each other and the EOC via frame relay. While
 messaging services stayed up. EOC personnel adapted            the two centers house the same equipment, they serve dif-
 quickly, texting requests and responses, and using text mes-   ferent application sets.
 sages to set up meetings. For several days after the storm,      The iSeries system in the East Bank data center, located
 text messaging was the only reliable way to communicate        on the ninth floor of the Joseph S. Yenni Building in Jef-
 with people beyond that of the parish radio network. Still,    ferson, just a mile north of the Huey P. Long Bridge, hous-
 some agencies refused to adapt.                                es utility billing operations. That data center also includes
   “The state office of Emergency Preparedness, they have       a Linux server running an Oracle GIS database for map-
 a program called E Team that works on the Internet, and        ping and geo-location services. The West Bank data center,
 they wanted us to go over that,” Capdeville says of requests   housed in a building on a parish campus in Gretna, powers
 her team tried to place with the state. “And we kept holler-   critical financial and payroll applications. There are also 35
 ing at them on an 800MHz radio that we don’t have Inter-       Windows servers hosting Exchange, Internet Information



                                                  7 | Redmondmag.com
St rm Warning
 Server, DNS and applications for several departments. The         when Katrina washed ashore. During the storm, high winds
 parish operates about 1,400 total Windows clients.                ripped the large air conditioning units off the top of the
   “Ever since the early ’90s the plan was to have two data        Yenni Building, tearing open the roof. Water from the
 centers with redundant capabilities,” explains Ridley             storm and from ruptured air conditioning chiller pipes
 Boudreaux, director of Electronic Information Systems for         poured into the upper floors of the building. “Even if you
 the parish. “We didn’t set them up redundantly, it’s not like     have generator power—with no air conditioning, forget it,”
 a hot situation where I can instantly say now we’re doing         Boudreaux says. “When you have something as catastrophic
 our payroll on the East Bank. But we always bought identi-        as the roof coming off and water cascading through the
 cal iSeries machines—or AS/400s—with enough capacity              building, a whole lot isn’t going to stay operational.”
 to run all of the applications on one of them.”                     Even with the East Bank site disabled, Boudreaux felt
   That decision, which helped the parish shave equipment,         lucky. The storm had largely spared the West Bank facility,
 software and management costs, got put to a severe test           where critical financial and payroll applications were host-



  The 3-Day Assumption                                               In Jefferson Parish, emergency planners turned to com-
                                                                   puter simulations to break some of those assumptions. As
  How could so many IT organizations fail to plan for the          part of an ongoing pre-storm educational push, the parish
  wholesale devastation of Hurricane Katrina? Tulane Univer-       created a video showing residents what they could expect
  sity in New Orleans, for instance, had a disaster plan that      from a major storm.
  called for backup tapes to be moved to a secure center just        “We used the technology, particularly the computer simu-
  four-and-a-half blocks away. That facility ended up under-       lations and the model, to heavy, heavy use,” says Maestri.
  water. Jefferson Parish locally hosted infrastructure like e-    “The film … actually shows the major commercial area of
  mail, Web and application servers, and lost service to all       the East Bank of Jefferson Parish—Metairie, Lakeside Shop-
  three for a time.                                                ping Center, the biggest mall in metropolitan New Orleans,
    Paul Barron, interim CIO at Tulane University, calls it “the   underwater. We showed them what it is going to look like if
  three-day assumption.” New Orleans hadn’t experienced a          this storm comes. And don’t be surprised if water is to the
  direct hit since Hurricane Betsy in 1965. Local planners,        roof. We’re showing 14, 15, 16 feet of water.”
  having seen dozens of storms strike Florida, Alabama, Texas        Others learned the lesson from afar. As president of the
  and elsewhere, assumed recovery from a major storm would         Jefferson Parish Employees Federal Credit Union, Vicki
  parallel those events. People would evacuate, wait out the       O’Brien was supposed to be at the bank’s headquarters to
  storm and return two or three days later to clean up, Bar-       oversee operations during a hurricane. She happened to be
  ron says.                                                        on vacation in California when Katrina struck.
    “That mindset pervaded not just the university, but the          “In hindsight that was the very best place in the world for me
  whole community and probably the region as well,” Barron         to be, because I was not stopped by communications. I had to
  explains. “And as a result, our disaster planning was more       make decisions based upon news media, but I could call my
  about getting ready for a hurricane that wasn’t going to         processors, I could call all of my third-party vendors,” O’Brien
  actually come, than actually preparing for what we do when       says. “I had a phone line. I had a computer, I had Internet.
  the hurricane gets here.”                                        Nobody here had any of that. But I couldn’t talk to the
    The difference, of course, was geography. When the             [Jefferson Parish] EOC. I couldn’t get through to the parish.”
  levees broke, New Orleans and other areas filled up like a         Hurricane Katrina put an end to the three-day assump-
  bathtub. IT staff that had evacuated could not re-enter the      tion. All across the region, IT managers are beefing up
  city. Rising waters completely disabled the infrastructure.      remote data operations and reassessing emergency plans.
  What would have been a three-day evacuation in Miami or          Jefferson Parish has moved Web serving out of the area and
  Houston stretched into a three-month ordeal in New               may relocate e-mail operations as well. Tulane has con-
  Orleans. Tulane, for instance, didn’t regain access to its       tracted Sungard for availability and business continuity
  downtown data center until November—and that was only            services to allow smooth failover for the next storm. And
  with generator power.                                            O’Brien is applying a few lessons of her own.
    “We were operating on a series of false assumptions,” Bar-       “Some of the things that in planning stages sound like
  ron says. “If the hurricane hit, we would have access to the     they will work perfectly, in real life, when you have a com-
  city and campus buildings—not true. Utility power would          munications issue, it doesn’t work,” she says. “We had put
  be restored, and the limited emergency power would be            together a booklet on hurricane preparedness for our credit
  sufficient—not true. Communications, particularly emer-          union members, to tell them what we were doing and how
  gency communications, would work—it was virtually impos-         to handle things in the event of a storm. After Katrina, I
  sible to use your cell phone.”                                   threw them all away.”                                    — M.D.


                                                     8 | Redmondmag.com
St rm Warning
 ed. “We were extremely fortunate that the building on the          and immediately started working on payroll, which was
 West Bank came through with flying colors. The generator           due the following Friday.”
 worked. Power remained to that building so that system               Boudreaux says that the AS/400-based iSeries platform
 stayed up,” he says.                                               eased migration of East Bank data center applications to
   Daily procedure calls for the backup tapes produced at           the sibling midframe in Gretna. In just a few hours his
 each data center to be transported to the other site—              team had dumped the bits onto the working AS/400 and
 ensuring that data will survive a fire or catastrophic event.      was up and running. “You take a backup of Exchange and
 With Katrina’s approach, the disaster plan called for up-to-       try to put it on a box that’s not identical to the one you
 date backup tapes to be transported from the two data cen-         took it off of,” he says with a laugh. “The only issue was
 ters to the hardened EOC site in Marrero. But the tech             the connectivity from the client. People who are mapped
 responsible for the transfer was running late, trying to get       to the East Bank location, for example, would have to be
 his family out of the area. The tapes ended up in Houston.         mapped to the West Bank location when East Bank goes
   “It probably worked out better that way,” says Boudreaux,        down. That was a minor issue, but it made us run around a
 who was able to have the tapes shipped to his temporary            little bit.”
 location in Baton Rouge. When it became clear that the               Boudreaux himself was on the north shore of Lake
 Gretna site and its critical applications had ridden out the       Pontchartrain when the storm hit. Rather than return to
 storm, Boudreaux’s team got to work.                               the stricken parish, he headed west “through downed trees
   “We got one of my people and somebody from payroll               and power lines” to Baton Rouge, where the parish was
 operations in that building by Thursday after the storm            establishing an emergency seat of government. Katrina’s


  Lessons in the Wind
  Interoperate: Following Katrina, Jefferson Parish experi-         it took creative and dedicated people to adapt and work
  enced a macabre repeat of the snarled communications that         around problems—whether it was a critical antenna cut by
  hampered response during 9/11. The lack of interoperable          FEMA or vital data and equipment trapped in the ninth
  radio and communications gear stymied coordination with           floor of a damaged building. Non-digital backup procedures
  local, state and federal agencies. Area parishes are now          need to be considered and developed.
  working with the state to implement a unified standard for        Stay or Go: Many of the most effective managers were
  radio communications.                                             housed outside of the New Orleans area, where they had
  Local backup is not enough: Many businesses and gov-              access to communications, Internet, power and transporta-
  ernment agencies dutifully backed up data ahead of the            tion. Still, it takes boots on the ground—like Walt Barowka
  storm, then housed the disks or tapes less than 10 miles          carrying hardware out of the Yenni Building—to get things
  away. Likewise, Jefferson Parish locally hosted both its e-mail   done. In other cases, pulling staff out of the area can
  and Web servers. The result: Parish staffers had to take heroic   invite catastrophe—as was the case with the Jefferson
  measures to recover and move these systems out of the area.       Parish pump operators.
  Seek alternatives: When all communication systems                 Challenge assumptions: During inspections of sensor
  failed, one surprisingly stayed online—text messaging. Jef-       stations and antenna mounts following the storm, Jason
  ferson Parish and other organizations quickly learned to use      Phillips learned that wooden poles ably weathered the high
  text messaging to communicate. Many also procured cell            winds. Galvanized steel poles, by contrast, tended to bend
  phones with out-of-state area codes, since these proved           at a 90-degree angle or snap entirely. No surprise, the
  much more reliable than local accounts. Text messaging is         parish has switched to wood poles.
  now a formal part of the emergency plan for the parish.           Produce hard copy: Critical data such as contact lists,
  Forge-trusted partnerships: Ridley Boudreaux was                  emergency documentation, authentication data and the like
  able to restore Web and e-mail service within two days of         should be printed and distributed, so it’s accessible during
  the hurricane, thanks to the help of the VAR that set up the      extensive power or network outages. Also consider distrib-
  parish’s IBM iSeries systems. The company provided server         uting this documentation on compact USB keys, so remote
  resources, office space in Baton Rouge and a skilled pro-         staffers have easy and portable access.
  grammer to help restore services.                                 Reject wishful thinking: Many IT plans worked within
  Don’t overvalue technology: Katrina pulled down the               the framework of a three-day evacuation and failed to
  entire working infrastructure of the New Orleans area. And        account for the effects of a direct hit by a hurricane. As a
  yet, the state urged Jefferson Parish to submit its requests      result, IT and emergency crews often had to take heroic
  via the E-Team Web-based logistics application. Ultimately,       measures to recover vital data and systems.          — M.D.


                                                      9 | Redmondmag.com
St rm Warning
 eastward turn had spared Louisiana’s capital city. There,          coming down from the north shore, he saw some young
 Boudreaux could find what he needed to recover his IT              bucks cutting trees along the side of the road,” Boudreaux
 operation: Power, communications, and access to systems,           recalls. “He said ‘Come on, I’m gonna pay you,’ threw
 partners, and vendors.                                             them into the back of his truck, and brought four guys
  “When I left I took all of my contact lists in hard copy—I        down, plus a couple of his own people and a couple of my
 printed it all out,” he says. “I had phone numbers for all of      people. They just hauled equipment down all day long,
 our vendors, I had passwords with me.”                             hauled it over to the Gretna building.”
  Even as Boudreaux was getting busy in Baton Rouge, one              By the end of the day, the impromptu IT rescue team had
 of his contractors, Walt Barowka, had struck out from the          delivered to the surviving data center everything but the
 north shore, heading south toward the stricken Yenni               AS/400 midframe system—which was too heavy to carry
 Building. His mission: To recover as much equipment and            down the nine flights of stairs.
 data from the disabled data center as he could.                      “It was heroic,” Boudreaux says. “That’s the word I used
  “On his way down, I think it was the Thursday after,              over and over again—absolutely heroic. Everything worked.”



  A Personal Toll                          Emergency staffers stayed on the job amid personal loss.
  As emergency coordinator of Jefferson Parish, Tom Rodrigue        and lo and behold the storm went right over Picayune. I didn’t
  scrambled to recover communications, procure fuel and             know until the following Friday that they were even OK.”
  supplies, and deliver services that might save lives across         For Dr. Walter Maestri, director of Emergency Management
  the parish. He recalls discovering a Navy ship, docked for        at Jefferson Parish at the time, the tragedy that struck
  repairs, that was loaded with 750,000 gallons of diesel           Rodrigue eventually touched him. His father had evacuated
  fuel. For emergency crews in Jefferson Parish, it was an          and survived the storm. A few months later, he was dead.
  unbelievable find.                                                  “One of the phenomena that we’re seeing now, is that
    “There was no way to get fuel and repeated requests for         there are a lot of deaths occurring secondary to that, being
  fuel from FEMA only ran into bureaucratic nightmares,” says       forced to evacuate,” Maestri says. “We’ve got research stud-
  Rodrigue. “In order to survive, we commandeered some fuel         ies going on in this area that indicate that it’s a significant
  trucks from a petroleum company. We put firemen in them           phenomenon that is happening.
  and we sent them to the ship, and we worked out with the            “My dad lost everything and he was 85 years old,” Maestri
  captain to be able to pump the fuel from the ship over the        continues. “He lost everything in the storm. His home was in
  levy into the trucks. And the firemen just did a continuous       New Orleans and his home was completely demolished. He
  rotation to all the different locations, such as the hospi-       lost all of his possessions, he lost all of his pictures. And four
  tals, nursing homes, that still were in operation because         months later, he died. Dad just … it was just too much.”
  they didn’t evacuate. Had it not been for that we would             Virtually the entire staff struggled with the emotional toll
  have completely [gone dark].”                                     of the calamity. Maestri recounts the stress put on his
    As a result of that effort, three of the five major hospitals   staff, as they fielded hundreds of desperate phone calls
  in Jefferson Parish stayed open and operational throughout        from family members seeking help for stranded relatives,
  the storm and its aftermath. By contrast, every hospital in       many of whom were too old or infirm to get out.
  New Orleans had shuttered, Rodrigue says.                           “Let me tell you, the toll that takes on the staff is incred-
                                                                    ible. We had people crash here, literally, as we went
  ‘I Had a Pretty Good Idea’                                        through this with those kinds of phone calls. And most of
  The work occurred even as Rodrigue, himself a resident of         our people are trained to expect that, to know that that is
  devastated St. Bernard Parish, suspected that his mother          going to happen. And yet, when you get those calls,”
  had been lost in the 15-foot high floodwaters. She was a          Maestri says, his voice trailing off.
  resident at St. Rita’s nursing home in St. Bernard Parish,          “Caroline and I sat by Tom Rodrigue, who has been in
  and was among the 34 people who died at the facility. The         emergency management for 20 years, as he made the phone
  nursing home owners were later charged with negligent             calls leading up to the storm, warning everyone in the St.
  homicide, for failing to evacuate ahead of the storm.             Bernard area where his mother was in a nursing home, that
    “I didn’t know for sure until the Friday after [the storm].     they had to get her out. They had to get all those people
  Based on what I saw on reports as far as storm surge was          out. And we sat next to him when he got the phone call—
  concerned and the fact that I knew they didn’t evacuate, I        she hadn’t gotten out. I mean, that kind of stuff. I don’t
  had a pretty good idea that was the case,” Rodrigue says.         want to overplay it or over-dramatize it, but you can imag-
  “But I didn’t know anything about my wife either, until the       ine what it’s like.”
  following Friday, because they went to Picayune, Mississippi,                                               Continued on page 11



                                                     10 | Redmondmag.com
St rm Warning
  Continued from page 10
                                                                 ship,” Rodrigue says. “I just bought another home in
  Out of House                                                   Metairie, but I lived in St. Bernard. There was about 15 feet
  As with others in the emergency management department,         of water. I don’t have much left to go back to.”
  the bunker-like EOC building became a temporary home.            Rodrigue credits his background in the military for help-
  The placed was packed, with staffers sleeping in hallways      ing him stay focused in the days and weeks after the
  and sharing scarce restroom facilities—Maestri recounts        storm. He says it’s a unique mindset shared by emergency
  setting up decontamination showers from the EOC’s hazmat       first responders.
  stores, so people had a way to take showers. For Rodrigue,       “Staying busy keeps your mind off of the personal things
  who was displaced by flooding, his stay at the EOC was the     that you have to deal with,” Rodrigue says. “I think that’s
  beginning of a six month sojourn.                              something that’s instilled in emergency personnel to begin
    “I was here for a month. I lived in this dormitory on the    with. And myself being ex-military, the mission came first.
  third floor for maybe a month. And then I lived on a navy      Everything else took a backseat.”


 IT Lessons                                                      Solutions. So we had a Web site up that afternoon.”
 Well, not quite everything. Boudreaux says the parish got         The site was used to host a registration page that let
 caught flat-footed when its Web site went dark during the       parish personnel confirm their whereabouts with EOC
 storm. A local company on the East Bank hosted the              staffers. Boudreaux also used the site to point staffers to a
 servers in its building, but when that building was dam-        private page containing his latest address information and
 aged, the servers were lost. It took the company more than      phone number. This helped keep him in constant touch
 a week to shuttle operations to Birmingham, Ala., and           even as he bounced between locations.
 restore service. In retrospect, Boudreaux says locally host-      In addition to setting up networks and communications,
 ing the Web site was a mistake.                                 Boudreaux funneled all parish e-mail—which was com-
   “In the parish we try our best to support local business-     pletely unavailable in Jefferson Parish—to his site in Baton
 es,” he says. “You learn lessons. Fortunately, we were regis-   Rouge. While he could not e-mail staffers in the afflicted
 tered with Network Solutions—it took a matter of minutes        area, he could review correspondence and pass on urgent
                                                                 items by fax, cell phone or text message.
                                                                   As for the AS/400 iSeries midframe left in the Yenni
                                                                 Building, Boudreaux’s team was finally able to access it
                                                                 four weeks later, once electricity and air conditioning had
                                                                 been restored. For nearly a month, the system sat in broil-
                                                                 ing temperatures and high humidity—still, it booted up
                                                                 normally, even with one hard drive failed and another
                                                                 showing failure indications.
                                                                   “So, the system worked. It came up and worked,”
                                                                 Boudreaux says. “When people talk about moving our
                                                                 financial, payroll, and utility running applications to a
                                                                 Wintel box, I tell them over my dead body.”
  Former Jefferson Parish EOC Director
  Walter Maestri warns that damage from                          Work to Do
  Katrina could make the next major storm                        Boudreaux is first to admit that he and his team had a lot
  much harder to cope with.
                                                                 to learn from the storm and its aftermath. Six months after
                                                                 Katrina had struck, major changes were already being
 to switch it over.”                                             made to the electronic infrastructure and planning across
   The switch over was made possible by CMA Technology           the parish. For instance, Maestri says a remote data center
 Solutions, the value-added reseller that had helped set up      has been established outside of the area. It’s a decision he
 the parish’s two iSeries systems.                               hopes others will adopt.
   “I showed up at their door 1 o’clock on Tuesday and said        “That’s what we recommend to businesses and industries
 ‘I need some help.’ They gave me an office, they gave me        in the area,” says Maestri. “Don’t move it six blocks. Don’t
 space on their iSeries, they gave me a brilliant program-       put it in a warehouse around the corner. Send your data
 mer, who helped me set up a one-page Web site that after-       out of town. If you’ve got multiple locations for your firm,
 noon and get our address changed with Network                   send it to an installation somewhere.



                                                  11 | Redmondmag.com
St rm Warning
   “We had Ridley running all over the place, physically,        location, even if none occurred during Katrina. He expects
 because we didn’t want to trust this data to the lines, to      the building will have a future as a dormitory or other
 electronic transfer, because we weren’t sure if there was a     function for the emergency management staff.
 breach in the line somewhere.”
   Similar improvements are being made to harden the             Respect the Plan
 damaged Yenni Building against another major storm.             There is no doubt that members of the Jefferson Parish
 Shatterproof glass designed for hurricane winds is being        Emergency Management Department were bitterly disap-
 installed. The roof is also being retrofitted to prevent a      pointed by state and federal response after the storm.
 repeat of the severe damage that occurred when the air          Staffers at the EOC repeat darkly comical tales of FEMA
 conditioning chiller units were torn off.                       crews tearing down parish communication gear and stop-
   Changes are coming for the EOC building, as well.             ping urgently needed shipments of fuel and water that
 The parish is working on plans to move into a larger            parish officials had arranged.
 facility, after having struggled with the Marrero site            But most of all, they do not understand why other agen-
 during Katrina.                                                 cies failed to follow the agreed upon disaster plan. “That
   “It was horrible. Those of us on this staff had six           plan said the government would be here in 72 hours,”
 months of hell,” Maestri says. “There were no shower            says Maestri.
 facilities. We had to set up tents in the backyard—it was         The plan, established by the Southeast Louisiana Hurri-
 in fact our decontamination tents because the hazmat            cane Task Force, draws together the resources of 16 area
 unit is part of emergency management, or it was. We             parishes, and provides a blueprint for government interac-
 had 100 people living in this building and they didn’t          tion during a calamity.
 have a place to take a shower. That’s why we’re looking           “When we lost communications with the state and with
 now for a larger facility with separate sleeping facilities,    FEMA for about a five to seven day period, we simply
 a larger kitchen.”                                              pulled the plan off the shelf and started looking at it. We
   Maestri also worries about flooding at the current EOC        said, ‘This is what we are going to do. This is what we

                                           Even the U.S. Coast Guard underestimated the challenge of oper-
  Missions Critical                        ating after a hurricane. Here’s how they weathered the event.

  When Hurricane Katrina struck the Gulf Coast at the end of     found out where they needed to connect to reach their
  the August, the timing for members of the U.S. Coast           data and were able to get to work.
  Guard sector command in New Orleans could hardly be              Laying the groundwork for effective command and control
  better. The unit had just completed the finishing touches      took more than planning and some well-crafted Visual Basic
  on its new Continuity of Operations Plan (COOP), which         scripts to prep systems. The team worked with major tele-
  defined how systems and communications would be kept           com carriers to ensure that bandwidth would be available
  viable during a catastrophe.                                   at the secondary site.
    “We had just finished going through the requirements and       “One week before the hurricane, I had met with MCI, Bell
  how we were going to do it in the week that Katrina hit,”      South and SBC, and explained to them our reliance on their
  recalls Commander Mark Johnston. “We were going to test        infrastructure for search and rescue,” says Johnston. “We
  it. So we got a live test.”                                    definitely formed a partnership. Right after the storm, we
    As it turned out, Johnston’s crew passed with flying col-    called them up and ordered three T1 lines for that hotel…
  ors. More than 500 people and 200 systems, including a         and they installed those three T1 lines in about 20 hours.
  variety of Dell PowerEdge Servers, were relocated from the     That was basically because we knew who to call and how to
  New Orleans Coast Guard office on Lake Pontchartrain to a      expedite those installations.”
  conference center in Alexandria, Louisiana, just north of
  Baton Rouge. “As the storm was hitting the beach they          Behind the Storm
  headed up north and took their teams with them. And            Within hours, Katrina had moved inland, leaving behind
  when they got up there, teams were ready to go,” John-         churning seas and a devastated landscape. Several Coast
  ston says.                                                     Guard cutters steamed toward shore behind the storm, pro-
    Johnston’s group relied on the Remote Desktop function-      viding high frequency radio and satellite communication
  ality of Windows to let client systems access data stored on   links for units in the area. The ships also helped coordinate
  servers in either Alexandria, Louisiana, or Alexandria, Vir-   air support, and served as floating front offices for Coast
  ginia. In the days and hours before the storm, IT managers     Guard officers in the area.
  had shipped backup tapes to each location. Users simply                                              Continued on page 13




                                                  12 | Redmondmag.com
St rm Warning
  Continued from page 12                                          The Best Laid Plans
    “If people weren’t connected via satellite they went to       “I would say this: The commercial infrastructure, I was
  the nearest Coast Guard cutter and got their command and        shocked the entire system was down,” admits Johnston.
  control,” says Johnston. “They have the same infrastruc-          In fact, the lessons from Katrina prompted the Coast Guard
  ture—with Microsoft Office and Windows.”                        to reconfigure its regional network operation, which had
    Those cutters also helped fill in the gaps when the Coast     employed a hub-and-spoke model with New Orleans at its center.
  Guard found that its network of tall VHF radio towers lining      “All those regions depended on their connections to New
  the Gulf Coast and Lake Pontchartrain had been disabled.        Orleans, and from there they went out to the rest of the
  When working, those towers let the Coast Guard communi-         world. That design has gone by the wayside,” says John-
  cate with off-shore vessels.                                    ston. “We always knew it was a single point of failure, but
    Restoring the towers was a top priority. The Coast Guard      we get our funding from Congress.
  deployed communication trailers to connect directly to the        “Now each sector gets its connection directly to the Coast
  tower stations, and then link via satellite to the Coast        Guard network, via an MPLS cloud. It’s all redundant,”
  Guard network. The teams used WAVE software, from Twist-        Johnston explains, “so if any one of those things goes
  ed Pair, to stitch together the disparate communication         down, it just reroutes through the cloud.”
  networks. One wrinkle: The solution had never been used           Despite the limitations, the Coast Guard’s COOP weathered
  over satellite before, so they worked with the vendor’s         Katrina pretty well, but Johnston says more work lies ahead.
  engineers to make it work.                                      He warns that the next calamity may not offer two or three
    “They could control it, they could hear if somebody was       days to prepare. He hopes to close that vulnerability with
  calling for help,” says Johnston. “And they could communi-      “some pretty neat backup technologies” that his group will
  cate through the radio to whoever was calling.”                 deploy as part of an upgrade to Windows Server 2003.


 committed to do. I can’t talk to them, but I know or I           were going to make entry into their facilities, and we were
 believe that this is what they are doing,’” says Maestri.        going to take what we needed, we were going to provide a
   “What we later found out—that was all the moaning and          list of everything that we did, and we would pay for what-
 groaning that you heard on the local level relative to           ever we took. But we were going to do that, because we
 FEMA—was that wasn’t what they did. They didn’t follow           needed to get food, we needed to get water.”
 the plan. So we all sat here sucking our thumbs for a number       That bit of improvisation—and countless others like
 of days, saying they’re coming to help. And they didn’t come.”   it—helped the EOC and key facilities like area hospitals
   The plan had called for the parish to expect a govern-         remain functional. But the breakdown was devastating.
 ment response within three days. 72 hours. “After that, all      With communications down, the parish simply had no
 our stock will be replenished, because the cavalry will be       way to know that the federal government would fail to
 here,” Maestri says.                                             respond within the agreed-upon window. It wasn’t until
   The cavalry, of course, didn’t come. Fuel supplies ran         that third day had passed that the parish knew it was on
 short. Water and ice became precious commodities. EOC            its own.
 staffers found themselves short of food, clothing and toi-         “You better be creative then,” says Maestri, “because in
 letries as what was assumed to be a three-day event              essence you gotta survive. You have to survive.”
 stretched into a week, and then two.
   “That’s when you get real creative, after that 72 hours,”      Conclusion: The Gathering Storm
 says Capdeville.                                                 By the time you read this report, the official start of the
   Getting creative entailed things like breaking into local      2006 hurricane season will be upon us. Yet, the entire
 Wal-Mart and Sams Club stores to confiscate essential            region remains fragile. Maestri says about a third of Jeffer-
 goods. It was a plan that had come to mind during one of         son Parish lives in temporary FEMA trailers, usually
 the group’s frequent “what-if” sessions, designed to help        mounted on blocks and secured with nylon tie-down
 the parish manage unforeseen calamity. Those prepara-            straps. Many will almost certainly topple in Category 1
 tions paid off, since management at the stores had been          hurricane winds, meaning that even an approaching tropi-
 apprised of the plan and were not surprised when they got        cal storm may require a full evacuation. Yet the storm-
 a call several days after the storm.                             related financial struggles of many families in the area
   “Once we realized that FEMA wasn’t coming, we knew             makes evacuation—with its steep costs in fuel, hotel bills
 where we had to go to get what we needed to survive,” says       and lost work—prohibitively expensive.
 Maestri. “We have the managers’ phone numbers and                  The infrastructure is also at issue. Maestri describes local
 names and where they were going to be. We told them we           electrical service as being held together “with spit, bailing



                                                   13 | Redmondmag.com
St rm Warning
 An Inside Job                      A small business keeps running by taking its show on the road.
 For Thomas Brown, Hurricane Katrina was an inside job. As        transaction,” Brown recalls.
 president of E-Claim.com, Brown’s firm offers Web-based            Brown had exceeded his monthly transaction limit,
 claims processing services to adjusters and agents who           prompting the company to freeze the account. Even worse,
 flock to disaster stricken regions to assess the damage. But     the company demanded three years of tax records before it
 what happens when the disaster is in your own back yard?         would lift the freeze. But that paperwork was in Brown’s
   “The biggest thing for me is that I decided a long time        basement, which had flooded during the storm.
 ago to host my servers in an area that wouldn’t be affected
 by any of this,” says Brown, whose servers reside in a data
 center in Maryland. “Being based in the New Orleans area,
 we certainly had a good deal of experience on what to
 expect and what to anticipate when a hurricane bears down
 on the area. I didn’t leave too much to chance.”
   It’s a good thing, too. The storm destroyed Brown’s Gret-
 na office—”The roof came off,” he says—and forced him
 and his assistant to bounce between locations as they
 struggled to find a place to set up shop.
   “We stayed in Munroe for almost a week, and worked from
 our office there,” Browns says. Unable to return to the New
 Orleans area, the two stayed briefly with Brown’s parents
 before setting up in a renovated condominium in Fort Wal-
 ton Beach, Florida. He would run E-Claim.com out of that
 building for a month.
   With his servers in Maryland and all his files on a laptop,
 Brown’s business didn’t miss a beat. Even his financial          Thomas Brown displays the water damaged remains of
 records, which he maintains using QuickBooks Online, were        his tax records. A few weeks after the storm, Brown’s
 always accessible.                                               credit-card company had frozen his account, citing
                                                                  unusual transaction activity.
 Night at the Improv
 Still, Brown was forced into a series of improvisations. For      “I took a picture of [the damaged returns] and sent it to
 instance, cellular service remained spotty for weeks after       him and said, here’s my tax records,” he says bitterly. “That
 the storm. “One of the things that came out of this was          was very frustrating. I had a really, really hard time dealing
 when you couldn’t make a voice call you could text mes-          with those people.”
 sage somebody. It wasn’t a lot of fun and gave us some            It took a week to finally free up the account, during
 sore thumbs, but it worked,” Brown says.                         which time bills were stacking up. Brown says he was fortu-
   With Sprint still struggling to restore service three months   nate he wasn’t delinquent on his accounts.
 after the hurricane, Brown purchased a cell phone with a
 Florida number to get service. It’s become a permanent part      ‘The Luckiest Guy in the World’
 of his business. “I will always keep [this phone] for that       Despite the dislocations, E-Claim.com ramped up from
 reason. When I go now I know that I have two different           fewer than 200 registered claims adjusters to about 1,000
 cell phones with two different area codes, and one of them       during the weeks after the storm. He also scaled up from
 will always be working.”                                         four servers to seven. He credits his hosting company,
   Disrupted postal service and Brown’s constant movements        Atlantic Computer Systems, for making sure his service
 made payments an issue. Brown says he resorted to paying one     kept pace.
 programmer “through PayPal” for a time. Most of his regular        “The thing about what we do is, it’s a Web-based manage-
 bills had already been set up for automatic payment via credit   ment application for catastrophe adjusters,” says Brown.
 card, preventing missed payments that could disrupt services.    “When these things happen, they’re working 24x7, which
   Customers, unable to send checks, were asked to make cred-     means we have to be up 24x7. And if they can’t reach us,
 it card payments at the E-Claims.com Web site. That worked       we’re in trouble.”
 fine, until the credit card company stopped his account            Just as important, prior to the 2004 hurricane season,
   “Two or three weeks into the thing I get a call from a         Brown had used RoboHelp to update documentation for his
 merchant account saying they can’t authorize a $10,000                                                Continued on page 15



                                                   14 | Redmondmag.com
St rm Warning
                                                                         That holds dire consequences for the next evacuation.
  Continued from page 14                                               With Katrina, old and infirm patients could be sheltered in
  Web-based applications. That eliminated a flood of tech              hospitals, where they had access to care, supplies and
  support calls—calls he would not have been able to han-              power for medical devices. Those patients will probably be
  dle after Katrina.                                                   forced to evacuate ahead of the next storm.
    “I probably fielded 10 support calls from adjusters,”                “We have to move those people completely out of the
  Brown says of the weeks following the storm. “The previ-             area. And some of those people are going to tell you, and
  ous year, even though we had fewer adjusters using the               they’re right, that that’s a death sentence,” Maestri says, his
  system, we were getting calls every day.”                            voice rising. “They can’t make the trip, even a hundred
    Brown says E-Claim.com is on track to service 2,000                miles. Physically they simply cannot do it.”
  adjusters this summer and is adding hardware to meet the               The grim calculus of evacuation demands that Maestri,
  demand. But like so many in the region, he’s just happy              Capdeville and other managers at the Jefferson Parish
  to have survived.                                                    EOC shuffle the deck, re-tune their models and reassess
    “We’re back in our house, we’re back in the office, and            how they conduct the next evacuation. Capdeville has been
  life is slowly returning to normal,” he says. “Every day I           tweaking the computer-based evacuation models, lowering
  look at it. It’s tough, but I look at what’s happened to             her assumptions about road capacities and taking into
  everyone else, and I feel like I’m the luckiest guy in               account issues like a higher percentage of vehicles becom-
  the world.”                                                          ing disabled on roadways.
                                                                         At the end of the day, however, Maestri is realistic. There
 wire and scotch tape.” He warns that many utilities in the            is only so much that technology can do. “I think one of the
 area are now operating under bankruptcy. And then there               problems is that we, 21st century Americans, are totally
 is the problem of the devastated medical system.                      dependent, in ways they don’t even think about, on tech-
   “If there is a piece of the critical infrastructure that is still   nology,” Maestri says.
 paralyzed, limping, I don’t know how to describe it, it’s the           “There is a picture that we have that somebody snapped,
 medical systems. We have gone from 11 major hospitals                 one of our own photographers, of the CEO and chairman
 functioning to three. A million two hundred thousand peo-             of the board of a Fortune 50 corporation who lives here,
 ple are now dependent on three hospitals, in essence,”                sitting on the curb eating an MRE (meal ready to eat),”
 Maestri warns. “We had a level 1 trauma center—it’s gone.             Maestri says. “I mean, he has planes at his disposal and
 Charity Hospital of Louisiana has closed. We had two                  God knows what. And he’s sitting at a curb eating an MRE,
 major medical schools functioning—Tulane and LSU.                     because all of his money, he couldn’t get to it.”—
 Those doctors, those students are now no longer here.
 They’ve moved out.”                                                   Michael Desmond is Redmond’s Editor at Large.

                                                                       Vicky O’Brien helps credit union customers cope
  Credit Where Credit’s Due                                            with the aftermath.
  When the largest disaster ever to strike the citizens of Jef-          “You’ve got the money in the bank, but if you can’t get it,
  ferson Parish occurred on Aug. 29, 2005, Vicky O’Brien               you can’t buy the groceries, you can’t buy the medicines, or
  knew that her credit union would play a critical role in the         whatever. If your debit card strip is not being picked up,
  recovery. As president of the $50 million Jefferson Parish           you might have millions in the bank but you can’t get it.”
  Employees Federal Credit Union, she realized that many of              For O’Brien, the message was clear. The credit union had
  her 10,000 members would need cash—and quickly.                      to find a way to put cash into the pockets of its members.
    “A lot of people don’t have credit cards in their pockets—           “Any member with a checking account with the credit
  a lot of our members don’t. And they couldn’t have paid for          union, we allowed all their accounts to overdraw so they
  their hotel if they didn’t,” recalls O’Brien.                        could have access to cash, because in this immediate
    Hurricane Katrina pulled down the financial infrastructure         area cash was an issue,” O’Brien says. “ We had over 300
  of the entire New Orleans area. ATM machines were dark.              people that overdrew their accounts. We had one member
  Stores could not handle credit or debit card transactions.           say he evacuated 18 people because of the cash we
  Even cash was made scarce by the closure of the New                  allowed him to have.”
  Orleans Federal Reserve. It posed a challenge for residents            The effort was complicated by the closure of the Federal
  and rescuers alike.                                                  Reserve in New Orleans. Like everyone else in the region,
    “No vendor that was open in this area for emergency per-           the credit union needed to get its hands on cash.
  sonnel would take anything but cash,” O’Brien says.                    “We were borrowing cash and we were getting cash from
     For Dr. Walter Maestri, director of the Jefferson Parish          everywhere. We were taking cash from our other branches
  Emergency Management Department, the cash crisis had                 and moving them in,” O’Brien says.
  serious repercussions.                                                                                     Continued on page 16

                                                       15 | Redmondmag.com
St rm Warning
 Continued from page 15                                         severe. But the credit union operations on the second and
   Maestri helped coordinate a larger effort to allow local     third floors escaped nearly unscathed.
 bank customers to withdraw cash from accounts. “We had           “We looked like we went out to lunch,” says O’Brien of the
 to send federal marshals to branches of the federal reserve    evacuated offices. “We were just so lucky.”
 to pick up—literally pick up—millions of dollars [of cash],      The Jefferson Parish credit union was also fortunate to
 transport it back, and set up a location where you could       have a trusted partner in USERS Incorporated. The credit
 walk in [and] get some kind of a telephone line,” he says.     union had deployed the company’s DataSafe core process-
 “And that way you could go and get money.”                     ing system in-house and maintained a hot site at USER’s
                                                                data center in Valley Forge, Pennsylvania. When the storm
 Stay at Home                                                   disabled the local infrastructure, O’Brien was able to shift
 The Jefferson Parish Employees Federal Credit Union, estab-    operations to the remote site.
 lished in 1959, has 10,000 members and prior to Katrina          Just as important, USERS helped O’Brien line up resources
 reported cash deposits of more than $50 million dollars. The   to keep the credit union on track. “I would come up with
 institution has seen its deposits grow since the storm, as     an idea and throw it out to them and they would help make
 members gathered funds to rebuild homes and businesses.        it happen. It gave me a large group of people at my dispos-
   “We were a $50 million credit union, but we’re a $70 mil-    al when I didn’t have anybody here.”
 lion credit union now,” says O’Brien.                            For instance, the company put O’Brien in touch with peo-
                                                                ple at Continental Federal Credit Union, who set her up
                                                                with a laptop until she could return to Louisiana. Another
                                                                credit union, La Capitol, provided temporary office space in
                                                                Baton Rouge for O’Brien’s employees.
                                                                  “There’s been an overwhelming amount of generosity in
                                                                the credit union community,” says O’Brien, who also credits
                                                                her vendor relationships. “If you are a small financial insti-
                                                                tution or a small company, the best thing you can do is
                                                                have good third-party partners. That’s what helped us
                                                                recover and recover well.”
                                                                  Today, the Jefferson Parish Employees Federal Credit
                                                                Union hosts its applications on servers at the USERS Incor-
                                                                porated data center in Valley Forge. The decision to move
                                                                away from local operations was easy.
                                                                  “We were investigating the On-Line [service] and we
                                                                decided to definitely make that move,” says O’Brien, who
                                                                sees benefits beyond disaster recovery for her bank. “When
                                                                you are a small shop you can’t keep up with buying the
                                                                technology anyway.”

                                                                ‘The Bayou Is What You Get’
                                                                Outsourcing applications makes even more sense when
                                                                you consider the struggle that area businesses face in
                                                                keeping employees. Like businesses across the stricken
                                                                Gulf Coast, the credit union has lost a lot of skilled labor
                                                                since the storm.
 The Jefferson Parish Employees Federal Credit Union,             O’Brien describes one employee who left the area for
 located on the second and third floors of the Yenni Build-     good. “The ordeal was so much she resigned and is moving
 ing, was spared the damage of the stricken upper floors.       to Georgia with her brother, to have more family support.
                                                                When you look at your management team, you’ve got to
   The storm disrupted bank operations, but O’Brien says her    consider the children of the team and the jobs of the
 firm was lucky. The bank’s processing offices are in the       spouses. In this area a lot of people work in oil compa-
 Yenni Building, which had its roof torn open during the        nies—a lot of people moved to Houston. I lost two
 storm. The resulting water damage to the upper floors,         [employees] because of that.”
 including to a parish data center on the ninth floor, was                                           Continued on page 17




                                                  16 | Redmondmag.com
St rm Warning
 Continued from page 16
   Many other employees simply didn’t return after the evac-
 uation. The reason: schools. Or in the case of Jefferson
 Parish, the lack there of. “I never thought about schools—I
 never gave that a thought,” O’Brien says.
   Even as many families have stayed out of the region, oth-
 ers are determined to rebuild. O’Brien is one of them. A 33-
 year resident of Jefferson Parish, she had evacuated just
 once prior to Katrina, and understands why so many others
 were reluctant to leave as well. And like so many in the
 region, she remains fiercely loyal to New Orleans and the
 surrounding area.
   “We’ve been a city a long time. We aren’t going any-
 where,” O’Brien says. “When you are talking to me, the
 bayou is what you get.”
   Now, she is committed to helping people stay. “We don’t       Businesses like retailer Casual Home Furniture have been
 know how to separate from our jobs, because we know peo-        forced to deal with labor shortages, even as they do busi-
 ple are depending on us. We can’t. I asked a friend, ‘How       ness out of a makeshift tent.
 can I be doing this?’ And she said, ‘Because you have to.’”
   Ironically, the best way for O’Brien to help keep people in     “Now I’ll always go. What I learned was I need to talk to
 the community is by making sure she’s among the first out       the vendors that support the credit union to keep it up and
 when the next storm threatens.                                  running. And I can’t do that without communications.”




                                                  17 | Redmondmag.com

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:5
posted:12/20/2011
language:English
pages:18