How Jefferson Parish planned for and
responded to the worst disaster in
U.S. History, and the lessons IT
professionals can learn from it.
IT Influencer Series
Storm Warning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1
Cascading Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2
3. Communications Collapse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3
4. Tulane’s Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3
Planes, Trains & Automobiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4
Home Away from Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4
5. ‘We couldn’t find them’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6
6. Pain at the Pumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6
7. Data Center Two-Step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
8. The 3-Day Assumption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
9. Lessons in the Wind . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9
10. A Personal Toll . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
‘I Had a Pretty Good Idea’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
Out of House . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11
11. IT Lessons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11
12. Work to Do . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11
13. Respect the Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12
14. Missions Critical . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12
Behind the Storm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12
The Best Laid Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13
15. Conclusion: The Gathering Storm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13
16. An Inside Job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14
Night at the Improv . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14
‘The Luckiest Guy in the World’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14
17. Credit Where Credit’s Due . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15
Stay at Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
‘The Bayou Is What You Get’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
How Jefferson Parish planned for and responded to
the worst disaster in U.S. history, and the lessons IT
professionals can learn from it. STORY AND PHOTOS BY MICHAEL DESMOND
T he Jefferson Parish Emergency Operat-
ing Center (EOC) squats over the low,
grassy plain south of the Mississippi River.
It’s a brutish, four-story block of a building that
sits in the shadow of a massive radio tower. A
brick smokestack stabs at the Louisiana sky.
great crescent, from the shores of Lake
Pontchartrain in the northwest to the Mississip-
pi River-bounding communities of Marrero and
Gretna to the south and southeast. Inside the
EOC building, emergency personnel worked to
evacuate, assist and rescue citizens across this
If the EOC building looks like a bomb shelter, expansive swath of earth.
it’s because it once was. On Aug. 29, 2005, that The Jefferson Parish EOC was well equipped
structure survived the greatest bomb ever dropped for the task. There were hardened landlines,
on the United States: Hurricane Katrina. 800MHz radios, and cell and satellite phones
Nearly 100 souls had crammed into the forti- tapping multiple networks. There was a fully
fied structure on the morning of Katrina’s land- redundant computer network, a fleet of remote
fall. Most came from across Jefferson Parish. sensors and cameras, and a detailed disaster plan
With a population of more than 450,000, the promising a multi-layered government
parish cradles the city of New Orleans in a response. And, of course, there was a system of
1 | Redmondmag.com
St rm Warning
canals, levees and pumping stations designed to channel shelters with supplies and backup power. His team was in
flood waters out of the area. place at the EOC. Key personnel stood by in neighboring
Within hours of Katrina coming ashore, every one of parishes to provide day one communications and recovery.
those systems would fail catastrophically. “Up to the breaching of the levees, this was a perfect
This is the story of the Jefferson Parish EOC and the model,” Maestri says from his office on the fourth floor.
people who struggled to run it as Katrina laid waste to the “We knew what we thought could happen. We knew what
region. Despite the meticulous planning, robust systems
redundancy and heroic efforts of parish personnel, the Jef-
ferson Parish EOC team—indeed, the entire region—
struggled to accomplish the most basic tasks. Nine months
later, it is not at all clear that the parish and its businesses
are any more prepared for another major event than they
were on that last Sunday of August 2005.
Dr. Walter S. Maestri had been director of the Jefferson
Parish Emergency Management Department since 1996.
The Sunday before landfall, he and nearly 100 parish
staffers packed into the bunker-like EOC building, with its
thick concrete walls and windowless rooms. Katrina boiled
to a frightening crescendo just 200 miles to the south.
Now a Category 5 storm, among the most powerful ever
recorded, it steamed directly toward New Orleans.
But when the storm weakened and turned unexpectedly to Jason Phillips monitors water-level and pump-station status
the east just hours before landfall, Maestri was confident. from his workstations at the Jefferson Parish EOC in Marrero.
More than 90 percent of parish residents had evacuated. we had to deal with. We had prepared the plans, spread the
Those unable to evacuate were housed at hospitals and message. Everything was according to plan. No surprises,
Maestri’s voice trails off. He sounds almost reluctant to
go on. “Then, of course, the storm comes in—and things
The power failed first, victim of the thrashing winds that
toppled poles, tangled lines and threw down transformers
and other gear. The EOC, like the parish’s two data cen-
ters, is supported by diesel generators that kicked in as
soon as service was disrupted. In the EOC, as well as at
local hospitals and other critical facilities, the lights stayed
on and the computers kept humming.
Out in the field, the parish’s network of nearly 200
radio-based remote sensor stations switched to battery
backup power. Jason Phillips, supervisory control and data
acquisition manager for the parish, watched as the real-
time feeds displayed on his twin monitors began to drop
off one by one. Most batteries lasted from 45 to 120 min-
utes before going dark. Wind meters, shorn of their direc-
tional vanes, stopped transmitting telemetry on wind
speeds, while many lakeside sensors kept working until
the waters swallowed them. The last sensor quit with a
reading of 29.6 feet.
“The ones we lost went underwater. Where we had them
mounted we never would think the water on the lake
Nearly 100 people crammed
into the bunker-like EOC building
would get that high,” Phillips says. “But it did.”
during Hurricane Katrina. Voice and data communications were next to fail. When
the power quit, general Internet access went with it.
2 | Redmondmag.com
St rm Warning
Phone service throughout the area faltered or failed
entirely within hours of landfall. At the EOC, staff mem-
bers could receive calls throughout the storm, but were
unable to dial out. It would take five days for BellSouth to
restore service to the EOC, one of the highest-priority
service points in the parish.
Of course, the parish disaster plan anticipated these out-
ages. Staffers had been issued cell phones from a diverse
set of service providers to help ensure voice communica-
tions. “Verizon, Sprint, Nextel, Cingular, T-Mobile,” The gutted interior of Casual Home Furniture, which
Maestri counts off in rapid succession. took on five feet of water when New Orleans flood
waters flowed down Airline Highway. The store today
In fact, most service failed as cellular towers sustained does business from a large tent in the parking lot.
damage, switching stations lost power and call volume
swamped the networks. Even the dozen or so EOC-oper- na on the 21-story Galleria building blew away. That sev-
ated Motorola satellite phones faltered during the storm, ered links to police, fire, hospital and key emergency per-
their signals blocked by thick cloud cover and heavy sonnel situated on the north side of the Huey P. Long
rain—though they would prove invaluable in the days Bridge. When crews rushed to fix the damage after the
that followed. storm, they discovered that T1 service between the Galle-
The one system that worked—the parish’s nearly 2,000 ria and the EOC was dead as well.
radios—suffered a crippling blow when the repeater anten- Like an army slicing through the weakened flank of an
Six months after a glancing blow from Hurricane Katrina
flooded New Orleans, the city’s downtown is a work in
progress. Street lights remain dark and power in office
buildings fails regularly. A half block from the Superdome,
its savaged roof now repaired, stands the dark monolith of
the Hyatt Hotel. The building’s façade is an ugly patchwork
of blown out windows and white plastic sheeting, a caustic
visual reminder of just how much work remains to be done.
Paul Barron, interim CIO of Tulane University, has a birds’
eye view of it all from his 14th floor office on Poydras
Street, right across from the Superdome. Like New Orleans
itself, Hurricane Katrina nearly finished off the 172-year-
old university, Barron says. “During the first 60 days, it was
unclear if Tulane was going to survive or not.”
Calamity struck quickly. “That Friday night I was at a pre-
season Saints game, and people were talking a little bit
about the hurricane. But nobody was talking about evacu-
ating,” Barron says.
Certainly not Tulane’s IT department. Tim Deeves, director
of network services, recalls kicking off full system backups
Friday evening. It was a fortunate thing, in hindsight, that
the storm struck after the weekend—it gave the IT staff More than half a year after the storm, battered facades,
time to fully back up their systems. That work went on intermittent power failures and non-working traffic
until Sunday at 2 p.m., when Deeves himself shut down the lights plague downtown New Orleans.
school e-mail system.
“I was a little bit worried,” Deeves says. “At one point I Continued on page 4
3 | Redmondmag.com
St rm Warning
Continued from page 3 They stopped at the Jefferson Parish operations center set
was the only person in the building, I was worried about up by the West Bank Expressway. There, McGinity and his
the elevator losing power. I said, ‘Chris, if you don’t hear team climbed into a deuce-and-a-half troop truck, along
about me by 1 o’clock, call public safety and tell them to with a contingent of heavily armed police. The transport
come get me out of the elevator.’” rumbled past the nearly-empty Superdome and circled the
The plan called for a courier from the Tulane security cen- building on Poydras Street, looking for an approach where
ter to accept the tapes from Deeves and transfer them to a the water was shallow. National guardsmen, patrolling the
nearby security center site. But when the courier failed to streets on Sea-Doos, looked on as McGinity dismounted,
show up, Deeves was told to leave the tapes out in the waded to the entrance and reached into the murky waters
14th floor hallway for pickup. When the courier finally for the lock at the bottom of the doors.
arrived, he found the doors locked—the building staff had The walk up to the 14th floor was uneventful, but the
fled. Those tapes would spend more than a week stewing in team found the entrance to the data center locked tight—
the late-summer Louisiana heat. the security protocol that should have opened the door
As it turned out, it’s a good thing the tapes never got to when power went out had failed. One group searched near-
the security center, which ended up largely underwater. by offices for a key, while the other began hauling down
“The interesting thing is the security center is four and the backup tapes found in the outer hall. A few minutes,
one-half blocks from here. That was not very smart,” Barron McGinity and his guys were in the data center room, tear-
says. “We lucked out in terms of getting our tapes, because ing open disk arrays and servers in the sweltering heat.
the security center is much lower than we are, and there “We pulled disk drives and the tapes, up until the point
was water. We got lucky.” where we just couldn’t pull anymore stuff because time had
run out,” McGinity says, estimating that the air in the
Planes, Trains & Automobiles outer hallways was about 110 degrees.
With the region devastated, Tulane President Scott Cowen The tapes in the boxes were a priority. After that, the drives
made a decision—delivering payroll to school employees from the systems were gathered in case they needed them for
would be the first mission. With credit card and other hardware being set up remotely. “We left one machine half
electronic transactions unavailable for weeks after the open with a board almost out of it,” McGinity says.
storm, people needed immediate access to cash. To provide By nine that night, the team was on a plane out of Baton
that, the university needed to recover the tapes Deeves Rouge on its way to Houston. “Well, everything is going
had left on the 14th floor of the Poydras Street building. according to plan, right?” McGinity jokes, drawing a laugh.
It fell to Rick McGinity, “We get our tapes and we take it to the hardware. We just
director of operating didn’t know where the hardware was.”
system and database
services, to lead a Home Away from Home
group into New Orleans The hardware, it turned out, was in Vorhees, New Jersey.
a week later to recover Barron had arranged for mainframe space and resources
the data. with Sungard’s offices in Vorhees, and McGinity and several
What follows reads others flew up with the tapes in a chartered jet Wednesday
like something from afternoon. The group worked frantically to get its
the script of the movie applications set up on a logical partition (LPAR) of an IBM
“Planes, Trains & Auto- mainframe operated by availability services provider
mobiles.” McGinity and Sungard. Twenty-four hours later, at 8 p.m. Thursday,
his team couldn’t fly Tulane’s payroll system was up and running. The university
into New Orleans Air- Tulane’s interim CIO Paul was back in business.
port—that was under Barron says the school erred “They had an environment, DASD (direct attached storage
military control—so when it moved its data center device) and things, running. They were able to carve up the
they landed in Baton downtown. resources and things we would need running up there,”
Rouge, picked up two McGinity recalls. “We just did some tweaks that morning to
staff members and drove to one of their homes. There they get everything in. There are always differences, but they were
picked up a second car. Once on the interstate, they negligible enough that Mary’s team was able to take over.”
dropped in behind what McGinity describes as “a whole As head of the software application services team, Mary
stream of ambulances going about 100 miles per hour.” The Walsh configures and manages the payroll system. In the
group talked its way past several roadblocks. Continued on page 5
4 | Redmondmag.com
St rm Warning
Continued from page 4 • Ensure that recovery plans are implemented uniformly
days following the storm, President Cowen had made across all departments, not just the university in general.
restoring payroll operations a top priority. University • Create a team of first responders to handle assigned
employees had to get paid, he reasoned, if the university tasks, such as preservation of irreplaceable research assets
was going to survive. like lab specimens.
With the software quickly in order, Walsh turned to maxi- • Require submission of cellular phone numbers and mul-
mizing the cash income of Tulane’s employees. She turned tiple e-mail addresses for all staff.
off income deductions and stopped monthly payments for • Create listserves using Tulane and personal e-mail
things like campus parking. Direct deposit payments went addresses in order to broadcast to the community.
out to employees, but the complete suspension of mail • Create a remote call center outside of the 504
service in New Orleans and the prolonged evacuation of area code.
staffers to points unknown often meant that paper checks Challenges will remain. For instance, Tulane had moved its
failed to find a home. data center into the 22-story Poydras Building in July, less
“The hard copy checks—that took a lot of perseverance. than two months before the storm struck. Prior to that, the
We had to track people down,” Walsh recalls. “It was an data center had been located in the university’s uptown
ongoing effort.” campus, which has recovered more quickly than the area
The effort paid off. In January, the campus reopened to around the Superdome.
students. Barron says more than 80 percent of students “This had been the corporate offices for some oil company
returned, despite many spending the previous semester at and they had a data center here,” says Barron, “so we
other schools. While many problems and challenges remain, decided to use the data center [facility]. Was it a good
the school that very nearly died in the rising waters of Kat- decision? Probably not. Will we keep it here? I think so.”
rina is back on track. In fact, the IT staff is planning to Barron says if the university could do it all over, the data
migrate the aging mail server infrastructure to Microsoft center would probably have ended up in Covington,
Exchange 2003 once classes let out for the summer. Louisiana, on the north shore of Lake Pontchartrain. “It is
In the meantime, school administrators are working to generally viewed as safe from any storm surge,” he says.
beef up Tulane’s readiness for another storm. Among So will Tulane move its data center again?
the changes: “Sunk costs have a big impact on people, even though it
• Immediately review operations with an eye toward shouldn’t,” Barron answers. “I don’t see us in the short
short- and long-term improvements in the school’s hurri- term doing that, but we may in the longer term.”
cane response. — M.D.
enemy, Katrina had cut off the EOC from the East Bank when he learned that the freshly mounted antenna had
communities pinned between Lake Pontchartrain and the been disconnected by FEMA personnel.
Mississippi River. “We had an instance where we got our antenna back up
These were some of the hardest hit areas in Jefferson and someone at FEMA came in and put their communica-
Parish. Flooding devastated portions of Metairie, particu- tions up, because of the operation that was going on [near
larly in the south, where floodwaters from New Orleans
followed Airline Highway into the parish. Powerful north
winds rotating off the retreating storm battered lakeside
communities. Vital resources like the New Orleans airport
and the I-10 highway linking the area to Baton Rouge
were a top priority.
Technicians rushed to replace the antenna and set up a
point-to-point microwave link between the EOC in
Marrero and the Galleria building—about an 8-mile
shot. That enabled 800MHz radio communication
parish-wide for first responders, but less than two days
later the link failed again.
As emergency coordinator for Jefferson Parish, Tom
Rodrigue is responsible for keeping communications up
during a storm. When Katrina came ashore, he was ready Katrina tore open the Yenni Building roof, disabling the data
to battle wind, water and line damage. But he was stunned center and damaging the upper floors.
5 | Redmondmag.com
St rm Warning
the Galleria site]. At which time the sheriff went back up “It was the pony express—people going from place to
there and put [on] armed guards, took their antenna down, place carrying messages.”
put ours back up, and says, ‘Anybody tries to take it down
...’” Rodrigue pauses as a co-worker finishes the thought, ‘We couldn’t find them’
“… they are going to shoot you.” Actually, Jefferson Parish was fortunate. Unlike surround-
Even with its radio service restored to the East Bank, the ing communities, it could radio state agencies.
EOC struggled to communicate with state, federal and “St. Bernard Parish, for whatever reason, couldn’t reach
other parish agencies. Jammed frequencies and incompati- the state,” Maestri says of the neighboring parish to the
ble radio networks stymied cooperation—Rodrigue east, which was utterly devastated by flooding. “So what
describes the system at the time as “inoperable.” Those dif- happened was St. Bernard Parish could reach us and we
ficulties enhanced the isolation created by rising floodwa- could relay messages to the state. We were relaying mes-
ters and broken roads. sages for multiple parishes.”
“We had to resort to runners,” says Carolyn Capdeville, The city of New Orleans, however, simply dropped off
Jefferson Parish coordinator for emergency management. the map. “They abandoned their EOC and set up their
Pain at the Pumps
In New Orleans, studies have shown that structural flaws Tom Rodrigue says similar issues plague the design of the
and poor maintenance most likely caused the breeches that canals that stretch miles inland from Lake Pontchartrain.
flooded 80 percent of the city. Perhaps no single element When barriers like the 17th Street Canal fail upstream from
was more devastating to the area than the Mississippi River the pumping station, there’s nothing to prevent the lake
Gulf Outlet (MR-GO), a shipping shortcut designed to cut 37 from rushing freely through the gap.
twisting miles out of the trip from the Port of New Orleans’ “When you think about it, they allowed the lake to intro-
Inner Harbor Navigation Channel to the Gulf of Mexico. duce itself all the way into the city, by having the pumping
station not at the lake, but back here,” says Rodrigue, stab-
bing a finger at a map of New Orleans, at a point about two
miles south of the lake. “The lake goes all the way into here,
then you pump the water through that canal to the lake.”
In Jefferson Parish, the lakeside pumps are situated right
on the shore. The problem was, they weren’t manned.
When parish President Aaron Broussard invoked the so-
called Doomsday Plan on August 28, the day before Katrina
struck, it required that all 236 pump operators evacuate to
Washington Parish, about 100 miles north. Lacking a safe
place to house the workers, the parish ordered its person-
nel out and shut down the pumps. The result: Flooding
developed into major events, as waters pooled in Metairie,
Kenner and elsewhere.
The parish learned a hard lesson that day. Broussard has
since announced that pump operators will stay local, and
that safe houses will be built at pump stations to house
Rodrigue says a flawed canal system helped create the workers during storms. In addition, cities in the parish,
disastrous scenes, like this one two blocks from the such as Kenner, are cross-training personnel to take over at
17th Street Canal breech. the pumps should parish staff be unavailable.
The result is a more robust—and more realistic—emergency
Opinions differ, but many engineers believe the MR-GO plan. The original plan simply abandoned one of the most
channeled gulf waters into a fast-moving surge that under- vital defenses of the parish—its pumping stations—and put
mined levees and led directly to the catastrophic flooding thousands of residents and emergency personnel at risk. The
in New Orleans’ 9th Ward and St. Bernard Parish. In short: lesson: Make sure your emergency procedures, however well
MR-GO may have been a gun-barrel thrust into the ribs of intentioned, don’t do more damage than they avoid.
the metropolitan area. — M.D.
6 | Redmondmag.com
St rm Warning
net. You’ve got to take it
this way or you are going to
lose us. There was no way
to get to E Team.”
The EOC team finally
convinced the state to
accept requests by phone.
But with cell and landline
service both out, even that
was dicey. “Eventually the
only communications sys-
tem to stay up during the
entire ordeal was satellite
Rodrigue. “Had it not
been for that we wouldn’t
have been able to make
any requests to the state.”
Parish workers soon
learned that cell phones
with out-of-state area codes
actually worked—even in
Canal breaches in the Lower 9th
Ward and at the 17th Street the stricken region. In the
Canal (shown) devastated large weeks after Katrina, team
sections of New Orleans. Many members began using cell
areas remain uninhabitable.
phones with numbers based
in Texas and other states. It
was just one of many
improvisations that helped
keep critical systems and
temporary operating facilities in different buildings down- communications functioning during the crisis.
town, and we couldn’t find them. Their towers went down, “It’s not the technology that saves anybody, it’s the peo-
their radio systems went down, or whatever, and we could- ple who use the technology,” Capdeville says. “That’s
n’t find them. missed a lot of times, the human part of this.”
“Communication interoperability is a tremendous prob-
lem, and nobody learned the lesson from 9/11. That Data Center Two-Step
problem, through Katrina, still exists,” Maestri warns. Jefferson Parish maintains two discreet data centers, locat-
“Our challenge is not interoperability, it’s operability. ed more than 10 miles apart. Identically configured, each
The commitment has been made that everybody will includes a single IBM AS/400 iSeries midframe server,
share the same system.” which hosts financial, payroll, utility billing and other
The EOC staff did get one pleasant surprise: Cellular applications. The servers connect to a 2TB SAN and are
voice service collapsed within hours of landfall, but text- linked to each other and the EOC via frame relay. While
messaging services stayed up. EOC personnel adapted the two centers house the same equipment, they serve dif-
quickly, texting requests and responses, and using text mes- ferent application sets.
sages to set up meetings. For several days after the storm, The iSeries system in the East Bank data center, located
text messaging was the only reliable way to communicate on the ninth floor of the Joseph S. Yenni Building in Jef-
with people beyond that of the parish radio network. Still, ferson, just a mile north of the Huey P. Long Bridge, hous-
some agencies refused to adapt. es utility billing operations. That data center also includes
“The state office of Emergency Preparedness, they have a Linux server running an Oracle GIS database for map-
a program called E Team that works on the Internet, and ping and geo-location services. The West Bank data center,
they wanted us to go over that,” Capdeville says of requests housed in a building on a parish campus in Gretna, powers
her team tried to place with the state. “And we kept holler- critical financial and payroll applications. There are also 35
ing at them on an 800MHz radio that we don’t have Inter- Windows servers hosting Exchange, Internet Information
7 | Redmondmag.com
St rm Warning
Server, DNS and applications for several departments. The when Katrina washed ashore. During the storm, high winds
parish operates about 1,400 total Windows clients. ripped the large air conditioning units off the top of the
“Ever since the early ’90s the plan was to have two data Yenni Building, tearing open the roof. Water from the
centers with redundant capabilities,” explains Ridley storm and from ruptured air conditioning chiller pipes
Boudreaux, director of Electronic Information Systems for poured into the upper floors of the building. “Even if you
the parish. “We didn’t set them up redundantly, it’s not like have generator power—with no air conditioning, forget it,”
a hot situation where I can instantly say now we’re doing Boudreaux says. “When you have something as catastrophic
our payroll on the East Bank. But we always bought identi- as the roof coming off and water cascading through the
cal iSeries machines—or AS/400s—with enough capacity building, a whole lot isn’t going to stay operational.”
to run all of the applications on one of them.” Even with the East Bank site disabled, Boudreaux felt
That decision, which helped the parish shave equipment, lucky. The storm had largely spared the West Bank facility,
software and management costs, got put to a severe test where critical financial and payroll applications were host-
The 3-Day Assumption In Jefferson Parish, emergency planners turned to com-
puter simulations to break some of those assumptions. As
How could so many IT organizations fail to plan for the part of an ongoing pre-storm educational push, the parish
wholesale devastation of Hurricane Katrina? Tulane Univer- created a video showing residents what they could expect
sity in New Orleans, for instance, had a disaster plan that from a major storm.
called for backup tapes to be moved to a secure center just “We used the technology, particularly the computer simu-
four-and-a-half blocks away. That facility ended up under- lations and the model, to heavy, heavy use,” says Maestri.
water. Jefferson Parish locally hosted infrastructure like e- “The film … actually shows the major commercial area of
mail, Web and application servers, and lost service to all the East Bank of Jefferson Parish—Metairie, Lakeside Shop-
three for a time. ping Center, the biggest mall in metropolitan New Orleans,
Paul Barron, interim CIO at Tulane University, calls it “the underwater. We showed them what it is going to look like if
three-day assumption.” New Orleans hadn’t experienced a this storm comes. And don’t be surprised if water is to the
direct hit since Hurricane Betsy in 1965. Local planners, roof. We’re showing 14, 15, 16 feet of water.”
having seen dozens of storms strike Florida, Alabama, Texas Others learned the lesson from afar. As president of the
and elsewhere, assumed recovery from a major storm would Jefferson Parish Employees Federal Credit Union, Vicki
parallel those events. People would evacuate, wait out the O’Brien was supposed to be at the bank’s headquarters to
storm and return two or three days later to clean up, Bar- oversee operations during a hurricane. She happened to be
ron says. on vacation in California when Katrina struck.
“That mindset pervaded not just the university, but the “In hindsight that was the very best place in the world for me
whole community and probably the region as well,” Barron to be, because I was not stopped by communications. I had to
explains. “And as a result, our disaster planning was more make decisions based upon news media, but I could call my
about getting ready for a hurricane that wasn’t going to processors, I could call all of my third-party vendors,” O’Brien
actually come, than actually preparing for what we do when says. “I had a phone line. I had a computer, I had Internet.
the hurricane gets here.” Nobody here had any of that. But I couldn’t talk to the
The difference, of course, was geography. When the [Jefferson Parish] EOC. I couldn’t get through to the parish.”
levees broke, New Orleans and other areas filled up like a Hurricane Katrina put an end to the three-day assump-
bathtub. IT staff that had evacuated could not re-enter the tion. All across the region, IT managers are beefing up
city. Rising waters completely disabled the infrastructure. remote data operations and reassessing emergency plans.
What would have been a three-day evacuation in Miami or Jefferson Parish has moved Web serving out of the area and
Houston stretched into a three-month ordeal in New may relocate e-mail operations as well. Tulane has con-
Orleans. Tulane, for instance, didn’t regain access to its tracted Sungard for availability and business continuity
downtown data center until November—and that was only services to allow smooth failover for the next storm. And
with generator power. O’Brien is applying a few lessons of her own.
“We were operating on a series of false assumptions,” Bar- “Some of the things that in planning stages sound like
ron says. “If the hurricane hit, we would have access to the they will work perfectly, in real life, when you have a com-
city and campus buildings—not true. Utility power would munications issue, it doesn’t work,” she says. “We had put
be restored, and the limited emergency power would be together a booklet on hurricane preparedness for our credit
sufficient—not true. Communications, particularly emer- union members, to tell them what we were doing and how
gency communications, would work—it was virtually impos- to handle things in the event of a storm. After Katrina, I
sible to use your cell phone.” threw them all away.” — M.D.
8 | Redmondmag.com
St rm Warning
ed. “We were extremely fortunate that the building on the and immediately started working on payroll, which was
West Bank came through with flying colors. The generator due the following Friday.”
worked. Power remained to that building so that system Boudreaux says that the AS/400-based iSeries platform
stayed up,” he says. eased migration of East Bank data center applications to
Daily procedure calls for the backup tapes produced at the sibling midframe in Gretna. In just a few hours his
each data center to be transported to the other site— team had dumped the bits onto the working AS/400 and
ensuring that data will survive a fire or catastrophic event. was up and running. “You take a backup of Exchange and
With Katrina’s approach, the disaster plan called for up-to- try to put it on a box that’s not identical to the one you
date backup tapes to be transported from the two data cen- took it off of,” he says with a laugh. “The only issue was
ters to the hardened EOC site in Marrero. But the tech the connectivity from the client. People who are mapped
responsible for the transfer was running late, trying to get to the East Bank location, for example, would have to be
his family out of the area. The tapes ended up in Houston. mapped to the West Bank location when East Bank goes
“It probably worked out better that way,” says Boudreaux, down. That was a minor issue, but it made us run around a
who was able to have the tapes shipped to his temporary little bit.”
location in Baton Rouge. When it became clear that the Boudreaux himself was on the north shore of Lake
Gretna site and its critical applications had ridden out the Pontchartrain when the storm hit. Rather than return to
storm, Boudreaux’s team got to work. the stricken parish, he headed west “through downed trees
“We got one of my people and somebody from payroll and power lines” to Baton Rouge, where the parish was
operations in that building by Thursday after the storm establishing an emergency seat of government. Katrina’s
Lessons in the Wind
Interoperate: Following Katrina, Jefferson Parish experi- it took creative and dedicated people to adapt and work
enced a macabre repeat of the snarled communications that around problems—whether it was a critical antenna cut by
hampered response during 9/11. The lack of interoperable FEMA or vital data and equipment trapped in the ninth
radio and communications gear stymied coordination with floor of a damaged building. Non-digital backup procedures
local, state and federal agencies. Area parishes are now need to be considered and developed.
working with the state to implement a unified standard for Stay or Go: Many of the most effective managers were
radio communications. housed outside of the New Orleans area, where they had
Local backup is not enough: Many businesses and gov- access to communications, Internet, power and transporta-
ernment agencies dutifully backed up data ahead of the tion. Still, it takes boots on the ground—like Walt Barowka
storm, then housed the disks or tapes less than 10 miles carrying hardware out of the Yenni Building—to get things
away. Likewise, Jefferson Parish locally hosted both its e-mail done. In other cases, pulling staff out of the area can
and Web servers. The result: Parish staffers had to take heroic invite catastrophe—as was the case with the Jefferson
measures to recover and move these systems out of the area. Parish pump operators.
Seek alternatives: When all communication systems Challenge assumptions: During inspections of sensor
failed, one surprisingly stayed online—text messaging. Jef- stations and antenna mounts following the storm, Jason
ferson Parish and other organizations quickly learned to use Phillips learned that wooden poles ably weathered the high
text messaging to communicate. Many also procured cell winds. Galvanized steel poles, by contrast, tended to bend
phones with out-of-state area codes, since these proved at a 90-degree angle or snap entirely. No surprise, the
much more reliable than local accounts. Text messaging is parish has switched to wood poles.
now a formal part of the emergency plan for the parish. Produce hard copy: Critical data such as contact lists,
Forge-trusted partnerships: Ridley Boudreaux was emergency documentation, authentication data and the like
able to restore Web and e-mail service within two days of should be printed and distributed, so it’s accessible during
the hurricane, thanks to the help of the VAR that set up the extensive power or network outages. Also consider distrib-
parish’s IBM iSeries systems. The company provided server uting this documentation on compact USB keys, so remote
resources, office space in Baton Rouge and a skilled pro- staffers have easy and portable access.
grammer to help restore services. Reject wishful thinking: Many IT plans worked within
Don’t overvalue technology: Katrina pulled down the the framework of a three-day evacuation and failed to
entire working infrastructure of the New Orleans area. And account for the effects of a direct hit by a hurricane. As a
yet, the state urged Jefferson Parish to submit its requests result, IT and emergency crews often had to take heroic
via the E-Team Web-based logistics application. Ultimately, measures to recover vital data and systems. — M.D.
9 | Redmondmag.com
St rm Warning
eastward turn had spared Louisiana’s capital city. There, coming down from the north shore, he saw some young
Boudreaux could find what he needed to recover his IT bucks cutting trees along the side of the road,” Boudreaux
operation: Power, communications, and access to systems, recalls. “He said ‘Come on, I’m gonna pay you,’ threw
partners, and vendors. them into the back of his truck, and brought four guys
“When I left I took all of my contact lists in hard copy—I down, plus a couple of his own people and a couple of my
printed it all out,” he says. “I had phone numbers for all of people. They just hauled equipment down all day long,
our vendors, I had passwords with me.” hauled it over to the Gretna building.”
Even as Boudreaux was getting busy in Baton Rouge, one By the end of the day, the impromptu IT rescue team had
of his contractors, Walt Barowka, had struck out from the delivered to the surviving data center everything but the
north shore, heading south toward the stricken Yenni AS/400 midframe system—which was too heavy to carry
Building. His mission: To recover as much equipment and down the nine flights of stairs.
data from the disabled data center as he could. “It was heroic,” Boudreaux says. “That’s the word I used
“On his way down, I think it was the Thursday after, over and over again—absolutely heroic. Everything worked.”
A Personal Toll Emergency staffers stayed on the job amid personal loss.
As emergency coordinator of Jefferson Parish, Tom Rodrigue and lo and behold the storm went right over Picayune. I didn’t
scrambled to recover communications, procure fuel and know until the following Friday that they were even OK.”
supplies, and deliver services that might save lives across For Dr. Walter Maestri, director of Emergency Management
the parish. He recalls discovering a Navy ship, docked for at Jefferson Parish at the time, the tragedy that struck
repairs, that was loaded with 750,000 gallons of diesel Rodrigue eventually touched him. His father had evacuated
fuel. For emergency crews in Jefferson Parish, it was an and survived the storm. A few months later, he was dead.
unbelievable find. “One of the phenomena that we’re seeing now, is that
“There was no way to get fuel and repeated requests for there are a lot of deaths occurring secondary to that, being
fuel from FEMA only ran into bureaucratic nightmares,” says forced to evacuate,” Maestri says. “We’ve got research stud-
Rodrigue. “In order to survive, we commandeered some fuel ies going on in this area that indicate that it’s a significant
trucks from a petroleum company. We put firemen in them phenomenon that is happening.
and we sent them to the ship, and we worked out with the “My dad lost everything and he was 85 years old,” Maestri
captain to be able to pump the fuel from the ship over the continues. “He lost everything in the storm. His home was in
levy into the trucks. And the firemen just did a continuous New Orleans and his home was completely demolished. He
rotation to all the different locations, such as the hospi- lost all of his possessions, he lost all of his pictures. And four
tals, nursing homes, that still were in operation because months later, he died. Dad just … it was just too much.”
they didn’t evacuate. Had it not been for that we would Virtually the entire staff struggled with the emotional toll
have completely [gone dark].” of the calamity. Maestri recounts the stress put on his
As a result of that effort, three of the five major hospitals staff, as they fielded hundreds of desperate phone calls
in Jefferson Parish stayed open and operational throughout from family members seeking help for stranded relatives,
the storm and its aftermath. By contrast, every hospital in many of whom were too old or infirm to get out.
New Orleans had shuttered, Rodrigue says. “Let me tell you, the toll that takes on the staff is incred-
ible. We had people crash here, literally, as we went
‘I Had a Pretty Good Idea’ through this with those kinds of phone calls. And most of
The work occurred even as Rodrigue, himself a resident of our people are trained to expect that, to know that that is
devastated St. Bernard Parish, suspected that his mother going to happen. And yet, when you get those calls,”
had been lost in the 15-foot high floodwaters. She was a Maestri says, his voice trailing off.
resident at St. Rita’s nursing home in St. Bernard Parish, “Caroline and I sat by Tom Rodrigue, who has been in
and was among the 34 people who died at the facility. The emergency management for 20 years, as he made the phone
nursing home owners were later charged with negligent calls leading up to the storm, warning everyone in the St.
homicide, for failing to evacuate ahead of the storm. Bernard area where his mother was in a nursing home, that
“I didn’t know for sure until the Friday after [the storm]. they had to get her out. They had to get all those people
Based on what I saw on reports as far as storm surge was out. And we sat next to him when he got the phone call—
concerned and the fact that I knew they didn’t evacuate, I she hadn’t gotten out. I mean, that kind of stuff. I don’t
had a pretty good idea that was the case,” Rodrigue says. want to overplay it or over-dramatize it, but you can imag-
“But I didn’t know anything about my wife either, until the ine what it’s like.”
following Friday, because they went to Picayune, Mississippi, Continued on page 11
10 | Redmondmag.com
St rm Warning
Continued from page 10
ship,” Rodrigue says. “I just bought another home in
Out of House Metairie, but I lived in St. Bernard. There was about 15 feet
As with others in the emergency management department, of water. I don’t have much left to go back to.”
the bunker-like EOC building became a temporary home. Rodrigue credits his background in the military for help-
The placed was packed, with staffers sleeping in hallways ing him stay focused in the days and weeks after the
and sharing scarce restroom facilities—Maestri recounts storm. He says it’s a unique mindset shared by emergency
setting up decontamination showers from the EOC’s hazmat first responders.
stores, so people had a way to take showers. For Rodrigue, “Staying busy keeps your mind off of the personal things
who was displaced by flooding, his stay at the EOC was the that you have to deal with,” Rodrigue says. “I think that’s
beginning of a six month sojourn. something that’s instilled in emergency personnel to begin
“I was here for a month. I lived in this dormitory on the with. And myself being ex-military, the mission came first.
third floor for maybe a month. And then I lived on a navy Everything else took a backseat.”
IT Lessons Solutions. So we had a Web site up that afternoon.”
Well, not quite everything. Boudreaux says the parish got The site was used to host a registration page that let
caught flat-footed when its Web site went dark during the parish personnel confirm their whereabouts with EOC
storm. A local company on the East Bank hosted the staffers. Boudreaux also used the site to point staffers to a
servers in its building, but when that building was dam- private page containing his latest address information and
aged, the servers were lost. It took the company more than phone number. This helped keep him in constant touch
a week to shuttle operations to Birmingham, Ala., and even as he bounced between locations.
restore service. In retrospect, Boudreaux says locally host- In addition to setting up networks and communications,
ing the Web site was a mistake. Boudreaux funneled all parish e-mail—which was com-
“In the parish we try our best to support local business- pletely unavailable in Jefferson Parish—to his site in Baton
es,” he says. “You learn lessons. Fortunately, we were regis- Rouge. While he could not e-mail staffers in the afflicted
tered with Network Solutions—it took a matter of minutes area, he could review correspondence and pass on urgent
items by fax, cell phone or text message.
As for the AS/400 iSeries midframe left in the Yenni
Building, Boudreaux’s team was finally able to access it
four weeks later, once electricity and air conditioning had
been restored. For nearly a month, the system sat in broil-
ing temperatures and high humidity—still, it booted up
normally, even with one hard drive failed and another
showing failure indications.
“So, the system worked. It came up and worked,”
Boudreaux says. “When people talk about moving our
financial, payroll, and utility running applications to a
Wintel box, I tell them over my dead body.”
Former Jefferson Parish EOC Director
Walter Maestri warns that damage from Work to Do
Katrina could make the next major storm Boudreaux is first to admit that he and his team had a lot
much harder to cope with.
to learn from the storm and its aftermath. Six months after
Katrina had struck, major changes were already being
to switch it over.” made to the electronic infrastructure and planning across
The switch over was made possible by CMA Technology the parish. For instance, Maestri says a remote data center
Solutions, the value-added reseller that had helped set up has been established outside of the area. It’s a decision he
the parish’s two iSeries systems. hopes others will adopt.
“I showed up at their door 1 o’clock on Tuesday and said “That’s what we recommend to businesses and industries
‘I need some help.’ They gave me an office, they gave me in the area,” says Maestri. “Don’t move it six blocks. Don’t
space on their iSeries, they gave me a brilliant program- put it in a warehouse around the corner. Send your data
mer, who helped me set up a one-page Web site that after- out of town. If you’ve got multiple locations for your firm,
noon and get our address changed with Network send it to an installation somewhere.
11 | Redmondmag.com
St rm Warning
“We had Ridley running all over the place, physically, location, even if none occurred during Katrina. He expects
because we didn’t want to trust this data to the lines, to the building will have a future as a dormitory or other
electronic transfer, because we weren’t sure if there was a function for the emergency management staff.
breach in the line somewhere.”
Similar improvements are being made to harden the Respect the Plan
damaged Yenni Building against another major storm. There is no doubt that members of the Jefferson Parish
Shatterproof glass designed for hurricane winds is being Emergency Management Department were bitterly disap-
installed. The roof is also being retrofitted to prevent a pointed by state and federal response after the storm.
repeat of the severe damage that occurred when the air Staffers at the EOC repeat darkly comical tales of FEMA
conditioning chiller units were torn off. crews tearing down parish communication gear and stop-
Changes are coming for the EOC building, as well. ping urgently needed shipments of fuel and water that
The parish is working on plans to move into a larger parish officials had arranged.
facility, after having struggled with the Marrero site But most of all, they do not understand why other agen-
during Katrina. cies failed to follow the agreed upon disaster plan. “That
“It was horrible. Those of us on this staff had six plan said the government would be here in 72 hours,”
months of hell,” Maestri says. “There were no shower says Maestri.
facilities. We had to set up tents in the backyard—it was The plan, established by the Southeast Louisiana Hurri-
in fact our decontamination tents because the hazmat cane Task Force, draws together the resources of 16 area
unit is part of emergency management, or it was. We parishes, and provides a blueprint for government interac-
had 100 people living in this building and they didn’t tion during a calamity.
have a place to take a shower. That’s why we’re looking “When we lost communications with the state and with
now for a larger facility with separate sleeping facilities, FEMA for about a five to seven day period, we simply
a larger kitchen.” pulled the plan off the shelf and started looking at it. We
Maestri also worries about flooding at the current EOC said, ‘This is what we are going to do. This is what we
Even the U.S. Coast Guard underestimated the challenge of oper-
Missions Critical ating after a hurricane. Here’s how they weathered the event.
When Hurricane Katrina struck the Gulf Coast at the end of found out where they needed to connect to reach their
the August, the timing for members of the U.S. Coast data and were able to get to work.
Guard sector command in New Orleans could hardly be Laying the groundwork for effective command and control
better. The unit had just completed the finishing touches took more than planning and some well-crafted Visual Basic
on its new Continuity of Operations Plan (COOP), which scripts to prep systems. The team worked with major tele-
defined how systems and communications would be kept com carriers to ensure that bandwidth would be available
viable during a catastrophe. at the secondary site.
“We had just finished going through the requirements and “One week before the hurricane, I had met with MCI, Bell
how we were going to do it in the week that Katrina hit,” South and SBC, and explained to them our reliance on their
recalls Commander Mark Johnston. “We were going to test infrastructure for search and rescue,” says Johnston. “We
it. So we got a live test.” definitely formed a partnership. Right after the storm, we
As it turned out, Johnston’s crew passed with flying col- called them up and ordered three T1 lines for that hotel…
ors. More than 500 people and 200 systems, including a and they installed those three T1 lines in about 20 hours.
variety of Dell PowerEdge Servers, were relocated from the That was basically because we knew who to call and how to
New Orleans Coast Guard office on Lake Pontchartrain to a expedite those installations.”
conference center in Alexandria, Louisiana, just north of
Baton Rouge. “As the storm was hitting the beach they Behind the Storm
headed up north and took their teams with them. And Within hours, Katrina had moved inland, leaving behind
when they got up there, teams were ready to go,” John- churning seas and a devastated landscape. Several Coast
ston says. Guard cutters steamed toward shore behind the storm, pro-
Johnston’s group relied on the Remote Desktop function- viding high frequency radio and satellite communication
ality of Windows to let client systems access data stored on links for units in the area. The ships also helped coordinate
servers in either Alexandria, Louisiana, or Alexandria, Vir- air support, and served as floating front offices for Coast
ginia. In the days and hours before the storm, IT managers Guard officers in the area.
had shipped backup tapes to each location. Users simply Continued on page 13
12 | Redmondmag.com
St rm Warning
Continued from page 12 The Best Laid Plans
“If people weren’t connected via satellite they went to “I would say this: The commercial infrastructure, I was
the nearest Coast Guard cutter and got their command and shocked the entire system was down,” admits Johnston.
control,” says Johnston. “They have the same infrastruc- In fact, the lessons from Katrina prompted the Coast Guard
ture—with Microsoft Office and Windows.” to reconfigure its regional network operation, which had
Those cutters also helped fill in the gaps when the Coast employed a hub-and-spoke model with New Orleans at its center.
Guard found that its network of tall VHF radio towers lining “All those regions depended on their connections to New
the Gulf Coast and Lake Pontchartrain had been disabled. Orleans, and from there they went out to the rest of the
When working, those towers let the Coast Guard communi- world. That design has gone by the wayside,” says John-
cate with off-shore vessels. ston. “We always knew it was a single point of failure, but
Restoring the towers was a top priority. The Coast Guard we get our funding from Congress.
deployed communication trailers to connect directly to the “Now each sector gets its connection directly to the Coast
tower stations, and then link via satellite to the Coast Guard network, via an MPLS cloud. It’s all redundant,”
Guard network. The teams used WAVE software, from Twist- Johnston explains, “so if any one of those things goes
ed Pair, to stitch together the disparate communication down, it just reroutes through the cloud.”
networks. One wrinkle: The solution had never been used Despite the limitations, the Coast Guard’s COOP weathered
over satellite before, so they worked with the vendor’s Katrina pretty well, but Johnston says more work lies ahead.
engineers to make it work. He warns that the next calamity may not offer two or three
“They could control it, they could hear if somebody was days to prepare. He hopes to close that vulnerability with
calling for help,” says Johnston. “And they could communi- “some pretty neat backup technologies” that his group will
cate through the radio to whoever was calling.” deploy as part of an upgrade to Windows Server 2003.
committed to do. I can’t talk to them, but I know or I were going to make entry into their facilities, and we were
believe that this is what they are doing,’” says Maestri. going to take what we needed, we were going to provide a
“What we later found out—that was all the moaning and list of everything that we did, and we would pay for what-
groaning that you heard on the local level relative to ever we took. But we were going to do that, because we
FEMA—was that wasn’t what they did. They didn’t follow needed to get food, we needed to get water.”
the plan. So we all sat here sucking our thumbs for a number That bit of improvisation—and countless others like
of days, saying they’re coming to help. And they didn’t come.” it—helped the EOC and key facilities like area hospitals
The plan had called for the parish to expect a govern- remain functional. But the breakdown was devastating.
ment response within three days. 72 hours. “After that, all With communications down, the parish simply had no
our stock will be replenished, because the cavalry will be way to know that the federal government would fail to
here,” Maestri says. respond within the agreed-upon window. It wasn’t until
The cavalry, of course, didn’t come. Fuel supplies ran that third day had passed that the parish knew it was on
short. Water and ice became precious commodities. EOC its own.
staffers found themselves short of food, clothing and toi- “You better be creative then,” says Maestri, “because in
letries as what was assumed to be a three-day event essence you gotta survive. You have to survive.”
stretched into a week, and then two.
“That’s when you get real creative, after that 72 hours,” Conclusion: The Gathering Storm
says Capdeville. By the time you read this report, the official start of the
Getting creative entailed things like breaking into local 2006 hurricane season will be upon us. Yet, the entire
Wal-Mart and Sams Club stores to confiscate essential region remains fragile. Maestri says about a third of Jeffer-
goods. It was a plan that had come to mind during one of son Parish lives in temporary FEMA trailers, usually
the group’s frequent “what-if” sessions, designed to help mounted on blocks and secured with nylon tie-down
the parish manage unforeseen calamity. Those prepara- straps. Many will almost certainly topple in Category 1
tions paid off, since management at the stores had been hurricane winds, meaning that even an approaching tropi-
apprised of the plan and were not surprised when they got cal storm may require a full evacuation. Yet the storm-
a call several days after the storm. related financial struggles of many families in the area
“Once we realized that FEMA wasn’t coming, we knew makes evacuation—with its steep costs in fuel, hotel bills
where we had to go to get what we needed to survive,” says and lost work—prohibitively expensive.
Maestri. “We have the managers’ phone numbers and The infrastructure is also at issue. Maestri describes local
names and where they were going to be. We told them we electrical service as being held together “with spit, bailing
13 | Redmondmag.com
St rm Warning
An Inside Job A small business keeps running by taking its show on the road.
For Thomas Brown, Hurricane Katrina was an inside job. As transaction,” Brown recalls.
president of E-Claim.com, Brown’s firm offers Web-based Brown had exceeded his monthly transaction limit,
claims processing services to adjusters and agents who prompting the company to freeze the account. Even worse,
flock to disaster stricken regions to assess the damage. But the company demanded three years of tax records before it
what happens when the disaster is in your own back yard? would lift the freeze. But that paperwork was in Brown’s
“The biggest thing for me is that I decided a long time basement, which had flooded during the storm.
ago to host my servers in an area that wouldn’t be affected
by any of this,” says Brown, whose servers reside in a data
center in Maryland. “Being based in the New Orleans area,
we certainly had a good deal of experience on what to
expect and what to anticipate when a hurricane bears down
on the area. I didn’t leave too much to chance.”
It’s a good thing, too. The storm destroyed Brown’s Gret-
na office—”The roof came off,” he says—and forced him
and his assistant to bounce between locations as they
struggled to find a place to set up shop.
“We stayed in Munroe for almost a week, and worked from
our office there,” Browns says. Unable to return to the New
Orleans area, the two stayed briefly with Brown’s parents
before setting up in a renovated condominium in Fort Wal-
ton Beach, Florida. He would run E-Claim.com out of that
building for a month.
With his servers in Maryland and all his files on a laptop,
Brown’s business didn’t miss a beat. Even his financial Thomas Brown displays the water damaged remains of
records, which he maintains using QuickBooks Online, were his tax records. A few weeks after the storm, Brown’s
always accessible. credit-card company had frozen his account, citing
unusual transaction activity.
Night at the Improv
Still, Brown was forced into a series of improvisations. For “I took a picture of [the damaged returns] and sent it to
instance, cellular service remained spotty for weeks after him and said, here’s my tax records,” he says bitterly. “That
the storm. “One of the things that came out of this was was very frustrating. I had a really, really hard time dealing
when you couldn’t make a voice call you could text mes- with those people.”
sage somebody. It wasn’t a lot of fun and gave us some It took a week to finally free up the account, during
sore thumbs, but it worked,” Brown says. which time bills were stacking up. Brown says he was fortu-
With Sprint still struggling to restore service three months nate he wasn’t delinquent on his accounts.
after the hurricane, Brown purchased a cell phone with a
Florida number to get service. It’s become a permanent part ‘The Luckiest Guy in the World’
of his business. “I will always keep [this phone] for that Despite the dislocations, E-Claim.com ramped up from
reason. When I go now I know that I have two different fewer than 200 registered claims adjusters to about 1,000
cell phones with two different area codes, and one of them during the weeks after the storm. He also scaled up from
will always be working.” four servers to seven. He credits his hosting company,
Disrupted postal service and Brown’s constant movements Atlantic Computer Systems, for making sure his service
made payments an issue. Brown says he resorted to paying one kept pace.
programmer “through PayPal” for a time. Most of his regular “The thing about what we do is, it’s a Web-based manage-
bills had already been set up for automatic payment via credit ment application for catastrophe adjusters,” says Brown.
card, preventing missed payments that could disrupt services. “When these things happen, they’re working 24x7, which
Customers, unable to send checks, were asked to make cred- means we have to be up 24x7. And if they can’t reach us,
it card payments at the E-Claims.com Web site. That worked we’re in trouble.”
fine, until the credit card company stopped his account Just as important, prior to the 2004 hurricane season,
“Two or three weeks into the thing I get a call from a Brown had used RoboHelp to update documentation for his
merchant account saying they can’t authorize a $10,000 Continued on page 15
14 | Redmondmag.com
St rm Warning
That holds dire consequences for the next evacuation.
Continued from page 14 With Katrina, old and infirm patients could be sheltered in
Web-based applications. That eliminated a flood of tech hospitals, where they had access to care, supplies and
support calls—calls he would not have been able to han- power for medical devices. Those patients will probably be
dle after Katrina. forced to evacuate ahead of the next storm.
“I probably fielded 10 support calls from adjusters,” “We have to move those people completely out of the
Brown says of the weeks following the storm. “The previ- area. And some of those people are going to tell you, and
ous year, even though we had fewer adjusters using the they’re right, that that’s a death sentence,” Maestri says, his
system, we were getting calls every day.” voice rising. “They can’t make the trip, even a hundred
Brown says E-Claim.com is on track to service 2,000 miles. Physically they simply cannot do it.”
adjusters this summer and is adding hardware to meet the The grim calculus of evacuation demands that Maestri,
demand. But like so many in the region, he’s just happy Capdeville and other managers at the Jefferson Parish
to have survived. EOC shuffle the deck, re-tune their models and reassess
“We’re back in our house, we’re back in the office, and how they conduct the next evacuation. Capdeville has been
life is slowly returning to normal,” he says. “Every day I tweaking the computer-based evacuation models, lowering
look at it. It’s tough, but I look at what’s happened to her assumptions about road capacities and taking into
everyone else, and I feel like I’m the luckiest guy in account issues like a higher percentage of vehicles becom-
the world.” ing disabled on roadways.
At the end of the day, however, Maestri is realistic. There
wire and scotch tape.” He warns that many utilities in the is only so much that technology can do. “I think one of the
area are now operating under bankruptcy. And then there problems is that we, 21st century Americans, are totally
is the problem of the devastated medical system. dependent, in ways they don’t even think about, on tech-
“If there is a piece of the critical infrastructure that is still nology,” Maestri says.
paralyzed, limping, I don’t know how to describe it, it’s the “There is a picture that we have that somebody snapped,
medical systems. We have gone from 11 major hospitals one of our own photographers, of the CEO and chairman
functioning to three. A million two hundred thousand peo- of the board of a Fortune 50 corporation who lives here,
ple are now dependent on three hospitals, in essence,” sitting on the curb eating an MRE (meal ready to eat),”
Maestri warns. “We had a level 1 trauma center—it’s gone. Maestri says. “I mean, he has planes at his disposal and
Charity Hospital of Louisiana has closed. We had two God knows what. And he’s sitting at a curb eating an MRE,
major medical schools functioning—Tulane and LSU. because all of his money, he couldn’t get to it.”—
Those doctors, those students are now no longer here.
They’ve moved out.” Michael Desmond is Redmond’s Editor at Large.
Vicky O’Brien helps credit union customers cope
Credit Where Credit’s Due with the aftermath.
When the largest disaster ever to strike the citizens of Jef- “You’ve got the money in the bank, but if you can’t get it,
ferson Parish occurred on Aug. 29, 2005, Vicky O’Brien you can’t buy the groceries, you can’t buy the medicines, or
knew that her credit union would play a critical role in the whatever. If your debit card strip is not being picked up,
recovery. As president of the $50 million Jefferson Parish you might have millions in the bank but you can’t get it.”
Employees Federal Credit Union, she realized that many of For O’Brien, the message was clear. The credit union had
her 10,000 members would need cash—and quickly. to find a way to put cash into the pockets of its members.
“A lot of people don’t have credit cards in their pockets— “Any member with a checking account with the credit
a lot of our members don’t. And they couldn’t have paid for union, we allowed all their accounts to overdraw so they
their hotel if they didn’t,” recalls O’Brien. could have access to cash, because in this immediate
Hurricane Katrina pulled down the financial infrastructure area cash was an issue,” O’Brien says. “ We had over 300
of the entire New Orleans area. ATM machines were dark. people that overdrew their accounts. We had one member
Stores could not handle credit or debit card transactions. say he evacuated 18 people because of the cash we
Even cash was made scarce by the closure of the New allowed him to have.”
Orleans Federal Reserve. It posed a challenge for residents The effort was complicated by the closure of the Federal
and rescuers alike. Reserve in New Orleans. Like everyone else in the region,
“No vendor that was open in this area for emergency per- the credit union needed to get its hands on cash.
sonnel would take anything but cash,” O’Brien says. “We were borrowing cash and we were getting cash from
For Dr. Walter Maestri, director of the Jefferson Parish everywhere. We were taking cash from our other branches
Emergency Management Department, the cash crisis had and moving them in,” O’Brien says.
serious repercussions. Continued on page 16
15 | Redmondmag.com
St rm Warning
Continued from page 15 severe. But the credit union operations on the second and
Maestri helped coordinate a larger effort to allow local third floors escaped nearly unscathed.
bank customers to withdraw cash from accounts. “We had “We looked like we went out to lunch,” says O’Brien of the
to send federal marshals to branches of the federal reserve evacuated offices. “We were just so lucky.”
to pick up—literally pick up—millions of dollars [of cash], The Jefferson Parish credit union was also fortunate to
transport it back, and set up a location where you could have a trusted partner in USERS Incorporated. The credit
walk in [and] get some kind of a telephone line,” he says. union had deployed the company’s DataSafe core process-
“And that way you could go and get money.” ing system in-house and maintained a hot site at USER’s
data center in Valley Forge, Pennsylvania. When the storm
Stay at Home disabled the local infrastructure, O’Brien was able to shift
The Jefferson Parish Employees Federal Credit Union, estab- operations to the remote site.
lished in 1959, has 10,000 members and prior to Katrina Just as important, USERS helped O’Brien line up resources
reported cash deposits of more than $50 million dollars. The to keep the credit union on track. “I would come up with
institution has seen its deposits grow since the storm, as an idea and throw it out to them and they would help make
members gathered funds to rebuild homes and businesses. it happen. It gave me a large group of people at my dispos-
“We were a $50 million credit union, but we’re a $70 mil- al when I didn’t have anybody here.”
lion credit union now,” says O’Brien. For instance, the company put O’Brien in touch with peo-
ple at Continental Federal Credit Union, who set her up
with a laptop until she could return to Louisiana. Another
credit union, La Capitol, provided temporary office space in
Baton Rouge for O’Brien’s employees.
“There’s been an overwhelming amount of generosity in
the credit union community,” says O’Brien, who also credits
her vendor relationships. “If you are a small financial insti-
tution or a small company, the best thing you can do is
have good third-party partners. That’s what helped us
recover and recover well.”
Today, the Jefferson Parish Employees Federal Credit
Union hosts its applications on servers at the USERS Incor-
porated data center in Valley Forge. The decision to move
away from local operations was easy.
“We were investigating the On-Line [service] and we
decided to definitely make that move,” says O’Brien, who
sees benefits beyond disaster recovery for her bank. “When
you are a small shop you can’t keep up with buying the
‘The Bayou Is What You Get’
Outsourcing applications makes even more sense when
you consider the struggle that area businesses face in
keeping employees. Like businesses across the stricken
Gulf Coast, the credit union has lost a lot of skilled labor
since the storm.
The Jefferson Parish Employees Federal Credit Union, O’Brien describes one employee who left the area for
located on the second and third floors of the Yenni Build- good. “The ordeal was so much she resigned and is moving
ing, was spared the damage of the stricken upper floors. to Georgia with her brother, to have more family support.
When you look at your management team, you’ve got to
The storm disrupted bank operations, but O’Brien says her consider the children of the team and the jobs of the
firm was lucky. The bank’s processing offices are in the spouses. In this area a lot of people work in oil compa-
Yenni Building, which had its roof torn open during the nies—a lot of people moved to Houston. I lost two
storm. The resulting water damage to the upper floors, [employees] because of that.”
including to a parish data center on the ninth floor, was Continued on page 17
16 | Redmondmag.com
St rm Warning
Continued from page 16
Many other employees simply didn’t return after the evac-
uation. The reason: schools. Or in the case of Jefferson
Parish, the lack there of. “I never thought about schools—I
never gave that a thought,” O’Brien says.
Even as many families have stayed out of the region, oth-
ers are determined to rebuild. O’Brien is one of them. A 33-
year resident of Jefferson Parish, she had evacuated just
once prior to Katrina, and understands why so many others
were reluctant to leave as well. And like so many in the
region, she remains fiercely loyal to New Orleans and the
“We’ve been a city a long time. We aren’t going any-
where,” O’Brien says. “When you are talking to me, the
bayou is what you get.”
Now, she is committed to helping people stay. “We don’t Businesses like retailer Casual Home Furniture have been
know how to separate from our jobs, because we know peo- forced to deal with labor shortages, even as they do busi-
ple are depending on us. We can’t. I asked a friend, ‘How ness out of a makeshift tent.
can I be doing this?’ And she said, ‘Because you have to.’”
Ironically, the best way for O’Brien to help keep people in “Now I’ll always go. What I learned was I need to talk to
the community is by making sure she’s among the first out the vendors that support the credit union to keep it up and
when the next storm threatens. running. And I can’t do that without communications.”
17 | Redmondmag.com