Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out
Get this document free

365

VIEWS: 16 PAGES: 2

									Generator Failure At Power Station of 365 Main Inc.

365 Main reports on root cause of generator failure during power outage
Company makes information available to help others avoid the same
problem.
Data center developer and operator 365 Main Inc. has issued information that details the
root cause behind why back-up power generators in the company's San Francisco facility
failed to start during a PG&E power outage last week, resulting in approximately 40
percent of customers in the facility losing power to their equipment for up to 45 minutes.
The problem
At 1:47 p.m. on Tuesday, July 24, 365 Main's San Francisco data center was impacted by
a power surge caused when transformer breakers at a local PG&E power station
unexpectedly opened. PG&E has still not determined what caused the breakers to open.
Typically when a power outage occurs, the outage triggers 365 Main's rigorously
maintained and tested back-up diesel generators to start-up and take over providing
power supply to customers. 365 Main's San Francisco facility has ten 2.1-megawatt back-
up generators to be used in the event of a loss of utility power. Eight primary generators
can successfully power the building, with two generators available on stand-by in case
there are any failures with the primary eight.
However, following the power outage last week, three of 365 Main's 10 back up power
generators, manufactured by Hitec, failed to complete their start sequence. A complete
investigation of the incident began immediately.
Within hours of the incident, an international team of specialists was deployed to 365
Main's San Francisco data center facility to join on-site technicians and began
systematically testing the generators in search of a root cause. After days of thorough
testing around the clock, the team discovered a weakness in an essential component of
the back-up generator system known as a DDEC (Detroit Diesel Electronic Controller).
The team discovered a setting in the DDEC that was not allowing the component to
correctly reset its memory. Erroneous data left in the DDEC's memory subsequently
caused misfiring or engine start failures when the generators were called on to start
during the power outage on July 24.
The fix
The investigation team discovered DDEC issues on each of the failed Hitec units and
were able to successfully simulate failure. A fix was introduced by altering the timing of
a command to the DDEC component, allowing more time between the engine shutdown
command and the DDEC reset command. Once this fix was introduced, the Hitec
generators successfully passed more than 50 consecutive start-up sequence tests without
incident.
The testing methodology was performed by Hitec specialists along with 365 Main's chief
technician and staff. Specialists from Cupertino Electric were present during all testing,
and EYP Mission Critical Facilities will provide independent verification of the findings
the week of 8/6/07.
365 Main has implemented the DDEC fix in its San Francisco and El Segundo facilities.
Of the five data centers in 365 Main's portfolio, the San Francisco and El Segundo
facilities are the only ones with Hitec generators containing DDECs. All other facilities
feature other brands of generators or have different models of Hitecs.
365 Main is sharing the discoveries of its investigation with other Hitec customers. In
addition, Hitec has expanded its preventative maintenance procedures as a direct result of
discoveries made during the 365 Main investigation.
In the wake of the outage, 365 Main published an apology to customers and daily updates
directly from the investigation team meeting minutes, allowing customers and the public
at large to track progress. A complete archive of these updates and more details about
today's update are available at: http://www.365main.com/status_update.html


Abhay Gujar , Shanti Ashram , Borivli , Mumbai.India.

								
To top