Sample Report Sample Report

Document Sample
Sample Report Sample Report Powered By Docstoc
					PowerCET Corporation

                                                                                                              Sample ReportReport
3350 Scott Blvd., Bldg. 55, Unit 1
Santa Clara, CA 95054-3124
Voice: 408/988-1346
Fax: 408/988-4869

                             Problem Investigation Report
                           UPSIDE UPS Transformer Failure
                         Income Management Company (IMC)
                                    Tropic, Island
IMC experienced a catastrophic failure of the output transformer associated with the three-month old
UPSIDE 12kVA uninterruptible power system (UPS). The transformer failure was a result of over heating
in the transformer windings filling the computer room with significant levels of smoke and activating the
circuit breaker removing power from the system and probably preventing an open flame fire in the
computer room.

One theory for the failure was excessive harmonic loading on the transformer as the installation had
reportedly experienced other circuit breaker overheating problems in general distribution panelboards
which was “corrected” through installation of an additional electrical distribution transformer.

In addition there was concern over the length of branch circuit runs, approximately 80 feet, and the impact
it might have on equipment and system performance.

PowerCET Corporation was retained to investigate these issues and provide and evaluation of (1) the
probable cause of the UPS failure, (2) general electrical system design, (3) harmonic loading, (4)
computer room electrical configuration, and (5) review of emergency/disaster recover plans and

A visual inspection of the electrical system and review of existing documentation was conducted.

Power measurements were made of the computer room electrical system as well as the electrical
panelboards serving the third-floor office area occupied by IMC. A Dranetz-BMI Power Platform PP1 was
utilized along with additional measuring equipment including Fluke 867 Graphical Multimeter, Tektronix
THS720A Digital Storage Oscilloscope, and associated accessories.

A single line drawing of the IMC computer room electrical system was developed to document the
existing configuration and to aid as a planning tool in evaluating any needed changes to improve
electrical equipment configuration.

Key Findings & Corrective Activities

1. UPS Output Transformer Failure
The UPS load was monitored to determine the total load as well as the harmonic content. The results are
contained in the Table 1 and were recorded using the PP1. This data reflects the actual load that had
been connected to the failed UPS. Readings were made at the input to the UPS (output) panelboard
powered from a single-phase 208V feeder.

The readings are not excessive or unusual for the types of loads found in computer room installations and
are well within the design capabilities of the UPSIDE UPS equipment.

5/23/1999                                                                                          Page 1
Problem Investigation Report

                                                                                                            Sample Report
UPSIDE UPS Transformer Failure
IMC, Tropic, Island

                               Table 1 - UPS Load (commercial power).
                                    Unit     A        B       Total
                                    V        117.5    118.2   117.8
                                    I        24.55    25.72   50.27
                                    kW       2.614    2.415   5.029
                                    KVA      2.884    3.040   5.890
                                    PF       0.906    0.795   0.854
                                    Vthd     2.53     3.26
                                    Ithd     46.24    74.80
Figure 1 shows representative waveforms of the connected UPS load operating from commercial
(building) power before any changes were made. The waveform shows the reason for the difference in
the Ithd values between sides A and B. The investigation revealed that the Compaq servers were all
powered from the A-side and appear to have power factor (harmonically) corrected power supplies, while
side-B is dominantly conventional switch-mode power supplies.

                                  Figure 1 - UPS Load (utility power).

An inspection of the failed UPS equipment was conducted and revealed that the failure of the output
transformer was apparently associated with an inner winding of the transformer. There was considerable
charring and discoloration of several of the wires connected to the inner windings. A detailed evaluation
of the transformer will be necessary, by UPSIDE, to determine the exact failure mechanism. A detailed
failure analysis should be requested from UPSIDE. In all likelihood the replacement system should be
considered a warranty service item.

5/23/1999                                                                                         Page 2
Problem Investigation Report

                                                                                                               Sample Report
UPSIDE UPS Transformer Failure
IMC, Tropic, Island

Figures 2 and 3 are photos of the failed UPS equipment and arrows indicate the area of failure. The most
likely cause of the failure is a poor connection of the input taps on the primary winding. A high resistance
connection even with the moderate load on the UPS would generate sufficient heat over time that could
account for the early life failure of the transformer.

                                   Figure 2 - Failed UPS Equipment.

                        Figure 3 - Close-up of USP Failed Output Transformer.

5/23/1999                                                                                           Page 3
Problem Investigation Report

                                                                                                               Sample Report
UPSIDE UPS Transformer Failure
IMC, Tropic, Island

2. Reconfiguration of UPS Loads and Electrical Service
A review of the IMC electrical configuration existing on 26 April 99 was performed. Figure 4 is a simplified
single-line drawing of the arrangement with the UPS equipment powered from a single-phase 208V
feeder. Operating the UPS equipment at the 208V input level decreases the available capacity by
approximately 13%

                    Figure 4 - IMC Electrical Configuration Existing on 26 April 99.

After reviewing the IMC electrical configuration the following recommendations were developed and

    •   Install a 25kVA 208V/240-120V isolation transformer on the electrical feed serving the computer
        room to provide a 240V input source to the UPS to allow the system to operate at its full capacity.

    •   Install a manual emergency transfer switch on the secondary of the isolation transformer.

    •   Establish an emergcncy panelboard to provide electrical circuits to the UPS and non-UPS
        equipment that would require generator back under emergency conditions.

Figure 5 represents the present electrical configuration for the IMC Tropic facilities. The 208V/240-120V
isolation transformer installed in the basement provides a 240-120V feeder for the computer room
emergency panelboard which feeds both the UPSIDE UPS and other critical non-UPS loads.

The transfer switch was installed on the output of the isolation transformer so that when a suitable
emergency generator is obtrained it can be interfaced to the facility without having to again reconfigure
the electrical service associated with the computer room. If IMC is unable to obtain a suitable trailer
mounted generator with a 240-120V split-phase output then it may be necessary to investigate using a 3-
phase generator and install an appropriate 3-phase transfer switch ahead of the 240-120V isolation
transformer. This approach would require some additional electrical rearrangement to provide a suitable
3-phase load for the generator.

5/23/1999                                                                                           Page 4
Problem Investigation Report

                                                                                                             Sample Report
UPSIDE UPS Transformer Failure
IMC, Tropic, Island

The manual bypass switch was installed in the computer to allow the UPS equipment to be bypassed for
maintenance or in the event of a UPS problem.

        Figure 5 - IMC Electrical Configuration After Computer Room Electrical Rearrangement.

The existing load distribution with all of the Compaq servers connected to one side (phase-to-neutral) of
the UPS output panelboard resulted in an imbalance of harmonic loading on the two output legs. The
power connections to the Compaq servers were rearranged so that two of the servers are powerered
from one leg and three are powered from the other leg. This change balanced the harmonic loading on
each leg and improved operational efficiencies.

In an effort to extend the UPS inverter battery operation each of the UPS output branch circuits was
evaluated with respect to the need to be connected to the UPS. It was determined that some of the loads
could be removed from the UPS output and a new emergency power bus established that would have
access to backup generator power (when installed). The following loads were removed from the UPS
panelboard on 29 April 99:

    •     Workstation outside the computer room    4.0 Amps
    •     Administrative telephone switch (system has existing back-up battery supply >6 hours)   2.0

Table 2 and Figure 6 reflect the UPS loading and waveforms assoicated with the UPSIDE UPS output
after completion of the changes discussed above—rearrangement of Compaq servers and transfer of
loads from the UPS to the emergency electrical panelboard (non-UPS loads).

5/23/1999                                                                                           Page 5
Problem Investigation Report

                                                                                                                Sample Report
UPSIDE UPS Transformer Failure
IMC, Tropic, Island

                           Table 2 – UPS Load After Circuit Rearrangements
                                      Unit     A        B        Total
                                      V        120.2    120.1    120.1
                                      I        24.0     23.3     47.3
                                      kW       2.3      2.3      4.6
                                      KVA      2.9      2.8      5.7
                                      PF       0.810    0.821    0.816
                                      Vthd     2.21     2.26
                                      Ithd     73.17    69.51

         Figure 6 - UPS output waveshape after computer room electrical reconfiguration.

3. Emergency Procedures and Capabilities
The existing UPSIDE UPS configuration is suitable to buffer the IMC operation from short duration power
interruptions. The connected load with the present battery supply will provide approximately 20 minutes
of operation. To extend the runtime with the currently installed battery supply it will be necessary to
immediately, upon a power interruption, initiate an aggressive load shedding procedure. This would
require shutting down all non-essential UPS loads—specifically three of the four trading positions and any
other systems in the computer room that are not required under a minimum configuration. How long the
runtime period can be extended will depend on how fast the load can be reduced, but it is unlikely that it
could be extended beyond 25 or 30 minutes as it takes time to perform an orderly shutdown of the

To extend the UPS battery runtime it will be necessary to expand the battery plant. To achieve a 2-hour
battery runtime three additional battery cabinets will probably be required and it will still be necessary to
implement an aggressive load shedding procedure.

5/23/1999                                                                                              Page 6
Problem Investigation Report

                                                                                                                 Sample Report
UPSIDE UPS Transformer Failure
IMC, Tropic, Island

The existing Emergency Plan called for using several small portable gasoline generators to support IMC
operation in the event of a longer duration electric power interruption. It unlikely that these generators
could be deployed successfully in the present third-floor office suite configuration... In addition to the
problem (and delays) of transport there is the issue of a suitable operating location and bringing gasoline
into the building.

The best alternative, as discussed previously, is to obtain a suitable trailer mounted generator set that
can be deployed in the driveway area (or parked next to the curb) and connected to the manual transfer
switch in the basement via flexible power cable.

4. General Office Electrical/Lighting Panelboards
The two 42-pole panelboards serving the IMC office suite were inspected and briefly monitored. The
inspection revealed that several of the circuit breakers in panelboard “A” are dual terminated. This is
generally not a good practice and there are sufficient pole positions in the panelboards to accommodate
the additional individual circuits.

Panelboard “A” is dominantly lighting and computer loads while panelboard “B” has the majority of the
HVAC equipment. Both panelboards are fed from the same feeder, which is split in a connection box just
below the panelboards. It was reported that the panelboard main breakers had overheated in the past
due to harmonics. The measurements taken during the week of April 26th did not indicate excessive
harmonic loading and the total load appeared to be well within the installed main breaker capacity. Both
voltage and current total harmonic distortion (THD) were at acceptable levels for this type of installation.
The loading on the feeder appears to be benefiting from the mixture of computer and motor loads severed
from the panelboards. (CAUTION: These observations are based on a very limited monitoring period
and may not be totally representative of a worse case condition.)

A 3-phase wye-wye (K-factor rated) transformer was installed on the panelboard “A” and “B” feeder in the
basement electrical equipment room. The reason given for the installation was to mitigate the “harmonic”
problem on the upper floors. It is not possible to comment on the previous “harmonic” and overheating
problems associated with panelboards “A” and “B”, as the readings were not made available.

It is unclear as to why a wye-to-wye transformer was installed in the basement area. If the problem being
addressed was associated with the excessive neutral currents from computer loads then a delta-to-wye
transformer would have been more appropriate. In addition locating a delta-wye transformer in the
electrical room on the third-floor would be the preferred location, as it would have eliminated the neutral in
the feeder running between the basement electrical room and the third-floor. The delta-wye transformer
would also have provided some harmonic filtering (conversion) of the 3rd harmonic into 5th and 7th

5. Energy Savings Opportunity
Measurements of the lighting load in panelboard “A” indicate that there is at least 8,000 Watts of lighting
load that could be controlled for a considerable energy savings. Assuming a 12-hour workday at 52
weeks per year the savings in simply turning off the lighting when the office is not occupied would save
approximately $10,150 per year in electricity. The savings could be considerably more if other energy
savings procedures were implemented with respect to printers and copiers.

6. Lightning and Transient Protection
The electrical service and telecommunications facilities are not equipped with any type of lightning or
surge protection equipment.

5/23/1999                                                                                             Page 7
Problem Investigation Report

                                                                                                        Sample Report
UPSIDE UPS Transformer Failure
IMC, Tropic, Island

The telecommunications facilities appear to be particularly vulnerable. Figure 7 shows the
telecommunications termination blocks in the basement area and Figure 8 the conduit entrance. This
area appears to be the main entrance point for the telecommunications cabling and the following
deficiencies were noted:

   •   Telephone company cable sheath was not grounded

   •   Station protectors were not installed

   •   Internal and external cables are bundled together—they need to be separated.

Even if Tropic Telephone Company brings the cable entry up to industry standards, IMC should consider
installation of secondary protection devices on all telecommunications and data circuits.

                 Figure 7 - Telecommunications terminal blocks (basement area).

                  Figure 8 - Telecommunications cable conduit (basement area).

5/23/1999                                                                                      Page 8
Problem Investigation Report

                                                                                                                 Sample Report
UPSIDE UPS Transformer Failure
IMC, Tropic, Island

7. Intermittent High Current Surges on Compaq Power Supplies
Abnormally high current surges apparently associated with the Compaq servers have been recorded.
The cause of these surges is unknown at this time, but the level of current increases by as much as 20 to
30 amps for up to 0.1 seconds. Figure 9 shows the associated waveforms for the equipment during one
of these occurrences--specifically channel "A" current increases from 24A to 58A. It appears that the
pulse width modulator (PWM) in the power supply is "gating" full on for a short period of time. Possible
causes are (1) component problem in the power supply, or (2) some unknown interference is confusing
the power supply control circuitry and causing the PWM to turn full on. The PP1 power monitor was left in
the computer room for an extended period which channels "C" and "D" connected to determine which
power supply was causing the problem. To date the problem has not reoccurred.

                    Figure 9 - UPS output showing current surge on channel "C".

8. Monitoring and Annunciator
There is little if any monitoring an alarming associated with the electrical environment and UPS
performance. The UPS is equipped with a 9-pin alarm connector, which brings out a variety of status
signals, which could be alarmed.

In addition there is no on-going monitoring of the incoming utility power or the output of the UPS

Long Branch Circuits (Trading Desk)

There was concern that the excessive length of the branch circuits feeding the trading desk area would
result in excessive neutral-to-ground voltages. The load on each branch circuit is approximately 4Arms
and results in a neutral-to-ground voltage drop of less than 1 volt. No corrective action is required for this

5/23/1999                                                                                              Page 9
Problem Investigation Report

                                                                                                                Sample Report
UPSIDE UPS Transformer Failure
IMC, Tropic, Island

a) Rearrange branch circuits in panelboard “A” to eliminate dual (double lugging) terminations. There
   should be sufficient spare circuit breaker pole positions to accommodate this activity. While there
   does not appear to be any problem with circuit loading at this time, future equipment additions in the
   office area could pose an interference (or interaction) problem. (Reference section 4 above for
   additional information.)

b) If “harmonic” problems reoccur with panelboards “A” and/or “B” then it may be necessary to locate a
   delta-to-wye distribution transformer in the third-floor electrical room. This action would serve multiple
   purposes by (1) eliminating the long neutral run to the basement that results in higher neutral-to-
   ground voltages and (2) would improve voltage levels for the equipment being served from the
   panelboards. If such a transformer is to be installed it may be advisable to split the HVAC and
   lighting loads from the electronic office loads. The HVAC loads would be served from the existing
   panelboard “B” directly connected to the feeder and the office equipment served from the wye
   secondary of the transformer via panelboard “A”. Figure 10 illustrates the existing electrical
   configuration of panelboards “A” and “B”.

                     Basement Electrical Room               4th Floor Electrical Closet

                                                                                           Pnl A
                                                                                          208V 3Ø
                                                                                          4W 125A
                                                                                           Pnl B
               6kA @ 208 3Ø 4-W            1:1
                                                                                          208V 3Ø
                                                                                          4W 125A

                       Figure 10 - Existing IMC panelboard A & B configuration.

c) Figure 11 shows the proposed configuration which calls for the installation of a new delta-wye
   transformer in the third-floor electrical closet and removal of the existing wye-wye transformer located
   in the basement electrical area—refer to Figure 10. Leaving the wye-wye transformer installed would
   add unnecessary impedance to the circuit and would not be required, as the new transformer would
   provide sufficient buffering for the electronic loads on the third-floor. (NOTE: This recommendation
   should only be implemented if problems reoccur with the loading on existing panelboards “A” and

                     Basement Electrical Room               4th Floor Electrical Closet

                                                                                           Pnl A
                                                                                          208V 3Ø
                                                                                          4W 125A
                                                                                           Pnl B
               6kA @ 208 3Ø 4-W                                                           208V 3Ø
                                                                                          4W 125A

  Figure 11 - Proposed IMC panelboard A & B configuration with delta-wye isolation transformer.

5/23/1999                                                                                           Page 10
Problem Investigation Report

                                                                                                                 Sample Report
UPSIDE UPS Transformer Failure
IMC, Tropic, Island

d) The present installation is vulnerable to lightning surges on both the power and telecommunications
   services. It is recommended that IMC contact the building owner and see if a building service
   entrance surge protection system can be installed. If this is not possible then IMC should, at
   minimum, install appropriate surge protection on all feeders and/or panelboards.

e) The telecommunications circuits should also be protected with appropriate protection technology. At
   minimum the telephone company should install industry standard protectors in the basement area
   where the cable enters the building. In addition secondary protection technology is available with
   clamping levels very close to application voltages that can easily be retrofitted to the existing 66-type
   punch-down blocks

f)   The incoming telephone cable (basement location, see figure 8) needs to be separated from the
     inside cabling. The existing installation in which the two types of cable are tightly bundled together
     provides a coupling mechanism that would bypass protection devices if installed. Even a few inches
     of separation would be a significant improvement over the present configuration.

g) Install ferrite (split) cores on all power cords and data cables associated with the Compaq servers. If
   the cause of the current surges (refer to item 7 above) is some type of high frequency interference
   then the ferrite cores may provide sufficient filtering to eliminate the problem. Recommended cores
   are Tokin ESD-QS-25 (or equivalent). The cores cost $7 each in small quantities and are available
   from PowerCET or a Tokin distributor. If the cable is flexible enough then one core per cable will be
   sufficient with a loop--i.e., the cable passing through the ferrite core twice--other wise four cores will
   be required per cable.

h) Install a Gordon Kapes Site Monitor Model 125 for the computer room. The Model 125 has the ability
   to monitor the necessary points and provide both local and dial-up remote reporting for after hours.

i)   In addition to the Gordon Kapes Site Monitor it is recommended that a power monitor be installed that
     will provide the necessary information to manage the UPS load and record the reliability and quality of
     the incoming electrical service. An Electro Industries Futura or Nexus monitor is recommended. The
     Electro Industries equipment and current probes are available from PowerCET or can probably be
     obtained from the local electrical contractor.

Emergency Generator and Disaster Plans
Unfortunately the building configuration does not lend itself to incorporating a conventional emergency
generator system. Exterior space is non-existent and interior (ground floor) space is built-out as office
suites. In addition the uncertainty of the business climate and duration of the existing lease make any
large building additions (or alternations) financially unattractive. The following is a brief discussion on
some possible alternatives.

Before making a significant investment in emergency power systems IMC should make a determination
as to the type of system availability that is required. It appears that IMC operations are more of a 5x12
(5-days, 12-hours per day) as opposed to a 7x24 (7-days, 24-hours per day) operation. There is a
considerable cost difference between providing 5x12 (or maybe 5x8 if only the trading day coverage is
required) versus a true 7x24 operation.

In all likelihood the minimum necessary coverage is probably the ability to sustain trading at a single
trading position and approximately 4kW of load in the computer room. The UPS is currently running at
approximately 5kW. It might be possible to obtain a small diesel generator (12kW) that is mounted on
small wheels--Honda Model EB12DAG, cost <$8000. This unit would be the absolute minimum you could
use. It would require that you have it stored on-site or very close, as the wheels are not sufficient for long
distance towing on roads. The weight is approximately 750 pounds and the diesel fuel is much less of a
problem than gasoline.

5/23/1999                                                                                             Page 11
Problem Investigation Report

                                                                                                      Sample Report
UPSIDE UPS Transformer Failure
IMC, Tropic, Island

Report prepared by:

Bruce B. Lonie
President, PowerCET Corporation

 The information contained in this document is provided for educational purposes only as an example
 on how to incorporate power monitoring data and other observations into a report format. It is not
 intended to provide consulting advice for any specific problem or situation. This is a copyrighted
 document and intended for individual use and should not be reproduced or distributed in any form
 without specific written permission from PowerCET Corporation.

                                      PowerCET Corporation

5/23/1999                                                                                   Page 12