Database Retail by lqf20778

VIEWS: 34 PAGES: 9

More Info
									ERCOT Retail Transaction Processing Outage & Degradation Summary
Annual Breakdown (July 2006 - Current)
                 2007
Outage Type                                               Degradation Type
                        Number of                                                            Number of
                        occurances   Duration (minutes)                                      occurances   Transaction Count
PaperFree                                                 PaperFree Duplicate Transactions        3             121

Siebel                      1               210           Siebel Duplicate Transactions          1              790

NAESB                       2               292           Lost Transactions

TIBCO                                                     TML Connectivity                       2

Human Error                 1               750           TML Functionality Error                1

TML                                                       MarkeTrak Connectivity

TML (report explorer)       1               50            Extract Delay                          2

Database                    1               70            Slow System Performance                2

Infrastructure              6               742           Transaction Processing Errors          3

                                                          Other



                 2006 (July - December)

Outage Type                                               Degradation Type
                        Number of                                                            Number of
                        occurances   Duration (minutes)                                      occurances   Transaction Count
PaperFree                                                 PaperFree Duplicate Transactions        2        Needs updating

Siebel                                                    Siebel Duplicate Transactions          7         Needs updating

NAESB                       2               55            Lost Transactions

TIBCO                                                     TML Connectivity                       1

Human Error                 2              1042           TML Functionality Error                2

TML                                                       MarkeTrak Connectivity                 2

TML (report explorer)       1               120           Extract Delays                         5

Database                                                  Slow System Performance

Infrastructure              3               909           Transaction Processing Errors          2

                                                          Other
                                                     July
                                                                                                                                                                                                             July
                                                                                                                                                                                                                           May
                                                                                                                                                                                                                                 April




                                                                                                                                                                                                                    June
                                                                                                                                                                                                                                         March




                                            August
                                                                                                                                                                                                    August




                      October
                                                                                                                                                                              October
                                                                                                                                                                                                                                                            January

                                                                                                                                                                                                                                                 February




December
                                                                                                                                                        December




           November
                                                                                                                                                                   November




                                September
                                                                                                                                                                                        September




                                                            2006
                                                                                                                                                                                                                                                                      2007
                                                            NA                                                                                                                                                                                                        NA




1
                      1
                                                                                                                                                                                                                                                            2
                                                              ES                                                                                                                                                                                                        ES
                                                                         B                                                                                                                                                                                                          B

                                                            Pa                                                                                                                                                                                                        Pa
                                                                 pe                                                                                                                                                                                                        pe




                                                                                                                                  Outage Summary
                                                                                                                                                                                                                                                                                                                                            Outage Summary
                                                                     rFr                                                                                                                                                                                                        rFr
                                                                        ee                                                                                                                                                                                                         e         e

                                                            Sie                                                                                                                                                                                                       Sie




                                                                                                                                                                                                                                                 1
                                                               be                                                                                                                                                                                                          be
                                                                 l                                                                                                                                                                                                                  l

                                                            TIB                                                                                                                                                                                                       TIB
                                                                   CO                                                                                                                                                                                                        CO


                                                            Hu                                                                                                                                                                                                        Hu




2
                                                                 ma                                                                                                                                                                                         1              ma
                                                                    n        Er                                                                                                                                                                                               n          Er
                                                                               ror                                                                                                                                                                                                         ror
                                                            TM                                                                                                                                                                                                        TM
                                                              L                                                                                                                                                                                                         L




1
                                                                                                                                                                                                                                                                                                                                                                  Monthly Breakdown (July 2006 - Current)




                                                            TM                                                                                                                                                                                                        TM
                                                                                                                                                                                                                                                 1




                                                              L(                                                                                                                                                                                                        L(
                                                                         Re                                                                                                                                                                                                         Re
                                                                            p     ort                                                                                                                                                                                                  p      ort
                                                                                        Ex                                                                                                                                                                                                            Ex
                                                                                             plo                                                                                                                                                                                                          plo
                                                            Da                                  rer                                                                                                                                                                   Da                                     rer
                                                                                                   )                                                                                                                                                                                                            )
                                                                                                                                                                                                                                                 1




                                                                 tab                                                                                                                                                                                                       tab
                                                                         as                                                                                                                                                                                                         as
                                                                              e                                                                                                                                                                                                          e

                                                            Inf                                                                                                                                                                                                       Inf
                                                                ra




                      2
                                1
                                                                                                                                                                                                                                                 1
                                                                                                                                                                                                                                                            5




                                                                     str                                                                                                                                                                                                  ras
                                                                        u     ctu
                                                                                                                                                                                                                                                                                tru
                                                                                                                                                                                                                                                                                         ctu
                                                                                  re                                                                                                                                                                                                             re
                                                                                                                                                                                                                                                                                                                                                                  ERCOT Retail Transaction Processing Outage & Degradation Summary




                                                     July
                                                                                                                                                                                                             July
                                                                                                                                                                                                                           May
                                                                                                                                                                                                                                 April




                                                                                                                                                                                                                    June
                                                                                                                                                                                                                                         March




                                            August
                                                                                                                                                                                                    August




                      October
                                                                                                                                                                              October
                                                                                                                                                                                                                                                            January

                                                                                                                                                                                                                                                 February




December
                                                                                                                                                        December




           November
                                                                                                                                                                   November




                                September
                                                                                                                                                                                        September




                                                            2006
                                                                                                                                                                                                                                                                      2007




                                                            Pa                                                                                                                                                                                                        Pa




                                1
                                                     1
                                                                                                                                                                                                                                                 1
                                                                                                                                                                                                                                                            2




                                                                 pe                                                                                                                                                                                                        pe
                                                                     rFr                                                                                                                                                                                                        rFr
                                                                        ee                                                                                                                                                                                                         e
                                                                           D            up
                                                                                                                                                                                                                                                                                             eD
                                                                                                                                                                                                                                                                                                    up
                                                                                             lica                                                                                                                                                                                                        lica
                                                            Sie                                      te                                                                                                                                                               Sie                                         te




           1
                      2
                                2
                                            2
                                                                                                                                                                                                                                                 1




                                                               be                                       Tra                                                                                                                                                                be                                        Tra
                                                                 lD                                         ns                                                                                                                                                                      lD                                   ns
                                                                              up                               a                                                                                                                                                                         up                                 a
                                                                                 li   ca
                                                                                                                      cti
                                                                                                                         on                                                                                                                                                                      lica                             cti
                                                                                                                                                                                                                                                                                                                                     o
                                                                                                                                  Degradation Summary
                                                                                                                                                                                                                                                                                                                                            Degradation Summary




                                                                                         te                                   s                                                                                                                                                                         te                             ns
                                                            Lo                                   Tra                                                                                                                                                                  Lo                                   T      ran
                                2                              st                                     ns                                                                                                                                                                   st                                           sa
                                                                  T      ran                                ac
                                                                                                                 tio                                                                                                                                                            Tra                                        cti
                                                                                                                    ns                                                                                                                                                                   ns                                   on
                                                                               sa
                                                                                     cti                                                                                                                                                                                                         ac                                s
                                                                                           on                                                                                                                                                                                                        tio
                                                                                                 s                                                                                                                                                                                                       n    s
                                                            TM                                                                                                                                                                                                        TM

1
                                                                                                                                                                                                                                                 1
                                                                                                                                                                                                                                                            1




                                                              LC                                                                                                                                                                                                        LC
                                                                         on                                                                                                                                                                                                             on
                                                                               ne                                                                                                                                                                                                            ne
                                                                                     cti                                                                                                                                                                                                          cti
                                                                                           vit                                                                                                                                                                                                          vit
                                                            TM                                   y                                                                                                                                                                    TM                                      y
1
                                1
                                                                                                                                                                                                                                                            1




                                                              LF                                                                                                                                                                                                        LF
                                                                         un                                                                                                                                                                                                             un
                                                                              cti                                                                                                                                                                                                            cti
                                                                                 on                                                                                                                                                                                                             on
                                                                                    ali                                                                                                                                                                                                               ali
                                                                                        ty                                                                                                                                                                                                                  ty
                                                            Ma                                       Er                                                                                                                                                               Ma                                       E   rro
2
           1




                                                               rke                                     ror                                                                                                                                                               rk     eT                                     r
                                                                   Tra
                                                                       k           Co
                                                                                                                                                                                                                                                                                  rak
                                                                                                                                                                                                                                                                                                 Co
                                                                                     nn                                                                                                                                                                                                            nn
                                                            Ex                                   ec                                                                                                                                                                   Ex                                      ec
                                                                                                     tiv                                                                                                                                                                                                          tiv
1
                      2
                                1
                                            2
                                                                                                                                                                                                                                                 1
                                                                                                                                                                                                                                                            2




                                                               tra                                         ity                                                                                                                                                             tra                                       ity
                                                                   c     tD                                                                                                                                                                                                    c        tD
                                                                               ela                                                                                                                                                                                                           ela
                                                                                      y                                                                                                                                                                                                           y
                                                            Slo                                                                                                                                                                                                       Slo
                                                               w                                                                                                                                                                                                           w
                      1
                                2




                                                                      Sy                                                                                                                                                                                                        Sy
                                                                        ste                                                                                                                                                                                                              ste
                                                                           m                                                                                                                                                                                                                     m
                                                                                          Pe                                                                                                                                                                                                          Pe
                                                                                                 rfo                                                                                                                                                                                                          rfo
                                                            Tra                                      rm                                                                                                                                                               Tra                                         rm
1
                                1
                                                                                                                                                                                                                                                            3




                                                                 ns                                    an                                                                                                                                                                   ns                                      an
                                                                         ac                                      ce                                                                                                                                                                 ac                                       ce
                                                                             tio                                                                                                                                                                                                         tio
                                                                                 n    Pr                                                                                                                                                                                                     n     Pr
                                                                                        oc                                                                                                                                                                                                           oc
                                                            Ot                                   es                                                                                                                                                                   Ot                                      es
                                                                                                      sin                                                                                                                                                                                                          sin
                                1




                                                               he                                                                                                                                                                                                        he
                                                                     r                                   gE                                                                                                                                                                     r                                       gE
                                                                                                                  rro                                                                                                                                                                                                         rro
                                                                                                                      rs                                                                                                                                                                                                          rs
ERCOT Retail Systems Incident Log
July 26, 2006 - February 27, 2007 Outages
 Issue      Notification                      Start               Duration                                                                                                    Service                                                                               Date (to be)
  Date         Date        Notification ID    Time     End Time    (mins)    Application         Issue Description                   Outage Type     Identification Method    Impact      Market Impact                 Root Cause              Resolution         Implemented                        Notes                                     Current Status
                                                                                                                                                                                          Cell will contain
                                                                                                                                                                                         market participant
              2/14/07,       In use after                                                                                                                                     Planned    and end customer
2/25/07       2/25/07           4/1/07       8:00AM    6:00PM       600        Retail      Planned Maintenance Outage                               Planned Maintenance       Outage     impact - if known    n/a                                     n/a             2/25/07                                                 Complete - Green status
                                                                                                                                                                                          Cell will contain                                       Additional
                                                                                                                                                                                         market participant                                      storage was
                                                                               TIBCO       Inbound and outbound                                                              Unplanned   and end customer     Table space issue with the       allocated to the
2/23/07       2/23/07                        11:05AM 12:15PM        70        Database     transaction processing halted   Database                 Monitoring                Outage     impact - if known    database                             database           2/23/07      Stabilization from RBP implementation      Complete - Green status
                                                                                                                                                                                          Cell will contain
               2/8/07,                                                                                                                                                                   market participant
              2/16/07,                                                                                                                                                        Planned    and end customer
2/18/07       2/18/07                        8:00AM    8:00PM       720        Retail      Planned Maintenance Outage                               Planned Maintenance       Outage     impact - if known    n/a                                    n/a              2/18/07                                                 Complete - Green status
              1/16/07,                                                                                                                                                                    Cell will contain
              1/30/07,                                                                                                                                                                   market participant
2/10/07 -      2/9/07,                         2/10      2/11                              Planned Maintenance Retail                                                         Planned    and end customer
 2/11/07      2/11/07                        12:00PM   5:00PM      1,740       Retail      Release                                                  Planned Maintenance       Outage     impact - if known    n/a                                    n/a              2/11/07                                                 Complete - Green status
                                                                                                                                                                                          Cell will contain                                                                        ERCOT solution in the process of being
                                                                                                                                                                                         market participant   Issue has been escalated to                                          implemented as discussed in the Jan '07
                                                                                           Connectivity issue between                                                        Unplanned   and end customer     SAN vendor for root cause        ERCOT effort                        RMS Retail Transaction Processing          In implementation - Completion date to be
 2/7/07       2/8/07                         9:15PM    10:40PM      85        Paperfree    application and SAN             Infrastructure           Monitoring                Outage     impact - if known    analysis                          underway               TBD         presentation                               determined - Yellow status/monitoring in progress
                                                                                                                                                                                          Cell will contain
                                                                                TML        TML - Report Explorer                                                                         market participant                                      Server and                        Proactive monitoring and automated
                                                                              (Report      unavailable during normal                                                         Unplanned   and end customer     Connectivity issue across         services were                      recovery is being implemented to correct
 2/7/07       2/7/07                         11:05AM 11:55AM        50        Explorer)    business hours                  TML (report expolorer)   Market Notification       Outage     impact - if known    applications                        restarted           2/7/07       this issue.                                  Complete - Green status
                                                                                                                                                                                                                                                                                   The overlying issue was a performance
                                                                                                                                                                                                                                                 Database                          issue in Siebel causing transactions to take
                                                                                                                                                                                          Cell will contain                                    maintenance to                      longer to process than normal at certain
                                                                                           Inbound transaction processing                                                                market participant                                       resolve                          times of the day. This was resolved with
              2/2/07,                                                                      was interrupted following Siebel                                                  Unplanned   and end customer Siebel workflow locked due to         performance                        database maintenance during a planned
 2/2/07       2/5/07                         6:00AM    9:30AM       210        Siebel      Batch.                           Siebel                  Monitoring                Outage     impact - if known incomplete data entering Siebel.        issues             2/11/07      maintenance window.                          Complete - Green status
                                                                                                                                                                                          Cell will contain
                                                                                           NAESB was unable to send                                                                      market participant                                                                                                                   Complete - SCR745 Phase 1 would have handled
                                                                                           transactions to Paperfree for                                                     Unplanned   and end customer                                      Already included                                                               this issue - Implementation 11-27-06 Scheduled
1/24/07       1/24/07                        12:00PM   3:00PM       180        NAESB       processing                      NAESB                    Monitoring                Outage     impact - if known    Application frozen              in another project      1/24/07      System rebooted.                           completion Feb 16, 2007 - Green status
                                                                                                                                                                                          Cell will contain
                                                                                           The NAESB application locked                                                                  market participant                                                                                                                   Complete - SCR745 Phase 1 would have handled
                                                                                           up and required a reboot. All                                                     Unplanned   and end customer                                      Already included                                                               this issue - Implementation 11-27-06 Scheduled
1/22/07       1/22/07                        10:06AM 11:59AM        112        NAESB       processing suspended.         NAESB                      Monitoring                Outage     impact - if known    Application frozen              in another project      1/22/07      System rebooted.                           completion Feb 16, 2007 - Green status
                                                                                                                                                                                          Cell will contain
              1/10/07,                                                                                                                                                                   market participant
              1/19/07,                                                                                                                                                        Planned    and end customer
1/21/07       1/21/07                        8:00AM    1:30PM       330        Retail      Planned Maintenance Outage                               Planned Maintenance       Outage     impact - if known    n/a                                    n/a              1/21/07                                                 Complete - Green status
                                                                                                                                                                                          Cell will contain
                                                                                           Find ESIID and Find                                                                           market participant
                                                                                           Transaction components of                                                         Unplanned   and end customer     Server crash caused the            Server was
1/18/07       1/18/07                        8:00AM    8:45AM       45          TML        TML unavailable                 Infrastructure           Monitoring                Outage     impact - if known    disruption of service               restarted           1/18/07                                                 Complete - Green status
                                                                                                                                                                                          Cell will contain
                                                                                                                                                                                         market participant
              1/12/07,                                                                                                                                                        Planned    and end customer                                                                          This outage was used to implement a
1/14/07       1/14/07                        8:00AM    11:30AM      210        Retail      Planned Maintenance Outage                               Planned Maintenance       Outage     impact - if known    n/a                                    n/a              1/14/07      portion of the Paperfree solution          Complete - Green status
                                                                                                                                                                                          Cell will contain
                                                                                                                                                                                         market participant
                                                                                                                                                                             Unplanned   and end customer                                     Replaced server
 1/9/07       1/9/07                         2:00PM    2:17PM       17          TML        Failed memory module            Infrastructure           Monitoring                Outage     impact - if known    Memory failure                      memory              1/9/07                                                  Complete - Green status
                                                                                           Attempted to replace the                                                                       Cell will contain                                    Server backed                       ERCOT solution in the process of being
                                                                                           Paperfree architecture as                                                                     market participant                                       out and                          implemented as discussed in the Jan '07
                                                                                           identified by the on-going                                                        Unplanned   and end customer                                     implementation                       RMS Retail Transaction Processing          In implementation - Completion date to be
 1/8/07       1/8/07                         5:00PM    8:15PM       195       Paperfree    Paperfree issues analysis       Infrastructure           Monitoring                Outage     impact - if known    Server would not come online      rescheduled            TBD         presentation                               determined - Yellow status/monitoring in progress
                                                                                                                                                                                          Cell will contain                                                                        ERCOT solution in the process of being
                                                                                                                                                                                         market participant   ERCOT has identified a                                               implemented as discussed in the Jan '07
                                                                                           Problem pulling data from                                                         Unplanned   and end customer     communication protocol issue     ERCOT effort                        RMS Retail Transaction Processing          In implementation - Completion date to be
 1/5/07       1/5/07                         3:00AM    7:30AM       270       Paperfree    NAESB                           Infrastructure           Monitoring                Outage     impact - if known    between Paperfree servers         underway               TBD         presentation                               determined - Yellow status/monitoring in progress

                                                                                                                                                                                                              Memory issue caused a portion
                                                                                                                                                                                                              of the clustered environment to     The new
                                                                                                                                                                                                              be replaced. The clustered        hardware was
                                                                                                                                                                                          Cell will contain   hardware had specific            correctly setup                     Veritas clustering software is a new
                                                                                                                                                                                         market participant   identification required by the   and transaction                     solution being used to implement SCR745.
                                                                                           Memory failure in the clustered                                                   Unplanned   and end customer     cluster software which cause the processing was                      This issue was a result of learning how to
 1/3/07       1/4/07                         5:21PM    7:31PM       130       Paperfree    environment                     Infrastructure           Monitoring                Outage     impact - if known    clustered solution to fail.      able to resume         1/3/07       work with this software.                   Complete - Green status
                                                                                                                                                                                          Cell will contain
              1/2/07,                                                                                                                                                                    market participant                                     Index rebuilds
              1/3/07,                          1/2       1/3                   TIBCO                                                                                         Unplanned   and end customer     Indexing failed on the TIBCO    were required to                     Siebel Batch transactions were delayed on
 1/2/07       1/4/07                         3:20PM    9:50AM       750       Database     TIBCO database indexing issue Human Error                Monitoring                Outage     impact - if known    database                        resolve the issue       1/3/07       1/2/07 due to this issue                  Complete - Green status


                                                                                                                                                                                                                                              Pulse monitoring
                                                                                                                                                                                                                                                   has been
                                                                                                                                                                                                                                              implemented to
                                                                                           Following an emergency                                                                         Cell will contain                                   alert Operations
                                                                                           migration to correct the                                                                      market participant                                    if transactions
                                              12/21     12/22                              Customer Care process, not all                                                    Unplanned   and end customer An inbound processing                     are not
12/21/06     12/22/06                        5:45PM    8:45AM       540        TIBCO       components were reinstated     Human Error               Monitoring                Outage     impact - if known component was not turned on           processing          12/22/06                                                 Complete - Green status

                                                                                                                                                                                          Cell will contain                                   Partitioning rules
                                                                                                                                                                                         market participant                                      changed to
                                                                               TIBCO       TIBCO database partitioning                                                       Unplanned   and end customer Table space on the TIBCO              ensure table
12/14/06     12/14/06                        12:00AM   8:22AM       502       Database     error                           Human Error              Monitoring                Outage     impact - if known database was consumed              space is available     12/14/06      Stabilization from RBP implementation      Complete - Green status
           11/14/06,
           11/28/06,                                                                                                                                     Cell will contain
           11/29/06,                                                                                                                                    market participant                                                           Implementation of Retail Business
            12/7/06,     12/8     12/9                       Planned Maintenance Retail                                                      Planned    and end customer                                                             Processes - PR50121, PR50025,
12/8/06    12/10/06    7:00AM   10:10PM   2350    Retail     Release                                                Planned Maintenance      Outage     impact - if known n/a                               n/a           12/8/06    PR40007, PR50030, PR50059, PR50060              Complete - Implementation of RBP - Green status
                                                                                                                                                         Cell will contain
                                                             The NAESB application locked                                                               market participant                                                                                                           Complete - SCR745 Phase 1 would have handled
                                                             up and required a reboot. All                                                  Unplanned   and end customer                              Already included                                                               this issue - Implementation 11-27-06 Scheduled
12/5/06    12/5/06     1:10PM   1:47PM    37     NAESB       processing suspended.         NAESB                    Monitoring               Outage     impact - if known Application frozen         in another project   12/5/06    System rebooted.                                completion Feb 16, 2007 - Green status
                                                                                                                                                         Cell will contain
                                                   TML       ERCOT TML Report Explorer                                                                  market participant
                                                 (Report     unavailable and Lodestar Batch                                                  Planned    and end customer                                                             Pre-migration activities to prepare for the
12/4/06    12/4/06     7:00PM   9:00PM    120    Explorer)   Extracts delivered late        TML (report explorer)   Planned Maintenance      Outage     impact - if known n/a                               n/a           12/4/06    RBP release on 12/8/06                          Complete - Green status
                                                                                                                                                         Cell will contain
                                                                                                                                                        market participant
           10/30/06,                                         Planned Maintenance Retail                                                      Planned    and end customer                                                             SCR745 Phase 2 - Paperfree
11/11/06   11/11/06    12:00PM 10:49PM    649     Retail     Release                                                Planned Maintenance      Outage     impact - if known n/a                               n/a             n/a      implementation - PR60006                        Complete - Green status
                                                                                                                                                         Cell will contain
                                                             The NAESB application locked                                                               market participant                                                                                                           Complete - SCR745 Phase 1 would have handled
                                                             up and required a reboot. All                                                  Unplanned   and end customer                              Already included                                                               this issue - Implementation 11-27-06 Scheduled
10/17/06   10/17/06    10:39AM 10:57AM    18     NAESB       processing suspended.         NAESB                    Monitoring               Outage     impact - if known       Application frozen   in another project   10/17/06   System rebooted.                                completion Feb 16, 2007 - Green status
                                                                                                                                                         Cell will contain
           10/4/06,                                                                                                                                     market participant
           10/9/06,                                                                                                                          Planned    and end customer
10/15/06   10/13/06    8:00AM   8:00PM    720     Retail     Planned Maintenance Outage                             Planned Maintenance      Outage     impact - if known n/a                               n/a             n/a                                                      Complete - Green status
                                                                                                                                                         Cell will contain
                                                                                                                                                        market participant                                                           Conversion to the IBM P series server will
                                                             Critical memory issue on a                                                     Unplanned   and end customer                              Already included               allow for hot swap capabilities for this type
10/11/06   10/11/06    5:08PM   6:10PM    62      Retail     Retail Database Server          Infrastructure         Monitoring               Outage     impact - if known Hardware failure           in another project   10/11/06   of issue.                                       Complete - Green status
                                                                                                                                                         Cell will contain                                                           ERCOT solution in the process of being
                                                                                                                                                        market participant                                                           implemented as discussed in the Jan '07
                                                             Paperfree File Server network                                                  Unplanned   and end customer                              ERCOT effort                   RMS Retail Transaction Processing               In implementation - Completion date to be
10/2/06    10/3/06     7:46PM   7:56PM    18     Paperfree   outage                          Infrastructure         Monitoring               Outage     impact - if known Hardware failure             underway           10/3/06    presentation                                    determined - Yellow status/monitoring in progress
                                                                                                                                                         Cell will contain                                                           ERCOT solution in the process of being
                                                                                                                                                        market participant                                                           implemented as discussed in the Jan '07
                                                             Paperfree File Server not                              Market Notification /   Unplanned   and end customer                              ERCOT effort                   RMS Retail Transaction Processing               In implementation - Completion date to be
9/25/06    9/25/06     1:58AM   3:47PM    829    Paperfree   responding                      Infrastructure         Monitoring               Outage     impact - if known NIC driver error             underway           9/25/06    presentation                                    determined - Yellow status/monitoring in progress
ERCOT Retail Systems Degradation Log
July 26, 2006 - February 27, 2007 Degradations
 Issue      Notification                     Start                                                                                                  Identification      Service                                                                                          Date (to be)
  Date         Date        Notification ID   Time     End Time     Application            Issue Description              Degradation Type              Method           Impact          Market Impact                Root Cause                    Resolution           Implemented                         Notes                                     Current Status

                                                                                                                                                                                        Cell will contain
                                                                                                                                                   Monitoring /       Intermittent     market participant Domain controller and DNS
                                                                                   Intermittent TML and                                            Market             accessibility    and end customer issues due to planned
2/26/07       2/26/07                        9:45AM   12:15PM          TML         MarkeTrak connectivity issues   TML Connectivity                Notification          issues        impact - if known maintenance                             Server replaced           2/26/07                                                  Complete - Green status
                                                                                                                                                                                                          A subset of extracts were not
                                                                                                                                                                                                          generated due to the planned
                                                                                                                                                                                        Cell will contain outage unexpectedly affecting         Extract runs were
              2/12/07,                                                                                                                                                                 market participant ERCOT’s ability to catch up on      executed following the
2/10/07 -     2/13/07,                                                TML                                                                                                              and end customer the data lag issues (1/23/07            completion of the                       ERCOT has implemented processes to
 2/11/07      2/16/07                         n/a       n/a      (Report Explorer) Delay in extract posting        Extract Delay                   Monitoring         Extract delay    impact - if known market notice)                          planned outage            2/12/07      minimize this type of impact on the market. Complete - Green status

                                                                                                                                                                                        Cell will contain                                                                               ERCOT solution in the process of being
                                                                                                                                                                                       market participant ERCOT has identified a                                                        implemented as discussed in the Jan '07
                                                                                                                   Paperfree Duplicate                                Transaction      and end customer communication protocol issue              ERCOT effort                          RMS Retail Transaction Processing            In implementation - Completion date to be
 2/6/07       2/7/07                          n/a       n/a         Paperfree      Duplicate transactions (29)     Transactions                    Monitoring         Duplication      impact - if known between Paperfree servers                 underway                  n/a        presentation                                 determined - Yellow status/monitoring in progress
                                                                                                                                                                                                                                                                                        The overlying issue was a performance
                                                                                                                                                                                                                                                                                        issue in Siebel causing transactions to take
                                                                                                                                                                                        Cell will contain                                                                               longer to process than normal at certain
                                                                                   'DUP' reject transactions (790)                                                       Reject        market participant                                    Database maintenance                       times of the day. This was resolved with
 2/5/07,      2/7/07,                                                              sent due to multiple processing                                                    transactions     and end customer Siebel processed the same            to resolve performance                     database maintenance during a planned
 2/7/07       2/9/07                          n/a       n/a           Siebel       of the same transactions        Siebel Duplicate Transactions   Monitoring         sent in error    impact - if known transactions multiple times                  issues               2/11/07      maintenance window.                          Complete - Green status


                                                                                                                                                                                        Cell will contain                                                                               ERCOT solution in the process of being
                                                                                                                                                                                       market participant ERCOT has identified a                                                        implemented as discussed in the Jan '07
                                                                                                                   Paperfree Duplicate                                Transaction      and end customer communication protocol issue              ERCOT effort                          RMS Retail Transaction Processing           In implementation - Completion date to be
1/31/07       2/1/07                          n/a       n/a         Paperfree      Duplicate transactions (32)     Transactions                    Monitoring         Duplication      impact - if known between Paperfree servers                 underway                  n/a        presentation                                determined - Yellow status/monitoring in progress

                                                                                                                                                                                        Cell will contain                                                                               ERCOT solution in the process of being
                                                                                                                                                                                       market participant ERCOT has identified a                                                        implemented as discussed in the Jan '07
1/28/07,                                                                                                           Paperfree Duplicate                                 Transaction     and end customer communication protocol issue              ERCOT effort                          RMS Retail Transaction Processing           In implementation - Completion date to be
1/29/07       1/30/07                         n/a       n/a         Paperfree      Duplicate transactions (60)     Transactions                    Monitoring          Duplication     impact - if known between Paperfree servers                 underway                  n/a        presentation                                determined - Yellow status/monitoring in progress
                                                                                                                                                                         Delay in
                                                                                   ERCOT has been experiencing                                                         processing                                                                                                       The overlying issue was a performance
                                                                                   periodic failures of some                                                               until                                                                                                        issue in Siebel causing transactions to take
                                                                                   transactions as they process to                                                   reprocessing is    Cell will contain                                                                               longer to process than normal at certain
                                                                                   Siebel in some cases resulting                                                        complete      market participant                                    Database maintenance                       times of the day. This was resolved with
                                                                                   in “OtherPermissionDenied”                                                           (generally     and end customer Siebel database performance          to resolve performance                     database maintenance during a planned
1/25/07       1/25/07                         n/a       n/a       TIBCO/Siebel     reject responses                Transaction Processing Errors Monitoring             same day)      impact - if known issues                                       issues               2/11/07      maintenance window.                          Complete - Green status

                                                                                                                                                                                      Cell will contain
                                                                                                                                                                                     market participant
                                                                      TML          Siebel Service Order Extracts                                                                     and end customer                                                                                                                               More information to be presented the next TDTWG
1/23/07       1/23/07                         n/a       n/a      (Report Explorer) delay                           Extract Delay                   Monitoring         Extract delay impact - if known Slow system performance                           TBD                  n/a                                                    - Yellow status
                                                                                                                                                                                                        ERCOT determined a data              ERCOT isolated the
                                                                                                                                                                     Confusion and Cell will contain synchronization issue existed           times for the missing
              1/18/07,                                                             Naming error on Siebel Service                                                        delay in    market participant between the Lodestar source          data and repaired the
1/16/07 -     1/19/07,                                                TML          Orders Extracts and missing                                     Market              information and end customer system and the Lodestar                  data set in the Lodestar                   Operating processes are being updated to
 1/17/07      1/22/07                         n/a       n/a      (Report Explorer) data on Load Extracts          Extract Delay                    Notification         availability impact - if known Operational Data Store (ODS)          ODS                           1/22/07      include additional monitoring               Complete - Green status

                                                                                                                                                                          Delayed     Cell will contain     Processing issue within                                                     ERCOT solution in the process of being
                                                                                   Outbound 814 transaction                                                          processing of a market participant   Paperfree where certain file  All of these transactions                       implemented as discussed in the Jan '07
1/17/07 -     1/19/07,                                                             processing issue impacting                                                            subset of   and end customer types within a directory were not have been reprocessed                           RMS Retail Transaction Processing           In implementation - Completion date to be
 1/19/07      1/23/07                         n/a       n/a         Paperfree      subset of transactions          Transaction Processing Errors Monitoring            transactions impact - if known         correctly transferred             by ERCOT                   1/23/07      presentation                                determined - Yellow status/monitoring in progress
                                                                                                                                                                                                        When the data conversion was
                                                                                                                                                                                                        done to support the
                                                                                                                                                                                                        implementation of RBP, a
                                                                                   Some transactions are not                                                                          Cell will contain change was made that is                                                         All accepted pre-RBP orders and all
                                                                                   displaying on the Texas Market                                                                    market participant causing transactions that were                                                  accepted or rejected post-RBP orders are
              1/10/07,                                                             Link (TML) Find Transactions                                    Market            Information not and end customer submitted and rejected prior to                                                   displaying correctly. All service order
1/10/07       2/15/07                         n/a       n/a            TML         function.                      TML Functionality Error          Notification          available   impact - if known RBP to not display on TML        A SIR has been written             2/11/07      transactions should be visible.             Complete - Green status

                                                                                                                                                                                        Cell will contain
                                                                                                                                                                      Intermittent     market participant
                                                                                   Intermittent TML and API                                        Market             accessibility    and end customer Emergency Lodestar database
 1/5/07       1/5/07                          n/a       n/a            TML         connectivity issues             TML Connectivity                Notification          issues        impact - if known partitioning maintenance            Partitioning completed        1/5/07                                                   Complete - Green status

                                                                                                                                                                         Delayed      Cell will contain The way in which the TIBCO           Once the services were
                                                                                                                                                                     processing of a market participant Siebel Adapter and Siebel            restarted transaction
                                                                                                                                                                        subset of    and end customer Services were restarted caused         processing returned to
 1/3/07       1/4/07                          n/a       n/a       TIBCO/Siebel     High exception counts           Transaction Processing Errors Monitoring           transactions impact - if known processing issues                       normal                        1/26/07      RBP stabilization                           Complete - Green status
                                                                                                                                                                                                                                               Market Participants
                                                                                                                                                                        TML only                                                               have been asked to
                                                                                   Entering a phone number while                                                         814_03         Cell will contain   The phone number entry is         leave the phone entry
             12/20/06,                                                             creating customer                                                                    outbound       market participant   causing the TE segment of the        field blank until a
             12/29/06,                                                             enrollments/move ins/drops is                                                       transaction     and end customer     814_03 to be created but blank     Priority 1 SIR can be
12/20/06      2/16/07                         n/a       n/a            TML         causing transaction failures  TML Functionality Error           Monitoring             failure      impact - if known    causing mapping status rejects          implemented            2/11/07                                                  Complete - Green status




                                                                                                                                                                        867s not
12/15/06,                                                                                                                                                              forwarded,
12/16/06,    12/18/06,                                                                                                                                               delay in Siebel
12/21/06,    12/28/06,                                                                                                                                               Service Order Cell will contain                                                                                    ERCOT solution in the process of being
12/22/06,    12/29/06,                                                                                                                                                 completion, market participant ERCOT has identified a                                                            implemented as discussed in the Jan '07
12/23/06,     1/2/07,                                                              TX Set validations not occurring                                                      delay in    and end customer communication protocol issue                ERCOT effort                          RMS Retail Transaction Processing           In implementation - Completion date to be
12/27/06      1/19/07                         n/a       n/a         Paperfree      on a subset of 867 transactions Transaction Processing Errors Monitoring           reprocessing impact - if known between Paperfree servers                     underway                 TBD         presentation                                determined - Yellow status/monitoring in progress
                                                                                                                                                               Market
                                                                                                                                                           Participants
                                                                                                                                                         utilizing the API Cell will contain
               12/14/06,                                                                                                                                 are receiving a market participant Processing time for submissions    Additional exception
12/14/06 -     12/15/06,                                                                                                               Market                  timeout    and end customer exceeded due to bad data           handling added to the
12/18/06       12/18/06          n/a      n/a        MarkeTrak      API timeout issues                 MarkeTrak Connectivity          Notification           response     impact - if known handling                           API functionality       12/18/06                                                  Complete - Green status

                                                                                                                                                                            Cell will contain
               12/11/06,                                                                                                                                  Intermittent     market participant Memory allocation issue due to a
               12/12/06,                                            Intermittent TML and API                                           Market             accessibility    and end customer process not closing connections Process was modified
12/11/06       12/13/06          n/a      n/a         TML/API       connectivity issues                TML Connectivity                Notification          issues        impact - if known properly                          and monitored            12/13/06                                                  Complete - Green status

                                                                                                                                                                            Cell will contain                                                                      Siebel Service Order Extract
                                                                                                                                                                           market participant                                                                      Load Extract
                                                       TML                                                                                                                 and end customer Planned reboots from 12/4           Report processing                  Generation Extract
12/5/06         12/5/06          n/a      n/a     (Report Explorer) Extract posting delays             Extract Delay                   Monitoring         Report Delay     impact - if known notification                          caught up              n/a      Settlement & Billing Extract                   Complete - Green status
                                                                    The MarkeTrak system was
                                                                    successfully implemented
                                                                    yesterday, November 13th.
                                                                    Throughout the day, ERCOT                                                                               Cell will contain
                                                                    Identified four recurring issues                                                       Intermittent    market participant                                 Configuration changes                Configuration, market user setups, etc
               11/13/06,                                            Market Participants                                                Market             accessibility    and end customer                                    and additional user                 were part of stabilization the day of market
11/13/06       11/14/06          n/a      n/a        MarkeTrak      experienced with MarkeTrak         MarkeTrak Connectivity          Notification           issues       impact - if known Stabilization                           training           11/14/06   go-live.                                       Complete - Green status
                                                                                                                                                         Transaction ID
                                                                                                                                                           Duplication
                                                                                                                                                         (Transactions
                                                                                                                                                          were caught
                                                                                                                                                          prior to being
                                                                                                                                                            sent to the     Cell will contain
                                                                                                                                                           market - no     market participant
                N/A (See                                            SBMI, SBMO, SBSW                                                                       notification    and end customer                                    Already included in                 Fix to be implemented with RBP,
11/9/06      Service Impact)   12:34PM   1:17PM        Siebel       transaction duplication            Siebel Duplicate Transactions   Monitoring            required)     impact - if known TBD                                 another project        12/9/06    Temporary fix implemented in October           Complete - Green status


                                                                                                                                                                            Cell will contain
                                                                                                                                                                           market participant
                                                                    SBMI, SBMO, SBSW                                                                      Transaction      and end customer                                    Already included in                 Fix to be implemented with RBP,
10/26/06        10/26/06       1:23AM    1:25AM        Siebel       transaction duplication            Siebel Duplicate Transactions   Monitoring         Duplication      impact - if known TBD                                 another project        12/9/06    Temporary fix implemented in October           Complete - Green status

                                                                                                                                                                            Cell will contain
                                                                    Outbound processing of 814                                                                             market participant
                                10/14     10/15                     and 824 transactions                                               Manual                              and end customer                                    Already included in                                                                Complete - SCR745 Phase 2 - implementation 11-
10/14/16        10/16/06       7:40AM    7:10AM      Paperfree      suspended                          Slow System Performance         Troubleshooting    Degradation      impact - if known Process hung                        another project        10/15/06   Process restarted.                             11-06 - Green status

                                                                    ERCOT was unable to                                                                                     Cell will contain
                                                                    successfully create and post all                                                                       market participant
                                                       TML          Siebel Service Order Full                                          Market                              and end customer
10/10/06        10/10/06         n/a      n/a     (Report Explorer) Extracts                         Extract Delay                     Notification       Report Delay     impact - if known Disk full on storage device      One time occurrence       10/10/06                                                  Complete - Green status
                                                                    Errors when downloading
                                                                    Settlement Statements and the
                                                                    following extracts:                                                                                     Cell will contain
                                                                    ESIID Service History and                                          Market             Settlements      market participant
                                                       TML          Usage, Settlement & Billing,                                       Notification /     and Extracts     and end customer
10/9/06         10/9/06          n/a      n/a     (Report Explorer) Load, Generation                 Extract Delay                     Monitoring         not available    impact - if known Unknown                          One time occurrence         n/a                                                     Complete - Green status

                                                                                                                                                                         Cell will contain
                                                                                                                                                                        market participant
                                                                    SBMI, SBMO, SBSW                                                                     Transaction ID and end customer                                       Already included in                 Fix to be implemented with RBP,
10/5/06         10/5/06        12:36AM 12:42AM         Siebel       transaction duplication            Siebel Duplicate Transactions   Monitoring         Duplication   impact - if known TBD                                    another project        12/9/06    Temporary fix implemented in October           Complete - Green status


                                                                                                                                                                            Cell will contain
                                                                                                                                                                           market participant
                                                                    Delayed 867_03 IDR and NIDR                                                                            and end customer                                    Already included in                 Increased Hawk monitoring to capture and
9/27/06         9/27/06          n/a      n/a         TIBCO         Activity Reports            Slow System Performance                Monitoring         Report Delay     impact - if known Data Exception                      another project        9/28/06    correct exception immediately            Complete - Green status

                                                                                                                                                                            Cell will contain
                                                                                                                                       Market                              market participant
                                                                                                                                       Notification /     Transaction      and end customer                                   Not already included in                                                             Complete - SCR745 Phase 2 - implementation 11-
9/26/06         9/29/06        12:02AM   4:04PM      Paperfree      997s not sent during outages       Lost Transactions               Daily Report          Loss          impact - if known TBD                                  another project         n/a      Result of Paperfree outage                     11-06 - Green status

                                                                                                                                                                            Cell will contain
                                                                                                                                       Market                              market participant
                                                                                                                                       Notification /     Transaction      and end customer                                   Not already included in                                                             Complete - SCR745 Phase 2 - implementation 11-
9/25/06         9/29/06        1:58AM    3:47PM      Paperfree      997s not sent during outages       Lost Transactions               Daily Report          Loss          impact - if known TBD                                  another project         n/a      Result of Paperfree outage                     11-06 - Green status

                                                                                                                                                                            Cell will contain                                                                      ERCOT solution in the process of being
                                                                                                                                                                           market participant ERCOT has identified a                                               implemented as discussed in the Jan '07
9/22/06 -       9/22/06 -                                           814 and 867 transactions           Paperfree Duplicate                                Transaction      and end customer communication protocol issue          ERCOT effort                     RMS Retail Transaction Processing              In implementation - Completion date to be
 9/26/06         9/27/06         n/a      n/a        Paperfree      duplicated                         Transactions                    Monitoring         Duplication      impact - if known between Paperfree servers             underway               n/a      presentation                                   determined - Yellow status/monitoring in progress

                                                                    Market Participants receiving                                                                         Cell will contain
                                                                    the “No ESI ID found” message                                                                        market participant
                                                                    when using the API and to                                          Market            Information not and end customer                                      Already included in
9/20/06         9/20/06          n/a      n/a         TML API       research                      TML Functionality Error              Notification          available   impact - if known Code issue                            another project        9/28/06                                                   Complete - Green status

                                                                                                                                                                         Cell will contain
                                                                    Siebel Batch continued                                                                              market participant
                                                                    following SBMI issue with                                          Market                Order      and end customer
9/21/06         9/21/06        ~2:00AM ~4:00AM         Siebel       future dated evaluation date       Other                           Notification       Cancellations impact - if known Incorrect date used                 One time occurrence       9/21/06    Process change and Training implemented Complete - Green status

                                                                                                                                                                         Cell will contain
                                                                                                                                                                        market participant
                                                                    SBMI, SBMO, SBSW                                                                     Transaction ID and end customer                                       Already included in
9/21/06         9/21/06        12:55AM   1:03AM        Siebel       transaction duplication            Siebel Duplicate Transactions   Monitoring         Duplication   impact - if known TBD                                    another project        12/9/06    Fix to be implemented with RBP                 Complete - Green status
                                                                                                                                                      Cell will contain
                                                                                                                     Market                          market participant
                                                    Annual Validation backlog of                                     Coordination /                  and end customer Annual Validation and system                                                                                Complete for 2006 - Green status - long term fix to
9/19/06     9/19/06     n/a     n/a     Paperfree   814_20 Maintains                 Slow System Performance         Monitoring          Backlog     impact - if known limitations                      Annual occurrence       n/a     current backlog is ~300K                  be implemented in Feb 07

                                                                                                                                                      Cell will contain
                                                                                                                                                     market participant
                                                    SBMI, SBMO, SBSW                                                                  Transaction ID and end customer                                   Already included in
9/11/06     9/11/06   12:46AM 12:49AM    Siebel     transaction duplication          Siebel Duplicate Transactions   Monitoring        Duplication   impact - if known TBD                                another project     12/9/06   Fix to be implemented with RBP            Complete - Green status

                                                                                                                                                      Cell will contain
                                                                                                                                                     market participant
8/15/06 -                                           ESIID address deleted from                                     Exception            Rejected     and end customer                                   Already included in
 9/8/06     9/8/06      n/a     n/a      Siebel     database                         Transaction Processing Errors Processing         Transactions   impact - if known Software Bug                       another project       n/a     Implementation date to be determined      Complete - Green status

                                                                                                                                                      Cell will contain
                                                                                                                     Market                          market participant
                                                    Posting delay for extracts and                                   Notification /                  and end customer                                   Already included in                                                       Complete - Architecture change in place - Green
 9/5/06     9/5/06      n/a     n/a      TIBCO      reports                          Extract Delay                   Monitoring       Report Delay   impact - if known Component Unresponsive             another project     9/6/06    Redefined Hawk monitoring agent rules     Status


                                                                                                                                                      Cell will contain
                                                                                                                                                     market participant
                                                    SBMI, SBMO, SBSW                                                                  Transaction ID and end customer                                   Already included in
8/30/06     8/31/06   9:17PM   9:28PM    Siebel     transaction duplication          Siebel Duplicate Transactions   Monitoring        Duplication   impact - if known TBD                                another project     12/9/06   Fix to be implemented with RBP            Complete - Green status

                                                                                                                                                      Cell will contain
                                                                                                                     Market                          market participant
                                                    Posting delay for extracts and                                   Notification /                  and end customer                                   Already included in                                                       Complete - Architecture change in place - Green
8/24/06     8/24/06     n/a     n/a      TIBCO      reports                          Extract Delay                   Monitoring       Report Delay   impact - if known Component Unresponsive             another project     8/24/06   Implementation of Hawk monitoring agent   Status

                                                                                                                                                      Cell will contain
                                                                                                                                                     market participant
                                                                                                                                                     and end customer                                   Already included in                                                       Complete - Architecture change in place - Green
 8/8/06     8/8/06      n/a     n/a      TIBCO      Delayed TDSP ESIID Report        Extract Delay                   Monitoring       Report Delay   impact - if known Code issue with implementation     another project     8/24/06                                             Status

                                                                                                                                                      Cell will contain
                                                                                                                                                     market participant
                                                    SBMI, SBMO, SBSW                                                                  Transaction ID and end customer                                   Already included in
 8/2/06     8/2/06    1:10AM   1:13AM    Siebel     transaction duplication          Siebel Duplicate Transactions   Monitoring        Duplication   impact - if known TBD                                another project     12/9/06   Fix to be implemented with RBP            Complete - Green status


                                                                                                                                                      Cell will contain                                                                 ERCOT solution in the process of being
                                                                                                                                                     market participant ERCOT has identified a                                          implemented as discussed in the Jan '07
                                                    814 and 867 transactions         Paperfree Duplicate                               Transaction   and end customer communication protocol issue        ERCOT effort                  RMS Retail Transaction Processing         In implementation - Completion date to be
7/27/06     7/27/06   1:00AM   9:30AM   Paperfree   duplicated                       Transactions                    Monitoring        Duplication   impact - if known between Paperfree servers           underway             n/a     presentation                              determined - Yellow status/monitoring in progress
2007 Retail Transaction Processing Service Availability
                                                 Gross Available        Planned Outage    Net Available      Unplanned          Percent
        Month                  Service              Minutes                 Minutes         Minutes        Outage Minutes      Availability

January         Retail Transaction Processing                  44,640               540           44,100               1,637        96.29%

February        Retail Transaction Processing                  40,320             3,060           37,260                365         99.02%

March           Retail Transaction Processing                  44,640                             44,640                           100.00%

April           Retail Transaction Processing                  43,200                             43,200                           100.00%

May             Retail Transaction Processing                  44,640                             44,640                           100.00%

June            Retail Transaction Processing                  43,200                             43,200                           100.00%

July            Retail Transaction Processing                  44,640                             44,640                           100.00%

August          Retail Transaction Processing                  44,640                             44,640                           100.00%

September       Retail Transaction Processing                  43,200                             43,200                           100.00%

October         Retail Transaction Processing                  44,640                             44,640                           100.00%

November        Retail Transaction Processing                  43,200                             43,200                           100.00%

December        Retail Transaction Processing                  44,640                             44,640                           100.00%


                                                           84,960

Year to Date    Retail Transaction Processing   (to Feb. 28)                      3,600          81,360        2,002               97.54%
2006 Retail Transaction Processing Service Availability
                                                 Gross Available        Planned Outage       Net Available      Unplanned           Percent
        Month                  Service              Minutes                 Minutes            Minutes        Outage Minutes       Availability

January         Retail Transaction Processing                  44,640              1620              43,020               105           99.76%

February        Retail Transaction Processing                  40,320               990              39,330               556           98.59%

March           Retail Transaction Processing                  44,640               960              43,680                    0          100%

April           Retail Transaction Processing                  43,200               715              42,485                    0          100%

May             Retail Transaction Processing                  44,640             3,255              41,385               402           99.03%

June            Retail Transaction Processing                  43,200                    0           43,200                    0          100%

July            Retail Transaction Processing                  44,640               390              44,250               304           99.31%

August          Retail Transaction Processing                  44,640               885              43,755                    0          100%

September       Retail Transaction Processing                  43,200                    0           43,200             1,071           97.52%

October         Retail Transaction Processing                  44,640               720              43,920               183           99.58%

November        Retail Transaction Processing                  43,200               569              42,631                    0          100%

December        Retail Transaction Processing                  44,640             3,790              40,850             1,079           97.36%


                                                          525,600

Year to Date    Retail Transaction Processing   (to Dec. 31)                    13,894             511,706              3700           99.27%

								
To top