Docstoc

TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked

Document Sample
TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked Powered By Docstoc
					       National Smart Metering Program




                                        National Stakeholder
                                        Steering Committee
                                        National Testing Framework
                                        – Testing Strategy


                                                                                                         Version 0.11    Deleted: 0
                                        Version number:

                                        Status:                                                   Draft for Discussion

                                        Author:                                                         Debbie Dodd

                                                                                                      31 March 2010      Deleted: 11
                                        Date published:

                                        File name:                                TFWG workshop 05 outcome -
                                                                            Testing Framework Testing Strategy
                                                                                      V0.11 changemarked.doc




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc          Page 1 of 29
Security Classification: Unrestricted
                                         NSSC – National Smart Metering Program
                                       National Testing Framework – Testing Strategy




Table of Contents
1       DOCUMENT CONTROL ................................................................................................................................ 3
    1.1         VERSION CONTROL..................................................................................................................................... 3
    1.2         APPROVAL .................................................................................................................................................. 3
    1.3         REFERENCES ............................................................................................................................................... 4
2       INTRODUCTION ............................................................................................................................................. 5
    2.1         BACKGROUND ............................................................................................................................................ 5
    2.2         SCOPE OF THE TESTING FRAMEWORK ......................................................................................................... 6
    2.3         TESTING FRAMEWORK DOCUMENTATION SUITE ........................................................................................ 7
    2.4         PURPOSE - TESTING STRATEGY................................................................................................................... 8
3       TESTING FRAMEWORK GOVERNANCE ................................................................................................. 9
    3.1         INTRODUCTION ........................................................................................................................................... 9
    3.2         STAKEHOLDERS .......................................................................................................................................... 9
    3.3         TESTING FRAMEWORK DOCUMENTATION GOVERNANCE ......................................................................... 10
    3.4         TESTING FRAMEWORK EXECUTION GOVERNANCE ................................................................................... 12
4       TESTING PRINCIPLES ................................................................................................................................ 16
    4.1         BACKGROUND ON TESTING....................................................................................................................... 16
    4.2         OVERVIEW OF TESTING PRINCIPLES ......................................................................................................... 16
    4.3         SCOPE OF TESTING ACTIVITIES .................................................................................................................. 18
    4.4         NSMP TEST LEVELS................................................................................................................................. 23
5       APPENDIX A – GLOSSARY ......................................................................................................................... 24
6       APPENDIX B - INDUSTRY TESTING STRUCTURE............................................................................... 26



Table of figures
Figure 1 – Testing Framework Scope ........................................................................................................................... 6
Figure 2 – NSMP Testing Framework Document Structure ......................................................................................... 7

Table of tables
Table 1 – NSMP Testing Framework Stakeholders ...................................................................................................... 9
Table 2 – Test Levels to Test Activities Matrix .......................................................................................................... 23




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc                                                       Page 2 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy




1                  Do c u m e n t Co n tro l
     1.1         Ve rs io n Co n tro l

 Version       Date                     Description                             Amended by

 0.1           7 Sep 2009               First draft – internal review at IV&V   Debbie Dodd
                                        Australia and Harry Koller
 0.2           11 Sep 2009              Updated with review comments by         Debbie Dodd
                                        Harry Koller and Hannah Williams
                                        PwC.
 0.3           15 Sep 2009              Draft for TFWG Meeting #1               Debbie Dodd
 0.4           02 Oct 2009              Updates based on TFWG Meeting #1        Debbie Dodd
                                        comments
 0.5           16 Nov 2009              Updates based upon the TFWG             Debbie Dodd
                                        Meeting #2 agreements on the
                                        member feedback.
 0.6           27 Nov 2009              Updates based upon the TFWG             Debbie Dodd and Harry
                                        Meeting #3 comments.                    Koller
 0.7           4 Dec 2009               Updates based upon review               Debbie Dodd
                                        comments from AEMO,
                                        EnergyAustralia, Origin Energy and
                                        Harry Koller.
 0.8           7 Dec 2009               Final review comments from the          Debbie Dodd
                                        TFWG members
 0.9           5 Mar 2010               Updated wording on volume and           Debbie Dodd
                                        stress testing, sections 4.3.3, 4.3.5
                                        and 4.3.6.
                                        Added wording to sections 3.4.1 and
                                        3.4.2 with regard to supporting
                                        volume and stress testing should they
                                        be deemed necessary.
 0.10          11 Mar 2010              Updated with review comments from       Debbie Dodd
                                        Dale Rawlinson.                         Dale Rawlinson (Origin)
 0.11          30 Mar 2010              Updated with review comments on         Debbie Dodd
                                        Volume Testing by the TFWG at
                                        Meeting #5.
                                        Replaced the framework
                                        documentation diagram



     1.2         Ap p ro va l

 Authorised by                            Signature                                       Date

 NSSC Program Director




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc      Page 3 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy


     1.3         Re fe re n c e s
The following documents are referred to in this document.

 Reference          Document Name                                 Version
 Number
      1.            Testing Framework Working Group –             Version 3.0 approved 7 December                Deleted: 2.1
                    Terms of Reference                            2009                                           Deleted: 19 August

      2.            B2B Industry Testing Strategy Tranche 1       Version 2.0
      3.            B2B Tranche 1 Post Implementation             Version 1.1, dated 1 December 2006
                    Report
      4.            Victorian AMI - Industry Testing Strategy     v2-1 090525_ draft




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc   Page 4 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy




2                  In tro d u c tio n
     2.1         Ba c kg ro u n d
            The National Stakeholder Steering Committee (NSSC) for the National Smart Metering Program
            (NSMP) has been established by the Ministerial Council on Energy (MCE) to develop a national
            framework for the rollout of smart metering infrastructure (SMI) including technical and
            operational requirements and changes to regulatory rules and procedures in the National
            Electricity Market (NEM).
            The inclusion of a Testing Framework work stream within the NSMP was agreed by the NSSC
            via the approval of a NSMP Change Request (18 March 2009) to facilitate:
                     •    better efficiency with a single coordinated effort that is reused as jurisdictions make
                          the decision to roll out smart meters
                     •    reduction of effort by cross-jurisdictional bodies (e.g. some Retailers, Metering Data
                          Providers (MDPs)) by application of a common framework
            The testing framework forms part of the national framework. The Testing Framework Working
            Group (TFWG) has been established by the NSSC to provide stakeholder recommendations on
            a testing framework to facilitate, subsequent to individual jurisdictions scheduling a smart meter
            roll-out, the following:
                     •    industry participants' ability to validate their smart metering related system builds and
                          processes
                     •    parties to bilaterally validate their smart metering systems and processes
                     •    protection of the national market processes i.e. validation that current retail market
                          and settlement processes will not be disrupted by those participants' new back office
                          systems implemented to handle the very large volumes of interval data and bi-
                          directional smart meter information transfers
                     •    validation of the operation of national procedures to support smart metering
                     •    validation of the operation of smart metering functions between participant interfaces
                          through central systems
                     •    validation that national smart metering functions perform to the required service
                          levels




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc      Page 5 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy




      2.2        S c o p e o f th e Te s tin g Fra m e wo rk
            The testing identified by the NSMP Testing Framework is the minimum expected to be
            undertaken in order to ensure compliance to NEM smart metering procedures and processes
            and that the market continues to operate post the implementation of smart metering.
            As per the TFWG Terms of Reference (TOR) Ref [1], the scope of the testing processes and
            procedures should include:
                     •    testing of participant interfaces to central IT systems (B2B and MSATS transactions
                          and meter data)
                     •    testing of business process functionality from the participant interface, through the
                          central systems, to other participants (Retailers, Australian Energy Market Operator
                          (AEMO), Distribution Network Service Providers (DNSPs), metering service
                          providers, MDPs)
                     •    testing to confirm that systems operate to the required performance levels given
                                                    1
                          volumes of data (AEMO , metering service providers, MDPs, DNSPs and Retailers)
            The figure below provides a pictorial representation of the scope.




                                    Boundary of
                                 Testing Framework
                                                                                                  Retailer
                                                                                                  systems

                                                                    Central Systems –
                                                                     B2B Gateway,
                                        Data                       MSATS Transactions,                AEMO
                                        Comms;                       Metering Data                   systems
                                        Meters                          Registered                  (MSATS; B2B)
                                                                   Participant External
                                                                        Interfaces

                                                                                                   DNSP
                                                        MDP                                       systems


                                                       AEMO
                                                      Accredited




                                          Figure 1 – Testing Framework Scope



1
    Participation in Volume Testing is not currently agreed with AEMO and is still under discussion.

File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc       Page 6 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy




     2.3         Te s tin g Fra m e wo rk Do c u m e n ta tio n S u ite
            The suite of documents will be developed in stages, as per the TFWG Terms of Reference. The
            stages are as follows:
                 •     Stage 1 – Testing Strategy (this document)
                 •     Stage 2 - Development of national procedures and processes
                 •     Stage 3 - Development of test plans, detailed test cases and test scripts


            Figure 2 below shows the list documents to be developed at each stage.




                  STAGE 1                                            Testing Strategy:
                                                              Governance and Testing Principles




                  STAGE 2
                                                          Process                           Procedures
                                                        Documentation                      Documentation




                  STAGE 3
                                   Industry Technical         Industry Business     Industry Volume        Industry Regression
                                      Test Plan(s)               Test Plan(s)          Test Plan               Test Plan(s)




                                  Industry Technical
                                                             Industry Business    Industry Volume Test     Selected test cases,
                                     Test Cases &
                                                               Test Scenarios          Scenarios            scripts, scenarios
                                       Scripts




                            Figure 2 – NSMP Testing Framework Document Structure




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc                Page 7 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy




     2.4          P u rp o s e - Te s tin g S tra te g y
            This Testing Strategy document represents the Stage 1 deliverable of the development of the
            NSMP Testing Framework and provides:
                 •       Governance arrangements for the NSMP Testing Framework ongoing development and
                         maintenance.
                 •       Governance arrangements for use of the tests developed as part of the NSMP Testing
                         Framework in an implementation program.
                 •       Testing principles to define:
                     o     The scope of industry testing for the various test activities of the NSMP
                     o     Clear objectives for each level of testing to be undertaken
                     o     How the levels fit together resulting in a test program that provides efficient coverage
                           of mandatory requirements.
            Together, the governance arrangements and testing principles covered in this document will
            detail the agreement and commitment of AEMO and all registered participants to comply with
            the industry testing needs of the National Smart Metering Program.
            This document is intended as a focus for detailed planning of the activities, schedule,
            dependencies and resources associated with the testing tasks. It will also highlight current
            testing risks for which a mitigation strategy should be considered.




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc        Page 8 of 29
Security Classification: Unrestricted
                                      NSSC – National Smart Metering Program
                                    National Testing Framework – Testing Strategy


3                    Te s tin g Fra m ewo rk Go ve rn a n c e
     3.1         In tro d u c tio n
            Governance arrangements will cover two aspects:
                 •      Testing Framework Documentation Governance
                 •      Testing Framework Execution Governance
            It should be noted that currently there are no formal obligations in the National Electricity Rules
            (NER) or NEM Procedures for registered participants to participate in industry testing associated
            with changes to the NEM systems. It is proposed that the obligations of AEMO and registered
            participants for the Testing Framework Executive Governance (sections 3.4.1 and 3.4.2) be
            formalised in the NER or subsidiary Procedures, in order to make them effective.
            The past experience with industry testing, particularly B2B, was highly unsatisfactory for most
            registered participants because there were no effective sanctions to make sure all registered
            participants were certified as ready to go live. Many registered participants had to incur
            unnecessary costs because some registered participants went live with critical errors that
            impacted other registered participants’ backend systems and processes.

     3.2         S ta ke h o ld e rs
                 As stated in the TFWG TOR, the stakeholders for the Testing Framework program of work
                 are identified in the following table. The NSMP Testing Framework documentation will be
                 presented to the NSSC for endorsement and approval.
                                          Table 1 – NSMP Testing Framework Stakeholders


                     Role                                   Stakeholders

                     Sponsor                                National Stakeholder Steering Committee (NSSC)

                     Stakeholders: Critical                 Ministerial Council on Energy (MCE)

                                                            AEMO

                                                            Regulation Working Group (established under the
                                                            NSSC)

                     Stakeholders: Essential                Information Exchange Committee (IEC)
                                                            IEC reference group - the B2B Procedures
                                                            Reference Group (BPRG)
                                                            Retail Market Executive Committee (RMEC)
                                                            RMEC reference groups - the Metrology Reference
                                                            Group (MRG) and Business
                                                            Process and Data Reference Group (PDRG)

                     Stakeholder:                           Victorian Advanced Metering Infrastructure (AMI)
                                                            program
                     Interested parties




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc   Page 9 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy




     3.3         Te s tin g Fra m e wo rk Do c u m e n ta tio n Go ve rn a n c e
            These governance arrangements will include establishing the following:
                 •     Testing Baseline Definition
                 •     Testing Framework Documentation Management
                 •     Confidentiality Arrangements

                 3.3.1        Te s tin g Ba s e lin e De fin itio n
            The testing covered by the NSMP Testing Framework will have as its baseline the additional
            and modified NEM processes and procedures resulting from the implementation of smart
            metering. It will also cover the testing required of any associated processes and procedures that
            may not have changed but could be impacted by the smart metering changes, as identified by
            the NSSC Business Processes and Procedures Working Group (BPPWG).
            At the time of writing this document the content of the baseline for this NSMP testing framework
            is being developed by the NSSC BPPWG for approval by the National Electricity Market (NEM)
            participants. The format that this document will take will be influenced by the TFWG as testable
            requirements are required as the basis for the detailed tests.

                 3.3.2        Te s tin g Fra m e wo rk Do c u m e n ta tio n Ma n a ge m e n t
            There are three (3) parts to the testing framework documentation that will require management:
                 •     The suite of documents developed as part of this Testing Framework (Stages 1, 2 & 3)
                 •     Build and Test library (TFWG Stage 3), hosted in the selected test management tool.
                 •     Post implementation – management of the test artefacts from each jurisdiction’s testing
            The following sections deal with identifying the document management aspects, including:
                 •     Documentation Ownership
                 •     Change Management
                 •     Test Artefacts Management

                  3.3.2.1      Do c u m e n ta tio n Own e rs h ip
            It is recommended that this testing framework documentation suite (see Figure 2) ultimately be
            owned and maintained by AEMO working with registered participants, in accordance with
                                              2
            current AEMO internal practices. .

                  3.3.2.2      Ch a n g e Ma n a g e m e n t
            This Testing Strategy is a living document. Accordingly, any changes identified during the
            course of Stages 2 and 3 of the Testing Framework development or subsequently, will result in
            an update to this document.
            Prior to the handover of the Testing Framework to the eventual owner, this document will be
            updated and managed by the TFWG. Subsequently, changes to the Testing Framework



2
 This is in line with the recommendations from the TFWG to the NSSC as outlined in the NSMP Testing
Framework Scoping Discussion Paper_V0.9. This TFWG recommendation is also being explored with
AEMO/IEC/RMEC.

File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc   Page 10 of 29
Security Classification: Unrestricted
                                      NSSC – National Smart Metering Program
                                    National Testing Framework – Testing Strategy

            documentation will be in accordance with the practices of the eventual owner and agreed by the
            registered participants.
            As the testing baseline will be established and agreed, scope creep is not likely. However,
            changes to the testing baseline will be governed by the arrangements made by the NSSC and
            could result in impact on the Testing Framework documentation.

                  3.3.2.3       Te s t Arte fa c ts Ma n a g e m e n t
            The testing framework documentation suite owner will be responsible for archiving all testing
            artefacts and making them available to Industry. Testing artefacts are defined as the outputs of
            testing execution and will include, but not be limited to, the following:
                 •     Completed review checklists
                 •     Test results from each round of testing
                 •     Test Reports, including progress and final reports
                 •     Test data
                 •     Test schedules
                 •     Matrix of Testing Partners
                 •     Any reports or files produced while testing

                 3.3.3        Co n fid e n tia lity Arra n g e m e n ts
            The following information has been agreed by registered participants to be confidential:
                 •     Compliance obligations with the Privacy Act to be met by all registered participants and
                       AEMO. Customer personal details will include, but not be limited to, information such
                       as:
                            o     Name
                            o     Address
                            o     Location
                            o     Telephone number(s)
                 •     Commercially sensitive information to be protected within the industry and not in the
                       public domain.
                 •     Test Results will be transparent between participating registered participants, but not in
                       the public domain.
                 •     Vendor details
            Please note, the use of NMIs without customer personal details is not confidential information.




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc    Page 11 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy




     3.4           Te s tin g Fra m e wo rk Exe c u tio n Go ve rn a n c e
            Governance arrangements will be required to cover industry participation when the tests
            developed as part of this Testing Framework are actually implemented by a jurisdiction in a
            mandated rollout. These arrangements will include:
                   •    Obligations of AEMO                                                                           Deleted: Regulatory

                   •    Obligations of Industry Registered Participants                                               Deleted: Regulatory

                   •    Industry Testing Roles and Responsibilities
                   •    Software and System Release Criteria
                   •    Risk Management
            The IEC/RMEC may, subsequent to becoming the owners of this framework, consider the
            obligations on industry participants to undertake industry testing activities in other
            circumstances, such as:
                   •    Changes to NEM procedures                                                                     Formatted: Bullets and Numbering

                   •    New industry participants
                   •    Participants with changes to backend systems
                                                                                                                      Deleted: As noted earlier, it is
                   3.4.1       Ob lig a tio n s o f AEMO                                                              proposed to formalise the testing
                                                                                                                      framework execution obligations for
                                                                                                                      AEMO and registered participants in
            The following obligations are placed upon AEMO when industry testing is undertaken in a                   the NER or subsidiary Procedures.
            mandated jurisdictional rollout. This list includes current obligations under the NER as well as          Without effective sanctions (e.g. a Rule
                                                                                                                      breach), this governance framework
            voluntary obligations. AEMO will:
                                                                                                                      has no real teeth and diligent
                                                                                                                      companies will continue to incur
               •       Provide and maintain a Pre-production environment that:                                        unnecessary costs because of a lack of
                                                                                                                      diligence by other registered
                           o    Enables participants to test all smart metering processes and supports end to         participants.¶
                                end testing of transactions.                                                          Re g u la to ry

                           o    Supports Industry Technical Testing                                                   Deleted: regulatory
                                                                                                                      Deleted: are proposed for AEMO
                           o    Supports Participant and Industry Volume Testing and Industry Stress Testing
                                should any of them be considered necessary                                            Deleted: , where they don’.
                                                                                                                      Deleted: 3
                           o    Is available for the duration of any industry testing required by the industry.
                           o    Operates mechanisms or tools to measure the performance of its Pre-production
                                central systems during all Industry testing and share the results with industry.
               •       Communicate any known variations or deviations between Pre-production and
                       Production central systems to the Industry Test Manager prior to commencement of
                       Industry Technical Testing.
               •       Communicate changes with the Pre-production environment to all market participants
                       prior to changes being implemented.
               •       Provide feedback to the Industry Participants regarding any issues that are with AEMO to
                       resolve.
               •       Provide sufficient information about any changes to market systems to Industry
                       Participants, including those already using smart metering production systems, to allow
                       those participants to decide on the extent of regression testing and industry testing
                       participation required.




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc       Page 12 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy

               •     Continue to offer a means of market participation for small users without internal
                     systems.
                                                                                                                     Deleted: Re g u la to ry
                   3.4.2      Ob lig a tio n s o f In d u s try Re g is te re d P a rtic ip a n ts                   Deleted: 4

            The following voluntary obligations are proposed for industry registered participants in a               Deleted: NER
            mandated jurisdictional rollout. Each industry registered participant will:
               •     Ensure that their backend systems have been sufficiently tested or assessed as having
                     minimal risk of failure, to give certainty to compliance with the modified and new NEM
                     procedures and processes. This may include undertaking in-house functional and non-
                     functional testing (including performance testing), where individual businesses test their
                     transactions, data management and systems internally.
               •     Provide proof of successful completion of functional testing of backend systems or
                     documented risk evaluations by qualified staff, indicating which elements of testing were
                     deemed unnecessary or low risk.
               •     Communicate any known variations or deviations from live systems to the Industry Test
                     Manager prior to the participant industry test readiness review.
               •     Undertake Industry testing with test systems that are comparable to live systems. This
                     means providing a test environment that enables testing of the critical and high risk
                     elements of the smart metering processes and supports end to end testing of
                     transactions. For example, for performance of high volume or criticality transactions the
                     test environment would need sufficient test data, measurement tools and equipment
                     capable of handling production level data loads or tools to simulate the data load.
               •     Undertake Participant and Industry Volume Testing and Industry Stress Testing should
                     any of them be considered necessary
               •     Whenever possible, use a test external interface during industry testing.
               •     If using the production external interface for industry testing, be able to distinguish
                     between the production and test transactions and measure the performance.
               •     Retest any changes to internal systems prior to use in industry testing, even if the
                     changes occur during an industry testing cycle.
               •     Participate in testing of critical and high risk market transactions.
               •     Provide sufficient suitable resources (people and equipment) to deliver the Industry
                     Testing.
               •     Inform market participants of when they plan to test new or materially changed systems,
                     in an accurate and timely manner. Timeframes for this will need to be agreed between
                     participants for any testing session.
               •     If an affected Retailer, participate in Regression Testing of high priority market
                     transactions when each DNSP implements new or materially changed systems.
               •     Provide accurate information prior to and during Industry Testing to avoid significant
                     operational, consumer and cost impacts.
               •     Inform the industry testing organisation of any risks to their full participation in any
                     industry testing that they are part of.
               •     Provide sufficient resources for browser-based participation, if required
               •     If required, agree to partner with another participant for the use of testing resources




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc      Page 13 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy

               •     Where possible, ensure that any changes to existing processes and procedures are              Formatted: Bullets and Numbering
                     minimised, to reduce the impact on businesses that have not yet implemented smart
                     metering.
                                                                                                                   Deleted: ¶
                   3.4.3      In d u s try Te s tin g Ro le s a n d Re s p o n s ib ilitie s
            Industry Testing will require a co-operative approach from all industry participants under the
            obligations established in the NER, with each participant having a clear understanding of the
            resources required, both technical and managerial. In order to satisfactorily conduct testing
            among industry participants and AEMO, a structure is needed that reflects the testing
            organisation required to undertake the Industry Testing with formalisation of roles and
            responsibilities for each organisation participating in the testing at any given time.
            The information in Section 6, Appendix B, identifies the roles and responsibilities of AEMO and
            each registered participant as part of the organisation structure required during an
            implementation program. Each participant business and AEMO will be expected to apply the
            appropriately skilled personnel to each role and responsibility identified.

            There are roles identified as required with each participant organisation. The number of
            personnel is not dictated here, only the responsibilities required. One person may play more
            than one role.

            These roles and responsibilities will need to be fulfilled by the businesses participating each
            time industry testing is run.

                   3.4.4      S o ftwa re a n d S ys te m Re le a s e Crite ria
            Management and reporting of errors will be undertaken at each phase of testing. Error severity
            will be defined and error limits set for each level of testing. This entire process, including an
            appeal process, will be documented in Stage 2 of the Testing Framework.

                   3.4.4.1     AEMO

            Release to Pre-production Environment

            The release of new and modified processes and procedures to the test environment for
            registered participants’ use will be controlled by AEMO in accordance with their internal
            processes.

            The release of new and updated Responders to the pre-production environment for registered
            participants’ use will be controlled by AEMO in accordance with their internal processes.

            AEMO will provide all participants with a list of any known problems relating to this testing at the
            time of release to pre-production and any subsequent problems that are raised against the pre-
            production environment during participant testing activities.

            Release to the Production Environment

            The release of the tested processes and procedures to the production environment occurs twice
            a year for scheduled releases. Other releases outside these two can be negotiated with AEMO.
            Release to production is controlled by AEMO in accordance with their internal processes.

            AEMO will provide all participants with a list of any known problems relating to this testing at the
            time of release to production and any subsequent problems that are raised against the
            production environment during market operation.




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc   Page 14 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy

                   3.4.4.2     Re g is te re d P a rtic ip a n ts

            Release of new software or systems during industry testing

            All registered participant systems are expected to have undergone successful testing prior to
            usage with Industry testing.

            Any changes made to participant systems during Industry testing must undergo internal testing
            prior to use in Industry testing.

            Registered participants will be allowed to progress to the next level of testing if issues remaining
            unfixed in their backend systems and external interfaces do not exceed the error limits defined
            in the Stage 2 process documentation.

            Registered participants will be unable to progress to the next level of testing with any smart
            metering process or service that does not meet the error limits defined in the Stage 2 process
            documentation.

            Release of new software or systems for use in Production

            At the completion of Industry testing a period of time will be allowed for registered participants to
            migrate any changes made to their test internal systems to the production systems. All changes
            must have completed Industry testing successfully before being used in production.

            Registered participants will be allowed to engage in smart metering market processes with any
            smart metering process or service if issues remaining unfixed in their backend systems and
            external interfaces do not exceed the error limits defined in the Stage 2 process documentation.

            Registered participants will be unable to engage in smart metering market processes with any
            smart metering process or service that does not meet the error limits defined in the Stage 2
            process documentation.

                   3.4.5      Ris k Ma n a g e m e n t

            There are two (2) aspects to risk management:

               •     Planning risk management
               •     Test execution risk management
            A formal risk management process will be employed to identify and manage risks during the
            planning phase of the testing. This process will be defined during Stage 2 of this Testing
            Framework.

            Test execution risk management may be applied to the execution order and priority of tests. In
            order to be able to apply a risk-based approach, functions will be prioritised and individual
            requirements within each function given a priority. Tests will be written for all requirements and
            the prioritisation will be used by the industry test manager to manage the test execution phase.




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc   Page 15 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy




4                  Te s tin g P rin c ip le s
     4.1         Ba c kg ro u n d o n Te s tin g
            As this document will potentially be read by a readership with varying degrees of knowledge on
            testing, it is worthwhile adding a short summary on why testing is done and outlining the
            benefits.
            Testing is undertaken to validate that what has been developed (software, systems, processes
            and procedures) actually works and to verify that what has been developed is what was
            requested.
            Testing has the most value when it is applied by the appropriate people and resources to each
            component (in this case, participant internal systems, external interfaces and central market
            systems) followed by testing of the whole as a system, utilising likely real use scenarios. These
            scenarios generally cover multiple functions and add a degree of complexity to the testing.
            Testing at the component level will include expected operation as well as exception operation
            i.e. what happens when things don’t go to plan. This testing finds low level problems and results
            in robust components of the system, as problems are found and fixed before the components of
            the system start to interact.
            Once components are working correctly, the interfaces between them need to be tested
            independently of each of the components and then the components are tested together. This
            testing approach allows for a gradual build up and finds and fixes problems when it is cost
            effective to do so. It also gives confidence that individual components can communicate
            correctly.
            Whilst the Testing Framework does not cover all of the testing that participants will undertake
            (e.g. the participant internal systems and AEMO market systems are excluded) this strategy has
            an expectation that this testing will be undertaken.
            The Test Strategy offered by this Testing Framework follows these principles.

     4.2         Ove rvie w o f Te s tin g P rin c ip le s
            This NSMP Testing Framework documents all the industry testing that will be undertaken by
            participants. It consists of a range of testing activities that will be structured to achieve the most
            efficient and effective amount of test coverage with the minimum risk. These activities will range
            from industry technical testing and participant volume testing using the controlled pre-production
            environment, followed by business testing and industry volume testing with other participants.
            AEMO will develop NEM Systems in accordance with the agreed consultation time frames.
            Testing baselines would be developed to meet release dates. The intended build schedule will
            have an impact on the test design and development, test data design and development and test
            execution planning.
            Analysis of the approved NEM changes will be required to discover which ones (if any) can be
            tested and implemented before smart meters are deployed by businesses and which ones will
            require smart meters to be deployed and production-ready. It may be that this testing framework
            will need to support running testing on trial numbers of smart meters prior to a full scale rollout,
            where businesses are attempting to carry out complete end to end testing prior to commencing
            that large scale rollout.
            Industry testing will also require a minimum number of each type of participant to be effective
            and will need to take account of any jurisdictional differences, especially where participants




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc   Page 16 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy

            such as Retailers are active in more than one jurisdiction. The minimum numbers and type of
            participant will be determined when the baseline is better understood.
            The test process described in this and subsequent testing framework documents is a disciplined
            test methodology that will use the practices of “plan, specify, execute” to produce thorough,
            repeatable tests relevant to the participants’ requirements. It promotes the concept of “end to
            end” testing throughout each testing phase, to ensure that the most appropriate type of testing
            is done at the most effective stage of development by the people in the best position to do it.
            Where applicable, a risk-based approach will be taken with the Industry Testing activities. This
            means that functions and tests within them will be prioritised based upon criteria such as cost to
            test, criticality, complexity of the function, resources required and likelihood of failure. All of
            these factors will be taken into account when assessing the risk. Tests will be written for all
            functions, but prioritised to allow selection of the highest priority tests in the event of time or
            resource constraints.




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc   Page 17 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy




      4.3          S c o p e o f te s tin g a c tivitie s
            The scope of the industry testing activities is provided below. These activities are sequential
            and entry to the next testing activity requires successful completion of those previous.
            Industry testing will include the following:
                     •     Participant industry test readiness review
                     •     Industry Technical Testing - testing of participant interfaces to central IT systems
                           (B2B and Market Settlement and Transfer Solution (MSATS) transactions and meter
                           data)
                     •
                                                             5
                           Participant Volume Testing - testing to confirm that systems operate to the required
                           performance levels given volumes of data (AEMO, metering service providers,
                           MDPs, DNSPs and Retailers). This testing will involve each participant business
                           testing through the pre-production environment, but not with each other. A risk
                           assessment will be undertaken to determine if participant volume testing will need to
                           be executed.
                     •     Business Testing - testing of business process functionality from the participant
                           interface, through the central systems, to other participants (Retailers, Australian
                           Energy Market Operator (AEMO), Distribution Network Service Providers (DNSPs),
                           metering service providers, MDPs).
                     •
                                                         6
                           Industry Volume Testing – To be most effective, Industry Volume Testing will utilise
                           the same strategy as Business Testing described above, but with volumes of data
                           designed to exercise the business scenarios end to end (including participant
                           systems, external interfaces and AEMO central systems). While the AEMO and
                           participant systems are not under test, they are considered test equipment and must
                           be utilised for this testing to be effective. A risk assessment will be undertaken to
                           determine if industry volume testing will need to be executed.
                     •
                                                     7
                           Industry Stress Testing – uses the same equipment as Industry volume testing and
                           will be designed to test the behaviour of the system under extreme conditions, such
                           as the build up of interval data following extended communications outages. A risk
                           assessment will be undertaken to determine if industry stress testing will need to be
                           executed.
                     •     Participant Go/No Go to production review

                   4.3.1      P a rtic ip a n t In d u s try Te s t Re a d in e s s
            All participants will undergo a test readiness review. This Industry Test Readiness Review will
            be conducted prior to the commencement of any Industry testing. The review will be conducted
            in accordance with an agreed checklist and procedure and will ensure that all entry criteria to
            Industry testing have been met, document any deviations from expected and discuss any non-
            conformance issues. This Test Readiness review will be conducted at a time scheduled as part
            of the agreed testing schedule.
            Entry Criteria will include, but not be limited to, the following:
               •     Successful completion of testing, including risk analysis if testing is not carried out, of
                     backend systems

5
    The capability of the AEMO Pre-production environment will be assessed as part of the risk analysis.
6
    The capability of the AEMO Pre-production environment will be assessed as part of the risk analysis.
7
    The capability of the AEMO Pre-production environment will be assessed as part of the risk analysis.

File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc     Page 18 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy

               •       Prior to the commencement of industry technical testing, all participants will be required
                       to enter into the NSMP Defect Tracking tool any unresolved defects from internal testing
                       that could affect the success of the industry testing.
               •        All severity 1 or 2 defects must be fixed or a satisfactory workaround provided that
                       reduces the severity level to a 3, prior to participation in industry technical testing.
               •       Waivers need to be sought for any variation from the expected production systems
            The content of the Participant Industry Test Readiness Review will be documented as part of
            Stage 2 of the Testing Framework.
                                                                                                                      Deleted: ¶
                   4.3.2      In d u s try Te c h n ic a l Te s tin g
            Industry Technical Testing will be undertaken by participant businesses utilising testing
            harnesses called Responders. These Responders will allow participants to test independently of
            each other and find and fix any issues with backend test systems and discovered before
            involving other testing partners.

                   4.3.3      P a rtic ip a n t Vo lu m e Te s tin g
            A risk assessment, including risk criteria, will be undertaken during Testing Framework -Stage 3
            by the TFWG, utilising the agreed Testing Risk Management process documented in Stage 2.
            Any risk assessment undertaken outside of the NSMP will be undertaken by the registered
            participants and AEMO, in that same manner. The risk assessment will take into account the
            following aspects to determine whether participant volume testing is required:
                   •    The consequences to each participant of a failure to meet required performance levels
                   •    The defensive measures in place to prevent problems relating to volume occurring
                   •    The criticality of each of the functions under consideration to market operation
                   •    Determining the amount of data likely to be generated by each new or modified NEM
                        process or procedure and add that too the current data volumes supported
                   •    The cost of testing versus the benefit from executing participant volume tests
                   •    Quantify and document the differences between test environments and production
                        systems for participants and AEMO
                   •    The efficacy of the tests to be performed based on limitations (e.g. test environment
                        capability)
                   •    The capability of the AEMO Pre-production environment, including Responders, to
                        support the testing
                   •    Manual analysis of volumes to be supported by participants
                   •    The status of each participant’s back office systems with regard to expected capacity for
                        handling smart metering functions
                   •    Analysis of any real evidence from the Victorian AMI program on the NEM
            The risk assessment should be reviewed and any additional information available analysed by
            each jurisdiction prior to commencement of Industry Testing.
            Note that an outcome from the risk assessment could be to undertake volume testing of all or
            some NEM processes and procedures.
            In the event that the decision is made to conduct participant volume testing, it will be undertaken
            in accordance with the following principles.




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc       Page 19 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy

            This testing will be undertaken to confirm that systems operate to the required performance
            levels given volumes of data (AEMO, metering service providers, DNSPs and Retailers), within             Deleted: 8
            the limitations of test systems. This testing will involve each participant business testing with test
            backend systems through the pre-production environment, but not with each other. The
            Responders will support this testing and appropriate test data will need to be developed by              Deleted: 9
            individual participants. It will allow participants to ascertain the capability of their external
            interfaces to sustain the data volumes needed to support smart metering in conjunction with
            ensuring that other market transactions have not been affected.
            This testing will be useful to participants and AEMO to ascertain any changes that may be
            required to external interfaces, including transaction limits and will cover, if feasible, any aspect
            of the communication that is specifically configured for each participant.
            It must only be performed using stable and tested participant internal systems.

                 4.3.4        Bu s in e s s Te s tin g
            This testing covers business process functionality from the participant interface, through the
            central systems, to other participants (Retailers, AEMO, Distribution Network Service Providers
            (DNSPs), metering service providers).
            To be effective this testing takes an end-to-end approach using as near to production systems
            as possible. It will use test environments, which by their nature may differ from production
            systems.
            This testing will involve a partnering arrangement and can be entered by a participant after
            successful completion of Industry Technical Testing and Participant Volume testing.

                 4.3.5        In d u s try Vo lu m e Te s tin g

            A risk assessment, including risk criteria, will be undertaken during Testing Framework -Stage 3
            by the TFWG, utilising the agreed Testing Risk Management process documented in Stage 2.
            Any risk assessment undertaken outside of the NSMP will be undertaken by the registered
            participants and AEMO, in the same manner. The testing baseline will be analysed to determine
            the extent of testing necessary to protect the market in production.
            The risk assessment will take into account the following aspects to determine whether industry
            volume testing is required:
                 •     The consequences to the market and its participants of a failure to meet required
                       performance levels
                 •     The defensive measures in place to prevent problems relating to volume occurring
                 •     The criticality of each of the functions under analysis to market operation
                 •     The cost of testing versus the benefit from executing industry volume tests
                 •     Quantify and document the differences between test environments and production
                       systems for participants and AEMO
                 •     The efficacy of the tests to be performed based on limitations (e.g. test environment
                       capability)
                 •     The capability of the AEMO Pre-production environment to support the testing
                 •     Manual analysis of volumes to be supported by participants
                 •     The status of each participant’s back office systems with regard to expected capacity for
                       handling new smart metering functions
                 •     Lessons learnt from the Victorian AMI experience


File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc      Page 20 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy

            The risk assessment should be reviewed and any additional information available analysed by
            each jurisdiction prior to commencement of any Industry Testing.
            In the event that the decision is made to conduct industry volume testing, it will be undertaken in
            accordance with the following principles.
            Volume testing will be performed by applying expected loads while measuring the performance
            and observing the behaviour of the systems under test for the particular functionality or process,
            or group of functions and processes, being exercised. Volume testing falls into the category of
            Non-functional testing.
            To be effective this testing will take an end-to-end approach using as near to production
            systems as possible. It will use test environments, which by their nature may differ from
            production systems.
            Industry volume testing will ensure that likely business scenarios are exercised and
            performance measured, within the limits of the capacity of test systems.

                 4.3.6        In d u s try S tre s s Te s tin g

            A risk assessment, including risk criteria, will be undertaken during Testing Framework -Stage 3
            by the registered participants and AEMO, utilising the agreed Testing Risk Management
            process documented in Stage 2. The risk assessment will take into account the following
            aspects to determine whether industry stress testing is required:
                 •     Document the scenarios that could predictably lead to above average expected load
                 •     Quantify the impact on the market of an overload event
                 •     Determine whether the testing would be useful in ascertaining the cut off point at which
                       performance starts to degrade
                 •     Consider the adequacy of the defensive measures to determine whether the scenarios
                       that could potentially lead to above average load can be prevented from occurring             Deleted: are

                 •     Consider the defensive measure to determine what the likely reaction to an overload
                       would be
                 •     The cost of testing versus the benefit from executing industry stress tests
            In the event that the decision is made to conduct industry stress testing, it will be undertaken in
            accordance with the following principles.
            Stress testing is volume testing where above-average expected load is applied to determine the
            robustness and behaviour of the system. It can be used to find obscure defects, identify weak
            points in the architecture or assess the system’s capacity for growth.
            Stress testing would be carried out after Industry Volume Testing has been successfully
            completed and will require sufficient test data or a test tool to provide or simulate the volumes
            required. To gain sufficient knowledge on the robustness of the system, stress testing should be
            run for a sufficient period of time to ensure that all activities are exercised in the usual
            timeframes.




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc      Page 21 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy




                 4.3.7        P a rtic ip a n t Go /No Go to P ro d u c tio n Re vie w
            At the completion of the Industry testing each participant will undergo a review of the results to
            assess the readiness of each participant to go live. The timing of this review is important, as
            businesses must be given an opportunity to rectify any issues found during the Industry Testing.
            A period of time after the completion of Industry Testing is allowed for businesses to update
            production systems to make them ready for production.
            If any aspect of the smart metering functionality implemented by a business has not completed
            the allocated mandatory tests or exceeds the error limits set, the participant will be required to
            complete the tests and/or rectify the problems and retest before participation with that smart
            metering service in the market is allowed.
            The content of the Participant Go/No Go to Production Review will be documented as part of
            Stage 2 of the Testing Framework.




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc   Page 22 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy




     4.4         NS MP Te s t Le ve ls
            The following testing will be undertaken to implement the testing activities described in section
            4.3. The application of these activities to the testing levels is covered by the figure below, and
            includes the following:
                 •     Requirements review and tracing
                 •     Functional and Non-functional testing
                 •     System Integration Testing
                 •     Regression Testing
                 •     Volume and Stress Testing
            Requirements review and tracing will be required for all identified testing activities. It will involve
            assessing the NEM process and procedure changes, including the non functional aspects such
            as service levels and volume testing requirements.
            The Functional and Non-functional test levels will be applied to the industry testing and cover
            exercising individual functions utilising specifically designed test data. This testing will cover
            exception handling as well as correct operation.
            System Integration testing will use scenario tests covering operational processes and utilising
            mixed transactions. It will be designed to exercise the participant and central systems in the
            manner expected from daily market involvement.
            Regression testing will apply to all testing activities and cover transactions and business
            processes potentially impacted by new and changed systems, but not changed as a result of
            AMI. The criteria to select the regression test suite will be documented in the individual test
            plans for each test activity, as described in section 4.3.
            The following table indicates the relationship between test levels and the associated test
            activities.
                                        Table 2 – Test Levels to Test Activities Matrix

   Test Levels                          Industry       Participant     Business       Industry       Industry
                                        Technical      Volume          Testing        Volume         Stress
                                        Testing        Testing                        Testing        Testing

   Requirements Review                        X              X              X               X               X
   and Tracing

   Functional and Non-                        X              X              X
   Functional Testing

   System Integration                                                       X               X               X
   Testing

   Regression Testing                         X              X              X               X               X

   Volume and Stress                                         X                              X               X
   Testing




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc     Page 23 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy




       5           Ap p e n d ix A – Glo s s a ry
The following acronyms are in the NSMP Glossary.

ACOSS           Australian Council of Social Services
AEMC            Australian Energy Market Commission
AEMO            Australian Energy Market Operator
AER             Australian Energy Regulator
AMI             Advanced Metering Infrastructure (Victorian smart metering program)
AS              Australian Standard
B2B             Business to Business
BPRG            B2B Procedures Reference Group (established under the IEC)
BPPWG           Business Processes and Procedures Working Group (established under the NSSC)
BRDRG           Business Requirements Definition Reference Group (established under the AMI
                program)
BRWG            Business Requirements Working Group (established under the NSSC)
CATS            Consumer Administration and Transfer Solution
COAG            Council of Australian Governments
DNSP            Distribution Network Service Provider
DRET            Commonwealth Department of Resources, Energy and Tourism (also referred to as
                RET)
EEEC            Equipment Energy Efficiency (E3) Committee
ENA             Energy Networks Association
ERAA            Energy Retailers Association of Australia
ESC             Essential Services Commission
FRC             Full Retail Contestability
FRMP            Financially Responsible Market Participant
HAN             Home Area Network
IEC             Information Exchange Committee (established under section 7.2A.2 of the Rules)
IHD             In-home Display
LNSP            Local Network Service Provider
MCE             Ministerial Council on Energy (established under the COAG)
MDA             Metering Data Agent
MDF             Metering Data File
MDFF            Metering Data File Format
MOU             Memorandum of Understanding
MRG             Metrology Reference Group (established under the RMEC)
MSATS           Market Settlement and Transfer Solution
MTWG            Metering Technology Working Group (established under the AMI program)
NCRE            National Consumer Roundtable on Energy



File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc   Page 24 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy

NECF            National Energy Consumer Framework
NEL             National Electricity Law
NEM             National Electricity Market which excludes Western Australia and Northern Territory
NEMMCO          National Electricity Market Management Company
NEO             National Electricity Objective (as set out in section 7 of the NEL)
NER             National Electricity Rules
NSMP            National Smart Metering Program
NSSC            National Stakeholder Steering Committee (National Smart Metering Program)
NT              Northern Territory
OMRV            Operating Model Requirements Version (Victorian AMI Program)
PDRG            Business Process & Data Reference Group (established under the RMEC)
PwC             PricewaterhouseCoopers
PTWG            Pilots and Trials Working Group (established under the NSSC)
RET             Commonwealth Department of Resources, Energy and Tourism
RIS             Regulatory Impact Statement
RFP             Request for Proposal
RMEC            Retail Market Executive Committee (an advisory committee to AEMO)
RP              Responsible Person
RPWG            Retail Policy Working Group (established under the MCE)
RWG             Regulation Working Group (established under the NSSC)
SCO             Standing Council of Officials (as established under the MCE)
SM              Smart Metering
SMCN            Smart Metering Communication Network
SME             Subject Matter Expert
SMI             Smart Metering Infrastructure
SMMS            Smart Metering Management System
SMWG            Smart Metering Working Group (established under the SCO)
SWIS            South Western Interconnected System in Western Australia
TFWG            Testing Framework Reference Group
TOR             Terms of Reference
TRWG            Technical and Regulatory Working Group (established under the AMI Program)
WA              Western Australia
WAIMO           Western Australian Independent Market Operator
WEM             Wholesale Electricity Market (Western Australia)
WIGS            Wholesale Inter-connector Generator and Sample
WG              Working Group




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc   Page 25 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy


6                  Ap p e n d ix B - In d u s try Te s tin g Stru c tu re
                 6.1.1        Ro le s a n d Re s p o n s ib ilitie s


        Business/            Role                Responsibilities                                 Reports to
        Group

        AEMO                 Develops and        Provide release documentation to all             Jurisdictional
                             manages the         participants                                     Body
                             software,
                                                 Taking ownership of AEMO defects
                             provides and
                             supports the        Providing a BAU InfoCentre/Helpdesk to
                             pre-production      answer B2B procedures, FAQs, and other
                             environment         issues related to testing.
                             and central IT
                                                 Providing Test Management Tool licenses
                             systems used
                                                 and the infrastructure to run the selected
                             in the National
                                                 Test Management Tool.
                             Electricity
                             Market              Provide support for the Hub and MSATS
                                                 pre-production environment

        Jurisdiction         Industry Test       Overall management and co-ordination of          Jurisdictional
                             Manager             the testing activities between the               Body
                                                 Participants and the Hub
                                                 Hosting and facilitating status conference
                                                 calls
                                                 Issuing agendas for meetings and
                                                 workshops
                                                 Record the actions and decisions from
                                                 meetings and workshops
                                                 Monitor the overall status of test execution
                                                 and defect resolution
                                                 Develop and update the Test Schedule
                                                 Organise Test Management Tool training as
                                                 required
                                                 Provide test and metrics reports and
                                                 communications as necessary in a timely
                                                 manner.
                                                 Host and facilitate the following conference
                                                 call meetings during industry test execution:
                                                 •   Testing Launch Meetings
                                                 •   Daily Start Up Calls
                                                 •   Daily Completion Meeting
                                                 •   Weekly Status Meetings
                                                 •   Cycle Startup Meeting
                                                 •   Cycle Completion Meeting
                                                 •   Testing Completion Meeting
                                                 •   Readiness of Go-Live Assessment


File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc      Page 26 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy


        Business/            Role             Responsibilities                                    Reports to
        Group

        Jurisdiction         Defect           The defect manager is responsible for               Industry Test
                             Manager          ensuring that:                                      Manager
                                              •   Defects and issues are appropriately
                                                  worded, have sufficient information to
                                                  repeat the problem and are
                                                  unambiguous
                                              •   Have the correct severity level assigned
                                              •   Are assigned to the correct organisation
                                                  or individual
                                              •   Appropriately tested prior to closure
                                              •   Producing defect tracking reports as
                                                  required for the daily meeting
                                              This role could be combined with the
                                              Industry Test Manager, but bear in mind it
                                              can be a full-time task during test execution
                                              phases

        Test                 Industry Test    The TMG is a group of Participant Test              Jurisdictional
        Management           Manager and      Managers, or their designees, chaired by the        Body
        Group (TMG)          Participant      Industry Test Manager.
                             Test Managers
                                              This group is responsible for industry test
                                              execution, agreeing the testing schedule, the
                                              first level for resolution of any conflicts e.g.
                                              defect severity, adequate resolution and
                                              ownership that can’t be resolved by the
                                              Defect Manager, testing partnerships and
                                              readiness review disputes.

        Committee            Conflict         The existing committee structure will be            Jurisdictional
        Structure            Resolution       used in the event of escalation from the            Body
        adopted by                            TMG of disputes or issues. This committee
        the                                   structure will be responsible for resolving
        Jurisdictional                        any testing conflict or issue that is unable to
        Body                                  be resolved by the Industry Test Manager
                                              and/or the Test Management Group, in a
                                              timely manner.
                                              The decision of this group will be final.

        Registered            General         Participate in the development of Test              Industry Test
        Participant                           Scenarios, Test Scripts and Test Cases              Manager
                                              (TFWG)
                                              Identifying Test Data and updating The Test
                                              Management Tool as required
                                              Execute assigned tests
                                              Input and update defects as required
                                              Attend meetings, teleconferences and
                                              workshops as required
                                              Fix identified defects in a timely manner
                                              Provide a suitable test


File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc      Page 27 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy


        Business/            Role             Responsibilities                                    Reports to
        Group
                                              environment/mechanism to enable testing
                                              that effectively reduces the risk of failure in
                                              production
                                              Identify Test Data and update The Test
                                              Management Tool as required

        Registered            Test Manager    Each participant is required to designate a         Industry Test
        Participant                           test manager. This Test Manager will                Manager
                                              provide input into all Test Documentation
                                              and will facilitate the review of test
                                              documentation within that participant’s
                                              organisation and provide meaningful
                                              feedback to the Industry Test Manager.
                                              Manage the industry testing activities within
                                              their participant organisation.
                                              Take ownership of defects assigned to that
                                              participant organisation for resolution.

        Registered            Testers         Each participant is required to staff testers       Participant
        Participant                           needed to facilitate the test execution             Test
                                              according to the agreed-upon timetable and          Manager
                                              schedule. This includes the input of test
                                              cases, the verification of results and analysis
                                              of defects.

        Registered           Test             Each participant is required to have a Test         Industry Test
        Participant          Management       Management Tool Coordinator. This person            Manager
                             Tool Co-         is required to keep The Test Management
                             ordinator        Tool up-to-date in terms of test case
                                              execution results (e.g., passes and failures)
                                              and the input and modification of defect
                                              reports. Small participants will be allocated a
                                              Test Management Tool Co-ordinator to save
                                              on infrastructure costs.




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc      Page 28 of 29
Security Classification: Unrestricted
                                     NSSC – National Smart Metering Program
                                   National Testing Framework – Testing Strategy


        Business/            Role             Responsibilities                                    Reports to
        Group

        Registered           Help Desk        Participants will be required to nominate a         Industry Test
        Participant                           representative from their organisation, who         Manager
                                              is involved in industry testing and
                                              understands the business process, to carry
                                              out a help desk function. This person will be
                                              responsible for responding to any calls
                                              received from other participants or AEMO.
                                              Participants need to be mindful that this
                                              could be a full time role.
                                              The types of call that should be directed
                                              towards this person include:
                                              •   When expecting to receive a
                                                  transaction/response from the
                                                  participant;
                                              •   When unable to track a test that involved
                                                  the participant;
                                              •   When attempting to determine the cause
                                                  or whereabouts of a defect encountered
                                                  during the testing.




File Name: TFWG workshop 05 outcome - Testing Framework Testing Strategy V0.11 changemarked.doc      Page 29 of 29
Security Classification: Unrestricted

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:1
posted:12/4/2011
language:English
pages:29