Docstoc

Evaluating your SQA process

Document Sample
Evaluating your SQA process Powered By Docstoc
					Evaluating your SQA process


    Presented to Red Earth QA
        by Robert Watkins
Model 1 – CMMI
   •   Level 1 -   Initial
   •   Level 2 -   Repeatable
   •   Level 3 -   Defined
   •   Level 4 -   Managed
   •   Level 5 -   Optimizing
SQA Processes Through
  the Lens of CMMI
       • Achieving Level 1 - Initial
          – Somebody said they ‘tested’.

       • Achieving Level 2 -
         Repeatable
          – There are test scripts written
            down for many of the tests
            for most projects.
SQA Processes Through
  the Lens of CMMI
       • Achieving Level 3 - Defined
          – A Test Plan was put together.

       • Achieving Level 4 – Managed
          – There are documented processes
            and standard templates used.
            These documents are reviewed as
            part of the process.

       • Achieving Level 5 – Optimizing
          – Results of the testing process are
            used to improve the process for
            each project.
Model 2 - TMM
   • Level 1 - Initial
   • Level 2 - Phase Definition
   • Level 3 - Integration
   • Level 4 - Management and
     Measurement
   • Level 5 -
     Optimization/Defect
     Prevention and Quality
     Control
TMM Level 1
  • A chaotic process
  • Not distinguished from
    debugging, ill-defined
  • The tests are developed ad hoc
    after coding is complete
  • Usually lack a trained
    professional testing staff and
    testing tools
  • The objective of testing is to
    show that the system and
    software work
TMM – Level 2
   • Identify testing as a separate
     function from debugging
   • Testing becomes a defined
     phase following coding
   • Standardized process to the
     point where basic testing
     techniques and methods are in
     place
   • The objective of testing is to
     show that the system and
     software meets specifications
TMM – Level 3
   • Integrate testing into the entire life
     cycle
   • Establish a formal testing organization
      – establishes formal testing technical
        trainings
      – controls and monitors the testing
        process
      – begins to consider using automated test
        tools
   • The objective of testing is based on
     system requirements
   • Major milestone reached at this level:
     management recognizes testing as a
     professional activity
TMM – Level 4
   • Testing is a measured and
     quantified process
   • Development products are now
     tested for quality attributes
     such as Reliability, Usability,
     and Maintainability
   • Test cases are collected and
     recorded in a test database for
     reuse and regression testing
   • Defects found during testing are
     now logged, given a severity
     level, and assigned a priority for
     correction
TMM – Level 5
   • Testing is institutionalized
     within the organization
   • Testing process is well defined
     and managed
   • Testing costs and effectiveness
     are monitored
   • Automated tools are a primary
     part of the testing process
   • There is an established
     procedure for selecting and
     evaluating testing tools
SQA Processes Through
   the Lens of TMM
       • Achieving Level 2 - Phase Definition
          – You define a specific time for testing as
            part of your project
          – You focus testing on ensuring the
            software meets specs. This means that
            you actually have some understanding
            of the specifications for the software
            being tested.
          – There are differing types of test
            artifacts such as test data, test scripts,
            test plans even if used differently by
            different testers.
          – You don’t require that the testing
            process starts at the same time the
            project starts.
          – QA may make the go/no go decision for
            a release.
SQA Processes Through
   the Lens of TMM
       • Achieving Level 3 – Integration
          – The QA process starts when the
            project starts.
          – The Test Plan (or similar document)
            is started at the same time that the
            project is being scoped out.
          – QA staff has a background in
            testing from both experience AND
            formal training.
          – QA progress is monitored as part of
            the project process.
          – QA staff provides the information
            to the project team to make the
            go/no go decision for a release.
          – The standards for joining the QA
            team are high.
SQA Processes Through
   the Lens of TMM
       • Achieving Level 4 - Management and
         Measurement
          – Most aspects of the testing process are
            measured. Test Requirements
            coverage, Code Coverage, Defect
            Analysis, etc.
          – Test Plans are required to include test
            types beyond Functional Requirements.
            This includes Load, Performance,
            Stress, Recoverability, etc.
          – There is a formal system and
            corresponding process for managing
            Test Scripts.
          – There is a formal system and
            corresponding process for managing
            defects.
SQA Processes Through
   the Lens of TMM
       • Achieving Level 5 -
         Optimization/Defect Prevention and
         Quality Control
          – All levels of the organization
            participate in the testing process.
            Support staff provides input on test
            requirements. Marketing provides input
            on Performance Requirements and
            Usability expectations. Etc.
          – Risks associated with testing are
            managed as a part of the project.
          – Reviews of the effectiveness of the
            testing are conducted and changes
            made accordingly.
          – All members of the QA team have
            access to and are well-versed in using
            the automated testing tools which
            make up the majority of their QA time.
 SQA Evaluation by Project Phase
              Level 2 –       Level 3 -        Level 4 –         Level 5 -
              Phase           Integration      Management        Optimization
              Definition                       and
                                               Measurement
              There is a      QA is involved   Metrics that      All areas of
Planning      time slot for   in creating      define            the company
              testing         project plan     ‘acceptability’   participate in
                                               are agreed to     setting goals
              Test Plans      Test Plan is a   Additional        Managing
Design        document        formal           test types are    Risks is built
              functional      document         included in       into the
              tests                            the Test Plan     process
              Keep testing    Test Cases and Formal              Automated
Development   and re-         data are re-   management          tools are the
              testing         used           of bugs and         norm
                                             test scripts
              Testing stops   Test Results     Go/no go          Effectiveness
Deployment    when no         are used for     decision is       metrics are
              time is left    go/no go         based on          fed back into
                              decision         meeting goals     the process
References
  • CMMI
   (formerly known as CMM)
   http://www.sei.cmu.edu/cmmi/

  • TMM
   http://www.cs.iit.edu/research/tmm.html

  • TMM Description
   http://www.cs.umd.edu/~atif/Teaching/F
   all2002/StudentSlides/Duy.pdf
Questions?

				
DOCUMENT INFO