Evaluating your SQA process Presented to Red Earth QA by Robert Watkins Model 1 – CMMI • Level 1 - Initial • Level 2 - Repeatable • Level 3 - Defined • Level 4 - Managed • Level 5 - Optimizing SQA Processes Through the Lens of CMMI • Achieving Level 1 - Initial – Somebody said they ‘tested’. • Achieving Level 2 - Repeatable – There are test scripts written down for many of the tests for most projects. SQA Processes Through the Lens of CMMI • Achieving Level 3 - Defined – A Test Plan was put together. • Achieving Level 4 – Managed – There are documented processes and standard templates used. These documents are reviewed as part of the process. • Achieving Level 5 – Optimizing – Results of the testing process are used to improve the process for each project. Model 2 - TMM • Level 1 - Initial • Level 2 - Phase Definition • Level 3 - Integration • Level 4 - Management and Measurement • Level 5 - Optimization/Defect Prevention and Quality Control TMM Level 1 • A chaotic process • Not distinguished from debugging, ill-defined • The tests are developed ad hoc after coding is complete • Usually lack a trained professional testing staff and testing tools • The objective of testing is to show that the system and software work TMM – Level 2 • Identify testing as a separate function from debugging • Testing becomes a defined phase following coding • Standardized process to the point where basic testing techniques and methods are in place • The objective of testing is to show that the system and software meets specifications TMM – Level 3 • Integrate testing into the entire life cycle • Establish a formal testing organization – establishes formal testing technical trainings – controls and monitors the testing process – begins to consider using automated test tools • The objective of testing is based on system requirements • Major milestone reached at this level: management recognizes testing as a professional activity TMM – Level 4 • Testing is a measured and quantified process • Development products are now tested for quality attributes such as Reliability, Usability, and Maintainability • Test cases are collected and recorded in a test database for reuse and regression testing • Defects found during testing are now logged, given a severity level, and assigned a priority for correction TMM – Level 5 • Testing is institutionalized within the organization • Testing process is well defined and managed • Testing costs and effectiveness are monitored • Automated tools are a primary part of the testing process • There is an established procedure for selecting and evaluating testing tools SQA Processes Through the Lens of TMM • Achieving Level 2 - Phase Definition – You define a specific time for testing as part of your project – You focus testing on ensuring the software meets specs. This means that you actually have some understanding of the specifications for the software being tested. – There are differing types of test artifacts such as test data, test scripts, test plans even if used differently by different testers. – You don’t require that the testing process starts at the same time the project starts. – QA may make the go/no go decision for a release. SQA Processes Through the Lens of TMM • Achieving Level 3 – Integration – The QA process starts when the project starts. – The Test Plan (or similar document) is started at the same time that the project is being scoped out. – QA staff has a background in testing from both experience AND formal training. – QA progress is monitored as part of the project process. – QA staff provides the information to the project team to make the go/no go decision for a release. – The standards for joining the QA team are high. SQA Processes Through the Lens of TMM • Achieving Level 4 - Management and Measurement – Most aspects of the testing process are measured. Test Requirements coverage, Code Coverage, Defect Analysis, etc. – Test Plans are required to include test types beyond Functional Requirements. This includes Load, Performance, Stress, Recoverability, etc. – There is a formal system and corresponding process for managing Test Scripts. – There is a formal system and corresponding process for managing defects. SQA Processes Through the Lens of TMM • Achieving Level 5 - Optimization/Defect Prevention and Quality Control – All levels of the organization participate in the testing process. Support staff provides input on test requirements. Marketing provides input on Performance Requirements and Usability expectations. Etc. – Risks associated with testing are managed as a part of the project. – Reviews of the effectiveness of the testing are conducted and changes made accordingly. – All members of the QA team have access to and are well-versed in using the automated testing tools which make up the majority of their QA time. SQA Evaluation by Project Phase Level 2 – Level 3 - Level 4 – Level 5 - Phase Integration Management Optimization Definition and Measurement There is a QA is involved Metrics that All areas of Planning time slot for in creating define the company testing project plan ‘acceptability’ participate in are agreed to setting goals Test Plans Test Plan is a Additional Managing Design document formal test types are Risks is built functional document included in into the tests the Test Plan process Keep testing Test Cases and Formal Automated Development and re- data are re- management tools are the testing used of bugs and norm test scripts Testing stops Test Results Go/no go Effectiveness Deployment when no are used for decision is metrics are time is left go/no go based on fed back into decision meeting goals the process References • CMMI (formerly known as CMM) http://www.sei.cmu.edu/cmmi/ • TMM http://www.cs.iit.edu/research/tmm.html • TMM Description http://www.cs.umd.edu/~atif/Teaching/F all2002/StudentSlides/Duy.pdf Questions?