Shared by: Jeronohnson
Testing Plan The Goal of Test Planning The IEEE Standard 829-1998: To prescribe the scope, approach, resources, and schedule of the testing activities. To identify the item being tested, the features to be tested, the testing tasks to be performed, the personnel responsible for each task, and the risk associated with the plan. The Goal of Test Planning To establish the list of tasks which, if performed, will identify all of the requirements that have not been met in the software. The main work product is the test plan. Test Methodology Test Plan Test Design Test Implementation Results Analysis and Documentation Test Plan The test plan is a by-product of the detailed planning process that’s undertaken to create it. It’s the planning process that matters, not the resulting document. (..cont) Test Plan The test plan documents the overall approach to the test. In many ways, the test plan serves as a summary of the test activities that will be performed. It shows how the tests will be organized, and outlines all of the tester needs which must be met in order to properly carry out the test. The test plan should be inspected by members of the engineering team and a senior manager. (..cont) Test Plan The initial test plan is abstract, and the final test plan is concrete (fuzzy to focus). The initial test plan contains high level ideas about testing the system without getting into the details of exact test cases. The most important test cases come from the requirement of the system. Steps for Test Plan Plan the test phase Define test strategy Planning the resource requirement Distribute tester assignment Plan test schedule Test Phase Based on the proposed development model and decide whether unique phases, or stages, of testing should be performed over the course of the project. In code-and-fix model, there’s probably only one test phase – test until someone yells stop. In waterfall and spiral model, there can be several test phases from examining the product spec to acceptance testing. (..cont) Test Phase Each phase must have criteria defined for it that objectively and absolutely declares if the phase is over and the next one has begun. Test Strategy Test strategy describes the approach that the test team will use to test the software both overall and in each phase. When you are presented with a product spec: you’ll need to decide if it is better to use black-box or white-box testing? If mix – when will you apply each and which part? Manually or use tools? If using tools, develop the tools or purchase? Resource Requirement The process of deciding what’s necessary to accomplish the testing strategy. Everything that could possibly be used for testing over the course of the project needs to be considered. (..cont) Resource Requirement Example: People: How many, what experience, what expertise? Should they be full-time, part-time, contract or student? Equipment: Computers, test hardware, printers, tools. Office and lab space: How big? How will they be arranged? Software: Word processors, databases, custom tools. What will be purchased? What need to be written? Outsource companies: Will they be used? What criteria? Cost? Tester Assignments Break out the individual tester assignments. The inter-group responsibilities discussed earlier dealt with what functional group (management, test, programmers, and so on) is responsible for what high-level task. Identify the testers responsible for each area of the software and for each testable feature. Example of Tester Assignments Tester Test Assignments Aliah Character formatting: fonts, size, color, style. Sarah Layout: bullets, paragraphs, tabs, wrapping. Luis Configuration and compatibility. Jolie UI: usability, appearance, accessibility. Test Schedule The test schedule takes all the information presented so far and maps it into the overall project schedule. Critical stage because a few highly desired features that were thought to be easy to design and code may turn out to be very time consuming to test. Gantt Chart (Tools: Microsoft Project). (..cont) Test Schedule An important consideration with test planning is that the amount of test work typically is not distributed evenly over the entire product development cycle. Some testing occurs early in the form of spec and code reviews, tool development, etc, but the number of testing tasks and the number of people and amount of time spent testing often increases over the course of the project. Example of Test Schedule Testing Task Date Test Plan Complete 05/03/2008 Test Cases Complete 01/06/2008 Test Pass #1 15/06/2008 – 01/08/2008 Test Pass #2 15/08/2008 – 01/10/2008 Test Pass #3 15/10/2008 – 15/11/2008 Test Design Definition by IEEE 829: Test design specification “refines the test approach (defined in the test plan) and identifies the features to be covered by the design and its associated tests. It is also identifies the test cases and test procedures, if any, required to accomplish the testing and specifies the feature pass/fail criteria”. (..cont) Test Design Based on IEEE 829 standard, test design should include: Identifiers Features to be tested Approach Test case identification Pass/fail criteria Identifiers A unique identifier that can be used to reference and locate the test design spec. The spec should also reference the overall test plan and contain pointers to any other plans or specs that it references. Features to be tested A description of the software feature covered by the test design spec. Example: “the addition function of calculator”, “font size selection and display in WordPad”, and “video card configuration testing of QuickTime”. Approach A description of the general approach that will be used to test the features. It should expand on the approach, if any, listed in the test plan, describe the technique to be used, and explain how the results will be verified. Example: “A testing tool will be developed to sequentially load and save pre-built data files of various sizes. The number of data files, sizes, and the data they contain will be determined through black- box technique and supplemented with white-box examples from the programmer. A pass or fail will be determined by comparing the saved file bit-for-bit against the original using a file compare tool”. Test case identification A high-level description and references to the specific test cases that will be used to check the feature. It should list the selected equivalence partitions and provide references to the test cases and test procedures used to run them. (..cont) Test case identification Example: Check the highest possible value Test case ID # 15326 Check the lowest possible value Test case ID # 15327 (..cont) Test case identification It’s important that the actual test case values aren’t defined in this section. For someone reviewing the test design spec for proper test coverage, a description of the equivalence partitions is much more useful than the specific values themselves. Pass/Fail Criteria Describe exactly what constitutes a pass and a fail of the tested feature. What is acceptable and what is not? Simple: a pass is when all the test cases run without finding a bug. Fuzzy: a failure is when 10% or more of the test cases fail. Test Case Planning – why? Organization Even a small software may have thousands of test cases. The test cases have been created by several testers over the course of several months or years. Proper planning will organize them so that all the testers and other project team members can review and use them effectively. (..cont) Test Case Planning – why? Repeatability The same test cases can be used several times to look for new bug, and makes sure old one get fixed. Without test case planning, it would be impossible to know what test cases were last run and exactly how they were run so that we can repeat the exact tests. (..cont) Test Case Planning – why? Tracking How many test cases did you plan to run? How many did you run on the last software release? How many passed and how many failed? (..cont) Test Case Planning – why? Proof of testing (or not testing) In high-risk industries, the software test team must prove that it did run the tests that it planned to run. It could actually be illegal, and dangerous, to release software in which few test caes were skipped. Proper test case planning and tracking provides a means for proving what was tested. Test Case - Definition IEEE 839: Test case specification documents the actual values used for input along with the anticipated outputs. A test case also identifies any constraints on the test procedures resulting from use of that specific test case. Test Case Specs IEEE 829 standard lists some important information that should be included in the test case specs: Identifiers Test item Input specification Output specification Environmental needs Special procedural requirements Intercase dependencies (..cont) Test Case Specs Identifiers A unique identifier is referenced by the test design specs and the test procedure specs. Test item Describes detailed feature, code module, etc that’s being tested. Provide references to product specs or other design docs from which the test case was based. (..cont) Test Case Specs Input specification List of all the inputs or conditions given to the software to execute the test case. If you are testing a file-based product, it would be the name of the file and a description of its contents. Output specification Results you expect from executing the test case. Did all the contents of the file load as expected? (..cont) Test Case Specs Environmental needs Hardware, software, test tools, facilities, staff, etc that are necessary to run the test case. Special procedural requirements Describe anything unusual that must be done to perform the test. Example: testing nuclear power plant..? (..cont) Test Case Specs Intercase dependencies If the test case depends on another test case or might be affected by another, that information should be included here. (..cont) Test Case Specs Example: Test case presented in the form of matrix or table Test Case ID Printer Mfg Model Mode Option WP0001 Canon BJC-700 B/W Text WP0002 HP LaserJet IV Color Auto . . WP00010 HP LaserJet IV High Draft Test Procedure / Implementation Execute the test cases. IEEE 829 defines the test procedure specification “ identifies all the steps required to operate the system and exercise the specified test cases in order to implement the associated test design.” Test Procedures How to perform the test cases. Information need to be defined: Identifiers Purpose Special requirement Procedure steps Test Procedure Identifiers A unique identifier that ties the test procedure to the associated test cases and test design. Purpose The purpose of the procedure and reference to the test cases that it will execute. Special requirements Other procedures, special testing skills, or special equipment needed to run the procedure. Test Procedure Procedure steps Detailed description of how the tests are to be run: Log – Tells how and by what method the results and observations will be recorded. Setup – How to prepare for the test. Start – Steps used to start the test. Procedure – Steps to run the test. Measure – How the results are to be determined? (..cont) Procedure Steps Shut down – Steps for suspending the test for unexpected reasons. Restart – How to pick up the test at certain point if there’s failure or after shutting down. Stop – Steps for an orderly halt to the test. Wrap up – How to restore the environment to its pre-test condition. Contingencies – What to do if things don’t go as planned. Example of test procedure Identifier: WinCalcProc98.1872 Purpose: This procedure describe the steps necessary to execute the Addition function test cases … through … Special Requirements: No special requirement (Individual test cases). Procedure Steps: Log: The tester will use WordPad with the Testlog template … Setup: The tester must install a clean copy of … Start: 1) Boot up Windows 98 2) Click the Start button 3) … Procedure: For each test case identified above, enter the test input data using the keyboard … Measure: … Result Analysis and Documentation Fundamental principles for reporting a bug: Report bugs as soon as possible Effectively describe the bugs Minimal, singular, obvious and general etc. Be non-judgmental in reporting bugs professionally– written against the product, not the person, state only facts. Follow up on your bug reports Make sure it reported properly, and given attention that it needs to be addressed. Get them fixed. Not all bugs are created equally A bug that corrupts a user’s data is more severe than one that’s a simple misspelling. But, what if the data corruption can occur only in a very rare instance that no user is ever likely to see it and the misspelling caused every user to have problems installing the software? Which one is more important to fix? How to classify bugs? General concept used: Severity Indicateshow bad the bug is; the likelihood and the degree of impact when the user encounters the bug. Priority Indicateshow much emphasis should be placed on fixing the bug and the urgency of making the fix. Example Severity 1. System crash, data loss, data corruption, security breach. 2. Operational error, wrong results, loss of functionality. 3. Minor problem, misspelling, UI layout, rare occurrence. 4. Suggestion Priority 1. Immediate fix, blocks further testing, very visible. 2. Must fix before the product is released. 3. Should fix when time permits. 4. Would like to fix but the product can be release as it is. Data corruption: Severity 1, Priority 3. Misspelling: Severity 3, Priority 2.