Software Testing Testing Strategies Software Testing Strategies ● First software quality assurance tools applied to control the software product's quality before its shipment ● November 1994 survey by Perry 24 % of project development budget was for testing 32 % of project management budget was for testing 27 % of project time was schedule for testing (actually they allocate 45 % of their schedule time for testing) Outlines ● 2.1 Introduction Testing Objectives ● 2.2 Software Testing Strategies ● 2.3 Software Test Classifications ● 2.4 White Box Testing ● 2.5 Black Box Testing Objective ● Explain testing objectives ● Discuss the difference between the various testing strategies ● Describe the concept of black box and white box testing Outlines ● 2.1 Introduction Testing Objectives ● 2.2 Software Testing Strategies ● 2.3 Software Test Classifications ● 2.4 White Box Testing ● 2.5 Black Box Testing Definition Revisited ● Myers Testing is the process of executing a program with intention of finding errors ● Paul Jorgensen Testing is obviously concerned with errors, faults, failures and incidents. A test is the act of exercising software with test cases with an objective of Finding failure and Demonstrate correct execution ● ISO Technical operation that consist of the determination of one or more characteristics of a given product, process or service according to a specified procedure What is the objectives of Software Testing? Direct Objectives ● To identify and reveal as many errors as possible in the tested software ● To bring the tested software, after correction of the identified errors and retesting to an acceptable level of quality ● To perform the required tests efficiency and effectively, within budgetary and scheduling limitation Indirect Objective ● To compile a record of software errors for use in error prevention (by corrective and preventive actions) Outlines ● 2.1 Introduction Testing Objectives ● 2.2 Software Testing Strategies ● 2.3 Software Test Classifications ● 2.4 White Box Testing ● 2.5 Black Box Testing Stages of Testing ● Module or unit testing. ● Integration testing, ● Function testing. ● Performance testing. ● Acceptance testing. ● Installation testing. Testing Strategies ● We began by 'testing-in-the-small' and move toward 'testing -in-the-large' ● For conventional software The module (component) is our initial focus Integration of module follows ● For OO software OO class that include attributes and operations and implies communication and collaboration Unit Testing • Program reviews. • Formal verification. • Testing the program itself. – black box and white box testing. Black Box or White Box? • Maximum # of logic paths - determine if white box testing is possible. • Nature of input data. • Amount of computation involved. • Complexity of algorithms. Unit Testing Details • Interfaces tested for proper information flow. • Local data are examined to ensure that integrity is maintained. • Boundary conditions are tested. • Basis path testing should be used. • All error handling paths should be tested. • Drivers and/or stubs need to be developed to test incomplete software. Generating Test Data • Ideally want to test every permutation of valid and invalid inputs • Equivalence partitioning it often required to reduce to infinite test case sets – Every possible input belongs to one of the equivalence classes. – No input belongs to more than one class. – Each point is representative of class. Integration Testing Strategies Test methodologies may vary but two basic testing strategies applied: ● Test the software in its entirety “Big Bang Testing” ● Test the software in modules - Unit Tests, Integration Tests, Systems Tests “Incremental testing” ● Two strategies for incremental testing Bottom - up testing (test harness). Top - down testing (stubs). ● Sandwich Testing Top-Down Integration Testing • Main program used as a test driver and stubs are substitutes for components directly subordinate to it. • Subordinate stubs are replaced one at a time with real components (following the depth-first or breadth-first approach). • Tests are conducted as each component is integrated. • On completion of each set of tests and other stub is replaced with a real component. • Regression testing may be used to ensure that new errors not introduced. Top Down Integration A top module is tested with stubs B F G stubs are replaced one at a time, "depth first" C as new modules are integrated, some subset of tests is re-run D E ● Stub is a piece of code emulating a called function Top down – Advantages & Disadvantages ● Top-down integration supports fault isolation; ● Major design flaws show up early. Modules of a product can be divided into two groups: The logic modules and operational modules. By coding and testing the logic modules before the operational modules, top-down integration will exploit any major design faults early in the development process ● The main disadvantage of top-down integration is that potentially reusable modules may not be adequately tested. Bottom-Up Integration Testing • Low level components are combined in clusters that perform a specific software function. • A driver (control program) is written to coordinate test case input and output. • The cluster is tested. • Drivers are removed and clusters are combined moving upward in the program structure. Bottom-Up Integration A B F G drivers are replaced one at a time, "depth first" C worker modules are grouped into builds and integrated D E cluster ● a driver is a piece of code emulating a calling function Bottom-up – Advantages & Disadvantages ● The operational modules are thoroughly tested when using a bottom-up strategy. It also provides fault-isolation, as does top-down integration. ● Major design faults will be left undetected until late in the development, since the logic module are integrated last. ● This may result in large costs in redesigning and recoding substantial portions of the project. Sandwich Testing ● Combine the two so as to capitalize upon their strengths and minimize their weaknesses ● neither top-down nor bottom-up implementation/integration is suitable for all the modules, the solution is to partition them Sandwich Testing A Top modules are tested with stubs B F G C Worker modules are grouped into builds and integrated D E cluster Outlines ● 2.1 Introduction Testing Objectives ● 2.2 Software Testing Strategies ● 2.3 Software Test Classifications ● 2.4 White Box Testing ● 2.5 Black Box Testing Software Test Classification ● Software test may be classified according to the testing concept or to the requirement classification According to Testing Concept ● What concept to test? Output ● Output is used to archive an acceptable level of quality Structural of the software ● Internal structural and calculations involved is included for satisfactory testing ● Two Classes have been developed Black box testing ● Identify bug according to software malfunctioning ● Functionality testing White box testing ● Examine internal calculation paths in order to identify bug ● Structural testing According to Requirement ● The test id carried out to ensure full coverage of the respective requirement Operation ● Correctness, Reliability, Efficiency, Integrity, Usability Revision ● Maintainability, flexibility, testability Transition ● Portability, reusabilily, interoperability ● From the requirement we can define the test classification ● White box and black box can be used implemented respectively Test Classification according to Requirement Correctness 1.1 Output correctness tests 1.2 Documentation tests 1.3 Availability tests 1.4 Data processing & calculations correctness tests 1.5 Software qualification tests Reliability 2. Reliability tests Efficiency 3. Stress test (load tests and durability tests) Integrity 4. Software system security tests Usability 5.1 Training usability tests 5.2 Operational usability tests Maintainability 6. Maintainability tests Flexibility 7. Flexibility tests Testability 8. Testability tests Portability 9. Portability tests Re usability 10. Re usability tests Interoperability 11.1 Software interoperability tests 11.2 Equipment interoperability test IEEE definitions Black box testing Testing that ignores the internal mechanism of the system or component and focuses solely on the outputs in response to selected inputs and execution conditions Testing conducted to evaluate the compliance of a system or component with specified functional requirements White box testing Testing that takes into account the internal mechanism of a system or component Outlines ● 2.1 Introduction Testing Objectives ● 2.2 Software Testing Strategies ● 2.3 Software Test Classifications ● 2.4 White Box Testing ● 2.5 Black Box Testing White Box Testing ● White Box Testing requires verification of every program statement and comment ● White box testing enables performance of Data processing and calculation correctness tests ● Every computation operation must be examined Software qualification tests, ● Software code (including comments) Maintainability test ● Failure causes detection, support adaption, software improvement Reusability test ● Reusable for future software packages Data processing and calculation correctness tests ● Checking the operation by each test cases Different path Which path is going to be tested? ● Two approaches Path coverage ● Path coverage of a test is measured by the percentage of all possible program paths included in planned testing. Line coverage ● Line coverage of a test is measured by the percentage of program code lines included in planned testing. Path coverage ● Different path is created by condition statements IF-THEN-ELSE DO-WHILE DO-UNTIL ● Consider a simple module with 10 conditional statement allowing two options each Creates 210 different path = 1024 path Examples: ● Taximeter Taxi meter 1. Minimal fare: $2. This fare covers the distance traveled up to 1000 yards and waiting time (stopping for traffic lights or traffic jams, etc.) of up to 3 minutes. 2. For every additional 250 yards or part of it: 25 cents. 3. For every additional 2 minutes of stopping or waiting or part thereof: 20 cents. 4. One suitcase: 0 change; each additional suitcase: $1. 5. Night supplement: 25%, effective for journeys between 21.00 and 06.00. 6. Regular clients are entitled to a 10% discount and are not charged the night supplement. Flow Chart 1 Charge the minimal fare D > 1000 D ≤ 1000 2 3 Distance 4 WT > 3 WT ≤ 3 5 6 Waiting time 7 S >1 8 S≤1 9 No.of suitcases 10 Yes 11 No 12 Regular client? 13 14 Night journey? 15 Yes No 16 17 Print receipt. Examples ● 24 different paths may be indicated ● In order to archive full path coverage of the software module, we have to prepare at least 24 test cases ● We need to find a minimum number of paths to cover all line of code Line coverage Flow Graph 1 2 4 3 R1 5 7 6 R2 8 R6 9 R3 10 11 12 13 R4 14 15 R5 16 17 The Minimum number of paths Minimun 3 1 path is required 2 4 3 R1 5 R2 7 6 8 R6 9 R3 10 11 12 13 R4 14 15 R5 16 17 Advantages of White box Testing Advantages: * Direct determination of software correctness as expressed in the processing paths, including algorithms. * Allows performance of line coverage follow up. * Ascertains quality of coding work and its adherence to coding standards. Disadvantages : * The vast resources utilized, much above those required for black box testing of the same software package. * The inability to test software performance in terms of availability (response time), reliability, load durability, etc. Outlines ● 2.1 Introduction Testing Objectives ● 2.2 Software Testing Strategies ● 2.3 Software Test Classifications ● 2.4 White Box Testing ● 2.5 Black Box Testing Test Classification according to Requirement Correctness 1.1 Output correctness tests 1.2 Documentation tests 1.3 Availability tests 1.4 Data processing & calculations correctness tests 1.5 Software qualification tests Reliability 2. Reliability tests Efficiency 3. Stress test (load tests and durability tests) Integrity 4. Software system security tests Usability 5.1 Training usability tests 5.2 Operational usability tests Maintainability 6. Maintainability tests Flexibility 7. Flexibility tests Testability 8. Testability tests Portability 9. Portability tests Re usability 10. Re usability tests Interoperability 11.1 Software interoperability tests 11.2 Equipment interoperability test Black Box Testing ● Apart from correctness tests(those 2), maintainability and re usability, most of other testing classes are unique to black box testing Explain the importance of black box testing ● However, due to the special characteristic of each testing strategy, black box testing cannot automatically substitute for white box testing Black-Box Testing ● Black-box testing is testing from a functional or behavioral perspective to ensure a program meets its specification ● Testing usually conducted without knowledge of software implementation - system treated as a “black box” ● Black-box test design techniques include: equivalence partitioning, boundary value analysis, cause-effect graphing, random testing How much testing is adequate? ● Completely validating IEEE 754 floating-point division requires 264 test-cases! float divide(float x, float y) ● From practical and economic perspectives, exhaustive testing is usually not possible Which software pieces should we test? Which test cases should we choose? Equivalence class partitioning (EC) ● A black box method aimed at increasing the efficiency of testing and, at the same time, improving coverage of potential error conditions. Boundary Value Analysis ● Based on experience / heuristics Testing boundary conditions of equivalence classes is more effective ● Choose input boundary values as equivalence classes representatives ● Choose inputs that invoke output boundary values ● Examples: (0, 10] ⇒ validate using 0, 1, 2, 9, 10, 11 Read up to 5 elements ⇒ validate reading 0, 1, 4, 5, 6 elements BVA as an equivalence partitioning extension ● Choose one (or more) arbitrary value(s) in each equivalence class ● Choose valid values exactly on lower and upper boundaries of equivalence class ● Choose invalid values immediately below and above each boundary (if applicable) Equivalence Class Partitioning (EC) ● An equivalence class (EC) is a set of input variable values that produce the same output results or that are processed identically. ● EC boundaries are defined by a single numeric or alphabetic value, a group of numeric or alphabetic values, a range of values, and so on. ● An EC that contains only valid states is defined as a "valid EC," whereas an EC that contains only invalid states is defined as the "invalid EC." ● In cases where a program's input is provided by several variables, valid and invalid ECs should be defined for each variable. Equivalence Class Partitioning (EC) According to the equivalence class partitioning method: ● Each valid EC and each invalid EC are included in at least one test case. ● Definition of test cases is done separately for the valid and invalid ECs. Equivalence Class Partitioning (EC) ● In defining a test case for the valid ECs, we try to cover as many as possible “new” ECs in that same test case. ● In defining invalid ECs, we must assign one test case to each “new” invalid EC, as a test case that includes more than one invalid EC may not allow the tester to distinguish between the program’s separate reactions to each of the invalid ECs. ● Test cases are added as long as there are uncovered ECs. Example – Ticket Price ● Ticket price depends on four variables Day (weekday, weekend) Visitor's status (OT = one time, M = members) Entry Hour (6 – 19, 19.01 – 24) Visitor's age (up to 16, 16.01 – 60, 60.01 – 120) Entrance Ticket Price table Day on., ue., ed., hu, M T W T Fri. Sat., Sun. Visitor’s Ot Ot Mem Mem Ot Ot Mem Mem status E try n 6.00- 19.01- 6.00- 19.01- 6.00- 19.01- 6.00- 19.01 o r h u 19.00 24.00 19.00 24.00 19.00 24.00 19.00 - 24.00 g A e: 0.00- $5 $6 $2.50 $3 $7.50 $9 $3.50 $4 16.00 g A e 16.01- $10 $17 $5 $6 $15 $18 $7 $8 60.00 g A e: 60.01- $8 $8 $4 $4 $12 $12 $5.50 $5.50 120.00 Test cases – The Ticket price T est T est Day Visitor’s Entry Visitor’s T case result est case case of status hour age type no. week V C alid E s 1 Mn o . Ot 7.55 8.4 $5 2 Sat. Mem 20.44 42.7 $8 3 Sat. Mem 22.44 65.0 $5.50 4 Sat. Mem 6.00 0.0 $3.50 5 Sat. Mem 19.00 16.0 $3.50 6 Sat. Mem 19.01 16.01 $8 7 Sat. Mem 19.01 60.0 $8 8 Sat. Mem 24.00 60.01 $5.50 9 Sat. Mem 24.00 120.0 $5.50 Inv C alid E s 10 o M x. Ot 7.55 8.4 Invalid day 11 o . Mn 88 7.55 8.4 Invalid visito statu r’s s 12 o . Mn Ot 4.40 8.4 Inv try ou alid en h r 13 o . Mn Ot 8@ 8.4 Inv try ou alid en h r 14 Mn o . Ot 7.55 T T R Invalid visito ag r’s e 15 o . Mn Ot 7.55 150.1 Invalid visito ag r’s e Advantage of Black Box Testing Advantages: ● Allows us to carry out the majority of testing classes, most of which can be implemented solely by black box tests, i.e. load tests and availability tests. ● For testing classes that can be carried out by both white and black box tests, black box testing requires fewer resources. Disadvantages of Black Box Testing Disadvantages: ● Possibility that coincidental aggregation of several errors will produce the correct response for a test case, and prevent error detection. ● Absence of control of line coverage. There is no easy way to specify the parameters of the test cases required to improve coverage. ● Impossibility of testing the quality of coding and its strict adherence to the coding standards.