Introduction To Software Testing

Introduction to Software Testing Introduction to Software Testing Introduction to Software Testing What is Testing ? ? Definition Process of exercising or evaluating a system or system component: ? To verify that it satisfies specified requirements ? To identify differences (failures) between: ? ? Expected results Actual results ? To reduce the risks of not working to acceptable levels Requirements Expected Result TESTS Comparison Actual Result 2 Introduction to Software Testing Testing v.s. Debugging Debugging The process of analyzing and locating bugs when software does not behave as specified Testing Testing The process of executing software in a controlled manner, in order to answer the question “Does the software behave as specified?” 3 Introduction to Software Testing Terminology ? Error ? Human action that results in a defect in the software ? What we did to put bugs there ? Fault ? The actual defectin the software as a result of an error. Faults are typically referred to as “bugs”. ? What the trouble shooter found ? Failure ? Inability of the software to perform its required function ? What the user saw 4 Introduction to Software Testing Goals of Testing 1950’s view: Show it Works (Demonstration) 1970’s view: Find Defects (Detection) 1980’s view: Measure Quality (Prediction) 1990’s view: Control Quality (Prevention) Today, all four of these perspectives are still valid as a reason to test. The issue is not which perspective to choose, but rather how to prioritize the set. The priorities are: ? prevention ? detection ? prediction ? demonstration (if ok) 5 Introduction to Software Testing The Traditional vs. Modern Approach Integration Detailed Design H.L. Design Requirements Code Test & Fix Effort Traditional Integration Detailed Design H.L. Design Requirements Code Identify and Resolve Defects Effort Release Today Release 6 Introduction to Software Testing W - Whole Life Cycle Write Req. Test Req. Install Acceptance Test Logical Design Test Design Build Software Build System System Test Physical Design Test Design Integration Test Code Unit Test 7 Introduction to Software Testing Software Testing Methodology What’s in it? A systematic discipline, akin to that of Software Development, with the aid of strategies, guidelines and procedures 8 Introduction to Software Testing Systematic Test and Evaluation Process STEP Introduction to Software Testing A Software Testing Methodology ? Systematic Test and Evaluation Process * A Systematic Software Testing Method since 1985 * Developed by Software Quality Engineering Co. ? ANSI/IEEE Testing Standards * ANSI/IEEE 829 Test Documentation Standard * ANSI/IEEE 1008 Unit Test Process Standard 10 Introduction to Software Testing STEP Phases ? Plan Strategy - for each level * Master test plan * Detailed test plans ? Acquire Testware * Inventory test objectives * Design tests * Implement tests development by plans and requirements ? Measure Behavior * Execute tests * Check for adequacy * Evaluate testing process 11 Introduction to Software Testing What is a Master Test Plan? ? ? ? The testing policy of the project Reference for all testing reviews It will: ? state test levels ? for each level it will state test targets, ? ? ? ? responsibilities, environments revisions time tables, etc. 12 Introduction to Software Testing Planning Phase Process Overall Resources Project Info DEVELOP Master Plan Clarifications Master Test Plan Master Test Plan • Identify & prioritize product risks • Determine resource constraints • Develop overall testing strategy • Determine features and their aspects to be tested • Determine resource & training rqmts • Identify responsibilities • Specify an overall schedule • Review the test plan Software Info Level Specific Resources DEVELOP Detailed Plans Level Specific Plans Level Specific Plans • Identify & prioritize feature risks • Refine testing strategy • Specify features and their aspects to be tested • Specify special resource & tool rqmts • Specify a detailed schedule • Review the test plan 13 Introduction to Software Testing Testing Levels ? Unit ? To ensure that each software module complies with the functional specifications for that module based on design specs. ? Integration ? To test gradually the interfaces among system modules, components and subsystems. ? System ? To test the system as a whole (End-to-End testing) including hardware and interfacing with all other related production systems. To test the functional and non-functional requirements based on customer requirements for the overall system. 14 Introduction to Software Testing Testing Levels (Cont.) ? Production ? Alpha (Internal) - To subject approved integrated software to test conducted in a production-like environment. ? Beta (External) - To conduct a live application test of the software at selected customer sites before releasing the software product to all customers. ? Acceptance ? To ensure that the software satisfies all business needs and objectives, as they are defined by functional and non-functional requirements, through a customer focus. Usually it is a subset of system tests agreed upon with the customer 15 Introduction to Software Testing Functional/Non-Functional Testing ? Functional ? To verify requirements, specifications, rules, objectives, algorithms, etc. ? Non-Functional (Operational) ? To verify that the system will work in production and will provi de consistent and predictable level of service. It includes measuring resource utilization, performing high volume stress tests, testing system break points and testing performance bottlenecks. 16 Introduction to Software Testing System Testing Types(1) ? ? ? ? ? ? Capacity Concurrence HW Config. Installation Interface Performance Recovery Reliability - Limits on input variables as volumes - Simultaneous execution of multiple processing - System running on full range of permission - Installation procedures - Interface to existing system - Response time, throughput rates and total job run times - Restart/recovery procedures - Extended periods of normal processing ? ? ? Resource Usage - Consumption of CPU and both main and secondary storage 17 Introduction to Software Testing System Testing Types (2) ? ? ? Security - Attempting to breach security mechanisms Adaptation to changing loads The range of permissible software system configuration Human factors aspects of the user interfaces including the user procedures Sensitivity SW Config. Usability - ? 18 Introduction to Software Testing Testing Principles Introduction to Software Testing Testing Principles Errorcentric Testing ? Prioritize tests so that whenever you stop testing you have done the best testing in the time available ? Building Block Approach ? ? Unit testing for technical validation errors ? Integration testing for test interfaces across system ? System testing for test of business functionality ? Experience Based & Formal Test Case Design 20 Introduction to Software Testing Testing Principles ? Scalability ? Thoroughness of tests related to the risks of failures ? Concentration on those parts which deliver the main benefits ? ? Front-Loading ? System test cases design as Requirements are prepared Independent Testers ? Test Automation 21 Introduction to Software Testing The TESTER’S View Bug - Quality Improvement Opportunity ? Testing is an “ART” ? Testing is the Single Largest Activity in Development ? Automating Chaos Gives Faster Chaos ? We Don’t Have Time to Get it Right, We Are Too Busy ? Testing Drives the Project ? 22 Introduction to Software Testing Introduction To Testing Techniques 23 Introduction to Software Testing Test Case Design Approaches Goal: Design a Necessary and Sufficient Set of Test Cases, To Ensure That All Functional Requirements & Non Functional Requirements Have Been Met! ? ? ? ? Experience Based Combinatorial Machines Rigorous Algorithms Code -Based 24 Introduction to Software Testing Types of Testing (Partial List) ? ? ? ? ? ? ? ? ? ? ? Static Testing Dynamic Testing Black Box Testing White Box Testing Gray Box Testing Ad Hoc Testing Exploratory Testing Function Testing Non-Function Testing Regression Testing Stress Testing 25 Introduction to Software Testing Static Testing Testing a work product without running it Static Analysis • Should be done for all documents during the development of a system • Adherence to coding standards • Identification of anomalities ?Control flow ?Data flow ?Semantics • Executable/ Non-Executable code • Requirements versus code • Design versus code Techniques • Walkthrough • Review • Inspection • Static analysis with the auxiliary of tools. It generates: ?Metrics (Complexity, size,…) ?Code/Data Cross references ?Call Trees ?Structure Charts 26 Introduction to Software Testing What is Software Inspection ? A type of Review, but we differentiate it as following: ? Walkthrough - Main purpose: understanding ? author guides the group through a document, so all understand the same thing, concensus on changes to make ? Review - Main purpose: decision making ? group discusses document and makes a decision about the content, e.g. how something should be done, go or no-go decision ? Inspection - Main purpose: find defects ? formal individual and group checking, using sources and standards, according to detailed and specific rules ? inspection is a well-defined process 27 Introduction to Software Testing What can be Inspected ? ? Anything written down can be inspected, i.e. ? policy, business plans, marketing material, contracts ? system requirements, systems designs ? software code ? test plans, test designs, test cases, test results ? user manuals, procedures, training material 28 Introduction to Software Testing The Inspection Process Software Development Stage Change Request Process Improvement Entry Kick off Ind Chk Meet Edit Exit Next Software Development Stage 29 Introduction to Software Testing Documents Used in Inspection Company Customer Project related Process related Rules Check Lists Procedures Source(s) (in writing) Inspection Product Log / Editing / Summary 30 Introduction to Software Testing Inspection Effectiveness & Return on Investment ? Most forms of testing are less than 30% efficient in finding errors or bugs The measured defect removal efficiency of both formal design inspections and formal code inspections is about 60% efficient, and according to Tom Gilb some of them reached 88%! The best defect removal efficiency level technique Mature Inspection - for 1 hour of inspection work you save 8 - 30 hrs of work ? ? ? 31 Introduction to Software Testing If Inspections are so Good Why doesn’t Everyone Use them ? ? ? ? Mainly “brain” work Almost no tools No company tries to “sell” inspections while there are many vendors selling testing tools Needs discipline Management issue ? ? 32 Introduction to Software Testing Industrial Trends ? Inspections become major engineering activity ? Motorola - Fagan inspections ? NASA - reduction of 90% rework ? Testing to complement inspection and to allow quality certification Inspection can be introduced with payoff in first project Main issue: Cost (Inspection + Testing (new) < Cost (Testing (old)) ? ? 33 Introduction to Software Testing Dynamic Testing Testing a program based on code ’s behavior while running code’ Black-Box Testing • Program is viewed as a Black-Box •Test cases are derived directly from specifications, not from internal logic White-Box Testing • Program is viewed as a White- Box • Test cases are derived from internal logic not from specifications • Also called structured testing 34 Introduction to Software Testing Kinds of Dynamic Testing ? EXHAUSTIVE TESTING I.e. to test A = B+C if B and C are in the range -32767 to32768, 4,294,836,225 test cases are needed to do exhaustive testing!!! ? RANDOM ? is ? does ? does TESTING (Ad-Hoc) ? a poor technique not provide an optimal subset of test cases not provide a high probability of detecting most errors ? perhaps, the most widely used technique INTELLIGENT TESTING ? Develop test cases using Black-Box techniques ? Supplement these test cases using White-Box techniques 35 Introduction to Software Testing Black-Box Testing Techniques ? Equivalence Partitioning ? Boundary Value Analysis ? State Transitions ? Rigorous Algorithms ? Exploratory Testing 36 Introduction to Software Testing Equivalence Partitioning Partition the input domain into a finite number of equivalence classes ? Create test cases to invoke as many different conditions as possible ? * It is based on the premise that if a test case in an equivalence class detects an error, all other test cases in the same class detect the same error 37 Introduction to Software Testing Equivalence Partitioning Step 1 ? Identify the input parameters ? Identify the allowable classes of input ? Select a representative value for each class ? Create test cases to cover as many classes as possible ? Step 2 ? Select invalid values for each class ? Create additional test cases to cover invalid classes ? Step 3 ? Identify ? Identify the output parameters the possible classes of output ? Create test cases to generate each possible output 38 Introduction to Software Testing Equivalence Partitioning Example External Condition 1. Input Hours a. Regular b. Over Time 2. Input Rate 3.5 to 13.5 3. Gross pay 10 to 2000 Valid Invalid Equivalence Equivalence Class Class 25 43 12 0 -1 2.1, -2.5 15.0, ax2 9, 2010 39 Boundary Value 450 Introduction to Software Testing Boundary Value Analysis Boundary Value Analysis technique tests conditions on, above and beneath the edges of input and output Equivalence Classes ? Test cases are created to test the edge of each Equivalence Class ? Test cases are created to test edges of both input and output classes ? 40 Introduction to Software Testing Boundary Value Analysis Steps For each equivalence class identified: a value on the class boundary ? Pick a value just under the boundary ? Pick a value just over the boundary ? Select 41 Introduction to Software Testing Boundary Value AnalysisInvalid Example Valid External Equivalence Equivalence Condition Class Class 1. Input Hours a. Regular 25 0 b. Over Time 43 -1 2. Input Rate 3.5 to 13.5 12 2.1, -2.5 15.0, ax2 3. Gross pay 10 to 2000 450 9, 2010 Boundary Value 40, 39.9, 40.1 59.9, 60, 60.1 3.5, 3.49, 3.51 13.5, 13.51, 13.49 10.0, 10.01, 9.99 2000, 2000.1, 1999 42 Introduction to Software Testing FINITE STATE TESTING Process: Create test cases to: ? Force each transition ? Force each action 43 Introduction to Software Testing Exploratory Testing Some people can design test cases that will discover failures based on their experience ? Allow these people to write test cases ? Address complex areas ? Address changes ? Report failures, faults and errors so that experience level may grow ? 44 Introduction to Software Testing White-Box Testing Techniques ? Control Flow ? ? ? ? ? ? Statement Coverage Decision (Branch) Coverage Condition Coverage Decision/Condition Coverage Multiple Condition Coverage Loop Coverage All Definitions All Predicates Uses All Computation Uses All P-uses/some C-uses All Uses 45 ? Data Flow ? ? ? ? ? Introduction to Software Testing White-Box Testing ? STATEMENT COVERAGE ? To write test cases such that every statement is executed at least once ? It is a weak criteria ? It will not test the validity of logical operators ? DECISION ? Decision COVERAGE Coverage (also called Branch Coverage) states that test cases must be written such that each decision has a True and False outcome at least once ? Decision Coverage emphasizes that each branch direction must be executed at least once ? It usually satisfies Statement Coverage 46 Introduction to Software Testing White-Box Testing ? CONDITION COVERAGE ? Condition Coverage technique states that test cases must be written such that every condition in a decision takes all possible outcomes at least once ? Condition Coverage technique may not cover all decision outcomes ? DECISION/CONDITION COVERAGE ? Decision/Condition coverage technique states that each condition should take on all possible outcomes at least once and each decision must take on all possible outcomes at least once ? MULTIPLE CONDITION COVERAGE ? Multiple Condition coverage technique states that test cases must be written such that all possible combinations of conditions in each decision are taken at least once 47 Introduction to Software Testing White-Box Testing ? LOOP COVERAGE ? Loop coverage technique states that test cases must be written to test the loop counters Process: • Treat the loop counter as an equivalence class • Test the boundary conditions 48 Introduction to Software Testing Testing Automation Introduction to Software Testing What’ s Wrong with Manual Testing ? ? Covers functionality only * * * * * * at human speed major investment in process and not in errors detection limited by resources availability and not necessity inexact repetition of tests inaccurate results checking difficult regression testing means NO regression testing POOR QUALITY SYSTEMS COSTLY MAINTENANCE 50 Introduction to Software Testing GUI Test Automation Techniques ? Capture/Replay ? Test Programming ? Data Driven 51 Introduction to Software Testing Computer Aided Software Testing Tools 52 Introduction to Software Testing Categories of CAST Tools CAST Tools CAST Tools Static Static Testing Testing Static Static Analysis Analysis Review Review Support Support Requirements Requirements Dynamic Dynamic Testing Testing Test Test Management Management Results Results Checking Checking Comparator Comparator Test Planning Test Planning Test Test Preparation Preparation Test Design Test Design Data Data Preparation Preparation Test Test Running Running Test Harness Test Harness Capture Capture Replay Replay Simulator Simulator Dynamic Dynamic Analysis Analysis Debug Debug Testware Testware Management Management Metrics Metrics Coverage Coverage Error Tracking Error Tracking Performance Performance 53 Introduction to Software Testing Summary Introduction to Software Testing The Integrated View Requirements Management Visual Modeling Development Tools Components Manual & Automated Software Testing Software Configuration Management 55 Introduction to Software Testing Q&A 56