AppLabs Manual Testing

ACE Course Module-1 Fundamentals of Software Testing Unit 1 Chief reviewer: Revision Version 1.0 (Draft) Introduction to Software Testing Issue Date 30-Jan-06 Author N. Geetha Approved by: Confidential V 1.0 Page 1 of 75 TABLE OF CONTENTS Objectives ............................................................................................................................................................................................ 4 Overview ............................................................................................................................................................................................... 4 Software Error Case Studies .......................................................................................................................................................... 5 Mars Climate Orbiter (1999) ....................................................................................................................................................... 5 Shooting Down of Airbus 320 (1988) ........................................................................................................................................ 6 Software Error leading a Russian spacecraft to a rare ballistic descent ........................................................................ 7 NIST study On CNN.com - April 27, 2003 .............................................................................................................................. 8 Why does Software bug occur? ...................................................................................................................................................... 8 Software Development Life Cycle .................................................................................................................................................. 9 Software Testing Terms................................................................................................................................................................. 15 What is Verification & Validation? ............................................................................................................................................... 18 What is Software testing? ............................................................................................................................................................ 20 Historical definitions of Testing: ................................................................................................................................................. 22 Principles of Software testing ...................................................................................................................................................... 25 The Essentials of Software testing............................................................................................................................................. 25 Testing Objectives ........................................................................................................................................................................... 26 Goals of testing ................................................................................................................................................................................. 27 Approaches to Software Testing ................................................................................................................................................. 29 Prevention-oriented ..................................................................................................................................................................... 30 Challenges in testing ........................................................................................................................................................................ 34 What does a tester do? .................................................................................................................................................................. 38 What are the desirable Qualities of a tester? ......................................................................................................................... 40 Confidential V 1.0 Page 2 of 75 Levels of Testing.................................................................................................................................................................................. 43 Testing techniques ........................................................................................................................................................................... 44 Introduction to various testing types ......................................................................................................................................... 45 Software Test Documentation IEEE 829 ................................................................................................................................... 58 Test Documents ................................................................................................................................................................................ 58 What is a test tool? ......................................................................................................................................................................... 59 Custom Test Tools used at AppLabs ............................................................................................................................................ 61 Summary ............................................................................................................................................................................................. 62 Quick Test.......................................................................................................................................................................................... 64 Discussion ........................................................................................................................................................................................... 71 References ......................................................................................................................................................................................... 74 Confidential V 1.0 Page 3 of 75 Objectives       Appreciate the need for Software testing Learn what is Software Testing? Learn various approaches to Software testing Learn the basic of Software testing Identify the various testing types Learn to find bugs in any Software Application Overview This unit “Introduction to Software testing” will introduce you to fundamentals of software testing. You will learn to immediately find problems in any computer program. It helps you understand exactly what a software bug is, how serious they can be, and why they occur. You‟ll learn what your goal is as a software tester and what traits will help make you a good one. You will get introduced to some commonly used terms. Confidential V 1.0 Page 4 of 75 Software Error Case Studies Mars Climate Orbiter (1999) Purpose: to relay signals from the Mars Polar Lander once it reached the surface of the planet Disaster: smashed into the planet instead of reaching a safe orbit Why: Software bug - failure to convert English measures to metric values $165M Fig 1: Mars Climate Orbiter (1999) Confidential V 1.0 Page 5 of 75 Shooting Down of Airbus 320 (1988) • US Vincennes shot down Airbus 320 • Mistook airbus 320 for a F-14 • 290 people dead • Why: Software bug - cryptic and misleading output displayed by the tracking software Fig 2: Airbus 320(1988) Confidential V 1.0 Page 6 of 75 All the above incidents only reiterate the significance of thorough testing of software applications and products before they are put on production. It clearly demonstrates that cost of rectifying defect during development is much less than rectifying a defect in production. Software Error leading a Russian spacecraft to a rare ballistic descent Fig 3: CNN.com report Confidential V 1.0 Page 7 of 75 NIST study On CNN.com - April 27, 2003 Fig 4: NIST study on CNN.com Why does Software bug occur? Bug occurs when one or more of the following fie rules is true: 1. The software doesn‟t do something that the product specification says it should do. 2. The software does something that the product specification says it shouldn‟t do. 3. The software does something that the product specification doesn‟t mention. 4. The software doesn‟t do something that the product specification doesn‟t mention that should. 5. The software is difficult to understand, hard to use, slow, or - in the software tester‟s eyes - will be viewed by the end user as just plain not right. Confidential V 1.0 Page 8 of 75 Software Development Life Cycle The process used to create a software product from its initial conception to its public release is known as the „Software Development Life Cycle model‟ ; There are many different methods that can be used for developing the software. The most frequently used ones are: • Big- Bang • Code-and-Fix • Waterfall • Spiral • V-Testing Confidential V 1.0 Page 9 of 75 Big Bang Model: There is little if any planning, scheduling or formal development process. All effort in spent in developing the software and in writing the code but the final outcome is not always sure to be perfect. Testing perspective: There is little to no formal testing done. Some testing might be squeezed in just before release. You will have a perfect spec (the final product). It‟s impossible to go back and fix things that are broken; your job is just to report what you find, so the customers can be told about the problems. Final Product Confidential V 1.0 Page 10 of 75 Code-and-Fix Model: The team starts with a rough idea of what they want, do some design, then proceeds into a long repeating cycle of coding, testing, and fixing bugs. At some point it is decided to release the product. Testing perspective: As a tester, you along with the developers will be in a constant state of cycling. As often as every day, you‟ll be given new or updated releases of the software and will set off to test it. You‟ll run your tests, report the bugs, and then get a new software release. You will get to test most of the features, when the bug count is found to be less, then the people responsible will decide to release the product. Informal Specification Code Final Product Fix Confidential V 1.0 Page 11 of 75 Waterfall Model : The software development process flows from one step to the next step (idea-analysis-design-development - test-final product) in the waterfall model. Testing perspective: By the time the software is delivered to the test group, every detail has been decided on, written down, and turned into software. From that, you can create an accurate plan & schedule. You know exactly what you are testing, and there‟s no question about whether something is feature or a bug. Idea Analysis Design Development Test Final Product Confidential V 1.0 Page 12 of 75 Spiral Model: This model starts small and gradually expands as the project becomes better defined and gains stability. Testing perspective: Since you are involved from design phase itself & you have been testing all long, so the last push only will be a validation that everything is okay. Confidential V 1.0 Page 13 of 75 “V” Testing concept: In this model, when the project starts, both the system development and system test process start at the same time with the same information. Testing perspective: The test and development teams work closely together. As project‟s do & check procedures slowly address the coverage from start to finish, the high level of risk at a project‟s inception will decrease to an acceptable level, by the project‟s conclusion. Confidential V 1.0 Page 14 of 75 Software Testing Terms What is Quality? The term Quality can be defined in any of the below ways Quality = Conformance with specification = Conformance with requirements = Conformance with user‟s requirements = Fitness for use = Customer Satisfaction = Quality is what the customer say it is How to achieve quality Shift in Focus Product (QC) Process (QA) System (QM) Confidential V 1.0 Page 15 of 75 Quality definitions from eminent sources •A predictable degree of uniformity and dependability at low cost and suited to market (Deming) • Conformance to the requirements (Crosby) • Fitness for use (Juran) • The degree to which a system, component, or processes meets customer or user need or expectation (IEEE 610.12) • Degree to which a set of inherent characteristics fulfills the requirements (ISO 9000) Quality Assurance: The set of support activities (including facilitation, training, measurement & analysis) needed to provide adequate confidence that processes are established and continuously improved to produce products that meet specifications and are fit to use. Confidential V 1.0 Page 16 of 75 Goal of Quality Assurance: The goal of Quality Assurance is to provide management with the data necessary to be informed about product or service quality, there by gaining insight and confidence that the product or service quality is meeting its goals. Quality Control: The process by which product quality is compared with applicable standards, and the action taken when a non conformance is detected. It focuses on defect detection and removal. This is a line functionperformance of these tasks is the responsibility of the people working within the process. Quality Management: It integrates fundamental management techniques, existing improvement efforts, teamwork, and technical tools in a disciplined approach focused on continuous process improvement. It is also called total Customer Focus, total Quality Control, Quality Assurance and a Leadership Program Confidential V 1.0 Page 17 of 75 What is Verification & Validation? Verification: It is the process of evaluating the system or component to determine whether the products of the given development phase specifies the conditions imposed at the start of the phase. “Are we building the product right?” Validation: It is the process of evaluating a system or component during or at the end of the of the development process to determine whether it satisfies specified requirements. “Are we building the right product?” Verification & Validation a Case study In April 1990, Hubble space telescope was launched into orbit around the Earth. A large mirror was used as a reflective telescope to magnify the objects it‟s aiming at in space. Owing to its huge size, it couldn‟t be positioned or even viewed through while it was on Earth; all its attributes were carefully measured and compared with specifications. Soon after it was put into orbit around the Earth, the images it had returned were found to be out of focus. Confidential V 1.0 Page 18 of 75 Why it failed: The mirror was extremely precise but wasn‟t accurate. Testing had confirmed that the mirror met the spec – verification – but it didn‟t confirm that it met the original requirement - validation Confidential V 1.0 Page 19 of 75 What is Software testing? A technical investigation done to expose quality-related information about the product under test. A technical –It and includes tools experimentation, (measuring logic, mathematics, models, event tools (testing-support etc.) programs), instruments, generators, Investigation –An organized and thorough search for information. This is an active process of inquiry. We ask hard questions ( run hard test cases) and look carefully at the results. Done to expose quality-related information –Different objectives require different testing strategies and will yield different tests, different test documentation and different test results. Confidential V 1.0 Page 20 of 75 • About the product under test. Help Files Error Messages Samples Samples & Examples Final Product Readme file Labels & stickers User Manuals Product Support Information Fig 7: Product Constituents and other related things Confidential V 1.0 Page 21 of 75 Historical definitions of Testing:       Establishing confidence that a program does what it is supposed to do (Hetzel,1973) The Process of executing a program or system with the intent of finding errors(Myers,1979) Detecting specification errors and deviations from the specifications. Any activity aimed at evaluating an attribute or capability of a program or system (Hetzel,1983) The measurement of software quality (Hetzel,1983) Verifying that a system satisfies its specified requirements or identifying the differences between expected and actual results. Confidential V 1.0 Page 22 of 75 Some more test definitions…  “Testing is an activity in which a system or component is executed under specified conditions; the results are observed and recorded and an evaluation is made of some aspect of the system or component“ - IEEE   Executing a system or component is known as dynamic testing. Review, inspection and verification of documents (Requirements, design documents Test Plans etc.), code and other work products of software is known as static testing.    Static testing is found to be the most effective and efficient way of testing. Successful testing of software demands both dynamic and static testing. Measurements show that a defect discovered during design that costs $1 to rectify at that stage will cost $1,000 to repair in production. This clearly points out the advantage of early testing. Confidential V 1.0 Page 23 of 75  Testing should start with small measurable units of code, gradually progress towards testing integrated components of the applications and finally be completed with testing at the application level.  Testing verifies the system against its stated and implied requirements, i.e., is it doing what it is supposed to do? It should also check if the system is not doing what it is not supposed to do, if it takes care of boundary conditions, how the system performs in production-like environment and how fast and consistently the system responds when the data volumes are high. Confidential V 1.0 Page 24 of 75 Before understanding the process of testing software, it is necessary to learn the basic principles of testing. Principles of Software testing       All tests should be traceable to customer requirements. Tests should be planned long before testing begins. The Pareto principle applies to software testing. Testing should begin “in the small” and progress toward testing “in the large‟. Exhaustive testing is not possible To be most effective, testing should be conducted by an independent third party. The Essentials of Software testing    The quality of the test process determines the success of test effort Prevent defect migration by using early life cycle testing techniques. Automate testing process using tools. Confidential V 1.0 Page 25 of 75    A Test Professional must take responsibility for improving the testing process. Testing is a professional discipline requiring trained, skilled people. Cultivate a positive team attitude of creative destruction. Let us now discuss briefly the Testing Objectives: Testing Objectives     Testing is a process of executing a program with the intent of finding an error. A good test is one that has a high probability of finding an as yet undiscovered error. A successful test is one that uncovers an as yet undiscovered error. The objective is to design tests that systematically uncover different classes of errors and do so with a minimum amount of time and effort. ”Primary role of testing is not demonstration of correct performance, but the exposure of hidden defects.” - G.J. Myers Confidential V 1.0 Page 26 of 75 Secondary benefits include: Demonstrate that software functions    Work according to specification. Meet the performance requirements. The data collected during testing provides a good indication of software reliability and some indication of software quality. Testing cannot show the absence of defects, it can only show that software defects are present. Goals of testing Let us now discuss the different goals of testing  Correctness: The extent to which a program satisfies its specifications and fulfills the user‟s mission Confidential V 1.0 Page 27 of 75 and goals.  Reliability: The extent to which a program can be expected to perform its intended function with required precision.    Efficiency: The amount of computing resources and code required by a program to perform a function. Integrity: The extent to which access to software or data by unauthorized persons can be controlled,. Usability: The effort required for learning, operating, preparing input and interpreting output of a program.  Maintainability: The effort required for locating and fixing an error in an operational program.    Testability: The effort required for testing a program to ensure it performs its intended function. Flexibility: The effort required for modifying an operational program Portability: The effort required for transferring a program from one hardware configuration and/or software system environment to another. Confidential V 1.0 Page 28 of 75  Reusability: The extent to which a program can be used in other applications- related to the packaging and scope of the functions that the programs  Interoperability: The effort required to couple one system with another Approaches to Software Testing There are many approaches to testing. The importance of any of the approaches depends on the type of the system in which you are testing. Some of the approaches are given below: Debugging-oriented This approach identifies the errors during debugging the program. There is no difference between testing and debugging. Demonstration-oriented Confidential V 1.0 Page 29 of 75 The purpose of testing is to show that the software works. Here most of the time, the software is demonstrated in a normal sequence/flow. All the branches may not be tested. This approach is mainly to satisfy the customer and no value added to the program. Destruction-oriented The purpose of testing is to show the software doesn‟t work. It is a sadistic process, which explains why most people find it difficult. It is difficult to design test cases to test the program. Evaluation-oriented The purpose of testing is to reduce the perceived risk of not working up to an acceptable value. Prevention-oriented It can be viewed as testing is a mental discipline that results in low risk software. It is always better to forecast the possible errors and rectify it earlier. In general, program testing is more properly viewed as the destructive process of trying to find the errors (whose presence is assumed) in a program. A successful test case is one that furthers progress in this direction by causing Confidential V 1.0 Page 30 of 75 the program to fail. However, one wants to use program testing to establish some degree of confidence that a program does what it is supposed to do and does not do what is not supposed to do, but this purpose is best achieved by a diligent exploration for errors. Confidential V 1.0 Page 31 of 75 Software Configuration Testing Test Results Evaluation Errors Debug Corrections Test Configuration Error Rate Data Expected Results Reliability Model Predicted Reliability Fig 8: Test Information Flow in a test life cycle Confidential V 1.0 Page 32 of 75 In the above figure:  Software Configuration includes a Software Requirements Specification, a Design Specification, and source code.   A test configuration includes a Test Plan and Procedures, test cases, and testing tools. It is difficult to predict the time to debug the code, hence it is difficult to schedule. Confidential V 1.0 Page 33 of 75 Challenges in testing Testing is a complex and challenging activity for a variety of reasons. Here are a few that we considered in this Module. The Strategy Problem There are many different objectives for testing. To be effective, testers have to design their strategy (how they‟ll test, what they‟ll look for, what test artifacts they‟ll create) to meet their stakeholders‟ objectives. The Oracle Problem An oracle is the principle or mechanism by which you recognize a problem. When you run a test, you need some way to decide whether the program passed the test or failed it, and all of the methods we have available are imperfect. Confidential V 1.0 Page 34 of 75 Impossibility of Complete Testing The program can fail in many ways. As a new tester, you might believe that you can approach a piece of Software, fully test it, find all the bugs, and assure that the Software is perfect. Unfortunately this is not possible, even with the simplest programs, due to four key reasons.     The number of possible inputs is very large. The number of possible outputs is very large. The number of paths through the Software is very large. The Software specification is subjective. You may say that a bug is in the eye of the beholder. Let us take the example of Windows XP. Carefully made programs have on an average 5 faults/1000 LOC. Which would mean 1M LOC will have 5000 faults. Windows XP has 45M LOC. Hence expected faults could be 45 x 5000 = 225,000. Hence the key challenge for a Software Tester is to learn how to reduce the huge domain of possible tests into manageable set, and how to make wise risk-based decisions on what‟s important to test and what‟s not. The below Figure shows the relationship between the amount of testing performed and the number of bugs found. Confidential V 1.0 Page 35 of 75 If you attempt to test everything, the cost go up dramatically and the number of missed bugs declines to the point that it‟s no longer cost effective to continue. If the cut the testing short or make poor decisions of what to test, the costs are low but you‟ll miss lot of bugs. The goal is to hit that optimal amount of testing so that you don‟t test too much or too little. Confidential V 1.0 Page 36 of 75 Number of Missed Bugs Cost of Testing Optimal Amount of Testing Quality Under Testing Over Testing Amount of Testing Fig 9 : Testing Cost graph Confidential V 1.0 Page 37 of 75 What does a tester do? Software Under Test Verification Tester Reports Documentation Fig 6: Tester activity in general Confidential V 1.0 Page 38 of 75 The goal of a software tester is to find bugs. If the tester misses the bugs, it costs the project and loss to the company. The tester shouldn‟t be content at just finding bugs – You as a Software tester should think about how to find them sooner in the development process, thus making them cheaper to fix. Even finding them early, is not enough; As the testers are the customer‟s eyes and the first one to see the software, he/she should speak for the customer and must seek perfection. Therefore, the goal of a software tester is to find bugs, find them as early as possible, and make sure they get addressed. Confidential V 1.0 Page 39 of 75 What are the desirable Qualities of a tester? It may appear that a Software tester‟s job would be easier than a programmer‟s. Isn‟t it easier to break the code and find bugs than writing code? Surprisingly, it‟s not. Testing requires methodical and disciplined approach. It requires the same hard work and dedication that programming does. It involves very similar skills, and although a software tester doesn‟t necessarily need to be a full-fledged programmer, having that knowledge is a great benefit. Here‟s a list of traits that the software testers should have: Explorer: software testers aren‟t afraid to venture into unknown situations. They love o get a new piece of software, install it on their PC, and see that what happens. Confidential V 1.0 Page 40 of 75   Troubleshooters: Software testers are good in figuring out why something doesn‟t work. They love puzzles. Relentless: Software Testers keep trying. They may see a bug that quickly vanishes or is difficult to recreate. Rather than dismiss it as a fluke, they will try every way possible to find it.  Creative: Testing the obvious isn‟t sufficient for software testers. Their job is to think up creative and even off-the-wall approaches to find bugs.  Perfectionist (mellowed): They strive for perfection, but they know when it becomes unattainable and they are okay with getting as close as they can.  Good Judgment: Software testers need to make decisions about what they will test, how long it will take, and if the problem they‟re looking at is really a bug.  Tactful and diplomatic: They are always the bearers of the bad news. They have to tell the programmers that their baby is ugly. Good software testers know how to do so tactfully and professionally and know how to work with programmers who aren‟t always tactful and diplomatic. Confidential V 1.0 Page 41 of 75 Persuasive: Bugs that testers find won‟t always be viewed as severe enough to be fixed. Testers need to be good at making their points clear, demonstrating why the bug does indeed need to be fixed, and following through on making it happen. Confidential V 1.0 Page 42 of 75 Levels of Testing Unit Testing Done by the developer at module level.  Integration testing Conducted by the project team in work parallelism  System testing Conducted by project team or by separate testing team if any.  Acceptance testing Conducted by client either in developer‟s site or at his site.  Confidential V 1.0 Page 43 of 75 Testing techniques Black Box (Test to the Spec.) White Box (Test to the Code) Fig 10 : Testing techniques Black box testing: Testing of a system or component whose inputs, outputs and general functions are known, but whose contents or implementation are known or irrelevant White box testing: Testing that takes into account internal mechanism of a system or component, types include branch testing, path testing, etc.,. Synonymous with glass box testing and Structural testing. Confidential V 1.0 Page 44 of 75 Introduction to various testing types Regression testing : Each time a new model is added as a part of integration testing, the software changes. New data flow paths are established, new I/O may occur, and new control logic is invoked. These changes may cause problems with functions that previously worked flawlessly. In the context of an integration test, strategy regression testing is the re-execution of subset of tests that have already been conducted to ensure that changes have not propagated unintended side effects. Regression testing is the activity that helps to ensure that changes do not introduce unintended behavior or additional errors.   Testing a defect fix Testing that a defect fix has not caused some other error Both are done by regression testing by running a set of test cases run previously. Confidential V 1.0 Page 45 of 75 System Testing: System testing is actually a series of different tests whose primary purpose is to fully exercise the computerbased system. Although each test has a different purpose, all Work to verify that all system elements have been properly integrated and perform allocated functions. System Testing is a predetermined combination of tests that, when executed successfully, satisfy IT management that the system meets requirement. Facility Testing: Facility Testing is to determine if every facility mentioned in objectives has been implemented. This is done by scanning the objectives line by line and compared with user manual. Examples: Prompt the user to select one of the alternatives user should be able to specify range of values. Confidential V 1.0 Page 46 of 75 Volume Testing: Volume Testing is to subject the program to heavy volumes of data. This is done by checking fields, records and files to see if their sizes can accommodate all expected data (use an automated tool to create records). Examples: Large volumes of data in Client / server applications. Load / Performance Testing: Performance testing is designed to test run-time performance testing occurs throughout all steps in the testing process. Even at the unit level, the performance of an individual module may be assessed as white- box tests are conducted. However, it is until not all system elements are fully integrated that the true performance of a system can be ascertained. Confidential V 1.0 Page 47 of 75 Performance tests are often coupled with stress testing and often require both hardware and software instrumentation. That is, it is often necessary to measure resource utilization in an exacting fashion. External instrumentation can monitor execution intervals, log events as they occur, and sample machine states on a regular basis. By incrementing a system, the tester can uncover situation that lead to degradation and possible system failure. Testing is done by loading the system with activity that simulates legitimate user activity. Statistics collected to predict what performance and response times, users are likely to get. The load testing is conducted by creating virtual users., using a load test tool and by creating typical scenarios to Simulate load and Think times is used to simulate user behavior. Stress Testing: Stress test is designed to confront programs with abnormal situations. In essence, the tester who performs stress testing asks: "How high can we crank this up before it fails"? Stress testing executes a system in a manner that demands resources in abnormal quantity, frequency, or volume. Confidential V 1.0 Page 48 of 75 For example 1. Special test may be designed that generate 10 interrupts per second when 1 or 2 is the average rate. 2. Input data rates may be increased by an order of magnitude to determine how input function will respond. 3. Test cases that require maximum memory or other resources may be executed 4. Test cases that may cause trashing in a virtual operating system may be designed. 5. Test cases that may cause excessive hunting for disk resident data may be created. Essentially the tester attempts to break the program. A variation of stress testing is a technique called sensitivity testing. In some situation, a very small range of data contained within the bounds of valid data for a program may cause extreme and even erroneous processing or profound performance degradation. This situation analogous to a singularity in a mathematical function. Sensitivity testing attempts to uncover data combinations within valid input classes that may cause instability or improper processing. Confidential V 1.0 Page 49 of 75 Testing is performed by subjecting the application under test to peak volume of data in a short time. This is important for systems that normally operate below capacity but may be severely stressed during peak demand. Stress test is conducted by creating virtual users. Using a test tool typical user scenarios can be simulated. Do not use think times, as the idea is to exercise the system to fullest extent. Usability Testing: Usability Testing is done to checks for human factor problems: Are outputs meaningful? Are error diagnostics straightforward? Does GUI have conformity of Syntax, Conventions, Format. Style. Abbreviations? Is it easy to use? Confidential V 1.0 Page 50 of 75 Security Testing: Security testing attempts to verify that protection mechanisms built into a system will in fact protect it from improper penetration. During security testing, the tester plays the role(s) of the individual who desires to penetrate the system. Anything goes! The tester may attempt to acquire passwords through external clerical means, may attack the system with custom software designed to break down any defenses that have been constructed; may overwhelm the system, thereby denying service to others; may purposely cause system errors, hoping to penetrate during recovery; may browse through insecure data, hoping to find the key to system entry; and so on. Given enough time and resources, good security testing will ultimately penetrate a system. The role of the system designer is to make penetration cost greater than the value of the information that will be obtained. Devise test cases that subvert the program‟s security checks: Confidential V 1.0 Obtain passwords Access idle terminals Page 51 of 75 - Imitate valid users Guess passwords Check permissions of different user groups / users. Check database security Create more users than allowed in user groups. Delete user groups like “Supervisor” / ADMIN Rename user groups like “Supervisor/ ADMIN Storage Testing: Storage Testing is done to   Detect amount of main and secondary storage requirements program Determine capacity of system to store transaction data on a disk or in other files e.g. 1000 records of 512 bytes in a single flexible disk? Confidential V 1.0 Page 52 of 75 Recovery Testing: Many computer-based systems must recover from faults and resume processing within a pre-specified time. In some cases, a system must be fault tolerant; that is, processing faults must not cause overall system function to cease. In other cases, a system; failure must be corrected within a specified period of time or severe economic damage will occur. Recovery testing is a system test that forces the software to fail in a variety of ways and verifies that recovery is properly performed. If recovery is automatic (performed by the system itself), re-initialization, check pointing, mechanism, data recovery, and restart are each evaluated for correctness. If recovery requires human intervention, the mean time to repair is evaluated to determine whether it is within acceptable limits.    Determine ability of user to recover data or restart after a failure? Verifies both recovery process and components of process of recovery. Examples of recovery testing; Confidential V 1.0 Page 53 of 75 - Loss of input capability Loss of communication capability Loss of database integrity Application system failure Operator mistake Procedure Testing: Test any procedure prescribed for humans such as system operator or DB Administrator or user. Determine clarity of documentation on operation and use of system by having users do exactly what the manuals request. Compatibility / Conversion Testing: Compatibility testing is done with the below intent  Software should not interfere with the operation of other software. Confidential V 1.0 Page 54 of 75   Software should adhere to the conventions and requirements of the host operating system Software should work with all required optional software / hardware components and configurations. Instability Testing: Installability Testing is to check the installation and configuration procedure as well as any missing dependencies. This tests the installation procedure on:       Clean system Over a previous version of itself. Uninstall it completely Check for Auto play functionality Check custom install, partial install, add install if any. Check installation of third party components. Serviceability Testing: Check the service aids to be provided with the system (eg : Diagnostic programs),mean time to debug a problem, Confidential V 1.0 Page 55 of 75 Maintenance procedures etc., Documentation Testing: Documentation testing is testing the program operations & examples given in the user Documentation. Configuration Testing; Configuration testing the program with each type of hardware device and with minimum service. Acceptance Testing: Acceptance testing is done in the Final stage, before handing over to the customer. It is usually carried out by the customer. The Test cases are executed with actual data. Alpha & Beta Testing; If software is developed as a product to be used by many customers, it is impractical to perform formal acceptance tests with each one. Most software product builders use a process called alpha & beta testing to uncover errors that only the end user seems able to find. Confidential V 1.0 Page 56 of 75 The alpha test is conducted at the developer's site by a customer software is used in a natural setting with the developer "Looking over the shoulder" of the user and recording errors and usage problems. Alpha tests are conducted in a controlled environment. The beta test is conducted at one or more customer sites by the end user(S) of the software . Unlike alpha testing the developer is generally not present therefore the beta test is a "live". Application of the software in an environment that cannot be controlled by the developer. The customer records all problems (real/imagined) that are encountered during beta testing and reports these to the developer at regular intervals. Because of problems reported during beta test, the software developer makes modification and then prepares for release of the software product to the entire customer base. Localization: Localization is the process of adapting software for a new place. This is not necessarily change in language only. It should be Adaptable to new markets also tested to check that no necessary changes are missed, Confidential V 1.0 Page 57 of 75 Software Test Documentation IEEE 829     Standard set of documents valid for documentation of testing activities in SDLC Applicable for commercial, scientific and military software No specific methodologies, techniques or tools suggested No methodology for documentation control, configuration management, Quality assurance. Test Documents  Test Plan Describes the overall method to be used to verify that Software meets the product specification and the customer‟s needs. It includes the quality objectives, resource needs, schedules, assignments, methods, and so forth.  Test Cases List the specific items that will be tested and describe the detailed steps that will be followed to verify the software. Confidential V 1.0 Page 58 of 75  Bug Reports Describe the problems found as the test cases are followed. These could be done on paper but are often tracked in a database.  Test tools and automation If the test team is using automated methods to test the Software, that tools that are used, either purchased or written in-house, must be documented.  Metrics, Statistics and summaries These convey the progress being made as the test work progresses. They take the form of graphs, charts, and written reports.  Test Summary report What is a test tool?  Software that aids in planning/developing/executing of the testing process for another software package with the intention of detecting errors, tracking & reporting them.  V 1.0 The tools aid in setting up an unsupervised test facility for software testing. Page 59 of 75 Confidential Advantages of S/W Test tools:      Testing is formalized. Testing progress is automatically documented Test Plans can be reused. Defect tracking is systematic. Efficient because test scripts are developed that can be used in subsequent builds. Types of Testing Tools: 1. Test Planning, Management & Error Tracking. 2. Reviews & Inspections 3. Test Generation. 4. Test Execution. Confidential V 1.0 Page 60 of 75 Custom Test Tools used at AppLabs 1. AppQAmanager 2. Appzilla 3. AppMeter Confidential V 1.0 Page 61 of 75 Summary Software testing is a critical job. With the size and complexity of today‟s software, it‟s imperative that Software testing be performed professionally and effectively. The next module we shall discuss the various Software development Life Cycle models, you will learn to perform critical review of the specification. Dwell in detail the Software testing techniques. A quick recap of this unit “Introduction to Software testing”   Various Software error case studies demonstrate the need for systematic and effective Software testing. The goal of a software tester is to find bugs, find them as early as possible, and make sure they get addressed.    Software failure is commonly termed as a bug. Verification means “Are we building the product right?” Validation means “Are we building the right product?” Confidential V 1.0 Page 62 of 75   Black box and White box testing are the Software testing techniques Different testing types are intended to expose the different aspects of the Software Application.  Test Plan, Test Cases, Bug Reports, Test tools and automation, metrics, statistics summary reports constitute test Documentation. The requirement for higher-quality software demands a more systematic approach to testing from testers. Confidential V 1.0 Page 63 of 75 Quick Test 1. State whether True or False: a. It‟s important what term your company or team calls a problem in its software Solution: False. It‟s not important, but the term used often reflects the personality of the team and how they approach the finding, reporting, and fixing of the problems. b. A good tester relentlessly strives for perfection Solution: False. A good tester knows when perfection isn‟t attainable and when “good enough” is reached. 2. Why is it impossible to test a program completely? Solution: With any software other than the smallest and simplest program, there are too many inputs, too Confidential V 1.0 Page 64 of 75 many outputs, and too many path combinations to fully test. Also, software specs can be subjective and be interpreted in different ways. 3. Mixed scenario CPU Usage Comparison Report: The following diagram shows the CPU usage by the Server for different number of concurrent users accessing the server. The report corresponds to this area of testing a. Compatibility Confidential V 1.0 Page 65 of 75 b. Performance c. Functionality d. Security Confidential V 1.0 Page 66 of 75 4. Here‟s an error report: An error message - „twain_32.dll missing‟ is displayed, when the OK button is clicked the Installation continues and successfully completes. The above screen shot corresponds to this area of testing a. Compatibility b. Performance c. Installation d. Security Confidential V 1.0 Page 67 of 75 5. Here‟s a screen shot of an issue raised by a tester The above screen shot corresponds to this area of testing a. Compatibility b. Performance c. Security d. Installation Confidential V 1.0 Page 68 of 75 6. Below is the screen shot with tester‟s comments in box The placement of page indicators looks odd to see in the top of the interactivity. Suggestion : It should look good below of the navigation. Which aspect of testing does the above screenshot corresponds to a. Compatibility b. Usability c. Installation Confidential V 1.0 Page 69 of 75 d. Security 7. Tester‟s bug report reads as “Dark animation window appears for the model- Acids and alkalis”. On MAC On Windows Confidential V 1.0 Page 70 of 75 Which aspect of testing does the above screenshot corresponds to a. Compatibility b. Performance c. Installation d. Security Discussion 8. Listed below are four of the major computer system failures, what can you infer from these testing perspective a) On June 4 1996 the first flight of the European Space Agency's new Arian 5 rocket failed shortly after launching, resulting in an estimated uninsured loss of a half billion dollars. It was reportedly due to the lack of exception handling of a floating-point error in a conversion from a 64-bit integer to a 16-bit signed integer. b) The computer system of a major online U.S. stock trading service failed during trading hours several times over a period of days in February of 1999 according to nationwide news reports. The problem was reportedly due to bugs in a software upgrade intended to speed online trade confirmations. Confidential V 1.0 Page 71 of 75 c) In November of 1997 the stock of a major health industry company dropped 60% due to reports of failures in computer billing systems, problems with a large database conversion, and inadequate software testing. It was reported that more than $100,000,000 in receivables had to be written off and that multi-million dollar fines were levied on the company by government agencies. d) Software bugs caused the bank accounts of 823 customers of a major U.S. bank to be credited with $924,844,208.32 each in May of 1996, according to newspaper reports. The American Bankers Association claimed it was the largest such error in banking history. A bank spokesman said the programming errors were corrected and all funds were recovered. Solution: Testing is an important activity of the Software Quality process. They reiterate the significance of thorough testing of software applications and products before they are put on production. Postproduction defects are very expensive. 9. What‟s wrong with just testing that a program works as expected? Solution: At most, that‟s only half the testing problem. Users don‟t always follow the rules, and testers need Confidential V 1.0 Page 72 of 75 to prove out what happens when they don‟t. Also, if testers don‟t approach their testing with a gottabreak-it attitude, they will miss bugs. 10. Here is a Casestudy on the Y2K (Year 2000) Bug, Circa 1974 Sometime in the early 1970s a computer programmer – let‟s suppose his name was Dave – was working on a payroll system for his company. The computer he was using had very little memory for storage, forcing him to conserve every last byte he could. Dave was proud that he could pack his programs more tightly than any of his peers. One method he used was to shorten dates from their 4-digit format, such as 1973, to a 2-digit format, such as 73. Because his payroll relied heavily on date processing, Dave could save lots of expensive memory space. He briefly considered the problems that might occur when the current year hit 2000 and his program began doing computations on years such as 00 and 01. He knew there would be problems but decided that his program would surely be replaced or updated in 25 years and his immediate tasks were more important than planning for something that far out in time. After all, he had a deadline to meet. In 1995, Dave‟s program was still being used, Confidential V 1.0 Page 73 of 75 Dave was retired, and no one knew how to get into the program to check if it was Y2K compliant, let alone how to fix it. It‟s estimated that several hundred billion dollars were spent worldwide, to replace or update computer programs such as Dave‟s, to fix potential Year 2000 failures. Did Dave do anything wrong? Solution: If Dave was a good programmer he would have questioned the “obvious” oversight and not just programmed his software to work through 1999. Because he didn‟t, a software tester should have tested for and found the bug. The team could have decided whether to fix it. References 1. Effective methods of Software Testing by William E.Perry 2. Software Engineering by Roger Pressman 3. Lessons Learned in Software testing by Cem Kaner, James Bach, Bret Pettichord 4. S/W Testing Techniques by Loveland/Miller/Prewitt/Shannon 5. Practical Software Testing by Ilene Burnstein 6. Software Testing by Ron Patton 7. Managing the testing Process by Rex Black 8. Software Testing Fundamentals by Marnie L. Hutcheson Confidential V 1.0 Page 74 of 75 9. CSQP study material - STQC 10. CBOK - CSQA Confidential V 1.0 Page 75 of 75