Software Testing

Document Sample
Software Testing Powered By Docstoc
					    Software Testing

          @                       p //                   /

1                                          KAIST CS550
    Before Beginning

             @                       p //                   /

2                                             KAIST CS550
                                                                     KAIST SE lab

    Error, Fault, and Failure

      –   difference between the actual output and the correct output.
      –   measurement of the difference between the actual and ideal
      –   a condition(potential) that causes the software to fail.
      –   'b '
      –                                                   function.
          the inability of a system to perform q required function
      –   is produced only when there is a fault
      –   (c.f) presence of a fault does not guarantee a failure.

3                            KAIST CS550
                                                     KAIST SE lab

    Software Reliability

           p          y            g
      The probability of not undergo a failure
      Essential levels of confidence
      Unreliability is caused by faults
                                disciplines( wear    tear )
      Unlike other engineering disciplines("wear and tear")

4                       KAIST CS550
                                                                              KAIST SE lab

      Dependability Properties
                                         Robust but not safe: catastrophic failures
                                         can occur
                                                                 Safe but not correct:
                                                                 annoying failures can
                                                Robust           occur

                    C     t
                    Correct                            Safe
                                                       S f

                                                              Reliability but not correct:
    Correct but not safe or robust:                           error conditions occur
    the specification is inadequate                           rarely

5                                     KAIST CS550
                                                      KAIST SE lab

    V(erification) & V(alidation)

       – the process of determining whether or not the
         product of given phase fulfill the spec. from the
         previous phase
       – the process of evaluating the software at the end of
         the development to ensure the compliance with the
              i     t

6                       KAIST CS550
                                                       KAIST SE lab

    Classification of V&V Activities

      Most of the V&V activities can be classified as static or
       – Static
            p g
            program pprovingg
            code reading
       – Dynamic
            testing(so called dynamic testing)

7                        KAIST CS550
                                                                 KAIST SE lab

    Static v.s. Dynamic: characteristics

      –   do not observe system behavior
      –   not looking for system failures
      –   faults are directly detected
      –            behavior is b
          system b h i i observed, and  d    d
      –   determine the existence of failures
      –                              faults,
          reveals the presence of faults but not their absence
      –   actual faults are identified by "debugging" activity
      –   no failure does not means no fault

8                           KAIST CS550
                                                        KAIST SE lab

    Static v.s. Dynamic : Cost

         g                         y
      In general the static activity is more cost-effective than
      the dynamic
      Is it always true ?
      Is there any trade-off ?
      How do we combine the two methods ?

9                        KAIST CS550
                                                                  KAIST SE lab

     Difficulties in determining failure

                               q            g       p
       Definition of failure requires unambiguous specification
       of system behavior.
        – The specification should specify the true and correct
          system behavior
        – But hard to satisfy
             The specification might be error prone : it is usually built in
             the early phase
       –   Disagreement between developer and client
             may need possible error and failure specification, but not

10                           KAIST CS550
     Software Testing
     S ft     T ti

           @                       p //                   /

11                                          KAIST CS550
                                                        KAIST SE lab

     What is Software Testing ?

                 g(     y g)     y             p
       Exercising(analyzing) a system or component
        – with defined inputs
        – capturing monitored outputs
        – comparing outputs with specified or intended
       For the following purposes:
                    y       p
        – to identify discrepancies between actual results and
          correct or expected behavior
        – to provide high assurance of reliability, correctness,
12                        KAIST CS550
                                                                       KAIST SE lab

     Fundamental Testing Questions

      What cases should we test?
       –   what test data should we use ?
       –   what aspects of the software should we test?
      Are the outputs correct?
       –   do they meet the specification?
       –   do they meet the client's/users' needs?
      Are we there yet? When can we stop testing?
       –   did we find(and correct) a sufficient number of failures?
       –   did we cover the product satisfactorily?
      How well did we d ?
      H     ll        do?
       –   did we provide high assurance?
      Infinite possibilities with limited resources?

13                             KAIST CS550
                                                     KAIST SE lab

     State of the Practice

         g                         g        g          g
       High cost of software testing has long been recognized
        – early studies showed that upwards of 50% of
          development costs were incurred by software
        – figure still holds today
       Yet operational software still fails

14                       KAIST CS550
                                                      KAIST SE lab

     Testing Steps

                            g    p
       Q : List the following steps in order
       – Select what is to be measured by the test
       – Decide how to test
       – Develop test case
       – Execute the test case
       – Determine the expected result of the test oracle
       – Compare the actual result and the expected result

15                       KAIST CS550
                                                       KAIST SE lab

     Testing Principles

                            p     p
   Q : Choose the correct principles
 (Y,N) Testing is the process of executing a program with the
   intention of proving the correctness of the program.
 (Y,N) It is possible to completely test any nontrivial module
   or any system.
 (Y,N) Testing takes creativity and hard work
 (Y,N) Testing can prevent errors from occurring
 (Y,N) Testing is best done by several independent testers.

16                       KAIST CS550
                                                       KAIST SE lab

     Goal of Testing

       To find faults
        – A Test is successful if the program fails
            (Goodeneogh, Gerhart, "Toward a Theory of Test
              Data Selection", IEEE TSE Jan. 85)
                           (      p
       Provide confidence(Acceptance Testing)  g)
        – of reliability
             (p          )
        – of(probable) correctness
        – of detection(therefore absence) of particular faults

17                       KAIST CS550
                                                                KAIST SE lab

     Program Testing

       –   The process of exercising or evaluating a system or system
       –   Manual or automated means.
       –   To verify that it satisfies specified requirements.
       –                                                       results.
           To identify differences between expected and actual results
      Testing v.s. Debugging
       –   Testing : the process of finding errors.
       –   Debugging: the process of removing errors.
18                           KAIST CS550
                                                    KAIST SE lab

     Who Should Test Your Program?

      Human beings are g goal-oriented.
      What if your goal were to demonstrate that your
      program has no errors?
      Most people are inclined to defend what they produce.
      Q: Thus, who?

19                      KAIST CS550
                                                           KAIST SE lab

     Fisherman's Dilemma

                          y           g
      You have three days for fishing and 2 lakes to choose from.
      Day 1 at lake X nets 8 fishes.
      Day 2 at lake Y nets 32 fishes.
      Q: Which lake do you return to for day 3?
      Fishing for errors
       – In general, the probability of the existence of more errors
         in a section of a program is proportional to the number of
         errors already found in that section.
      Problems of error-prone modules: A study showed that ..

20                        KAIST CS550
                                                           KAIST SE lab

     Test Cases

                   p                    y       g
      Two kinds of problems discovered by testing:
       –   The program DOES NOT do what it is supposed to do.
       –   The program DOES something it is NOT supposed to do.
      Test cases must be written for INVALID and
      UNEXPECTED, as well as valid and expected input
      What are parts of a test case?
       –                        conditions.
           Description of input conditions
       –   Description of expected results.
      Q: Why expected results should be included in a test
21                           KAIST CS550
                                                                KAIST SE lab

     Test Every Input Conditions?

        –   A module has two input integer parameters.
        –   Word size: 32 bits.
        –   Number of input conditions: ???
        –   Test completely automated.
        –   Time to execute one test case: 0.0000001 second.
        –   Total time required: 58,494 years!!
       Generally impossible, for even trivial examples, to test a program
       using every possible input conditions.
       Conclusion: Testing cannot guarantee the absence of errors
       Realistic goal of testing
        – Design a small number of test cases that can establish a
                   bl l     l f   fid          th b
           reasonable level of confidence on the absence of  f
           undiscovered errors.
22                             KAIST CS550
                                                     KAIST SE lab

     Terminology (1/2)

                        g (Functional testing)
       Black box testing (                  g)
        – A test derived solely from examining the
          requirements, specifications or user documents.
       White box testing (Structural testing)
        – A test derived from examining the actual code.
       Testing in the small
                 g                  ,
        – Testing of individual unit, or module.
       Testing in the large
        – Testing of a set of modules.

23                       KAIST CS550
                                                    KAIST SE lab


        – When an element under test cannot be executed by
          itself, a DRIVER is needed to invoke the element.
        – The DRIVER supply appropriate data and report the
        – When an element under test calls other elements
          which are not present, STUBs are needed to take
          their places.
          The STUB           t    l     d    d
        – Th STUBs can return planned, random or constantt t
24                      KAIST CS550
                                                                    KAIST SE lab

     Practical Issues in Test Case Selection

                                     p p
       Test cases selected for three purposes
       –   fault detection(and correction)
       –   fault prevention
       –   high assurance software
       Test Selection is a sampling technique
       –    h       finite
           choose a fi i set of i         from an i fi i d
                                 f inputs f                    i
                                                  infinite domain
       –   choose a finite partition of the structure
       Minimize the number of tests while maximizing
       assurance that remaining defects(that have not been
          g )           p
       caught) are acceptable
       –   test case selection id a development process
25                            KAIST CS550
                                                       KAIST SE lab

     Test Execution and Behavior Verification

                        q           preparation
       Test execution requires test p p
       –   test bed generation
               stubs, drivers
               test initialization, cleanup
       –   testing profiling
       –   test data capture
       Test execution requires behavior verification
       –   behavior comparison with oracle
       –   problem reporting
       –   progress monitoring

26                             KAIST CS550
     Taxonomy of Testing Techniques

                     @                       p //                   /

27                                                    KAIST CS550
                                                             KAIST SE lab

     Various Testing Techniques
       Unit Testing     Functional        Structural      Spec-based
                        Testing           Testing         Testing

       Integration      Incremental
       Testing          Top-down
                        Top down

       Acceptance       Bottom-up
       Regression         Thread                        Testing
                          Sandwich    Critical module
28                    KAIST CS550
     Categories of Testing Techniques
        : by testing levels (phases)

                      @                       p //                   /

29                                                     KAIST CS550
                                                                   KAIST SE lab

     Testing Phases (Levels)
     Unit Testing
     –   testing f   it       i    ith     i     t
         t ti of a unit comparing with requirements
     Integration Testing
     –   systematic combination and testing of software components
         to insure consistency of component interface
               y           g
     Software System Testing
     –   testing an integrated software system comparing it with
         software system requirements
     System Testing
     –   testing an integrated hardware and software system (in
         target environment)
     –   (c.f) Host-Target Testing
30                            KAIST CS550
                                            KAIST SE lab

     Testing Phases (Levels) -cont.

         g              g
       Regression Testing
       –   re-testing after modification
       Acceptance Testing
       Host-Target Testing

31                            KAIST CS550
                                                              KAIST SE lab

     Levels of Testing
       Attempt to detect different types of faults
              Client                           Acceptance
              Needs                            Testing
                                               T i



              Code                                        g
                                               Unit Testing

32                            KAIST CS550
                                                                  KAIST SE lab

     Unit v.s. Integration v.s. System Testing
     Unit Testing          Integration Testing     System Testing

     from module           from interface          from requirements
     specification         specification           specifications

     visibility of code    visibility of the       no visibility of code
     details               integration structure

     complex scaffolding   some scaffolding        no drives/stubs
     required              required                required

     attention to          attention to            attention to system
     behavior of single    integrations among      functionalities
     module                modules

33                            KAIST CS550
                                                                              KAIST SE lab

     Integration Test

       Testing needed as the elements of a system are being combined.
       –   Nonincremental (Big-Bang) testing
       –   Incremental testing
       Nonincremental testing
       –   All the elements are combined at once and tested.
       –   It is difficult because errors cannot be easily attributed to particular
       Incremental testing
       –   Elements are added and tested incrementally.
       –   In What orders should the elements be integrated.

34                                KAIST CS550
                                                  KAIST SE lab

     Top-Down Strategy

     The root module is tested first.
     Stubs take the place of other modules called by the
     root module.
     A new module called by one of the already
     integrated modules is added and tested with
     The process continues in this manner until all
     modules have been integrated and tested.

35                      KAIST CS550
                                                        KAIST SE lab

     Top-Down Strategy

       – Advantageous if major flaws occur toward the top of
         the program.
       – Early skeletal program allows partial demonstration.
       – Before the I/O modules are added, the
         representation of test cases in stubs can be difficult.
       – Test cases are possibly in a different form for each
         level and must be continuously altered.

36                       KAIST CS550
                                                                        KAIST SE lab

     Bottom-Up Strategy

      Lower level modules are tested first using drivers.
      Higher l    l   d l       dd d ft      ll the   d l they ll
      Hi h level modules are added after all th modules th call
      have been added.
      The process continues in this manner until all modules have been
      integrated and tested
       –   Advantageous if major flaws occur toward the bottom of the program.
       –                                  create.
           Test conditions are easier to create
       –   Observation of test results is easier.
       –                                                                    added.
           The program as an entity does not exist until the last module is added
       –   As system subtrees are integrated at higher levels, several modules may
           be brought together for the first time in one step.

37                              KAIST CS550
                                                          KAIST SE lab

     Thread Strategy

                        p            gy
       A variation of top-down strategy.
       Input and output modules are added as early as
       Choose a set of modules (the thread) that represents a
       normal, important case, or a critical path through the
       –   Other threads can be integrated in parallel.
       –   Early skeleton versions are available early.
       –   It also improves programmers morale.
               l i                          l

38                           KAIST CS550
                                        KAIST SE lab

     Other Integration Strategies

       Critical Module First
       I/O First, etc

39                        KAIST CS550
     Categories of Testing Techniques
      : by test case selection criteria

                       @                       p //                   /

40                                                      KAIST CS550
                                                      KAIST SE lab

     Test Case Selection Criteria

       Test case selection criteria can be categorized as
        – Functional
             historically heuristics based on requirement
        – Structural
             path selection criteria
             data selection criteria
          Spec based
        – Spec-based
        – Fault-based

41                       KAIST CS550
                                                                KAIST SE lab

     Functional v.s. Structural

       Functional testing
       –   test cases selected based on functional specification(require
           document, high-level design)
       –    i     the           t    black box
           views th component as a bl k b
       Structural testing
       –   test cases selected based on software
       –   views the component as a white box
       Both functional and structural testing must be done.

42                           KAIST CS550
                                                       KAIST SE lab

     White Box Testing

            g         g                        (   program
       Using knowledge on internal structures(i.e. p g
       paths) and maybe even ignoring specification.
       Uses internal structures of the program to derive the
       test data
       Will "exhaustive(or very through) white-box testing"
       solve testing problem ?

43                       KAIST CS550
                                                        KAIST SE lab

     White Box Testing Steps

       Steps involved:
        – Examine the program logic.
        – Design test cases to satisfy logic coverage criteria.
        – Run the test cases.
        – Compare actual results and report errors.
        – Compare actual coverage to expected coverage.

44                        KAIST CS550
                                      KAIST SE lab

     Coverage Criteria in White-Box Testing

       Statement coverageg
       Decision coverage
       Condition coverage
       Decision/Condition Coverage
       Multiple condition Coverage

45                      KAIST CS550
                                                     KAIST SE lab

     Statement Coverage

      Every statement must be executed at least once.
      Q: Design a test case that can cover all the statements.
      Weakest logic coverage criterion.
      Possible undetected errors:

46                      KAIST CS550
                                                            KAIST SE lab

     Decision Coverage
     (Branch Coverage)

       Each branch must be taken at least once.
       –  If-then, if-then-else statement:
               True and False branches
        – Case statement:
               each branch and others or default.
        – While, repeat and for statements:
               Must enter loop body at least once and must exit
       Q: Design a set of test cases for White-box-test procedure,
       satisfying the decision coverage criterion:
       Possibly undetected errors:
       P     ibl      d t t d

47                         KAIST CS550
                                                        KAIST SE lab

     Condition Coverage

      Each condition in a decision must take on all possible
      outcomes at least once.
      Verify that the following test cases satisfy the condition
      Q: Design a set of test cases, satisfying the condition
      coverage criteria.
      Q: Is condition coverage the same as branch
      coverage? ?

48                       KAIST CS550
                                                      KAIST SE lab

     Decision/Condition Coverage

       Each condition in a decision must take on all possible
       outcomes at least once.
       Each decision must take on all possible outcomes at
       least once.

49                       KAIST CS550
                                                                KAIST SE lab

     Multiple-Condition Coverage

       Every possible combinations of condition outcomes in each
       decision must be taken on at least once.
       Combination of conditions in White-box-test procedure:
                                         multiple-condition coverage.
       Q: Design a set of test cases for multiple condition coverage

50                           KAIST CS550
                                                        KAIST SE lab

     Functional Testing Heuristics

       Test data are developed from documents that specify the
         ft     ' i t d db h i
       software's intended behavior
       Basic goal is to test each specified software
        – equivalence partitioning
        – boundary value analysis
        – syntax coverage
        – special value coverage
        – output domain coverage
        – cause /effect(decision table) coverage

51                        KAIST CS550
                                                     KAIST SE lab

     Equivalence Partitioning

                    p    p           q
       Divide the input space into equivalence class.
       Items in each equivalence class are treated the same
       in the sense that a test of any item in the class is
       equivalent to a test of any other item in the class
       Output space partitioning may be also considered.
       Both valid and invalid equivalence classes are
       Pick at least one element from each equivalence class
       to test.

52                      KAIST CS550
                                                                  KAIST SE lab

     Guidelines for Partitioning

               p              p             g           ,
       If an input condition specifies a range of values,
       –   e.g. -- the item count can be from 1 to 999.
       –   Identity one valid equivalent class and two invalid classes:
       –   Test cases:
       If an input conditions specifies a set of values and
       th     is          to believe th t each i h dl d
       there i a reason t b li       that    h is handled
       differently by the program, identify a valid equivalent
       class for each value and one invalid equivalence class
       –   e.g. --type of the variable must be integer or real.
       –   Test cases:

53                            KAIST CS550
                                                      KAIST SE lab

     Guidelines for Partitioning-cont.

     If an input condition specifies a "must be" situation,
     identify one valid and one invalid equivalence class.
      – e.g. -- type of index must be positive integer.
      – Test cases:
     If there is any reason to believe that elements in an
     equivalence class are not handled in an identical
     manner by the program, split the equivalence class into
     smaller pieces.

54                        KAIST CS550
                                                   KAIST SE lab

     Coverage Matrix

                                   g           q
      Show each test case's coverage in valid equivalence
      classes and invalid equivalence classes.

55                     KAIST CS550
                                                      KAIST SE lab

     Identifying Test Cases

                        q                                   y
       Until all valid equivalence class have been covered by
       test cases, make a new test case covering as many of
       the uncovered valid equivalence classes as possible.
       Until all invalid equivalence classes have been covered
       by test cases, make a new test case that covers ONE
          d                f th          di    lid   i l
       and ONLY ONE of the uncovered invalid equivalence

56                       KAIST CS550
                                                     KAIST SE lab

     Boundary Value Analysis

            p                   g payoff         g
       A simple method with a high p y in finding errors.
       Focus on the elements near the boundary of the
       equivalence class.
       Consider boundaries in both input and output spaces.

57                      KAIST CS550
                                                                          KAIST SE lab

     Guidelines for Boundary Value Analysis

       If an input condition specifies a range of values,
        –   e.g. [-1.0, +1.0]
        –   Test the boundary:
        –             j                         y
            Test the just outside of the boundary:
       If an input condition specifies a number of values,
        –   e.g. 1--255
        –   Test the maximum and minimum values:
        –   Test the value one beneath the minimum and the value one beyond the
                      set                              element.
       For an ordered set, focus on the first and last element
       Also check the output boundaries:
        –   Test maximum(minimum) output.
        –   Attempt to exceed maximum(minimum).
        –   Test no output.
58                                 KAIST CS550
                                                      KAIST SE lab

     Syntax Coverage

      Based on module interface
      Derives test data from syntactic representation of input
       – for example, input data may be generated from a
         representation such as a BNF grammar
                                    p g
      Ensures robustness--that the program p              p
                                             parses its input
      and handles incorrectly formatted data

59                      KAIST CS550
                                                            KAIST SE lab

     Output Range Coverage

      Based on module interface and function to be tested
      Requires expertise in the function to be tested
      Test data is choose to cover the output space
       –   typical and extreme values
       –   valid and invalid values
       –   For example, for an output with a specified range, inputs
           would be tested to produce output just inside, and jus
           outside the boundaries as well as interior value
       –     p              p          p
           Inputs that compute the outputs must be derived
       –   Ensures that functions are tested for range of output
           conditions and that all possible exceptions and error
           messages have been p
                  g               produced

60                          KAIST CS550
                                                       KAIST SE lab

     Special Value Coverage

       Based on the function to be tested
       Derive test data form algorithmic characteristics and
       well-known sensitive values
        – properties of function to be tested can aid in
          selecting that indicate the accuracy of the computed
        – for example, the periodicity of the sine function
                  t      d t    hi h differs b multiples of 360
          suggests use data which diff       by   lti l    f

61                       KAIST CS550
                                                               KAIST SE lab

     Cause and Effect Graph Coverage

                    g p provide a systematic means of
      Cause-effect graph p           y
      translating natural language specification into decision
      Decision tables are a concise method of representing
      equivalence classes and combining classes
       –             in h decision bl          if ll h       di i     h
           top rows i the d i i table specify all the conditions on the
       –                  p   y                         y
           Bottom rows specify all the actions that may occur
       –   Columns in the decision table match a set of conditions that
           are satisfied(causes) with a set of actions that should be
      Each column of the table suggests significant test data.
62                           KAIST CS550
                                                    KAIST SE lab

     Error Guessing

      Based on intuition and experience.
      Try to guess what the programmer may have
      Make test cases for assumptions not made explicit in
      the spec.

63                      KAIST CS550
                                                     KAIST SE lab

     An Example

        p y      program is to have two inputs.
      A payroll p g                         p
      The first input is a five digit id-number.
      The second input is a real number that is the number of
      hours worked
           p y                                 per
      The pay for the first 40hours is $6.00 p hour.
      After 40 hours, the person is paid time-and-half for
      Maximum hours per week is 56 hours
      The output is to be the name and gross pay for the
64                      KAIST CS550
                                         KAIST SE lab

     Partition Input Space

       Will be done during the lecture

65                       KAIST CS550
                                        KAIST SE lab

     Coverage Matrix

      Will be done during the lecture

66                      KAIST CS550
                                        KAIST SE lab

     Test Cases

      Will be done during the lecture

67                      KAIST CS550
                                                           KAIST SE lab

     So far, we have studied Software Testing, mainly dynamic
       testing approaches.

     Now,                               techniques, mainly
     Now we look at the static analysis techniques mainly, software
       inspection technique.

     Some of the verification techniques associated with formal
       methods are not covered in this class, such as model
       checking, etc
       checking etc. For those who are interested in such formal
       methods, please see the textbook.

68                         KAIST CS550
                                                              KAIST SE lab

     Static Analysis

       Inspecting the code to understand its properties and
                             verification i.e. testing.
       Complement dynamic verification, i e testing
       Informal analysis techniques:
       –   Code walk-through
        –  Code inspection
       Formal correctness proof.
       F    l       t         f

69                         KAIST CS550
                                                                         KAIST SE lab

     Code Walk-Through

      Select some test cases.
      Simulate execution of the code by hand.
      Si   l t      ti    f th    d b h d
       –   The number of people involved should be small(3-5).
       –   The participants should receive written documentation from the designer
           a few days before the meeting.
       –   The meeting should last a predefined amount of time.
       –                                                 y         ,         g     ,
           Discussion should be focused on the discovery of errors, not fixing them,
           nor on proposing alternative design decisions.
       –   Key people should be
               designer, presenting the rationale of the work.
               moderator for the discussion.
               secretary for writing a report.
       –   Managers should not participate in the meeting.

70                                 KAIST CS550
                                                                  KAIST SE lab

     Code Inspections

      Similar to the code walk-through.
      The analysis is aimed at the discovery of commonly made errors
      A list of some classical programming errors:
       –   Use of uninitialized variables.
       –   J
           Jumps i      loops.
                   into l
       –   Incompatible assignments.
       –                     loops.
           Nonterminating loops
       –   Array indexes out of bound.
       –      p p          g
           Improper storage allocation-deallocation.
       –   Actual-formal parameter mismatches.
       –   Comparisons of equality for floating point values.

71                           KAIST CS550
                             KAIST SE lab


72             KAIST CS550