Document Sample
testing Powered By Docstoc
					Testing can show the presence of errors,
but not their absence.
                                              E. Dijkstra

Trying to improve software quality by increasing
the amount of testing is like try to lose weight by
weighing yourself more often.
                                             S. McConnell
        Software Testing

Testing is the process of exercising a
program with the specific intent of finding
errors prior to delivery to the end user.
What Testing Shows

            requirements conformance


                                an indication
                                  of quality
  Who Tests the Software?

     developer                 independent tester

Understands the system         Must learn about the system,
but, will test "gently"        but, will attempt to break it
and, is driven by "delivery"   and, is driven by quality
• Testing is a critical process within any software life
  cycle model.
• Testing has implications on the quality of the
  produced software and the reputation of the
  producing company.
• Testing implementations is often referred to as
  software testing, or simply testing.
• Before:
  ▫ Testing was performed in ad hoc and informal manners.
  ▫ Testers were not properly trained and unmotivated.
  ▫ Testing often downplayed and little resources were allocated.
• Now: Reputation and marketability of the software is affected
  by the type and amount of errors discovered after release.
  ▫ Errors should be resolved and fixed before the software is
    released in the market.
  ▫ Revealing all failures and hence removing their resulting
    errors or faults is a very optimistic idea.
    GO TO Slide 46
Test coverage
• Only exhaustive (brute force) testing can guarantee
  full coverage, but it is impractical.

• Consider testing the software when a first name text field of
  20 characters maximum is filled by a user. Exhaustive testing
  requires that we test the software for all possible correct and
  incorrect field input values.
  ▫ Assume 80 possible correct and incorrect characters, the number of
    possible combinations for a valid or invalid first name would exceed
    8020. Using a computer that takes, say, 50 nanoseconds to process one
    combination, we would need about 1011 years to complete the testing
    for all possible combinations!
The reality
• With the absence of exhaustive testing, there are no
  formal guarantees that software is error-free.
• In practice even when all errors have been detected
  during testing, they may not all be fixed due to the
  lack of time.
• Unfixed errors may be fixed in future releases of the
• The difficulty in software testing is the inability to
  predict how many errors are left undiscovered.
Test coverage experiment
• Random testing experiment: errors of different levels of
  complexity were inserted in the software under test
• All errors except for one were discovered progressively.
• No indication on how much more testing is needed
• Finding errors is an asymptotic nonlinear function of time.
Ethics for software organizations
• It is the ethical responsibility of the software development
  company to ensure that
   ▫ adequate and appropriate resources are allocated to the
      testing activities,
   ▫ employees are properly trained to perform their testing
   ▫ state-of-the-art testing tools are available for use by
      trained employees, and
   ▫ an environment emphasizing the importance of software
      quality and dependability is being promoted internally.
Ethics for testers

• It is the ethical responsibility of the software tester
  to ensure that
  ▫ they remain up-to-date in their testing knowledge, and
  ▫ on the look for better tools and techniques, and
  ▫ that they put their best efforts and mental capabilities
    to discover as many errors as possible.
• After all, human well-being and physical safety may
  be at stake.
Dynamic and static testing
• Dynamic testing
  ▫ Test the software by executing it and applying test inputs
  ▫ Observe the output and make a judgment

• Static testing
  ▫ Check code without executing it
  ▫ Automatic static testing: Code as an input to a static analysis
  ▫ Non-automatic static testing: review, walkthrough, inspection
Test case
• Test plan includes a test suite
• A test suite is a set of organized test cases
  ▫ Organize around use cases for example
• Template to describe a test case
  ▫ Id, name, reference, priority, preconditions, preamble,
    input, expected output, observed output, verdict and
• Test cases are obtained using known testing
  techniques and strategies
Test case design vs execution
• Test case is developed first – verdict and remarks are
• Test case is executed when the actual testing is
  performed – the verdict is obtained
• Typically test cases are developed (designed) before
  even writing any code
  ▫ Acceptance test cases developed after requirements
    specification phase
  ▫ Unit and integration test plans developed after the design
    is completed and prior to coding
White-box, black-box and grey-box
testing techniques
• White-box: module or unit testing
• Black-box: module testing and system testing
Black-box unit/module testing
•   Boundary value analysis
•   Equivalence class testing
•   Decision table / decision tree based testing
•   State based testing
•   Robustness testing
Boundary value analysis
• BVA: test inputs obtained from the boundaries
   ▫ Serious errors hide at the boundaries.
• For example, simple boundary values for an input of integer
  data type would include: -INT_MAX, -INT_MAX + 1, -1, 0, +1,
  INT_MAX - 1 and INT_MAX.
• For example: 5am, 4:59pm for day rate; 5pm, 10:59pm for
  evening; 11pm, 4:59am for night rate.
• For complex and composite data types, boundary values
  become more difficult to generate and may require more
  ingenuity to obtain them.
Example 8.2 – Testing search function
  Suppose you are given a function called
  search, whose functional specification
  states that it will take an array A of 10
  integers and an integer value v to search
  for inside array A. The function search
  returns an integer r representing the
  index at which v was found in array A;
  otherwise it returns -1 if v is not found in
  array A.
Equivalence class testing
• Partitioning input domain into disjoint subsets or
• Partition includes related values
• Consider one test input from each partition
• Equivalence relation among partition elements
  ▫   ‘having the same number of digits’
  ▫   ‘all positive integers’
  ▫   ‘all even numbers’
  ▫   ‘all prime numbers’
  ▫   ‘all power of 2 numbers’
  ▫   …..
Decision table based testing

• Decision table: each column denoting a rule
  becomes the basis of a test case
  ▫ Conditions produce inputs
  ▫ Actions produce expected outputs
  ▫ In table 4.1: input variables are age, history and speed,
    and output variables are penalty and points
Decision tree-based testing

• Each distinct path from the root node to a leaf node
  is the basis of a test case
• If along the path, a variable input value does not
  exist, the test input for that variable can be assigned
  any valid value.
• For example, in fig 4.23, for the test case
  corresponding to the leftmost path, the speed value
  can be any illegal speed
State-based testing

• An implemented finite state machine conforms to its
  specification, if the observed output sequence
  matches the expected output according to the
• Test sequence must cover all states transitions
• Used to do black box testing of systems
  ▫ GUI, protocols, web page connectivity, …
• Transition tour: a sequence covering all transitions
  and returning back to the initial state
  ▫ Can only cover output errors
Black box-based robustness testing
• Test a system using one or preferably more black box
  based testing method
  ▫ For boundary value testing, test for illegal values
    beyond the boundaries – the system should survive
  ▫ In equivalence class testing, select one representative
    bad input from every partition of the bad input values
  ▫ For FSM based systems, apply unexpected inputs at
    each state – the system should survive
• Test for incorrect and illegal inputs also
White-box testing techniques

•   Given the source code of the module to test
•   Control flow based testing techniques
•   Data flow based testing techniques
•   Path testing
Control flow based testing

• Given source code of a module, transform it to a
  control flowgraph
• Nodes and edges – decision nodes, joining nodes
Example: digitCount

1 int digitCount(int n)
3 int count = 0;
4      while (n != 0)
5 { n = n / 10;
6     count++;
7 }
8     return count;

• When we have a multiple condition decision
• Reduced number of test cases but …
• A typo error may not be detected – because of short
• Solution – break the decision into many nodes
Path expression

• Representing in a generic way an execution
• Can represent the complete program
• Use operators:
  + for choice
  . For a concatenation
  {} * for zero or more repetition
  {}+ for one or more repetition
  () for grouping
Path expression

• Useful to study properties of test sequences
• Annotate the transitions with desirable property
  then analyze the resulting expression
• For example: assign a probability to each transition –
  allows you to evaluate the probability of each path;
  helps you decide on testing the most probable paths
Testing Strategies
  unit test          integration

 system                  validation
   test                     test
Integration Testing
• After unit testing all modules individually using both
  white and black box testing techniques
• Options:
  • the “big bang” approach
  • an incremental construction strategy
     Top-down integration
     Bottom-up integration
     Sandwich / hybrid integration
Top down integration
• Start with top level modules and integrate left to
• Good for testing GUI heavy modules
• Less time testing low level utility modules (high
  reusability modules)
• Variation: integrate i/o modules to get real input and
  outputs as early as possible
• Need stubs – fake modules to represent called
Bottom up integration

•   Start from low level modules
•   Good testing of low level utility modules
•   Can see the GUI late in the process
•   Require driver modules – to call the integrated
Sandwich integration testing

• Two teams – one starting bottom up and the other
  starting top down
• They meet in the middle
• Good testing of both high level GUI modules and low
  level utility modules
• Requires coordination and good management to
  ensure middle level modules also get a good share of
  the testing time
                   Smoke Testing
• A common approach for creating “daily builds” for product software
• Smoke testing steps:
  ▫ New software components are integrated into a “build.”
      A build includes all data files, libraries, reusable modules,
       and engineered components that implement one or more
       product functions.
  ▫ A series of tests is designed to expose errors that will keep the build from
    properly performing its function.
      The intent should be to uncover “show stopper” errors that
       will throw the software project behind schedule.
  ▫ The build is integrated with other builds and the entire product (in its
    current form) is smoke tested daily.
      The integration approach may be top down or bottom up.
More test selection strategies

•   Random testing
•   Probabilistic testing
•   Mutation testing
•   Regression testing
   High Order Testing Strategies
• Validation testing
  ▫ Focus is on software requirements
• System testing
  ▫ Focus is on system integration
• Alpha/Beta testing
  ▫ Focus is on customer usage
• Recovery testing (NFRs)
  ▫ forces the software to fail in a variety of ways and verifies that recovery is properly
• Security testing (NFRs)
  ▫ verifies that protection mechanisms built into a system will, in fact, protect it from
    improper penetration
• Stress testing (NFRs)
  ▫ executes a system in a manner that demands resources in abnormal quantity,
    frequency, or volume
• Performance Testing (NFRs)
  ▫ test the run-time performance of software within the context of an integrated
Software quality assurance
• Quality of the product and the processes by which
  the product is delivered
• SQA with respect to testing:
  ▫   Training of testers
  ▫   Documentation of test procedures
  ▫   Documentation and analysis of test results
  ▫   Review of test plans
  ▫   Proper management of testing activities
  ▫   Adherence to testing standards
  ▫   Develop and update of inspection checklists
  ▫   ….

Shared By: