Testing: A Roadmap
Mary Jean Harrold
Presented by : Atul Vij
CSCI 589 Fall 2004
A verification method that
applies a controlled set of
conditions and stimuli for
the purpose of finding
It’s a process that is performed to
support quality assurance
Making test Cases
Executing the application with those test
Examining the results produced by those
Why Further Research On
Testing accounts for 50% of the cost of software
Higher for critical systems
New tools need to be developed for easy and
What this paper is about
A report written in 2000
assesses the state of the art in software
outlines future directions in software
gives pointers to software testing
Software Testing Roadmap
Component Based Systems
Increased size and
complexity has led to
component based Component
Examples Provider User
Java beans, MFC, Java Views component Views Component
Applets, ActiveX independent Relevant to the
of the context Application
System Under Assemble
Availabilty of source code.
Component Trust Problem
Lack of testing mechanisms.
Theory of Test Adequacy by Rosenblum
Ongoing Research on providing testing tool
along with the component.
Efficient Techniques and Tools for:
Testing various aspects like security,
Representing and computing the types of
testing information that a component user
Storing and using testing information in the
Description of elements from which system is built.
Interaction among those elements.
Pattern that guide their composition and their
constraints on these patterns.
Testing based on Architecture
Chemical Abstract Machine (CHAM)
CHAM defined by a set of molecules(m1,m2…),
solutions(s1,s2,…) and transformation rules(t1,t2,…)
Transition Graph depicting all possible behavior of the
system is derived using software architecture
Derive a specific subgraph by selecting an
Derive a test plan to cover all the transitions in these
Example of CHAM
Lets say the criterion is concurrent
whatever scheduling policy the system will
adopt it will certainly have to consider this
subgraph (for detailed algorithms refer )
Transition Graph for sequential compiler Concurrent Reduced Graph
Techniques and tools for:
Test case requirements for accessing various
elements of the system
Systematic development of test cases early in
the development plan.
Automatic generation of test cases.
Evaluate software architecture for testability
and hence choose the best architecture early
in design phase.
Testing Evolving Software
Regression Testing : attempts to validate
modified software and ensure no new errors are
introduced into previously tested code.
Most expensive activity in a software system
Selective Regression Testing
Techniques and tools for:
Assist in managing the size of the test suite
Reduce the size of test suite
Identification of test cases that because of
modification is no longer needed.
Prioritize test cases.
Assess the testability of both software and
Demonstrating Effectiveness of
Increase confidence of developer
Provide visual interface
Identify classes of faults for a given test
Test complex boolean expression – Modified
Every point of entry and exit in the program
has been invoked atleast once
every condition in a decision in the program
has taken on all possible outcomes atleast
each condition has been shown to
independently affect the decisions outcome
Determine adequacy of test suites and test
cases that inspire confidence.
Determine interaction between the various test-
selection criteria and find ways to combine them
to perform more effective testing
Test cases based on data flow
use existing testing techniques to test visual
test complex Boolean expressions
Effective Processes for Testing
Process used for planning and
Microsoft uses a different approach
Synchronize and Stabilize as project
Build and Test a version every night.
Does Microsoft nightly rebuild,
minimize testing later ?
Test and Validate the existing models.
Generalized Results obtained by
Combine various techniques and tools
Combine static analysis with testing
Using Testing Artifacts
Artifacts: Execution Traces, Results of Test case
How to use Artifacts:
Dynamic Program Slides
Identifying the invariants in the Program.
Predicting the percentage of test suite that need to be
used after modification
Verifying that the existing techniques
provide useful information
New Techniques for testing to be used by
Some other techniques helpful in reaching end goal
Need for a scalable automatic test data generation
Static analysis required but expensive. Need for a
scalable analysis technique that can be used to compute
data-flow analysis expensive and hence need for
efficient instrumentation and recording techniques.
Methods and Tools
• Emphasizes the basic stages-Sets up the next paper.
• Examining Current Techniques to demonstrate
scalability & usefulness of techniques in practice-
• Relevance To Embedded Systems
• Talks about evolving systems-ES evolve by the second.
• Talks about component based testing-COTS.
• Emphasizes the need for Testing based on Precode Artifacts-
 Antonia Bertolino, Paola Inverardi, Henry Muccini,
Andrea Rosetti, “An Approach to Integration Testing
based on Architectural Descriptions”
 Antonia Bertolino, Andrea Polini, “A Framework for
Component Deployment Testing”
 M.J.Harrold, R.Gupta and M.L Soffa, “A methodology
for controlling the size of test suite”
 G. Rothermel , M.J. Harrold, “Analyzing Regression
Test Selection Techniques”
 A.Bertolino,P Inverardi,H.Muccini, “Software
Architecture and integration testing”