Evaluating effectiveness of risk evaluation and risk management

Document Sample
Evaluating effectiveness of risk evaluation and risk management Powered By Docstoc
					                Model for evaluating value at risk for
                        Risk Based Testing

Keshava Moorthy Chandrashekar                    Test Consultant

Abstract: Delivering high quality reliable software on time and within budget is the
main focus of software testing. The quality of the software delivered is decided based on
the test coverage, approach taken to discover the defects and effective risk evaluation.

The major amount of rework done in testing phase is due to poor risk evaluation and risk
management. The roles of development team, testing team and support teams are
crucial in evaluating the risk and the magnitude of the risk in testing. The risks which are
identified while testing is in progress consume 30-40% additional overall project effort
and cost. Though testing is one kind of risk mitigation for the overall project lifecycle,
there are lots of areas in testing where the risks are involved. Important areas can either
be functions or functional groups, or properties such as performance, capacity, security
etc. The result of this risk analysis is a list of functions and properties or combination of
both that need attention.

Risk based testing is getting popular since it is aimed at optimization of time and
resources available for testing without comprising on the quality of testing. However,
most approaches for Risk Based Testing today, suffer from the following drawbacks:
 Lack of precise definition of risk. Risks are often confused with issues
 Subjectivity in assessing the impact of risk, without any real, relevant quantification of
 The dynamic nature of risk, in terms of differing impact and probability of occurrence
   of the risk factor, over different stages of the project lifecycle, is not captured

This paper focuses on the deriving a model for risk evaluation and risk based testing,
based on a simple value at risk model, addressing limitations of the traditional models.


Software testing in many organizations is key and critical to delivering success in the
current business environment. Because of the complexity of software or systems, the
testing done on them is comprehensive but not exhaustive. Most of the test
management fails to handle the risks on time due to lack of knowledge in identifying the
risks and a risk based /modeled approach for testing.

Risk can be broadly defined as the degree of uncertainty about the project meeting its
goals, and is a function of probability of occurrence of the defined undesirable event and
the magnitude of severity. A risk can arise almost anywhere and have a significant
impact on the test schedules and quality. So it is important that testing is done in
operational frame work gives the ability to mitigate the risks early in the test life cycle.
Testing can reduce the probability of occurrence of certain risks by ensuring a wide
coverage of test risk management wide across all possible risks that may occur due to
development, testing approach, tools or test environment. Risk mitigation and
compliance reduces costs, ensures quality, ensures timeliness and protects reputation,
all of which are key factors to remain competitive

The higher and more critical risks are practically visualized in reality either very close to
the integration testing, system testing and performance testing phases, or when these
phases are in progress.

“Forward”[1] risks are those associated with the system operation and which involves the
consequences of the software failure. But there are “Backward risks “[1] which are more
of development life cycle originated risks.

Risk based testing is aimed at the optimization of the time and resources available for
testing without comprising on the quality of work. In standard approaches of testing all
aspects are given equal priority. In the later phases of the software life cycle such as the
System Testing phase there is a need to ensure completion testing in a short span of
time. In such situations standard approaches of testing are not effective as it is difficult to
take decisions about the aspects that have to be tested.

Risk based testing has a major advantage of being the most suitable kind of testing for
the System testing phase. The risk based testing procedure prioritizes risks and hence
the testing effort is focused which leads to completion of testing in a lesser time span.
Risk based testing also indicates the risk distribution across the application which is
helpful in taking decisions about the release probability of the software application.

To carryout this important factor is to understand the cost of each identified risk. The
value at risk model for software testing will provide access to efficiently manage the
testing risks.

Why Risk based testing?

Though risk based testing is contrasting from the traditional approach, it has got
numerous benefits to the project in terms of cumulative improvements in quality. Risk
based testing ensures that the applications are delivered to meet business and time
requirements. This also helps in optimizing the testing resources in a better manner.
Risk based testing allows the software organizations to manage and minimize the risks
by tracking and addressing the risks ahead.

Typical points of failure in new application development and testing effort are:
   1. Meet the timelines
   2. Meet the budget
   3. Application logic
   4. Keep up with business changes
   5. To realize and implement business cases as expected
   6. Measuring the reliability of the application
   7. Value addition to the application due to testing
Risk based testing outcome plays a major role in making a final decision to Go live or
not, objectively. This is because, risk based testing gives the access to understand the
status of the identified risks and depth of testing done. The more testing is done, the less
value it adds. Most organizations find it comfortable to invent workaround solutions
rather than addressing the risk. But these work around solutions in reality cost a lot
compared to addressing the risk itself.

The traditional risk model for testing

The traditional risk model has two elements

1) The probability of a fault being present
2) The impact of a defect /fault in the corresponding function if it occurs during normal

               RE(f) = P(f) * I(f)

Where RE(f) is the risk exposure function of f, P(f) is the probability of a fault occurring
in function f and I(f) is the impact when a fault is executed in a function f in operational

The cost of software failure is very high. More number of defects or problems and
defects can contribute to the software cost exponentially. To address this it might need a
rigorous risk management to comprehensively mitigate and make contingency plans.

“Heuristic reasoning is a reasoning not regarded as final and strict , but as provisional
and Plausible only, whose purpose is to discover the solution for the present problem “
written by George Polya , Greek Philosopher.

Using this approach, one need to identify the situations and related impacts by
repeatedly asking “what could go wrong here?”. The three factors that can be
considered are Vulnerabilities, threats and Victims.

Begin with a set of potential risks and match them to the details of the situation.
Basically the risks can be identified under three categories viz., Quality criteria,
Generic risks and risk catalogs.

Using following steps one can analyze the risks
   1. Select a component or a function you want to analyze
   2. Determine the severity of the concern
   3. Gather more information or inputs on the concern
   4. Visit each risk area on each list and determine its importance in the situation
   5. Record unknowns

In the traditional model, the risk exposure is seldom a quantified value but more a
subjective value like low, medium high, based on the corresponding probabilities and
Value at Risk Model
Value at Risk model is an extension of the traditional model, to of qualify, quantify and
mitigate the risks in software testing. It follows a 4-step process

   1.   Risk identification
   2.   Risk description
   3.   Risk estimation
   4.   Risk mitigation

Risk identification

Risk identification process provides the access to the projects/software developments
exposure to uncertainties that can occur during testing. This needs sound knowledge of
the entire application (software) and the other systems with which the application
interacts/interfaces and the deployment environment. This also requires a sound
knowledge of performance requirements and business user’s perspectives. Test risk
identification can be done systematically and methodically to ensure that the all the risks
are identified and all the activities linked to the risks are defined.

Developing a risk identification checklist :

Generic risks

           Complex: anything disproportionately large, intricate, or convoluted
           New: anything that has no history in the product
           Changed: anything that has been tampered with or “improved”
           Upstream Dependency: anything whose failure will cause cascading failure in
            the rest of the system
           Downstream Dependency: anything that is especially sensitive to failures in
            the rest of the system
           Critical: anything whose failure could cause substantial damage
           Precise: anything that must meet its requirements exactly
           Popular: anything that will be used a lot
           Strategic: anything that has special importance to your business, such as a
            feature that sets you apart from the competition
           Third-party: anything used in the product, but developed outside the project
           Distributed: anything spread out in time or space, yet whose elements must
            work together
           Buggy: anything known to have a lot of problems
           Recent Failure: anything with a recent history of failure

 Qualitative risks
        Capability: Can it perform the required functions?
        Reliability: Will it work well and resist failure in all required situations?
        Usability: How easy is it for a real user to use the product?
        Performance: How speedy and responsive is it?
        Installability: How easily can it be installed onto its target platform?
        Compatibility: How well does it work with external components and
            Supportability: How economical will it be to provide support to users of the
            Testability: How effectively can the product be tested?
            Maintainability: How economical will it be to build, fix, or enhance the
            Portability: How economical will it be to port or reuse the technology
            Localizability: How economical will it be to publish the product in another

Risk description

Risk description is a method of displaying risks in a structured format. A well designed
structure is necessary to ensure comprehensive risk identification, description and
assessment process. By considering the consequence and probability of each of the
risks set out in the table, it should be possible to prioritize the risks that would require
further analysis

#            Item                        Description
1            Risk name
2            Scope                       Description of the event,size,dependencies
3            Nature/ Area of impact      Quality,compliance,technical etc
4            Probability                 Probability of occurrence
5            Severity                    Severity of the risk(critical/High/Low)
6            Risk tolerance              Potential impacted areas and losses(gains)
7            Risk mitigation/control     Primary means of managing the risks,
             mechanism                   identification of protocols and other
                                         requirements/dependencies to control the risk
8            Policy development          Identify the group( testing /development) to
                                         develop the policies, suggestions

Risk Estimation

Identified risks can be quantified in terms of the magnitude of the impact and the
probability of occurrence. In the traditional model, they are typically represented in the
following tabular format:


Rating                Description                                Indicators
High                  Likely / can occur more frequently         Has occurred recently few
Medium                Likely / can occur less frequently         Has occurred some time
                                                                 back once/twice
Low                   Not likely to occur in the near future     Unlikely. No past

Rating              Description
High                Significant impact on delivery/user/stake holder
Medium              Less impact on the delivery/user/stake holder
Low                 Negligible or no impact on the delivery/user/stake

The next step is to calculate the magnitude of the risk by developing the Probability
/Impact matrix.

Probability             Impact                  RE (Risk Exposure)
High                    High                    High
High                    Medium                  High
High                    Low                     Medium
Medium                  High                    High
Medium                  Medium                  Medium
Medium                  Low                     Low
Low                     High                    Medium

This helps to understand and make decisions about the risks. This is called as risk
profiling or risk estimation.

Risk Estimation in the Value at Risk Model

Where Value at Risk model extends the traditional model is in terms of quantifying
high/medium/lows into just a single number – the loss associated to a given probability.
Organizations can further balance the cost of testing from the benefits derived through
statistical analysis of the derived metrics.

Value at risk has three components and they are
    Approximate cost of the risk
    Time over which the risk has to be monitored and measured
    Confidence level

Following factors play important role in identifying value at risk

Complexity of the software: The complexity of the software plays a major role in
detriming overall cost of the risk. Examples include complex control logic, data flow,
dependency on the subsystems etc. This means the one needs to consider different
aspects of complexity and problem areas. The amount and types of testing required
depends on the complexity of the software

Changed /Changing areas:
Frequent changes to the software have a significant impact on testing and overall quality
of the software.

Areas with many defects before (Historical experience) :
Defect fixes may potentially cause some other defects. The modules that had most
defects are likely to have more defects in the future.
Impact of number of people involved:
If the testing involves more people the overhead for communication, retesting and

Impact of time pressures
Time pressure makes people to take shortcuts and workarounds. This also may lead to
overtime work and will result in poor quality job done by the resources

Geographical usage
People working on the same project in different geographical locations potentially have
communication gaps. The cost contributed to this may be considerable

Impact on business
Due to system downtimes , unstable system or more iterations of the software releases ,
the cost to the business may be very high

By considering and factoring the above items along with the other identified risks to
determine the value at risk will make a significant difference in managing the risk.

Sl         Factors                                    Impact Value (in USD)
1          Complexity of the software                 5000
2          Changed /Changing areas                    3000
3          Areas with many defects before             10000
4          Impact of number of people involved        2000

5          Geographical usage                         1000

6          Impact of time pressures                   1000

7          Impact on business                         10000

Total cost of a risk related to testing is = 32000 USD

Obtain Value at Risk table

Sl no    Risk                        Value at Risk     Risk qualification(Qualified
                                                       /not qualified)
1        Unidentified business       32000 USD         Qualified

By calculating the Value at risk for each of the identified risks, the overall cost of the risk
in the testing area will be

CRe = ∑ Value at risk of R1+ Value of risk at R2 + ……..Value of risk at Rn
But in reality this is not true as some part of the Value at risk for each factors may be
embedded in other factor by extrapolating the risk factors.
            defects                            Market
                                   Complexity     pre



Hence we need to normalize the value at risk

Sl        Factors                                       Cost(in USD)
1           Complexity of the software                  1000
2           Changed /Changing areas                     Absorbed in complexity
3           Areas with many defects before              7000
4             Impact of number of people                2000

5             Geographical usage                        Absorbed in impact of
6             Impact of time pressures                  Absorbed in impact of
7             Impact on business                        10000

Now the actual value at risk = 11000 USD

The final (optional) step is to capture the dynamic nature of risk by plotting a VAR
distribution. There are three methods of calculating VAR distribution: the historical
method, the variance-covariance method and the Monte Carlo simulation. The historical
method takes into account past data for a risk factor, re-organizes actual historical
impact, putting them in order from worst to best. It then assumes that history will repeat
itself, from a risk perspective. The Variance-Covariance method assumes that impact is
normally distributed. In other words, it requires that we estimate only two factors - an
expected (or average) return and a standard deviation - which allow us to plot a normal
distribution curve. The Monte Carlo Simulation method involves developing a model for
future impact and running multiple hypothetical trials through the model.
Based on the confidence level, which would be different at different stages of the project,
one can calculate the overall value at risk and answer the key question – “what is the
worst loss that I can expect during a specified time period with a certain confidence

       Ilustrative plot of probabilities and VAR

Risk Mitigation:
    Projection of value of risk may provide the organization the visibility to the near
    Generation of the re-evaluated risk watch list based on the execution results of
      the risk based testing cycle. This watch list will also indicate the regression risks.
    Projection of the risk lowering factors and product release probability.

Case study

One of the world’s largest Insurance companies
Scenario : This organization has four applications which are interfaced and these
applications are being used in different geographies. These applications are financial
transaction critical applications. These applications were in the market for more than 10
years . The organization had recently re-platformed the application in J2EE technology.

Risks Identified in testing area :

   1. Shared UAT environment between all four applications
   2. Unavailability of performance test environment
   3. Technology upgrade was not done in the test environment
   4. Unavailability of strategy for end to end testing
   5. Unavailability of test data
   6. Poor test coverage
 The above factors are important because it will lead to obvious software failure and
rework. The values obtained after normalization given as below
Sl         Factors                              Cost of risk(in USD)
1          Shared UAT environment               2.5 million

2          Unavailability of performance        1.6 million
           test environment

3          Technology upgrade was not           0.5 million
           done in the test environment

4          Unavailability of strategy for end 1 million
           to end testing

5          Unavailability of test data          0.25 million

6          Poor Test coverage                   4 million

This helped the management to come up with a mitigation plan based on the cost

The benefits derived :
   1. Early mitigation of all the risks effectively
   2. Good Time and space for test preparation based on the risk factors
   3. The cost to business ( impact on business) is reduced drastically
   4. No fear of reputation loss in the market due to extensive test coverage


Testing in a situation where management cuts both budget and time is a bad game. We
have to endure and survive this game and turn it into a success. The general
methodology for handling this situation is not to test everything in bits, but to concentrate
on high risk areas and the worst impacted areas. The main objectives should be to
return the product as fast as possible to the developers with a list of critical defects or
deficiencies as possible and to make sure that best testing with good out put is achieved
with in the dead line.
So to do effective test risk management involves
     Anticipate the risks
     Asses the impact and plan to handle
     Calculate the value at risk in the testing area
     Use control /mitigation mechanisms to catch the ocuurance of risks
     Proactively handle the risk whenever it triggers

By doing the points discussed above a quality product can be delivered to any customer
with minimum pain

1. Redmill 2004: Redmill F: Exploring risk based test and its implications .
2. Technical report CMU/SEI-93-TR-6,ESC-TR-93-183, Taxonomy based risk
   identification ,1993
3. Specification based regression testing measurement with Risk analysis by
   yanping chen , October 2002
4. Heuristic risk- based testing by James Bach. Software quality engineering
5. Joachim Karlsson & Kevin Ryan “ A cost –Value approach for prioritizing the
   requirements” IEEE software,Sept 1997
6. James Bach “ Risk based Testing “,STQE 6/1999, www.stquemagazine.com
7. Evaluation of value at risk Models using historical data ,FRBNY economic policy
   review /Apr 1996