Docstoc

White-Box Test Reduction Using Case-Based Maintenance

Document Sample
White-Box Test Reduction Using Case-Based Maintenance Powered By Docstoc
					                                                           (IJCSIS) International Journal of Computer Science and Information Security,
                                                           Vol. 8, No. 6, September 2010




          White-Box Test Reduction Using Case-Based
                         Maintenance

             Siripong Roongruangsuwan                                                         Jirapun Daengdej
        Autonomous System Research Laboratory                                      Autonomous System Research Laboratory
Faculty of Science and Technology, Assumption University                   Faculty of Science and Technology, Assumption University
                    Bangkok, Thailand                                                           Bangkok, Thailand
                    P4919742@au.edu                                                          jirapun@scitech.au.edu


Abstract— Software testing has been proven that it takes around            test case reduction techniques. Also, the study reveals that
50-70% of the costs associated with the large development of               coverage-based approaches are wildly used and researched.
commercial software systems. Many reduction techniques have                Therefore, we concentrate on an approach to reduce test cases
been proposed to reduce costs. Unfortunately, the cost is usually          based on the coverage factor. Many coverage factors have been
over budget and those techniques are failed to reasonably control          proposed over a long period of time. Unfortunately, existing
costs. The primarily outstanding research issues, motivated this           factors and test case reduction methods ignore the complexity
study, are a large number of redundancy test cases and a                   and impact of test cases. Thus, we propose to reduce a number
decrease of ability to detect faults. To resolve these issues, this        of test cases by considering both of test case complexity and
paper proposes new deletion algorithms to minimize a number of
                                                                           impact.
white-box test cases, while maximizing an ability to reveal faults,
by using a concept of case-based maintenance. Our evaluations                  Our study [5] shows that one of effective approaches that
have shown that the proposed techniques can significantly reduce           significantly reduce a number of redundancy test cases is to
a number of unnecessary test cases while preserving the                    apply the concept of artificial intelligent. There are many
capability of fault detection.                                             artificial intelligent concepts, such as neutral network, fuzzy
                                                                           logic, learning algorithms and case-based reasoning (CBR).
    Keywords-component; Test reduction, test case reduction,               CBR is one of the most popular and actively researched areas
deletion method, case based maintenance and test case deletion
                                                                           in the past. The researches [4], [5], [26], [35] show that CBR
                                                                           has identical problems as same as software testing topic, such
                       I.    INTRODUCTION                                  as a huge number of redundancy cases and a decrease of
    Software Testing is an empirical investigation conducted to            system’s ability to resolve problems.
provide stakeholders with information about the quality of the                 Fundamentally, there are four steps in the CBR system,
product or service under test [13], with respect to the context in         which are: retrieve, reuse, revise and retain. These steps can
which it is intended to operate. Software Testing also provides            lead to a serious problem of uncontrollably growing cases in
an objective, independent view of the software to allow the                the system. However, the study shows that there are many
business to appreciate and understand the risks of                         proposed techniques in order to control a number of cases in
implementation of the software. The software testing                       the CBR system, such as add algorithms, deletion algorithms
techniques include the process of executing a program or                   and maintenance approaches. CBR have been investigated by
application with the intent of finding software bugs. It can also          CBR researchers in order to ensure that only small amounts of
be stated as the process of validating and verifying that                  efficient cases are stored in the case base.
software meets the business and technical requirements that
guided its design and development, so that it works as                          The previous work [27] shows that deletion algorithms are
expected. Software Testing can be implemented at any time in               the most popular and effective approaches to maintain a size of
the development process; however, the most test effort is                  the CBR system. There are many researchers have proposed
employed after the requirements have been defined and coding               several deletion algorithms [4], [20], [35], such as random
process has been completed.                                                method, utility approach and footprint algorithm. These
                                                                           algorithms aim to: (a) remove all redundancy or unnecessary
    Many researchers [6], [7], [8], [9], [10], [19], [24], [25],           cases (b) minimize a number of cases and (c) maintain the
[26], [28], [30], [36], [37], [39] have proven that these test case        ability of solving problems. Nevertheless, each technique has
reduction methods can reserve the ability to reveal faults.                strength and weakness. Some methods are suitable for
However, there are many outstanding research issues in this                removing cases. Some methods are perfectly suitable for
area. The motivated research issues are: a large number of test            reducing time. Some may be used for reserving the problem
cases particularly redundancy test cases and a decrease of                 solving capability. Eventually, the previous work [27]
ability to detect faults. The study shows that test case reduction         discovered several effective methods (e.g. confidential case
methods have been researched over a long period of time, such              filtering method, coverage value algorithm and confidential
as test case prioritization, random approach and coverage-based            coverage approach) to remove those cases, minimize size of



                                                                      32                              http://sites.google.com/site/ijcsis/
                                                                                                      ISSN 1947-5500
                                                                    (IJCSIS) International Journal of Computer Science and Information Security,
                                                                    Vol. 8, No. 6, September 2010




CBR and reduce amount of time, while preserving the ability of                    B. Case-Based Maintenance (CBM)
CBR system’s problem solving skill. Therefore, this paper                             Due to the CBR’s life cycle [16], the case base size grows
applies those effective deletion techniques to resolve the                        rapidly. That is caused a serious problem directly, for instance,
problems of software testing.                                                     duplicate data, inconsistency data, incorrect data, and an
    In the light of software testing, the proposed techniques                     expense of searching for an appropriate case in a large case
focus on how to maintain the test case while maintaining the                      base size. CBR can be classified as one of the Artificial
capability of fault detection. It is appear that test cases in this               Intelligence algorithms. CBR solves new problem by retrieving
paper are treated as cases in the CBR system. Also, there is an                   the similar case from the existing case base and then adapts the
assumption that a given set of test cases are generated by a                      existing case according to the target problem. Over the time,
path-oriented test case generation technique. The path-oriented                   CBR is growing. When the uncontrollable case-based growth is
technique is widely used for a white-box testing that derives                     occurred, the performance of CBR is decreasing. Therefore, the
test cases from available source code.                                            maintenance process is required in order to preserve or improve
                                                                                  the performance of the system. The process of maintaining
    Section 2 discusses an overview of test case reduction                        CBR is called CBM. David C. Wilson [8] presented the overall
techniques and approach to maintain CBR. Section 3 provides                       concepts of CBR and case based maintenance. This paper
a definition of terminologies used in this paper. Section 4 lists                 focused on the case based maintenance (CBM) approach in
the outstanding research issues motivated this study. Section 5                   term of the framework. In other words, this paper described the
proposes deletion algorithms using the concept of CBR.                            type of data collection and how the case based maintenance
Section 6 describes an evaluation method and discusses an                         works. There were so many policies for CBM, for example,
evaluation result. The last section represents all source                         addition, deletion, and retain.
references used in this paper.
                                                                                  “CBM was defined as the process of refining a CBR system’s
                      II.   LITERATURE REVIEW                                     case-base to improve the system’s performance. It implements
                                                                                  policies for revising the organization or contents
    This section describes an overview of test case reduction                     (representation, domain content, accounting information, or
techniques and the concept of case based maintenance. The                         implementation) of the case-base in order to facilitate future
following describes those two areas in details.                                   reasoning for a particular set of performance objectives.”
                                                                                      These studies [2], [3], [4], [16], [17], [27], [35] reveal that
A. Test Case Reduction Techniques                                                 several deletion algorithms have been proposed. For example, a
    This section discusses and organizes test case reduction (or                  random approach (RD), utility deletion algorithm (UD),
TCR) techniques researched in 1995-2006. This study shows                         footprint deletion algorithm (FD), footprint utility deletion
that there are many researchers who proposed a method to                          algorithm (FUD) and iterative case filtering algorithm (ICF).
reduce unnecessary test cases (also known as redundancy test
                                                                                      RD is the simplest approach, which removes the case
cases), like Offutt [2], Rothermel [8], McMaster [25] and
                                                                                  randomly. UD deletes the case that has minimum utility value.
Sampth [31]. These techniques aim to remove and minimize a
                                                                                  Footprint algorithm uses the competence model and removes
size of test cases while maintaining the ability to detect faults.
                                                                                  the auxiliary case from the system. FUD is a hybrid approach
The literature review [6], [7], [8], [9], [10], [11], [19], [24],
                                                                                  between Utility algorithm and Footprint algorithm, and is
[25], [36], [37], [39] shows that there are two types of
                                                                                  concerned with the competence model and the utility value.
reduction techniques, which are: (a) pre-process and (b) post-
                                                                                  Finally, ICF focuses on the case, which the reachability set is
process. First, the pre-process is a process that immediately
                                                                                  greater than the coverage set [16], [27].
reduces a size of test cases after generating. Typically, it is
occurred before regression testing phase. Second, the post-
process is a process that maintains and removes unnecessary                                                III.     DEFINITION
test cases, after running the first regression testing activities.                   This section describes a definition of CBR terminologies
Although these techniques can reduce the size of test cases, but                  used in the software testing area.
the ability to reveal faults seems slightly to be dropped.
However, Jefferson Offutt [5] and Rothermel [6], [7], [8], [9],
[10], [19], [20], [21], [32] has proven that these test case                            TABLE I.       DEFINITIONS OF CBR FOR SOFTWARE TESTING
reduction techniques have many benefits, particularly during                          Element               CBR                     Software Testing
the regression testing phase, and most of reduction techniques
can maintain an acceptable rate of fault detection. The                             Coverage       Coverage Set is the           Coverage set means a
advantages of these techniques are: (a) to spend less time in                       set            set of target problems,       set    of    stages    /
executing test cases, particularly during the regression testing                                   which it can be used          elements, which they
phase (b) to significantly reduce time and cost of manually                                        to solve successfully         can be used to test
comparing test results and (c) to effectively manage the test                                      [4].                          successfully        and
data associated with test cases. This study proposes a new “2C”
                                                                                                                                 reveal faults.
classification of test case reduction techniques, classified based
on their characteristics, as follows: (a) coverage-based                            Reachability   Reachability Set is the       Reachability set means
techniques and (b) concept analysis-based techniques.                               set            set of case bases that        a set of test cases that
                                                                                                   can be used to solve          can be used to reveal
    Autonomous System Research Laboratory, Faculty of Science and
Technology, Assumption University.

                                                                             33                                   http://sites.google.com/site/ijcsis/
                                                                                                                  ISSN 1947-5500
                                                              (IJCSIS) International Journal of Computer Science and Information Security,
                                                              Vol. 8, No. 6, September 2010




                  the target problem [4].   faults.                              From the above figure, there are two test cases, with the
  Competence      Competence is the         Competence is         the
                                                                             duplicated sequence and expected result, designed to test a
                                                                             “withdraw” feature in ATM machine. The sequence of the first
  set             range of the target       range of ability       to        test case is: (a) insert ATM card (b) insert PIN (c) select
                  problem that can be       reveal faults that   can         “withdraw” (d) select account type (e) identify an amount (f)
                  solved     successfully   be used to           test        click “OK” button (g) receive money and (h) receive card. The
                  [4].                      successfully.                    sequence of the second test case is similar to the first one.
  Auxiliary       Auxiliary Case is a       Auxiliary case is a test
                                                                             However, the additional step in the second case is to inquiry a
                                                                             balance amount before withdrawing the money. Therefore, it is
  set             case that does not        case that does not have          appear that the first test case is a part of the second test case.
                  have a direct effect on   a direct effect on the           We call the first test case as a redundancy test case.
                  the competence of a       competence        of   a
                                                                                 The study shows that there are many proposed methods to
                  system when it is         system when it is                                                                       s
                                                                             delete those duplicated test cases such as McMaster' work [24]
                  deleted [4].              removed.                                    s                          s
                                                                             [25], Jeff' method [13] and Khan' approach [23]. Also, the
  Pivot set       Pivotal Case is the       Pivotal case is a test           study shows that one of the most interesting research issues is
                  case that does have a     case that does have a            to minimize those duplicated tests and reduce cost of executing
                                                                             tests. Although there are many proposed methods to resolve
                  direct effect on the      direct effect on the
                                                                             that issue, that issue is still remaining. Thus, it is a challenge
                  competence      of a      ability to reveal faults         for researchers to continuously improve the ability to remove
                  system if it is deleted   if it is deleted.                duplicated tests.
                  [1], [29].
                                                                                 Last, test cases are designed to reveal faults during software
                    IV.    RESEARCH PROBLEM                                  testing phase. The empirical studies [8], [10], [19], [20], [21],
                                                                             [30], [32], [39] describe that reducing test cases may impact to
    This section discusses the details of research issues                    the ability of detect faults. Many reduction methods decrease a
motivated this study. The literature review reveals that [13],               capability of testing and reveal those faults. Therefore, one of
[23], [24], [25], [31], [38] those research issues are: (a) a large          outstanding research challenges for researchers is to remove
number of redundancy test cases and (b) a decrease of the                    tests while preserving the ability to defect faults.
ability to reveal faults. These research issues can be elaborated
in details as follows:
                                                                                                 V.    PROPOSED METHOD
    First, the literature review shows that redundancy test cases                For evolving software, test cases are growing dramatically.
are test cases tested by multiple test cases. Many test cases that           The more test cases software test engineers have, the more time
are designed to test the same things (e.g. same functions, same              and cost software test engineers consume. The literature review
line of code or same requirements) are duplicated. Those                     shows that regression testing activities consume a significant
duplicated tests are typically occurred during testing activities,           amount of time and cost. Although, a comprehensive set of
particularly during regression testing activities [13], [23], [24],          regression selection techniques [8], [9], [10], [19] has been
[25], [31], [38]. Those duplicated tests can be eventually                   proposed to minimize time and cost, there is an available room
removed in order to minimize time and cost to execute tests.                 to minimize size of tests and clean up all unnecessary test
   The following shows an example of redundancy test cases.                  cases. Thus, removing all redundancy test cases is desirable.
                                                                                 There are many approaches to reduce redundancy test cases
                                                                             and applying an artificial intelligent concept in the test case
                                                                             reduction process is an innovated approach. The literature
                                                                             review [5], [27] shows that there are many areas of artificial
                                                                             intelligent concept, such as artificial neutral network, fuzzy
                                                                             logic, learning algorithms and CBR concept. Also, it reveals
                                                                             that CBR has a same research issue as software testing has. The
                                                                             issue is that cases in the CBR system will be consistency
                                                                             growing bigger and larger all the time. There are four steps in
                                                                             CBR that can uncontrollably grow a size of the system:
                                                                             retrieve, reuse, revise and retain. Therefore, many CBR papers
                                                                             aim to reduce all redundancy cases, known as “deletion
                                                                             algorithms”. The smaller size of CBR system is better and
                                                                             desirable. Due to the fact that CBR has the same problem as
                                                                             software testing and this paper focuses on reduction methods,
                                                                             therefore, this paper proposes to apply CBR deletion
                                                                             algorithms to the test case reduction techniques.
              Figure 1. An Example of Control Flow Graph
                                                                               This paper introduces three reduction methods that apply
                                                                             CBR deletion algorithms: TCCF, TCIF and PCF methods.




                                                                        34                               http://sites.google.com/site/ijcsis/
                                                                                                         ISSN 1947-5500
                                                           (IJCSIS) International Journal of Computer Science and Information Security,
                                                           Vol. 8, No. 6, September 2010




Those techniques aim to reduce a number of test cases                       TC5 = {s1, s3, s5}
generated by path-oriented test case generation technique. This
technique is used for white-box testing only. However, the                  TC6 = {s1, s4, s3}
generation methods are out of the scope of this paper.                      TC7 = {s1, s2, s3, s5}
    The limitation of the proposed deletion algorithms are: (a)             TC8 = {s1, s4, s3, s5}
those methods are perfectly suitable for only white-box testing
techniques and (b) path coverage may not be applicable for a                TC9 = {s2, s3}
large system that contains over million lines of code.
                                                                            TC10 = {s2, s3, s5}
                                                                            TC11 = {s3, s5}
A. Example of Test Cases
                                                                            TC12 = {s4, s3}
    Given a set of test cases generated, this study discusses the
use of a number of case maintenance techniques, which have                  TC13 = {s4, s3, s5}
been investigated by CBR researchers in ensuring that only
                                                                                From the figure 2, we assume the following: (a) each state
small amount of cases are stored in the case base, thereby
                                                                            represents a block of source code (b) each state can reveal only
reducing number of test cases should be used in software
                                                                            1 fault; the total ability to reveal faults is 5 and (c) every single
testing. Similar to what happen to software testing, a number of
                                                                            transaction in the control flow graph must be tested.
CBR researchers have focused on finding approaches
especially for reducing cases in the CBR systems’ storages.                     The following describes the proposed methods that apply
                                                                            the concept of CBR in details:
    This paper proposes to use the path coverage criteria in
order to reduce redundancy test cases. This is because path
coverage has a huge benefit of required very thorough testing               B. Path Coverage for Filtering (PCF)
activities. The following describes in details of the above path
coverage using in the software testing field. Let S = {s1, s2, s3,              Code coverage analysis is a structural testing technique
s4, s5} to be a set of stage in the control flow graph. The                 (also known as white box testing). Structural testing compares
control flow graph can be derived from the source-code or                   test program behaviour against the apparent intention of the
program. It is a white-box testing. Thus, each state represents a           source code. This contrasts with functional testing (also
block of code. The techniques that aim to generate and derive               referred to black-box testing), which compares test program
test cases from the control flow graph are well-known as path-              behaviour against a requirements specification. Structural
oriented test case generation techniques. These techniques are              testing examines how the program works, taking into account
widely used to generate test cases. There are many research                 possible pitfalls in the structure and logic. Functional testing
papers on this area. However, the test case generation                      examines what the program accomplishes, without regard to
techniques are out of scope in this paper.                                  how it works internally. Structural testing is also called path
                                                                            testing since you choose test cases that cause paths to be taken
                                                                            through the structure of the program. The advantage of path
                                                                            cover is that it takes responsible for all statements as well as
                                                                            branches across a method. It requires very thorough testing.
                                                                            This is an effective substitute of other coverage criteria. The
                                                                            path coverage is used as coverage value in this technique. The
                                                                            Coverage value is combined into the addition policy for adding
                                                                            significant case [12]. Within the adding algorithm along with
                                                                            the coverage weight value stated in the review, the concept of
                                                                            deletion algorithm and the coverage have been proposed. The
          Figure 2. An Example of Control Flow Graph
                                                                            coverage value can specify how many nodes that the test case
    From the above figure, this paper assumes that each state               can cover. In other words, the coverage value is an indicator to
can reveal a fault. Thus, an ability to reveal faults of five states        measure that each test case covers nodes. It means that the
is equal to 5. Also, it is assumed that every single transaction            higher coverage value is, the more nodes can be contained and
must be tested. This example is used in the rest of paper.                  covered in the test case.
    Let TCn = {s1, s2, …,sn} where TC is a test case and sn is                 Let Cov(n) = value where Cov is a coverage value, value is
a stage or node in the path-oriented graph that is used to be               a number of test cases in each coverage group and n is a
tested. From the above figure, a set of test cases can be derived           coverage relationship.
as follows:
                                                                                The procedure of this method can be elaborated briefly as
TC1 = {s1, s2}                                                              the following steps.
TC2 = {s1, s3}                                                                 The first step is to determine a coverage set. From figure 2,
                                                                            each coverage set can be identified as follows:
TC3 = {s1, s4}
                                                                                Coverage (1) = {TC1}
TC4 = {s1, s2, s3}




                                                                       35                                http://sites.google.com/site/ijcsis/
                                                                                                         ISSN 1947-5500
                                                        (IJCSIS) International Journal of Computer Science and Information Security,
                                                        Vol. 8, No. 6, September 2010



   Coverage (2) = {TC2}                                                 C. Test Case Complexity for Filtering (TCCF)
   Coverage (3) = {TC3}                                                     A complexity of test case is the significant criteria in this
   Coverage (4) = {TC1, TC4, TC9}                                       proposed method [1], [16]. In this paper, the complexity of test
                                                                        case measures a number of states included in each test case. We
   Coverage (5) = {TC2, TC5, TC11}
                                                                        define the test case complexity as follows:
   Coverage (6) = {TC3, TC6, TC12}
                                                                           Definition 1: Let Cplx(TC) = {High, Medium, Low} where
   Coverage (7) = {TC1, TC4, TC7, TC9, TC10, TC11}                      Cplx is a complexity of test case, TC is a test case and the
   Coverage (8) = {TC3, TC6, TC8, TC11, TC12, TC13}                     complexity value can be measured as:
   Coverage (9) = {TC9}                                                          •   High when a number of states are greater than an
                                                                                     average number of states in the test suite.
   Coverage (10) = {TC9, TC10, TC11}
   Coverage (11) = {TC11}
                                                                                 •   Medium when a number of states are equal to an
                                                                                     average number of states in test suites.
   Coverage (12) = {TC12}
                                                                                 •   Low when a number of states are less than an
   Coverage (13) = {TC11, TC12, TC13}                                                average number of states in the test suites.
    The second step is also to determine a reachability set. The            The procedures of this method can be described briefly in
reachability set can be figured out from the above coverage set,        the following steps. The first two steps are to identify coverage
based on the given definition in this paper. Therefore, the             and reachability set.
reachability set can be identified as follows:
                                                                            Next, the step is to define an auxiliary set. Test cases that
   Reachability (TC1) = {1, 4, 7}                                       can be included in the auxiliary set have a greater number of
   Reachability (TC2) = {2, 5}                                          members in the reachability set than a number of members in
                                                                        the coverage set. From figure 2, therefore, the auxiliary set can
   Reachability (TC3) = {3, 6, 8}
                                                                        be identified as follows:
   Reachability (TC4) = {4, 7}
                                                                           Auxiliary set = {TC1, TC2, TC3, TC9, TC11, TC12}
   Reachability (TC5) = {5}
                                                                            Afterward, the method computes a complexity value for all
   Reachability (TC6) = {6, 8}                                          test cases in the above auxiliary set. From figure 2 and test
   Reachability (TC7) = {7}                                             suites that contain 13 test cases, the average number of states is
                                                                        (2+2+2+3+3+3+4+4+2+3+2+2+3)/13, which is equal to 3.
   Reachability (TC8) = {8}                                             Based on the average number of states, the complexity value
   Reachability (TC9) = {4, 7, 9, 10}                                   for each test case can be computed as follows:
   Reachability (TC10) = {7, 10}                                           Cplx(TC1) = Low, Cplx(TC2) = Low, Cplx(TC3) = Low,
                                                                        Cplx(TC4) = Medium, Cplx(TC5) = Medium, Cplx(TC6) =
   Reachability (TC11) = {5, 7, 8, 10, 11, 13}
                                                                        Medium, Cplx(TC9) = Low, Cplx(TC10) = Medium, Cplx(TC11)
   Reachability (TC12) = {6, 8, 12, 13}                                 = Low, Cplx(TC12) = Low and Cplx(TC13) = Medium
   Reachability (TC13) = {8, 13}                                           Finally, the last step removes test cases with minimum of
    The next step is to calculate a coverage value. This paper          complexity value from the auxiliary set. Thus, TC1, TC2, TC3,
proposes to calculate a coverage value based on a number of             TC9, TC11 and TC12 are removed.
test cases in each coverage group. Therefore, the coverage
value can be computed as follows:
                                                                        D. Test Case Impact for Filtering (TCIF)
Cov(1) = 1, Cov(2) = 1, Cov(3) = 1, Cov(4) = 3, Cov(5) = 3,                 The study [21] shows that software is error-ridden in part
Cov(6) = 4, Cov(7) = 6, Cov(8) = 6, Cov(9) = 1, Cov(10) = 3,            because of its growing complexity. Software is growing more
         Cov(11) = 1, Cov(12) = 1 and Cov(13) = 3                       complex every day. The size of software products is no longer
    Afterward, the step is to determine potential removable test        measured in thousands of lines of code, but it measures in
cases. These test cases can be identified when their number of          millions. Software developers already spend approximately 80
members in the reachability set is greater than a number of             percent of development costs [18] on identifying and correcting
members in the coverage set. Therefore, the potential removal           defects, and yet few products of any type other than software
test cases are: TC1, TC2, TC3, TC9, TC11 and TC12.                      are shipped with such high levels of errors. Other factors
                                                                        contributing to quality problems include marketing strategies,
   The last step is to removes all test cases with minimum              limited liability by software vendors, and decreasing returns on
coverage value, in the potential removable test cases.                  testing and debugging, according to the study. At the core of
Unfortunately, TC1, TC2, TC3, TC9, TC11 and TC12 are removed            these issues is difficulty in defining and measuring software
due to that they have the minimum coverage value.                       quality. Due to the fact that defining and measuring a quality of
                                                                        software is important and difficult, the impact of inadequate




                                                                   36                               http://sites.google.com/site/ijcsis/
                                                                                                    ISSN 1947-5500
                                                          (IJCSIS) International Journal of Computer Science and Information Security,
                                                          Vol. 8, No. 6, September 2010




testing must not be ignorance. The impact of inadequate testing           evaluates a result by generating a comparison graph in order to
could be lead to the problem of poor quality, expensive costs             determine the most recommended reduction approach.
and huge time-to-market. In conclusion, software testing                      The following table lists the description of each test data
engineers require identifying the impact of each test case in             that need to be generated randomly.
order to acknowledge and understand clearly the impact of
ignoring some test cases.                                                                   TABLE II.    A FORM OF TEST CASES
    In this paper, an impact value is an impact of test cases in               Attribute                 Description                   Data
term of the ability to detect faults if those test cases are                                                                           Type
removed and not be tested. We define the test case impact as                Test Id              A unique index to reference          Numeric
follows:                                                                                            test data. The value is a
    Definition 2: Let Imp(TC) = {High, Medium, Low} where                                         sequence number, starting
Imp is an impact if a test case is removed, TC is a test case and                                              at 1.
the impact value can be measured as:                                                               A Set of Input Data
                                                                            Full Name              A first and last name who           String
         •   High when the test case has revealed at least one
                                                                                                    own the mobile phone.
             fault for many times.
                                                                            Name                 A mobile brand name. The              String
         •   Medium when the test case has revealed faults for                                   value is a range of iPhone,
             only one time.                                                                      BlackBerry, Nokia, LG,
         •   Low when the test case has never revealed faults.                                   Sony Ericsson and
                                                                                                 Samsung.
    The procedure of this method is similar to the previous                 Model                A mobile model.                       String
method. The only different is that this method aims to use an               Price                A price of mobile. The unit          Numeric
impact value instead of complexity value. Therefore, the fire                                    of price is baht.
three steps are to: identify coverage set, define reachability set          Weight               A weight of mobile. The              Numeric
and determine an auxiliary set. Afterward, the next step is to
                                                                                                 unit of weight is gram (g).
compute and assign an impact value. The method computes the
impact value for all test cases in the above auxiliary set. From            Height               A height of mobile. The              Numeric
figure 2, the impact value for each test case can be computed as                                 unit of height is centimeter
follows:                                                                                         (cm).
                                                                            Graphics               A graphics mode option.            Boolean
   Imp(TC1) = Low, Imp(TC2) = High, Imp(TC3) = Medium,                                              The value can be T or F
Imp(TC4) = Low, Imp(TC5) = High, Imp(TC6) = Medium,                         WAP                  A WAP mode. The value is             Boolean
Imp(TC9) = Low, Imp(TC10) = Low, Imp(TC11) = Low,                                                             T or F
Imp(TC12) = Low and Imp(TC13) = Low
                                                                            Color                 A mobile color. The color            String
   Finally, the last step removes test cases with minimum of                                      can be Black, Gold, Silver,
impact value from the auxiliary set. Thus, TC1, TC4, TC7, TC9,                                          Blue, and White
TC10, TC11, TC12 and TC13 are removed.                                      Game                 A game mode. The value is            Boolean
                                                                                                              T or F
                    VI.      EVALUATION                                     Warranty                A mobile warranty. The            Numeric
    This section describes an experiments design,                                                 unit of warranty is month.
measurement metrics and results.
    This paragraph designs an experiment used to evaluate and                The following table describes an approach to generate
determine the best reduction methods. This paper proposes the             random data using the above attributes respectively.
following three steps. First, the experiment proposes to
randomly generate 2,000 test data used in the                                  TABLE III.      APPROACH TO GENERATE RANDOM TEST CASES
telecommunication industry. In this experiment, the test data is             Attribute                        Approach
represented as test case. Second, the experiment executes
                                                                              Test Id            Generate randomly from the following
reduction methods with the generated test cases and compares
                                                                                                  combination: t + Sequence Number.
among the following reduction methods: RD, UD, FD, FUD,
ICF and three proposed methods (e.g. TCCF, TCIF and PCF).
                                                                                                     For example, t1, t2, t3, …, tn.
This step randomly simulates defects for each test case in
                                                                          Name                    Random from the following values:
order to determine an ability to reveal faults. Third, the
                                                                                                 iPhone, BlackBerry, Nokia, LG, Sony
experiment aims to run the above methods for 10 times in
                                                                                                            and Samsung.
order to calculate the average value for each metric. The
                                                                          Model                   Random from the following values:
metrics used in this experiment are described in details in next
                                                                                                iPhone – iPhone 2G and iPhone 3G.
section. Afterward, the experiment compares the values and
                                                                                                BlackBerry – BlackBerry Bold 9000,
                                                                                                    BlackBerry Bold 9700, BlackBerry




                                                                     37                                 http://sites.google.com/site/ijcsis/
                                                                                                        ISSN 1947-5500
                                                         (IJCSIS) International Journal of Computer Science and Information Security,
                                                         Vol. 8, No. 6, September 2010




                        Curve 8300, BlackBerry Curve 8520,               methods are: RD, UD, FD, FUD, ICF, TCCF, TCIF and PCF.
                        BlackBerry Curve 8900, BlackBerry                Additionally, this section shows a graph format. There are two
                        Pearl, BlackBerry Pearl Flip 8200 and            dimensions in the following graph: (a) horizontal and (b)
                        BlackBerry Storm.                                vertical axis. The horizontal represents three measurements
                   Nokia – Nokia N97 Mini, Nokia E90, Nokia              whereas the vertical axis represents the percentage value.
                       E72, Nokia 8800, Nokia N86 and Nokia
                       E66.
                   LG – Lotus Elite LX610, Accolade VX5600,
                       Chocolate Touch VX8575, Arena
                       GT950 and eXpo GW820.
                   Sony Ericsson – Xperia X10, Vivaz Pro,
                       Xperia X10 Mini Pro, Elm and Aspen.
                   Samsung – Samsung Gravity 2, Behold II,
                       Comeback, SGH-T139, SGH-T659 and
                       SGH-T239.
Price              Random from the following values:
                   High: 30,000 baht – 50,000 baht
                   Medium: 10,000 baht – 29,999 baht
                                                                                   Figure 2. A Graph Comparison of Deletion Methods
                   Low: 3,000 baht – 9,999 baht
Weight             Random from the following values: 4-15                    The above graph presents that both of FD and PCF
Height             Random from the following values: 5-15                minimize a number of test cases by far better than other
Graphics           Random from the following values:                     reductions methods, approximately over 15%. Meanwhile,
                   True or False                                         both of them are the worst methods for preserving an ability to
                                                                         reveal faults. FUD, TCCF and TCIF are best top three
WAP                Random from the following values:
                   True or False                                         methods to reserve a capability to detect faults. They are
                                                                         greater than other methods over 22%. Unfortunately, they are
Color              Random from the following values:
                                                                         also the worst three methods that require a lot of time during a
                   Black, Gold, Silver, Blue and White
                                                                         reduction process. In the mean time, both of RD and PCF take
Game               Random from the following values:
                                                                         the least total reduction time among other methods.
                   True or False
                                                                             From the above figure, this study determines and ranks the
Warranty           Random from the following values:
                                                                         above comparative methods into five ranking: 5-Excellent, 4-
                   Long: 12 months – 18 months
                                                                         Very good, 3-Good, 2-Normal and 1-Poor. This study uses a
                   Medium: 6 months – 11 months                          maximum and minimum value to find an interval value for
                   Short: 1 month – 5 months                             ranking those methods.
                                                                             For a number of test cases, the maximum and minimum
    The paragraph lists the measurement metrics used in the              percentage is 56% and 13%. The different between maximum
experiment. The first measurement is a number of test cases.             and minimum value is 43%. An interval value is equal to a
The large number of test cases consumes time, effort and cost            result of dividing the different values by 5. As a result, the
more than the smaller size of test cases. Many reduction or              interval value is 8.6. Thus, it can be determined as follows: 5-
minimization approaches [6], [7], [8], [9], [10], [11], [19],            Excellent (since 13% to 21.6%), 4-Very good (between 21.6%
[24], [25], [36], [37], [39] have been proposed to minimize              and 30.2%), 3-Good (between 30.2% and 38.8%), 2-Normal
size of test cases. This has proven that size is one of important        (between 38.8% and 47.4%) and 1-Poor (from 47.4% to 56%).
metrics in software testing area. The second is an ability to                For an ability to reveal faults, the maximum and minimum
reveal faults. It aims to measure the percentage of faults               percentage is 94% and 37%. The different value is 57%. The
detection. One of the goals of test case with a set of data is to        interval value is 11.4. Therefore, it can be determined as
find defects. Thus, this metric is important criteria to measure         follows: 5-Excellent (since 82.6% to 94%), 4-Very good
and determine which reduction methods can preserve the high              (between 71.2% and 82.6%), 3-Good (between 59.8% and
ability to reveal faults. The last measurement is a total of             71.2%), 2-Normal (between 48.4% and 59.8%) and 1-Poor
reduction time: It is the total number of times running the              (from 37% to 48.4%).
reduction methods in the experiment. This metric is related to               For a total reduction time, the maximum and minimum
time used during execution time and maintenance time of test             percentage is 64% and 21%. The different between maximum
case reduction methods. Therefore, less time is desirable.               and minimum value is 43%. An interval value is equal to a
    This paragraph discusses an evaluation result of the above           result of dividing the different values by 5. As a result, the
experiment. This section presents the reduction methods                  interval value is 8.6. Thus, it can be determined as follows: 5-
results in term of: (a) a number of test cases (b) ability to            Excellent (since 21% to 29.6%), 4-Very good (between 29.6%
reveal faults and (c) total reduction time. The comparative




                                                                    38                                http://sites.google.com/site/ijcsis/
                                                                                                      ISSN 1947-5500
                                                         (IJCSIS) International Journal of Computer Science and Information Security,
                                                         Vol. 8, No. 6, September 2010




and 38.2%), 3-Good (between 38.2% and 46.8%), 2-Normal                                        VII.     CONCLUSION
(between 46.8% and 55.4%) and 1-Poor (from 55.4% to 64%).                   This paper reveals that there are many research challenges
    Therefore, the experiment result of those eight comparative         and gaps in the test case reduction area. Those challenges and
methods can be shown below:                                             gaps can give the research direction in this field. However, the
                                                                        research issues that motivated this study are: a large number of
   TABLE IV.      A COMPARISON OF TEST CASE REDUCTION METHODS
                                                                        test cases and a decrease of ability to reveal faults. This paper
         Algorithm              A            Abili       Total          combines an approach to maintain CBR and test case
                          Number of        ty to     Reduction          reduction in order to minimize a number of redundancy tests
                          Test Cases /    Reveal       Time
                                                                        while maintaining an ability to detect faults.
                             Size         Faults
                                                                            In this paper, we proposed deletion algorithms to reduce a
      Random Deletion            2            3           5             number of test cases that generated by widely-used white-box
         (RD)                                                           testing techniques. As part of our research, we conducted an
       Utility Deletion          1            3           4             experiment with 2,000 test cases used in the
          (UD)                                                          telecommunication industry in Thailand. Consequently, our
          Footprint              5            1           3
                                                                        evaluation result reveal that the proposed method is one of the
      Deletion (FD)                                                     most recommendation techniques to maintain ability to reveal
                                                                        faults and minimize a number of redundancy test cases.
      Footprint Utility          2            5           1             However, the limitation of the proposed techniques is that path
     Deletion (FUD)
                                                                        coverage may be not an effective coverage factor for a huge
        Iterative Case           4            3           3             system that contains million lines of code. This is because it
   Base Filtering (ICF)                                                 requires an exhaustive time and cost of identify coverage from
          Test Case              1            5           1             a huge amount of codes.
     Complexity for                                                         In future research, we plan to develop deletion algorithms
    Filtering (TCCF)                                                    with other coverage factors, such as function coverage and
       Test Case Impact          2            5           2             block-statement coverage. Also, we aim to implement deletion
   for Filtering (TCIF)                                                 algorithms for multiple test suites. Finally, the evaluation
                                                                        experiment should be conducted for a large commercial
       Path Coverage             5            1           5             system
   for Filtering (PCF)
                                                                                                     REFERENCES
    The above result suggests that FD and PCF is perfectly              [1]   Barry Smyth and Keane. "Remembering To Forget: A
suitable for a scenario that does not directly concern about an               Competence Preserving Deletion Policy for Case-Based
                                                                              Reasoning Systems." Proceedings of the 14th International Joint
ability to reveal faults and total reduction time. Both of FD                 Conference on Artificial Intelligence. Montréal, Québec,
and PCF are two of the most excellent methods to minimize a                   Canada: Morgan-Kaufman Inc., 1995. 377-382.
number of test cases. Meanwhile, FUD, TCCF and TCIF are                 [2]   Beizer, Boris. Software Testing Techniques. New York, USA:
the most recommended methods to delete tests while                            Van Nostrand Reinhold Inc., 1990.
preserving the ability to detect faults. In addition, both of RD        [3]   BO Qu, Changhai Nie, Baowen Xu and Xiaofang Zhang. "Test
                                                                              Case Prioritization for Black Box Testing." Proceeding with 31st
and PCF are excellent in case that total reduction time is                    Annual International Computer Software and Applications
matter.                                                                       Conference (COMPSAC 2007). Beijing, China, 2007. 465-474.
    The above graph presents that both of FD and PCF                    [4]   Boehm, B.W. "A spiral model of software development and
minimize a number of test cases by far better than other                      enhancement." IEEE Software Engineering (IEEE Computer
reductions methods, approximately over 15%. Meanwhile,                        Society), 1988: 61-72.
both of them are the worst methods for preserving an ability to         [5]   Daengdej, Jirapun. Adaptable Case Base Reasoning Techniques
                                                                              for Dealing with Highly Noise Cases. PhD Thesis, The
reveal faults. FUD, TCCF and TCIF are best top three                          University of New England, Australia: The University of New
methods to reserve a capability to detect faults. They are                    England, 1998.
greater than other methods over 22%. Unfortunately, they are            [6]   Gregg Rothermel and Mary Jean Harrold. "A Safe, Efficient
also the worst three methods that require a lot of time during a              Regression Test Selection Technique." ACM Transactions on
                                                                              Softw. Eng. And Methodology, 1997: 173-210.
reduction process. In the mean time, both of RD and PCF take
                                                                        [7]   Gregg Rothermel and Mary Jean Harrold. "Analyzing
the least total reduction time among other methods.                           Regression Test Selection Techniques." IEEE Transactions on
    The evaluation result suggests that FD and PCF is                         Software Engineering, 1996: 529-511.
perfectly suitable for a scenario that does not directly concern        [8]   Gregg Rothermel, Mary Jean Harrold, Jeffery Ostrin and
about an ability to reveal faults and total reduction time. Both              Christie Hong. "An Empirical Study of the Effects of
of FD and PCF are two of the most excellent methods to                        Minimization on the Fault Detection Capabilities of Test
                                                                              Suites." Proceedings of IEEE International Test Conference on
minimize a number of test cases. Meanwhile, FUD, TCCF and                                                     98).
                                                                              Software Maintenance (ITCSM' Bethesda, Maryland, USA:
TCIF are the most recommended methods to delete tests while                   IEEE Computer Society, 1998. 33-43.
preserving the ability to detect faults. In addition, both of RD        [9]   Gregg Rothermel, Mary Jean Harrold, Jeffery von Ronne and
and PCF are excellent in case that total time is matter.                      Christie Hong. "Empirical Studies of Test-Suite Reduction."




                                                                   39                                 http://sites.google.com/site/ijcsis/
                                                                                                      ISSN 1947-5500
                                                                (IJCSIS) International Journal of Computer Science and Information Security,
                                                                Vol. 8, No. 6, September 2010



       Journal of Software Testing, Verification, and Reliability 12, no.        [26] —. "Fault Detection Probability Analysis for Coverage-Based
       4 (December 2002): 219-249.                                                      Test Suite Reduction." Proceedings of IEEE International
[10]   Gregg Rothermel, Roland H. Untch, Chengyun Chu and Mary                          Conference on Software Maintenance (ICSM 2007). Paris,
       Jean Harrold. "Prioritizing Test Cases For Regression Testing."                  France, 2007. 335-344.
       IEEE Transactions on Software Engineering, 2001: 929-948.                 [27]   Siripong Roongruangsuwan and Jirapun Daengdej. Techniques
[11]   Jefferson Offutt, Jie Pan and Jeffery M. Voas. "Procedures for                   for improving case-based maintenance. MS Thesis, Faculty of
       Reducing the Size of Coverage-based Test Sets." Proceedings of                   Science and Technology, Assumption University, Bangkok,
       the Twelfth International Conference on Testing Computer                         Thailand: Assumption University, 2003.
       Software. Washington D.C, USA, 1995. 111-123.                             [28]   Siripong Roongruangsuwan and Jirapun Daengdej. Test case
[12]   Jun Zhu and Quiang Yang. "Remembering To Add                                     reduction. Technical Report 25521, Bangkok, Thailand:
       Competence-preserving Case Addition Policies for Case Base                       Assumption University, 2009.
       Maintenance." Proceedings of the 16th International Joint                 [29]   Smyth, Barry. Case Based Design. PhD Thesis, Department of
       Conference in Artificial Intelligence. Stockholm, Sweden :                       Computer Science, Trinity College, Dublin, Ireland: Trinity
       Morgan Kaufmann Publishers Inc, 1999. 234-241.                                   College, 1996.
[13]   Kaner, Cem. "Exploratory Testing." Quality Assurance Institute            [30]   Sprenkle, S., S. Sampath and A. Souter. "An empirical
       Worldwide Annual Software Testing Conference. Orlando,                           comparison of test suite reduction techniques for user-session-
       Florida, USA: Florida Institute of Technology, 2006.                             based testing of web applications." Journal of Software Test.
[14]   Lehmann, E. and J. Wegener. "Test case design by means of the                    Verificat. Reliabil., 2002: 587-596.
       CTE XL." Proceedings of the 8th European International                    [31]   Sreedevi Sampath, Sara Sprenkle, Emily Gibson and Lori
       Conference on Software Testing,. Kopenhagen, Denmark: ACM                        Pollock. "Web Application Testing with Customized Test
       Press, 2000. 1-10.                                                               Requirements – An Experimental Comparison Study."
[15]   Mary Jean Harrold, Rajiv Gupta and Mary Lou Soffa. "A                            Proceedings of the 17th International Symposium on Software
       Methodology for Controlling the Size of A Test Suite." ACM                       Reliability Engineering (ISSRE’06). Raleigh, NC, USA: IEEE
       Transactions on Software Engineering (ACM) 2, no. 3 (July                        Computer Society, 2006. 266 - 278.
       1993): 270-285.                                                           [32]   Todd L. Graves, Mary Jean Harrold, Jung-Min Kim, Adam
[16]   Nicha Kosindrdecha and Jirapun Daengdej. A Deletion                              Porter and Gregg Rothermel. "An Empirical Study of
       Algorithm for Case-Based Maintenance Based on Accuracy and                       Regression Test Selection Techniques." ACM Transactions on
       Competence. MS Thesis, Faculty of Science and Technology,                        Software Engineering and Methodology (TOSEM), 2001: 184-
       Assumption University, Bangkok, Thailand: Assumption                             208.
       University, 2003.                                                         [33]   W. Eric Wong, J. R. Horgan, Saul London and Hira Agrawal.
[17]   Nicha Kosindrdecha and Siripong Roongruangsuwan.                                 "A Study of Effective Regression Testing in Practice."
       "Reducing Test Cases Created by Path Oriented Test Case                          Proceedings of 8th IEEE International Symposium on Software
       Generation." Proceedings of the AIAA Conference and                              Reliability Engineering (ISSRE’97). California, USA: IEEE
       Exhibition. Rohnert Park, California, USA: NASA AIAA, 2007.                      Computer Society, 1997. 264.
[18]   NIST. The economic impacts of inadequate infrastructure for               [34]   W. Eric Wong, Joseph R. Horgan, Saul London and Aditya P.
       software testing. Technical Report, USA: NIST, 2002.                             Mathur. "Effect of Test Set Minimization on the Fault Detection
                                                                                        Effectiveness of the All-Uses Criterion." Proceedings of the 17th
[19]   Rothermel, G., R.H. Untch, C. Chu and M.J. Harrold. "Test case                   International Conference on Software Engineering. Seattle,
       prioritization: An empirical study." Proceedings of the 15th                     USA: ACM, 1995. 41-50.
       IEEE International. Oxford, England, UK: IEEE Computer
       Society, 1999. 179-188.                                                   [35]   Wilson, David C. A Case-Based Maintenance: The husbandry of
                                                                                        experiences. PhD Thesis, Computer and Science, Indiana
[20]   S. Elbaum, A. G. Malishevsky and G. Rothermel. "Prioritizing                     University, USA: Indiana University, 2001.
       Test Cases for Regression Testing." Proceedings of the
       International Symposium on Software Testing and Analysis.                 [36]   Xiaofang Zhang, Baowen Xu, Changhai Nie and Liang Shi. "An
       2000. 102-112.                                                                   Approach for Optimizing Test Suite Based on Testing
                                                                                        Requirement Reduction." Journal of Software (in Chinese),
[21]   S. Elbaum, A. Malishevsky, and G. Rothermel. "Test Case                          2007: 821-831.
       Prioritization: A Family of Empirical Studies." IEEE Trans. on
       Software Engineering 28 (February 2002): 159-182.                         [37]   Xiaofang Zhang, Baowen Xu, Changhai Nie and Liang Shi.
                                                                                        "Test Suite Optimization Based on Testing Requirements
[22]   S. Elbaum, P. Kallakuri, A. G. Malishevsky, G. Rothermel, and                    Reduction." International Journal of Electronics & Computer
       S. Kanduri. "Understanding the effects of changes on the cost-                   Science 7, no. 1 (2005): 9-15.
       effectiveness of regression testing techniques." Journal of
       Software Testing, Verification, and Reliability 13, no. 2 (June           [38]   Xue-ying MA, Bin-kui Sheng, Zhen-feng HE and Cheng-qing
       2003): 65-83.                                                                    YE. "A Genetic Algorithm for Test-Suite Reduction."
                                                                                        Proceedings of IEEE International Conference on Systems, Man
[23]   Saif-ur-Rebman Khan and Aamer Nadeem. "TestFilter: A                             and Cybernetics. Hangzhou, China: ACM Press, 2005. 133-139.
       Statement-Coverage Based Test Case Reduction Technique."
       Proceedings of 10th IEEE International Multitopic Conference.             [39]   Yanbing Yu, James A. Jones and Mary Jean Harrold. "An
       Islamabad, Pakistan, 2006.                                                       Empirical Study of the Effects of Test-Suite Reduction on Fault
                                                                                        Localization." Proceedings of International Conference on
[24]   Scott McMaster and Atif Memon. "Call Stack Coverage for GUI                      Software Engineer (ICSE’08). Leipzig, Germany: ACM, 2008.
       Test-Suite Reduction." Proceedings of the 17th IEEE                              201-210.
       International Symposium on Software Reliability Engineering
       (ISSRE 2006). North Carolina, USA, 2006.
[25]   —. "Call Stack Coverage for Test Suite Reduction." Proceedings
       of the 21st IEEE International Conference on Software
       Maintenance (ICSM’05). Budapest, Hungary, 2005. 539-548.




                                                                            40                                   http://sites.google.com/site/ijcsis/
                                                                                                                 ISSN 1947-5500

				
DOCUMENT INFO
Description: IJCSIS is an open access publishing venue for research in general computer science and information security. Target Audience: IT academics, university IT faculties; industry IT departments; government departments; the mobile industry and computing industry. Coverage includes: security infrastructures, network security: Internet security, content protection, cryptography, steganography and formal methods in information security; computer science, computer applications, multimedia systems, software, information systems, intelligent systems, web services, data mining, wireless communication, networking and technologies, innovation technology and management. The average paper acceptance rate for IJCSIS issues is kept at 25-30% with an aim to provide selective research work of quality in the areas of computer science and engineering. Thanks for your contributions in September 2010 issue and we are grateful to the experienced team of reviewers for providing valuable comments.