A Black-Box Test Case Generation Method by ijcsis

VIEWS: 230 PAGES: 10

More Info
									                                                          (IJCSIS) International Journal of Computer Science and Information Security,
                                                          Vol. 8, No. 6, September 2010

             A Black-Box Test Case Generation Method

                  Nicha Kosindrdecha                                                              Jirapun Daengdej
        Autonomous System Research Laboratory                                      Autonomous System Research Laboratory
Faculty of Science and Technology, Assumption University                   Faculty of Science and Technology, Assumption University
                    Bangkok, Thailand                                                           Bangkok, Thailand
                    P4919742@au.edu                                                          jirapun@scitech.au.edu

Abstract—Test case generation techniques have been researched              generation) are widely-used for generating test cases in the
over a long period of time. Unfortunately, while many                      commercial industry.
researchers have found methods of minimizing test cases, there
are still a number of important related issues that need to be                 Moreover, the study [2], [4], [10], [11], [12], [15], [21],
researched. The primarily outstanding research issue is a large            [22] shows that the primary research issue is that existing
single test suite containing a huge number of test cases. Our study        black-box test case generation methods generate a huge single
shows that this can lead to other two problems: unable to identify         test suite with a number of possible tests. The number of
suitable test cases for execution and those test cases are lack of         possible black-box tests for any non-trivial software application
ability to cover domain specific requirement. Therefore, we                is extremely large. Consequently, it is unable to identify
proposed an additional requirement prioritization process during           suitable test cases for execution.
a test case generation process and an automated method to
generate multiple test suites while minimizing a number of test                Also, the study shows that the secondary research issue is
cases from UML Use Case diagram 2.0. Our evaluation result                 that the existing black-box test case generation methods ignore
shows that the proposed method is the most recommendation                  critical domain specific requirements [5] during a test case
method to minimize size of test cases while maximizing ability to          generation process. These requirements are one of the most
cover critical domain specific requirements.                               important requirements that should be addressed during test
    Keywords-component; Test generation, testing and quality, test
case generation, test generation technique and generate tests                  Therefore, we propose a new black-box test case
                                                                           generation, with requirement prioritization approach, from
                                                                           requirements captured as use cases, 2.0, [23], [24], [33]. A use
                       I.    INTRODUCTION                                  case is the specification of interconnected sequences of actions
    Software testing is known as a key critical phase in the               that a system can perform, interacting with actors of the
software development life cycle, which account for a large part            system. Use cases have become one of the favorite approaches
of the development effort. A way of reducing testing effort,               for requirements capture. Our automated black-box approach
while ensuring its effectiveness, is to generate a minimize                aims to generate a minimize number of suitable test cases while
number of test cases automatically from artifacts used in the              reserving critical domain specific requirements. Additionally,
early phases of software development. Many test case                       we introduce an automated test generation method derived
generation techniques have been proposed [2], [4], [10], [11],             from UML Use Case diagram, 2.0. Our approach is developed
[12], [15], [21], [22], [42], [47], [50], mainly random, path-             to automatically generate many test suites based on notions
oriented, goal-oriented and model-based approaches. Random                 announced in the latest version of UML.
techniques determine a set of test cases based on assumptions
                                                                               The rest of the paper is organized as follow. Section 2
concerning fault distribution. Path-oriented techniques
                                                                           discusses an overview of test case generation techniques.
generally use control flow graph to identify paths to be covered
                                                                           Section 3 describes motivated research issues. Section 4
and generate the appropriate test cases for those paths. Goal-
                                                                           introduces a new test generation process with requirement
oriented techniques identify test cases covering a selected goal
                                                                           prioritization step. Also, section 4 proposes a new black-box
such as a statement or branch, irrespective of the path taken.
                                                                           test generation method. Section 5 describes an experiment,
There are many researchers and practitioners who have been
                                                                           measurement metrics and results. Section 6 provides the
working in generating a set of test cases based on the
                                                                           conclusion and research directions in the test case generation
specifications. Modeling languages are used to get the
                                                                           field. The last section represents all source references used in
specification and generate test cases. Since Unified Modeling
                                                                           this paper.
Language (UML) 2.0 is the most widely used language, many
researchers are using UML diagrams such as UML Use Case
diagram, UML Activity diagram and UML Statechart diagram                                    II.     LITERATURE REVIEW
to generate test cases and this has led to model-based test case               The literature review is structured into two sections. The
generation techniques. The study shows that model-based test               first section gives an overview of previous studies. The second
generation methods (or also known as black-box test                        section provides the related works

                                                                      22                                http://sites.google.com/site/ijcsis/
                                                                                                        ISSN 1947-5500
                                                                    (IJCSIS) International Journal of Computer Science and Information Security,
                                                                    Vol. 8, No. 6, September 2010

A.        An Overview of Recent Researches                                        Whereas all requirements are mandatory, some are more
    Model-based techniques are popular and most researchers                       critical than others. For example, failure to implement certain
have proposed several techniques. One of the reasons why                          requirements may have grave business ramifications that would
those model-based techniques are popular is that wrong                            make the system a failure, while others although contractually
interpretations of complex software from non-formal                               binding would have far less serious business consequences if
specification can result in incorrect implementations leading to                  they were not implemented or not implemented correctly (b)
testing them for conformance to its specification standard [43].                  Help programs through negotiation and consensus building to
A major advantage of model-based V&V is that it can be easily                     eliminate unnecessary potential “requirements” (i.e., goals,
automated, saving time and resources. Other advantages are                        desires, and “nice-to-haves” that do not merit the mandatory
shifting the testing activities to an earlier part of the software                nature of true requirements) and (c) schedule the
development process and generating test cases that are                            implementation of requirements (i.e., help determine what
independent of any particular implementation of the design [7].                   capabilities are implemented in what increment). Additionally,
                                                                                  these researches in 1980-2008 [8], [27], [28], [29], [30], [38]
    The model-based techniques are method to generate test                        reveal that there are many requirement prioritization methods
cases from model diagrams like UML Use Case diagram [23],                         such as Binary Search Tree (BST), 100-point method and
[24], [33], UML Sequence diagram [7] and UML State diagram                        Analytic Hierarchy Process (AHP)
[5], [43], [22], [2], [21], [15], [32], [4]. There are many
researchers who investigated in generating test cases from
those diagrams. The following paragraphs show examples of                                          III.   RESEARCH PROBLEM
model-based test generation techniques that have been                                  This section discusses the details of research issues related
proposed for a long time.                                                         to test case generation techniques and research problems,
    Heumann [23] presented how using use cases, derived from                      which are motivated this study. Every test case generation
UML Use Case diagram 1.0, to generate test cases can help                         technique has weak and strong points, as addressed in the
launch the testing process early in the development lifecycle                     literature survey. In general, referring to the literature review,
and also help with testing methodology. In a software                             the following lists major outstanding research challenges.
development project, use cases define system software                                 The first research problem is that existing test case
requirements. Use case development begins early on, so real                       generation methods are lack of ability to identify domain
use cases for key product functionality are available in early                    specific requirements. The study [5] shows that domain
iterations. According to the Rational Unified Process (RUP), a                    specific requirements are some of the most critical
use case is used to describe fully a sequence of actions                          requirements required to be captured for implementation and
performed by a system to provide an observable result of value                    testing, such as constraints requirements and database specific
to a person or another system using the product under                             requirements. Existing approaches ignore an ability to address
development." Use cases tell the customer what to expect, the                     domain specific requirements. Consequently, software testing
developer what to code, the technical writer what to document,                    engineers may ignore the critical functionality related to the
and the tester what to test. He proposed three-step process to                    critical domain specific requirements. Thus, this paper
generate test cases from a fully detailed use case: (a) for each                  introduces an approach to priority those specific requirements
use case, generate a full set of use-case scenarios (b) for each                  and generates an effective test case.
scenario, identify at least one test case and the conditions that
will make it execute and (c) for each test case, identify the data                    The second problem is that existing black-box test case
values with which to test.                                                        generation techniques aim to generate a large single test suite
                                                                                  with all possible test cases which maximize cover for each
    Ryser [24] raised the practical problems in software testing                  scenario. Basically, they generate a huge number of test cases
as follows: (1) Lack in planning/time and cost pressure, (2)                      which are impossible to execute given limited time and
Lacking test documentation, (3) Lacking tool support, (4)                         resources. As a result, those unexecuted test cases are useless
Formal language/specific testing languages required, (5)                          and it is unable to identify suitable test cases for execution.
Lacking measures, measurements and data to quantify testing
and evaluate test quality and (6) Insufficient test quality. They                                   IV.    PROPOSED METHOD
proposed their approach to resolve the above problems. Their
approach is to derive test case from scenario / UML Use Case                      A. Test Case Generation Process
diagram 1.0 and state diagram 1.0. In his work, the generation
of test cases is done in three processes: (a) preliminary test case                    This section presents a new high-level process to generate a
definition and test preparation during scenario creation (b) test                 set of test cases introduced by using the above comprehensive
case generation from Statechart and from dependency charts                        literature review and previous works [43].
and (c) test set refinement by application dependent strategies.

B.        Related Works
    This section provides the related works used in this paper,
prioritize requirement methods. Donald Firesmith [16]
addressed the purpose of requirement prioritization as follows:
(a) Determine the relative necessity of the requirements.

    Autonomous System Research Laboratory, Faculty of Science and
Technology, Assumption University.

                                                                             23                                http://sites.google.com/site/ijcsis/
                                                                                                               ISSN 1947-5500
                                                               (IJCSIS) International Journal of Computer Science and Information Security,
                                                               Vol. 8, No. 6, September 2010

                                                                                 From the above figure, the horizontal axis presents a
                                                                             customer’s need while the vertical axis represents a customer
                                                                             satisfaction. There are four groups of requirements based on
                                                                             those two factors: delight, attractive, indifferent and basic.
                                                                             First, the delight requirement is known as ‘nice-to-have’
                                                                             requirement. If this requirement is well fulfilled, it will increase
                                                                             the customer satisfaction. Otherwise, it will not decrease the
                                                                             satisfaction. Second, the attractive requirement is called as
                                                                             ‘surprise’ or ‘know your customer’ requirement. This
                                                                             requirement can directly increase the customer satisfaction if it
                                                                             is fulfilled. Marketers and sales [53] believe that if we can
                                                                             deliver this kind of requirement, it will impress customers and
                                                                             significantly improve the customer satisfaction. Third, the
                                                                             indifferent requirement is a requirement that customer does not
                                                                             concentrate and it will not impress customers at all. In the
                                                                             competitive industry, this requirement may be fulfilled, but
                                                                             there are no any impacts to the customer satisfaction. Last, the
                                                                             basic requirement is a mandatory requirement that customers
                                                                             basically expect. Therefore, if this requirement is well
                                                                             delivered, it will not increase the customer satisfaction.
         Figure 1. A Proposed Process to Generate Test Cases                    Furthermore, our study reveals that the requirement can be
                                                                             simply divided into two types: functional and non-functional
    From the above figure, there are two test case generation                requirement. Our study also presents that functional
processes: existing and proposed process. The left-hand side                 requirements can be categorized into two groups: domain
shows an existing process to generate test cases directly from               specific requirement [5] (or known as constraints requirement)
diagrams. Meanwhile, the right-hand side proposes to add an                  and behavior requirement. The following shows the
additional requirement prioritization process before generating              requirement classification used in this paper:
test cases. The requirement prioritization process aims to be
able to effectively handle with a large number of requirements.
The objective of this process is to prioritize and organize
requirements in an appropriate way in order to effectively
design and prepare test cases [16], [25], [37]. There are two
sub-processes: (a) classify requirements and (b) prioritize
   Our study [51], [52], [53], [54] shows that a marketing
perspective concentrates on two factors: customer’s needs and
customer satisfaction. We apply that perspective to the
requirement prioritization and propose the following:

                                                                                       Figure 3. Classify Requirement on Software Engineer

                                                                                 From the above figure, functional requirement is a
                                                                             requirement that customers directly are able to provide. The
                                                                             non-functional requirement is a requirement that is given
                                                                             indirectly. The domain specific or constraints requirement is a
                                                                             requirement relative to any constraints and business rules in the
                                                                             software development. Meanwhile, the behavior requirement is
                                                                             a requirement that describes a behavior of system. Once the
                                                                             requirement is classified based on previous two perspectives,
                                                                             the next process is to prioritize requirements based on return on
                                                                             investment (or ROI) [51], [52], [53]. From business
                                                                             perspective, ROI is the most important factor to assess the
                                                                             important of each requirement. The following presents a
                                                                             ranking tree by combining those two perspectives.

       Figure 2. Classify Requirement on Marketing’s Perspective

                                                                        24                                 http://sites.google.com/site/ijcsis/
                                                                                                           ISSN 1947-5500
                                                          (IJCSIS) International Journal of Computer Science and Information Security,
                                                          Vol. 8, No. 6, September 2010

                                                                          •   CostCode is a cost of coding that is charged to customers.
                                                                              This paper applies the cost-value approach to identify the
                                                                              cost of coding for each requirement group (e.g. “Must-
                                                                              Have”, “Should-Have”, “Could-Have” and “Wish”). The
                                                                              unit is US dollar.
                                                                          •   EffTest is an estimated effort of testing for each
                                                                              requirement. The unit is man-hours.
                                                                          •   CostTest is a cost of testing that is charged to customers.
                                                                              The approach to identify this value is similar to
                                                                              CostCode’s approach. The unit is US dollar.
                                                                              In this paper, we assumed the following in order to
                                                                          calculate CostCode and CostTest. Also, this paper assumes that
                                                                          a standard cost for both activities is $100 per man-hours.
                                                                          •   A value is 1.5 of (“Must-Have”, “Should-Have”) – this
                                                                              means that “Must-Have” requirements have one and half
                                                                              times cost value than “Should-Have” requirements.
                                                                          •   A value is 3 of (“Must-Have”, “Could-Have”) – this
                                                                              means that “Must-Have” requirements have three times
                                                                              cost value than “Could-Have” requirements.
              Figure 4. Requirement Prioritization Tree                   •   A value is 2 of (“Should-Have”, “Could-Have”) – this
                                                                              means that “Should-Have” requirements have two times
    From the above figure, we give the highest priority for all
                                                                              cost value than “Could-Have” requirements.
‘basic requirements due to the fact that they must be
implemented even they do not increase the customer                        •   A value is approximately 3 of (“Could-Have”, “Wish”) –
satisfaction. We rank the lowest priority for all ‘indifferent’               this means that “Could-Have” requirements have three
requirements, because customers do not concentrate on.                        times cost value than “Wish” requirements.
Additionally, we prioritize both of all ‘delight’ and ‘attractive’
requirement based on ROI. In this paper, we propose to use a                 Therefore, the procedure of requirement prioritization
cost-value approach to weight and prioritize requirements. This           process can be shortly described below:
paper proposes to use the following formula:                                  1.   Provide estimated efforts of coding and testing for each
                 P(Req) = (Cost * CP)                         (1)                  requirement.
Where:                                                                        2.   Assign cost value for each requirement group based on
•   P is a prioritization value.                                                   the previous requirement classification (e.g. “Must-
                                                                                   Have”, “Should-Have”, “Could-Have” and “Wish”).
•   Req is a requirement required to be prioritized.
                                                                              3.   Calculate a total estimated cost for coding and testing,
•   Cost is a total estimated cost of coding and testing for                       by using the formula (2).
    each requirement.                                                         4.   Define a customer priority for each requirement.
•   CP is an user-defined customer priority value. This value                 5.   Compute a priority value for each requirement by
    is in the range between 1 and 10. 10 is the highest priority                   using the formula (1).
    and 1 is the lowest priority. This value aims to allow
    customers to identify how important of each requirement                   6.   Prioritize requirements based on the higher priority
    is from their perspective.                                                     value.

   To compute the above cost for coding and testing, this                     Once the requirements are prioritized, the next proposed
paper proposes to apply the following formula:                            step is to generate test scenario and prepare test case.

    Cost = (EffCode*CostCode)+(EffTest*CostTest)              (2)         B. Test Case Generation Technique
                                                                              This section presents an automated test scenario generation
Where                                                                     derived from UML Use Case diagram 2.0. The big different
                                                                          between UML Use Case diagram 1.0 and 2.0 is a package
•   Cost is a total estimated cost.                                       notion that can group each use case into each package.
•   EffCode is an estimated effort of coding for each                        The following shows an example of UML Use Case
    requirement. The unit is man-hours.                                   diagram 2.0.

                                                                     25                               http://sites.google.com/site/ijcsis/
                                                                                                      ISSN 1947-5500
                                                            (IJCSIS) International Journal of Computer Science and Information Security,
                                                            Vol. 8, No. 6, September 2010

                                                                              Case Case             y         Event       ve Events      ss
                                                                               Id Name                                                  Rules
                                                                               UC- Withdra To allow 1. Insert             1. Select    (a)
                                                                               001   w      bank'  s     Card              Inquiry Input
                                                                                           customer 2. Input              2. Select amount
                                                                                              s to       PIN              A/C Type <=
                                                                                           withdraw 3. Select             3. Check Outstan
                                                                                            money Withdraw                 Balance ding
                                                                                             from     4. Select                     Balanc
                                                                                             ATM A/C Type                               e
                                                                                           machines 5. Input                        (b) Fee
                                                                                           anywher Balance                           charge
                                                                                              e in      6. Get                      if using
                                                                                           Thailand Money                           differen
                                                                                                .    7. Get Card                    t ATM
                                                                               UC- Transfer To allow         1. Insert    1. Select Amoun
         Figure 5. An Example of UML Use Case Diagram 2.0
                                                                               002          users to           Card        Inquiry   t <=
    From the above figure, the new notion in UML Use Case                                   transfer         2. Input     2. Select 50,000
diagram 2.0 is a package that is used for grouping each                                     money to           PIN        A/C Type baht
function. There are three packages or releases. Each release                                  other         3. Select     3. Check
contains different functional requirement. The first release                                banks in         Transfer      Balance
contains two functions: inquiry and withdraw. The second                                    Thailand        4. Select
release is composed of: transfer own account and transfer to                                from all           bank
other banks. The last release has only one function to support                               ATM            5. Select
Thai (TG) airline tickets.                                                                  machines           "To"
    Our approach aims to generate three test suites to cover the                                             account
above three packages while existing test case generation                                                    6. Select
techniques do not concentrate on. The first test suite is                                                   A/C Type
developed for: inquiry and withdraw functions. The second test                                               7. Input
suite is used for transferring own banks and other banks. The                                                Amount
last suite aims to a TG airline ticket support.                                                               8. Get
    The approach is built based on Heumann’s algorithm [23].                                               9. Get Card
The limitation of our approach is to ensure that all use cases are
fully dressed. The fully dressed use case is a use case with the
comprehensive of information, as follows: use case name, use                 The above use cases can be extracted into the following use
case number, purpose, summary, pre-condition, post-condition,             case scenarios:
actors, stakeholders, basic events, alternative events, business
rules, notes, version, author and date.
                                                                                     TABLE II.      EXTRACTED USE CASE SCENARIOS
    The proposed method contains four steps, as follows: (a)
extract use case diagram (b) generate test scenario (c) prepare             Scenario Id           Summary                Basic Scenario
test data and (d) prepare other test elements. These steps can be
shortly described as follows:                                              Scenario-001                      s
                                                                                               To allow bank' 1. Insert Card
                                                                                                customers to      2. Input PIN
   1.   The first step is to extract the following information                              withdraw money from 3. Select Withdraw
        from fully dressed use cases: (a) use case number (b)                                  ATM machines 4. Select A/C Type
        purpose (c) summary (d) pre-condition (e) post-                                     anywhere in Thailand. 5. Input Balance
        condition (f) basic event and (g) alternative events.                                                     6. Get Money
        This information is called use case scenario in this                                                      7. Get Card
        paper. The example fully dressed use cases of ATM
        withdraw functionality can be found as follows:


     Use      Use     Summar      Basic     Alternati Busine

                                                                     26                                 http://sites.google.com/site/ijcsis/
                                                                                                        ISSN 1947-5500
                                                       (IJCSIS) International Journal of Computer Science and Information Security,
                                                       Vol. 8, No. 6, September 2010

Scenario-002                     s
                   To allow bank' 1. Insert Card                         TS-002                         s
                                                                                          To allow bank' 1. Insert Card
                    customers to      2. Input PIN                                         customers to      2. Input PIN
                withdraw money from 3. Select Inquiry                                  withdraw money from 3. Select Inquiry
                   ATM machines 4. Select A/C Type                                        ATM machines 4. Select A/C Type
                anywhere in Thailand. 5. Check Balance                                 anywhere in Thailand. 5. Check Balance
                                      6. Select Withdraw                                                     6. Select Withdraw
                                      7. Select A/C Type                                                     7. Select A/C Type
                                      8. Input Balance                                                       8. Input Balance
                                      9. Get Money                                                           9. Get Money
                                      10. Get Card                                                           10. Get Card
Scenario-003      To allow users to     1. Insert Card                   TS-003          To allow users to       1. Insert Card
                  transfer money to     2. Input PIN                                     transfer money to       2. Input PIN
                    other banks in      3. Select Transfer                                 other banks in        3. Select Transfer
                  Thailand from all     4. Select bank                                   Thailand from all       4. Select bank
                   ATM machines         5. Select "To"                                    ATM machines           5. Select "To"
                                        account                                                                  account
                                        6. Select A/C Type                                                       6. Select A/C Type
                                        7. Input Amount                                                          7. Input Amount
                                        8. Get Receipt                                                           8. Get Receipt
                                        9. Get Card                                                              9. Get Card
Scenario-004      To allow users to     1. Insert Card                   TS-004          To allow users to       1. Insert Card
                  transfer money to     2. Input PIN                                     transfer money to       2. Input PIN
                    other banks in      3. Select Inquiry                                  other banks in        3. Select Inquiry
                  Thailand from all     4. Select A/C Type                               Thailand from all       4. Select A/C Type
                   ATM machines         5. Check Balance                                  ATM machines           5. Check Balance
                                        6. Select Transfer                                                       6. Select Transfer
                                        7. Select bank                                                           7. Select bank
                                        8. Select "To"                                                           8. Select "To"
                                        account                                                                  account
                                        9. Select A/C Type                                                       9. Select A/C Type
                                        10. Input Amount                                                         10. Input Amount
                                        11. Get Receipt                                                          11. Get Receipt
                                        12. Get Card                                                             12. Get Card

  2.   The second step is to automatically generate test                3.   The next step is to prepare test data. This step allows to
       scenarios from the previous use case scenarios [23].                  manually prepare an input data for each scenario.
       From the above table, we automatically generate the
       following test scenarios:                                        4.   The last step is to prepare other test elements, such as
                                                                             expected output, actual output and pass / fail status

                                                                                            V.    EVALUATION
Test Scenario            Summary          Basic Scenario                The section describes the experiments design, measurement
      Id                                                             metrics and results.
   TS-001                        s
                   To allow bank' 1. Insert Card
                                                                     A. Experiments Design
                    customers to      2. Input PIN
                withdraw money from 3. Select Withdraw                  1. Prepare Experiment Data. Before evaluating the
                   ATM machines 4. Select A/C Type                         proposed methods and other methods, preparing
                anywhere in Thailand. 5. Input Balance                     experiment data is required. In this step, 50
                                      6. Get Money                         requirements and 50 use case scenarios are randomly
                                      7. Get Card                          generated.
                                                                        2.   Generate Test Scenario and Test Case. A comparative
                                                                             evaluation method has been made among the proposed
                                                                             test scenario algorithm, Heumann’s technique Jim [23],
                                                                             Ryser’s method [24], Nilawar’s algorithm [33] and the
                                                                             proposed method presented in the previous section. It

                                                                27                                http://sites.google.com/site/ijcsis/
                                                                                                  ISSN 1947-5500
                                                            (IJCSIS) International Journal of Computer Science and Information Security,
                                                            Vol. 8, No. 6, September 2010

           is included a prioritization requirement algorithm prior                                       basic1, basic2, …, basicn.
           to generate a set of test scenarios and test cases.
                                                                            B. Measurement Metrics
    3.     Evaluate Results. In this step, the comparative
                                                                                The section lists the measurement metrics used in the
           generation methods are executed by using 50
                                                                            experiment. This paper proposes to use three metrics, which
           requirements and 50 use case scenarios. These methods
                                                                            are: (a) size of test cases (b) total time and (c) percentage of
           are also executed for 10 times in order to find out the
                                                                            critical domain requirement coverage. The following describe
           average percentage of critical domain requirement
                                                                            the measurement in details.
           coverage, a size of test cases and total generation time.
           In total, there are 500 requirements and 500 use case            1.   A Number of Test Cases: This is the total number of
           scenarios executed in this experiment.                                generated test cases, expressed as a percentage, as follows:
    The following tables present how to randomly generate data                       % Size = (# Size / # of Total Size)*100                   (3)
for requirements and use case scenarios respectively.                                                   Where:
                                                                            •    % Size is a percentage of the number of test cases.
                                                                            •    # of Size is a number of test cases.
   Attribute                       Approach
                                                                            •    # of Total Size is the maximum number of test cases in the
Requirement           Randomly generated from the following
                                                                                 experiment, which is assigned 1,000.
ID                    combination: Req + Sequence Number.
                                                                            2.   A Domain Specific Requirement Coverage: This is an
                        For example, Req1, Req2, Req3, …,                        indicator to identify the number of requirements covered in
                                      ReqN.                                      the system, particularly critical requirements, and critical
Description           Randomly generated from the following                      domain requirements [5]. Due to the fact that one of the
                       combination: Des + Sequence Number                        goals of software testing is to verify and validate
                             same as Requirement ID.                             requirements covered by the system, this metric is a must.
                                                                                 Therefore, a high percentage of critical requirement
                                                                                 coverage is desirable.
                        For example, Des1, Des2, Des3, …,
                                      DesN.                                      It can be calculated using the following formula:
Type of               Randomly selected from the following
                                                                                       % CRC = (# of Critical / # of Total)*100        (4)
Requirement          values: Functional AND Non-Functional.
MoSCoW                Randomly selected from the following
                                                                            •    % CRC is the percentage of critical requirement coverage.
Criteria             values: Must Have (M), Should Have (S),
                       Could Have (C) and Won’t Have (W)                    •    # of Critical is the number of critical requirements
Is it a critical      Randomly selected from the following                       covered.
requirement               values: True (Y) and False (N)
(Y/N)?                                                                      •    # of Total is the total number of requirements.
                                                                            3.   Total Time: This is the total number of times the
                                                                                 generation methods are run in the experiment. This metric
          TABLE V.      GENERATE RANDOM USE CASE SCENARIO                        is related to the time used during the testing development
                                                                                 phase (e.g. design test scenario and produce test case).
     Attribute                        Approach                                   Therefore, less time is desirable.
    Use case ID              Randomly generated from the
                            following combination: uCase +                       It can be calculated using the following formula:
                            Sequence Number. For example,                                Total = PTime + CTime + RTime              (5)
                              uCase1, uCase2, …, uCasen.                                                Where:
         Purpose             Randomly generated from the                    •    Total is the total amount of times consumed by running
                             following combination: Pur +                        generation methods.
                          Sequence Number same as Use case
                            ID. For example, Pur1, Pur2, …,                 •    PTime is the total amount of time consumed by
                                         Purn.                                   preparation before generating test cases.
   Pre-condition             Randomly generated from the                    •    CTime is the time to compile source code / binary code in
                            following combination: pCon +                        order to execute the program.
                          Sequence Number same as Use case
                          ID. For example, pCon1, pCon2, …,                 •    RTime is the total time to run the program under this
                                        pConn.                                   experiment.
  Basic Scenario             Randomly generated from the
                            following combination: uCase +
                            Sequence Number. For example,

                                                                       28                               http://sites.google.com/site/ijcsis/
                                                                                                        ISSN 1947-5500
                                                               (IJCSIS) International Journal of Computer Science and Information Security,
                                                               Vol. 8, No. 6, September 2010

C. Results and Discussion                                                        For an ability to cover critical domain specific requirement,
    This section discusses an evaluation result of the above                 the maximum and minimum percentage is 53.20% and 19%.
experiment. This section presents a graph that compares the                  The different value is 34.2%. The interval value is 6.84.
above proposed method to other three existing test case                      Therefore, it can be determined as follows: 5-Excellent (since
generation techniques, based on the following measurements:                  46.36% to 53.2%), 4-Very good (between 39.52% and
(a) size of test cases (b) critical domain coverage and (c) total            46.36%), 3-Good (between 32.68% and 39.52%), 2-Normal
time. Those three techniques are: (a) Heumman’s method (b)                   (between 25.84% and 32.68%) and 1-Poor (from 19% to
Ryser’s work and (c) Nilawar’s approach. There are two                       25.84%).
dimensions in the following graph: (a) horizontal and (b)                       For a total time, the maximum and minimum percentage is
vertical axis. The horizontal represents three measurements                  31.82% and 30.20%. The different between maximum and
whereas the vertical axis represents the percentage value.                   minimum value is 1.62%. An interval value is equal to a result
                                                                             of dividing the different values by 5. As a result, the interval
                                                                             value is 0.324. Thus, it can be determined as follows: 5-
                                                                             Excellent (since 30.2% to 30.524%), 4-Very good (between
                                                                             30.524% and 30.848%), 3-Good (between 30.848% and
                                                                             31.172%), 2-Normal (between 31.172% and 31.496%) and 1-
                                                                             Poor (from 31.496% to 31.82%).
                                                                                Therefore, the experiment result of those comparative
                                                                             methods can be shown below:

                                                                                TABLE VI.       A COMPARISON OF TEST CASE REDUCTION METHODS
                                                                                    Algorithm              A              Cover             Total
                                                                                                     Number of         Critical           Time
                                                                                                     Test Cases        Domain
                                                                                                                     Specific Req.
                                                                               Heumann’s                   1                 1                   5
       Figure 6. An Evaluation Result of Test Generation Methods               Ryser’s Method              1                 1                   1

    The above graph shows that the above proposed method                       Nilawar’s                   1                 1                   1
generates the smallest set of test cases. It is calculated as                  Method
80.80% where as the other techniques is computed over 97%.
Those techniques generated a bigger set of test cases, than a set              Our    Proposed             5                 5                   5
generated by the proposed method. The literature review                        Method
reveals that the smaller set of test cases is desirable. Also, the
graph shows that the proposed method consumes the least total
time during a generation process, comparing to other                             In the conclusion, the proposed method is the best to
techniques. It used only 30.20%, which is slightly less than                 generate the smallest size of test cases with the maximum of
others. Finally, the graph presents that the proposed method is              critical domain coverage and the least time consumed in the
the best techniques to coverage critical domains. Its percentage             generation process.
is much greater than other techniques’ percentage, over 30%.
   From the above figure, this study determines and ranks the                                       VI.   CONCLUSION
above comparative methods into five ranking: 5-Excellent, 4-                     In this paper, we introduced a new test case generation
Very good, 3-Good, 2-Normal and 1-Poor. This study uses a                    method and process, with an additional requirement
maximum and minimum value to find an interval value for                      prioritization process. The approach inserts an additional
ranking those methods.                                                       process to ensure that all domain specific requirements are
                                                                             captured during the test case generation. Also, the approach is
    For a number of test cases, the maximum and minimum                      developed to minimize a number of test cases in order to be
percentage is 98% and 80.80%. The different between                          able to select suitable test cases for execution. Additionally, we
maximum and minimum value is 17.2%. An interval value is                     proposed an automated approach to generate test cases from
equal to a result of dividing the different values by 5. As a                fully described UML use cases, version 2.0. Our generated
result, the interval value is approximately 3.4. Thus, it can be             method can generate many test suites derived from UML Use
determined as follows: 5-Excellent (since 80.80% to 84.2%), 4-               Case diagram, 2.0. Existing test case generation methods
Very good (between 84.2% and 87.6%), 3-Good (between                         generate only a large single test suite that contains a lot of
87.6% and 91%), 2-Normal (between 91% and 94.4%) and 1-                      numbers of test cases.
Poor (from 94.4% to 97.8%).
                                                                                Furthermore, we conducted an evaluation experiment with
                                                                             a random requirements and fully described use cases. Our

                                                                        29                                http://sites.google.com/site/ijcsis/
                                                                                                          ISSN 1947-5500
                                                            (IJCSIS) International Journal of Computer Science and Information Security,
                                                            Vol. 8, No. 6, September 2010

evaluation result reveals that the proposed method is the most               [15] David C. Kung, Chien-Hung Liu and Pei Hsia “An Object-
recommendation automated test case generation methods for                         Oriented Web Test Model for Testing Web Applications.” In
maximizing critical domain requirement coverage. Also, the                        Proceedings of the First Asia-Pacific Conference on Quality
result present that the proposed method is one of best methods                    Software (APAQS’00), page 111, Los Alamitos, CA, 2000.
to minimize a number of test cases.                                          [16] Donald Firesmith “Prioritizing Requirements. Journal of Object
                                                                                  Technology”, Vol.3, No8, 2004.
    The future research, we plan to enhance an ability to
prioritize requirements and conduct a large experiment for a                 [17] D. Harel “On visual formalisms.” Communications of the ACM,
large system development                                                          vol. 31, no. 5, pp. 514-530, 1988.
                                                                             [18] D. Harel. “Statecharts: A Visual Formulation for Complex
                           REFERENCES                                             System.” Sci.Comput. Program. 8(3):232-274, 1987.
[1] Ahl, V. “An Experimental Comparison of Five Prioritization               [19] Flippo Ricca and Paolo Tonella “Analysis and Testing of Web
    Methods” Master' Thesis, School of Engineering, Blekinge
                       s                                                          Applications.” Proc. of the 23rd International Conference on
    Institute of Technology, Ronneby, Sweden, 2005.                               Software Engineering, Toronto, Ontario, Canada. pp.25-34,
[2] Alessandra Cavarra, Charles Crichton, Jim Davies, Alan
    Hartman, Thierry Jeron and Laurent Mounier. “Using UML for               [20] Harel, D. “Statecharts: a visual formalism for complex system.”
    Automatic Test Generation.” Oxford University Computing                       Science of Computer Programming, v. 8, p. 231-274, 1987.
    Laboratory, Tools and Algorithms for the Construction and                [21] Hassan Reza, Kirk Ogaard and Amarnath Malge “A Model
    Analysis of Systems, TACAS' 2000, 2000.                                       Based Testing Technique to Test Web Applications Using
[3] Amaral. “A.S.M.S. Test case generation of systems specified in                Statecharts.” Fifth International Conference on Information
    Statecharts.” M.S. thesis – Laboratory of Computing and                       Technology, 2008.
    Applied Mathematics, INPE, Brazil, 2006.                                 [22] Ibrahim K. El-Far and James A. Whittaker “Model-based
[4] Annelises A. Andrews, Jeff Offutt and Roger T. Alexander.                     Software Testing”, 2000.
    “Testing Web Applications. Software and Systems Modeling.”,              [23] Jim Heumann “Generating Test Cases From Use Cases.”
    2004.                                                                         Rational Software, 2001.
[5] Avik Sinha, Ph.D and Dr. Carol S. Smidts. “Domain Specific               [24] Johannes Ryser and Martin Glinz “SCENT: A Method
    Test Case Generation Using Higher Ordered Typed Languages                     Employing Scenarios to Systematically Derive Test Cases for
    fro Specification.” Ph. D. Dissertation, 2005.                                System Test”, 2000.
[6] A. Bertolino. “Software Testing Research and Practice.” 10th             [25] Karl E. Wiegers “First Things First: Prioritizing Requirements.”
    International Workshop on Abstract State Machines                             Published in Software Development, 1999.
    (ASM'  2003), Taormina, Italy, 2003.
                                                                             [26] Karlsson, J. “Software Requirements Prioritizing.” Proceedings
[7] A.Z. Javed, P.A. Strooper and G.N. Watson “Automated                          of the Second International Conference on Requirements
    Generation of Test Cases Using Model-Driven Architecture.”                    Engineering (ICRE'  96). Colorado Springs, CO, April 15-18,
    Second International Workshop on Automation of Software Test                  1996. Los Alamitos, CA: IEEE Computer Society, p 110-116,
    (AST’07), 2007.                                                               1996.
[8] Beck, K. & Andres, C. “Extreme Programming Explained:                    [27] Karlsson, J. “Towards a Strategy for Software Requirements
    Embrace Change", 2nd ed. Boston, MA: Addison-Wesley, 2004.                    Selection.” Licentiate. Thesis 513, Linköping University, 1995.
[9] Boehm, B. & Ross, R. “Theory-W Software Project                          [28] Karlsson, J. & Ryan, K. “A Cost-Value Approach for
    Management: Principles and Examples.” IEEE Transactions on                    Prioritizing Requirements.” IEEE Software September/October,
    Software Engineering 15, 4: 902-916, 1989.                                    p67-75, 1997.
[10] B.M. Subraya, S.V. Subrahmanya “Object driven performance               [29] Leffingwell, D. & Widrig, D. “Managing Software
     testing in Web applications.” in: Proceedings of the First Asia-             Requirements: A Use Case Approach”, 2nd ed. Boston, MA:
     Pacific Conference on Quality Software (APAQS' pp. 17-26,                    Addison-Wesley, 2003.
     Hong Kong, China, 2000.
                                                                             [30] Leslie M. Tierstein “Managing a Designer / 2000 Project.”
[11] Chien-Hung Liu, David C. Kung, Pei Hsia and Chih-Tung Hsu                    NYOUG Fall’97 Conference, 1997.
     “Object-Based Data Flow Testing of Web Applications.”
                                                                             [31] L. Brim, I. Cerna, P. Varekova, and B. Zimmerova
     Proceedings of the First Asia-Pacific Conference on Quality
                                                                                  “Component-interaction automata as a verification oriented
     Software (APAQS' pp. 7-16, Hong Kong, China, 2000.
                                                                                  component-based system specification.” In: Proceedings
[12] C.H. Liu, D.C. Kung, P. Hsia, C.T. Hsu “Structural testing of                         05),
                                                                                  (SAVCBS' pp. 31-38, Lisbon, Portugal, 2005.
     Web applications.” in: Proceedings of 11th International
                                                                             [32] Mahnaz Shams, Diwakar Krishnamurthy and Behrouz Far “A
     Symposium on Software Reliability Engineering (ISSRE 2000),
                                                                                  Model-Based Approach for Testing the Performance of Web
     pp. 84-96, 2000.
                                                                                  Applications.” Proceedings of the Third International Workshop
[13] Davis, A. “The Art of Requirements Triage.” IEEE Computer                    on Software Quality Assurance (SOQUA’06), 2006.
     36, 3 p: 42-49, 2003.
                                                                             [33] Manish Nilawar and Dr. Sergiu Dascalu “A UML-Based
[14] Davis, A. “Just Enough Requirements Management: Where                        Approach for Testing Web Applications.” Master of Science
     Software Development Meets Marketing.” New York: Dorset                      with major in Computer Science, University of Nevada, Reno,
     House (ISBN 0-932633-64-1), 2005.                                            2003.

                                                                        30                                http://sites.google.com/site/ijcsis/
                                                                                                          ISSN 1947-5500
                                                            (IJCSIS) International Journal of Computer Science and Information Security,
                                                            Vol. 8, No. 6, September 2010

[34] Moisiadis, F. “Prioritising Scenario Evolution.” International              International Transactions in Operational Research, 9, 321-336,
     Conference on Requirements Engineering (ICRE 2000), 2000.                   2002.
[35] Moisiadis, F. “A Requirements Prioritisation Tool.” 6th                 [45] Wiegers, K. “E. Software Requirements”, 2nd ed. Redmond,
     Australian Workshop on Requirements Engineering (AWRE                        WA: Microsoft Press, 2003.
     2001). Sydney, Australia, 2001.                                         [46] Xiaoping Jia, Hongming Liu and Lizhang Qin “Formal
[36] M. Prasanna S.N. Sivanandam R.Venkatesan R.Sundarrajan “A                    Structured Specification for Web Application Testing”. Proc. of
     Survey on Automatic Test Case Generation.” Academic Open                     the 2003 Midwest Software Engineering Conference
     Internet Journal, 2005.                                                              03).
                                                                                  (MSEC' Chicago, IL, USA. pp.88-97, 2003.
[37] Nancy R. Mead “Requirements Prioritization Introduction.”               [47] Yang, J.T., Huang, J.L., Wang, F.J. and Chu, W.C.
     Software Engineering Institute, Carnegie Mellon University,                  “Constructing an object-oriented architecture for Web
     2008.                                                                        application testing.” Journal of Information Science and
[38] Park, J.; Port, D.; & Boehm B. “Supporting Distributed                       Engineering 18, 59-84, 2002.
     Collaborative Prioritization for Win-Win Requirements Capture           [48] Ye Wu and Jeff Offutt “Modeling and Testing Web-based
     and Negotiation.” Proceedings of the International Third World               Applications”, 2002.
     Multi-conference on Systemics, Cybernetics and Informatics              [49] Ye Wu, Jeff Offutt and Xiaochen “Modeling and Testing of
     (SCI' Vol. 2. 578-584, Orlando, FL, July 31-August 4, 1999.                  Dynamic Aspects of Web Applications, Submitted for
     Orlando, FL: International Institute of Informatics and Systemic             publication.”    Technical       Report    ISE-TR-04-01,
     (IIIS), 1999.                                                                www.ise.gmu.edu/techreps/, 2004.
[39] Rajib “Software Test Metric.” QCON, 2006.                               [50] Zhu, H., Hall, P., May, J. “Software Unit Test Coverage and
[40] Robert Nilsson, Jeff Offutt and Jonas Mellin “Test Case                      Adequacy.” ACM Comp. Survey 29(4), pp 366~427, 1997.
     Generation for Mutation-based Testing of Timeliness.”, 2006.            [51] Kano Noriaki, Nobuhiku Seraku, Fumio Takahashi,Shinichi
[41] Saaty, T. L. “The Analytic Hierarchy Process.” New York, NY:                 Tsuji. “Attractive Quality and Must-Be Quality.” Journal of the
     McGraw-Hill, 1980.                                                           Japanese Society for Quality Control. 14(2), pp 39~48, 1984.
[42] Shengbo Chen, Huaikou Miao, Zhongsheng Qian “Automatic                  [52] Cadotte, Ernest R., Turgeon, Normand “Dissatisfiers and
     Generating Test Cases for Testing Web Applications.”                         Satisfiers: Suggestions from Consumer Complaints and
     International Conference on Computational Intelligence and                   Compliments.”       Journal   of    Consumer       Satisfaction,
     Security Workshops, 2007.                                                    Dissatisfactions and Complaining Behavior. 1, pp 74~79, 1988.
[43] Valdivino Santiago, Ana Silvia Martins do Amaral, N.L.                  [53] Brandt, D. Randall “How service marketers can identify value-
     Vijaykumar, Maria de Fatima, Mattiello-Francisco, Eliane                     enhancing service elements.” Journal of Services Marketing.
     Martins and Odnei Cuesta Lopes “A Practical Approach for                     2(3), pp 35~41, 1988.
     Automated Test Case Generation using Statecharts”, 2006.                [54] Herzberg, Frederick, Mausner, B., Snyderman, B.B. “The
[44] Vijaykumar, N. L.; Carvalho, S. V.; Abdurahiman, V. “On                      motivation to work.” New York: Wiley, 2nd edition, 1959.
     proposing Statecharts to specify performance models.”

                                                                        31                                http://sites.google.com/site/ijcsis/
                                                                                                          ISSN 1947-5500

To top