A Black-Box Test Case Generation Method

Report as inappropriate Your report has been received

Description

IJCSIS is an open access publishing venue for research in general computer science and information security. Target Audience: IT academics, university IT faculties; industry IT departments; government departments; the mobile industry and computing industry. Coverage includes: security infrastructures, network security: Internet security, content protection, cryptography, steganography and formal methods in information security; computer science, computer applications, multimedia systems, software, information systems, intelligent systems, web services, data mining, wireless communication, networking and technologies, innovation technology and management. The average paper acceptance rate for IJCSIS issues is kept at 25-30% with an aim to provide selective research work of quality in the areas of computer science and engineering. Thanks for your contributions in September 2010 issue and we are grateful to the experienced team of reviewers for providing valuable comments.

Shared by: ijcsis

Stats

views:: 198
posted:: 10/10/2010
language:: English
pages:: 10

Public Domain

Document Sample

(IJCSIS) International Journal of Computer Science and Information Security,
Vol. 8, No. 6, September 2010

A Black-Box Test Case Generation Method

Nicha Kosindrdecha Jirapun Daengdej
Autonomous System Research Laboratory Autonomous System Research Laboratory
Faculty of Science and Technology, Assumption University Faculty of Science and Technology, Assumption University
Bangkok, Thailand Bangkok, Thailand
P4919742@au.edu jirapun@scitech.au.edu

Abstract—Test case generation techniques have been researched generation) are widely-used for generating test cases in the
over a long period of time. Unfortunately, while many commercial industry.
researchers have found methods of minimizing test cases, there
are still a number of important related issues that need to be Moreover, the study [2], [4], [10], [11], [12], [15], [21],
researched. The primarily outstanding research issue is a large [22] shows that the primary research issue is that existing
single test suite containing a huge number of test cases. Our study black-box test case generation methods generate a huge single
shows that this can lead to other two problems: unable to identify test suite with a number of possible tests. The number of
suitable test cases for execution and those test cases are lack of possible black-box tests for any non-trivial software application
ability to cover domain specific requirement. Therefore, we is extremely large. Consequently, it is unable to identify
proposed an additional requirement prioritization process during suitable test cases for execution.
a test case generation process and an automated method to
generate multiple test suites while minimizing a number of test Also, the study shows that the secondary research issue is
cases from UML Use Case diagram 2.0. Our evaluation result that the existing black-box test case generation methods ignore
shows that the proposed method is the most recommendation critical domain specific requirements [5] during a test case
method to minimize size of test cases while maximizing ability to generation process. These requirements are one of the most
cover critical domain specific requirements. important requirements that should be addressed during test
activities.
Keywords-component; Test generation, testing and quality, test
case generation, test generation technique and generate tests Therefore, we propose a new black-box test case
generation, with requirement prioritization approach, from
requirements captured as use cases, 2.0, [23], [24], [33]. A use
I. INTRODUCTION case is the specification of interconnected sequences of actions
Software testing is known as a key critical phase in the that a system can perform, interacting with actors of the
software development life cycle, which account for a large part system. Use cases have become one of the favorite approaches
of the development effort. A way of reducing testing effort, for requirements capture. Our automated black-box approach
while ensuring its effectiveness, is to generate a minimize aims to generate a minimize number of suitable test cases while
number of test cases automatically from artifacts used in the reserving critical domain specific requirements. Additionally,
early phases of software development. Many test case we introduce an automated test generation method derived
generation techniques have been proposed [2], [4], [10], [11], from UML Use Case diagram, 2.0. Our approach is developed
[12], [15], [21], [22], [42], [47], [50], mainly random, path- to automatically generate many test suites based on notions
oriented, goal-oriented and model-based approaches. Random announced in the latest version of UML.
techniques determine a set of test cases based on assumptions
The rest of the paper is organized as follow. Section 2
concerning fault distribution. Path-oriented techniques
discusses an overview of test case generation techniques.
generally use control flow graph to identify paths to be covered
Section 3 describes motivated research issues. Section 4
and generate the appropriate test cases for those paths. Goal-
introduces a new test generation process with requirement
oriented techniques identify test cases covering a selected goal
prioritization step. Also, section 4 proposes a new black-box
such as a statement or branch, irrespective of the path taken.
test generation method. Section 5 describes an experiment,
There are many researchers and practitioners who have been
measurement metrics and results. Section 6 provides the
working in generating a set of test cases based on the
conclusion and research directions in the test case generation
specifications. Modeling languages are used to get the
field. The last section represents all source references used in
specification and generate test cases. Since Unified Modeling
this paper.
Language (UML) 2.0 is the most widely used language, many
researchers are using UML diagrams such as UML Use Case
diagram, UML Activity diagram and UML Statechart diagram II. LITERATURE REVIEW
to generate test cases and this has led to model-based test case The literature review is structured into two sections. The
generation techniques. The study shows that model-based test first section gives an overview of previous studies. The second
generation methods (or also known as black-box test section provides the related works

22 http://sites.google.com/site/ijcsis/
ISSN 1947-5500
(IJCSIS) International Journal of Computer Science and Information Security,
Vol. 8, No. 6, September 2010

A. An Overview of Recent Researches Whereas all requirements are mandatory, some are more
Model-based techniques are popular and most researchers critical than others. For example, failure to implement certain
have proposed several techniques. One of the reasons why requirements may have grave business ramifications that would
those model-based techniques are popular is that wrong make the system a failure, while others although contractually
interpretations of complex software from non-formal binding would have far less serious business consequences if
specification can result in incorrect implementations leading to they were not implemented or not implemented correctly (b)
testing them for conformance to its specification standard [43]. Help programs through negotiation and consensus building to
A major advantage of model-based V&V is that it can be easily eliminate unnecessary potential “requirements” (i.e., goals,
automated, saving time and resources. Other advantages are desires, and “nice-to-haves” that do not merit the mandatory
shifting the testing activities to an earlier part of the software nature of true requirements) and (c) schedule the
development process and generating test cases that are implementation of requirements (i.e., help determine what
independent of any particular implementation of the design [7]. capabilities are implemented in what increment). Additionally,
these researches in 1980-2008 [8], [27], [28], [29], [30], [38]
The model-based techniques are method to generate test reveal that there are many requirement prioritization methods
cases from model diagrams like UML Use Case diagram [23], such as Binary Search Tree (BST), 100-point method and
[24], [33], UML Sequence diagram [7] and UML State diagram Analytic Hierarchy Process (AHP)
[5], [43], [22], [2], [21], [15], [32], [4]. There are many
researchers who investigated in generating test cases from
those diagrams. The following paragraphs show examples of III. RESEARCH PROBLEM
model-based test generation techniques that have been This section discusses the details of research issues related
proposed for a long time. to test case generation techniques and research problems,
Heumann [23] presented how using use cases, derived from which are motivated this study. Every test case generation
UML Use Case diagram 1.0, to generate test cases can help technique has weak and strong points, as addressed in the
launch the testing process early in the development lifecycle literature survey. In general, referring to the literature review,
and also help with testing methodology. In a software the following lists major outstanding research challenges.
development project, use cases define system software The first research problem is that existing test case
requirements. Use case development begins early on, so real generation methods are lack of ability to identify domain
use cases for key product functionality are available in early specific requirements. The study [5] shows that domain
iterations. According to the Rational Unified Process (RUP), a specific requirements are some of the most critical
use case is used to describe fully a sequence of actions requirements required to be captured for implementation and
performed by a system to provide an observable result of value testing, such as constraints requirements and database specific
to a person or another system using the product under requirements. Existing approaches ignore an ability to address
development." Use cases tell the customer what to expect, the domain specific requirements. Consequently, software testing
developer what to code, the technical writer what to document, engineers may ignore the critical functionality related to the
and the tester what to test. He proposed three-step process to critical domain specific requirements. Thus, this paper
generate test cases from a fully detailed use case: (a) for each introduces an approach to priority those specific requirements
use case, generate a full set of use-case scenarios (b) for each and generates an effective test case.
scenario, identify at least one test case and the conditions that
will make it execute and (c) for each test case, identify the data The second problem is that existing black-box test case
values with which to test. generation techniques aim to generate a large single test suite
with all possible test cases which maximize cover for each
Ryser [24] raised the practical problems in software testing scenario. Basically, they generate a huge number of test cases
as follows: (1) Lack in planning/time and cost pressure, (2) which are impossible to execute given limited time and
Lacking test documentation, (3) Lacking tool support, (4) resources. As a result, those unexecuted test cases are useless
Formal language/specific testing languages required, (5) and it is unable to identify suitable test cases for execution.
Lacking measures, measurements and data to quantify testing
and evaluate test quality and (6) Insufficient test quality. They IV. PROPOSED METHOD
proposed their approach to resolve the above problems. Their
approach is to derive test case from scenario / UML Use Case A. Test Case Generation Process
diagram 1.0 and state diagram 1.0. In his work, the generation
of test cases is done in three processes: (a) preliminary test case This section presents a new high-level process to generate a
definition and test preparation during scenario creation (b) test set of test cases introduced by using the above comprehensive
case generation from Statechart and from dependency charts literature review and previous works [43].
and (c) test set refinement by application dependent strategies.

B. Related Works
This section provides the related works used in this paper,
prioritize requirement methods. Donald Firesmith [16]
addressed the purpose of requirement prioritization as follows:
(a) Determine the relative necessity of the requirements.

Autonomous System Research Laboratory, Faculty of Science and
Technology, Assumption University.

23 http://sites.google.com/site/ijcsis/
ISSN 1947-5500
(IJCSIS) International Journal of Computer Science and Information Security,
Vol. 8, No. 6, September 2010

From the above figure, the horizontal axis presents a
customer’s need while the vertical axis represents a customer
satisfaction. There are four groups of requirements based on
those two factors: delight, attractive, indifferent and basic.
First, the delight requirement is known as ‘nice-to-have’
requirement. If this requirement is well fulfilled, it will increase
the customer satisfaction. Otherwise, it will not decrease the
satisfaction. Second, the attractive requirement is called as
‘surprise’ or ‘know your customer’ requirement. This
requirement can directly increase the customer satisfaction if it
is fulfilled. Marketers and sales [53] believe that if we can
deliver this kind of requirement, it will impress customers and
significantly improve the customer satisfaction. Third, the
indifferent requirement is a requirement that customer does not
concentrate and it will not impress customers at all. In the
competitive industry, this requirement may be fulfilled, but
there are no any impacts to the customer satisfaction. Last, the
basic requirement is a mandatory requirement that customers
basically expect. Therefore, if this requirement is well
delivered, it will not increase the customer satisfaction.
Figure 1. A Proposed Process to Generate Test Cases Furthermore, our study reveals that the requirement can be
simply divided into two types: functional and non-functional
From the above figure, there are two test case generation requirement. Our study also presents that functional
processes: existing and proposed process. The left-hand side requirements can be categorized into two groups: domain
shows an existing process to generate test cases directly from specific requirement [5] (or known as constraints requirement)
diagrams. Meanwhile, the right-hand side proposes to add an and behavior requirement. The following shows the
additional requirement prioritization process before generating requirement classification used in this paper:
test cases. The requirement prioritization process aims to be
able to effectively handle with a large number of requirements.
The objective of this process is to prioritize and organize
requirements in an appropriate way in order to effectively
design and prepare test cases [16], [25], [37]. There are two
sub-processes: (a) classify requirements and (b) prioritize
requirements.
Our study [51], [52], [53], [54] shows that a marketing
perspective concentrates on two factors: customer’s needs and
customer satisfaction. We apply that perspective to the
requirement prioritization and propose the following:

Figure 3. Classify Requirement on Software Engineer

From the above figure, functional requirement is a
requirement that customers directly are able to provide. The
non-functional requirement is a requirement that is given
indirectly. The domain specific or constraints requirement is a
requirement relative to any constraints and business rules in the
software development. Meanwhile, the behavior requirement is
a requirement that describes a behavior of system. Once the
requirement is classified based on previous two perspectives,
the next process is to prioritize requirements based on return on
investment (or ROI) [51], [52], [53]. From business
perspective, ROI is the most important factor to assess the
important of each requirement. The following presents a
ranking tree by combining those two perspectives.

Figure 2. Classify Requirement on Marketing’s Perspective

24 http://sites.google.com/site/ijcsis/
ISSN 1947-5500
(IJCSIS) International Journal of Computer Science and Information Security,
Vol. 8, No. 6, September 2010

• CostCode is a cost of coding that is charged to customers.
This paper applies the cost-value approach to identify the
cost of coding for each requirement group (e.g. “Must-
Have”, “Should-Have”, “Could-Have” and “Wish”). The
unit is US dollar.
• EffTest is an estimated effort of testing for each
requirement. The unit is man-hours.
• CostTest is a cost of testing that is charged to customers.
The approach to identify this value is similar to
CostCode’s approach. The unit is US dollar.
In this paper, we assumed the following in order to
calculate CostCode and CostTest. Also, this paper assumes that
a standard cost for both activities is $100 per man-hours.
• A value is 1.5 of (“Must-Have”, “Should-Have”) – this
means that “Must-Have” requirements have one and half
times cost value than “Should-Have” requirements.
• A value is 3 of (“Must-Have”, “Could-Have”) – this
means that “Must-Have” requirements have three times
cost value than “Could-Have” requirements.
Figure 4. Requirement Prioritization Tree • A value is 2 of (“Should-Have”, “Could-Have”) – this
means that “Should-Have” requirements have two times
From the above figure, we give the highest priority for all
cost value than “Could-Have” requirements.
‘basic requirements due to the fact that they must be
implemented even they do not increase the customer • A value is approximately 3 of (“Could-Have”, “Wish”) –
satisfaction. We rank the lowest priority for all ‘indifferent’ this means that “Could-Have” requirements have three
requirements, because customers do not concentrate on. times cost value than “Wish” requirements.
Additionally, we prioritize both of all ‘delight’ and ‘attractive’
requirement based on ROI. In this paper, we propose to use a Therefore, the procedure of requirement prioritization
cost-value approach to weight and prioritize requirements. This process can be shortly described below:
paper proposes to use the following formula: 1. Provide estimated efforts of coding and testing for each
P(Req) = (Cost * CP) (1) requirement.
Where: 2. Assign cost value for each requirement group based on
• P is a prioritization value. the previous requirement classification (e.g. “Must-
Have”, “Should-Have”, “Could-Have” and “Wish”).
• Req is a requirement required to be prioritized.
3. Calculate a total estimated cost for coding and testing,
• Cost is a total estimated cost of coding and testing for by using the formula (2).
each requirement. 4. Define a customer priority for each requirement.
• CP is an user-defined customer priority value. This value 5. Compute a priority value for each requirement by
is in the range between 1 and 10. 10 is the highest priority using the formula (1).
and 1 is the lowest priority. This value aims to allow
customers to identify how important of each requirement 6. Prioritize requirements based on the higher priority
is from their perspective. value.

To compute the above cost for coding and testing, this Once the requirements are prioritized, the next proposed
paper proposes to apply the following formula: step is to generate test scenario and prepare test case.

Cost = (EffCode*CostCode)+(EffTest*CostTest) (2) B. Test Case Generation Technique
This section presents an automated test scenario generation
Where derived from UML Use Case diagram 2.0. The big different
between UML Use Case diagram 1.0 and 2.0 is a package
• Cost is a total estimated cost. notion that can group each use case into each package.
• EffCode is an estimated effort of coding for each The following shows an example of UML Use Case
requirement. The unit is man-hours. diagram 2.0.

25 http://sites.google.com/site/ijcsis/
ISSN 1947-5500
(IJCSIS) International Journal of Computer Science and Information Security,
Vol. 8, No. 6, September 2010

Case Case y Event ve Events ss
Id Name Rules
UC- Withdra To allow 1. Insert 1. Select (a)
001 w bank' s Card Inquiry Input
customer 2. Input 2. Select amount
s to PIN A/C Type <=
withdraw 3. Select 3. Check Outstan
money Withdraw Balance ding
from 4. Select Balanc
ATM A/C Type e
machines 5. Input (b) Fee
anywher Balance charge
e in 6. Get if using
Thailand Money differen
. 7. Get Card t ATM
machin
es
UC- Transfer To allow 1. Insert 1. Select Amoun
Figure 5. An Example of UML Use Case Diagram 2.0
002 users to Card Inquiry t <=
From the above figure, the new notion in UML Use Case transfer 2. Input 2. Select 50,000
diagram 2.0 is a package that is used for grouping each money to PIN A/C Type baht
function. There are three packages or releases. Each release other 3. Select 3. Check
contains different functional requirement. The first release banks in Transfer Balance
contains two functions: inquiry and withdraw. The second Thailand 4. Select
release is composed of: transfer own account and transfer to from all bank
other banks. The last release has only one function to support ATM 5. Select
Thai (TG) airline tickets. machines "To"
Our approach aims to generate three test suites to cover the account
above three packages while existing test case generation 6. Select
techniques do not concentrate on. The first test suite is A/C Type
developed for: inquiry and withdraw functions. The second test 7. Input
suite is used for transferring own banks and other banks. The Amount
last suite aims to a TG airline ticket support. 8. Get
Receipt
The approach is built based on Heumann’s algorithm [23]. 9. Get Card
The limitation of our approach is to ensure that all use cases are
fully dressed. The fully dressed use case is a use case with the
comprehensive of information, as follows: use case name, use The above use cases can be extracted into the following use
case number, purpose, summary, pre-condition, post-condition, case scenarios:
actors, stakeholders, basic events, alternative events, business
rules, notes, version, author and date.
TABLE II. EXTRACTED USE CASE SCENARIOS
The proposed method contains four steps, as follows: (a)
extract use case diagram (b) generate test scenario (c) prepare Scenario Id Summary Basic Scenario
test data and (d) prepare other test elements. These steps can be
shortly described as follows: Scenario-001 s
To allow bank' 1. Insert Card
customers to 2. Input PIN
1. The first step is to extract the following information withdraw money from 3. Select Withdraw
from fully dressed use cases: (a) use case number (b) ATM machines 4. Select A/C Type
purpose (c) summary (d) pre-condition (e) post- anywhere in Thailand. 5. Input Balance
condition (f) basic event and (g) alternative events. 6. Get Money
This information is called use case scenario in this 7. Get Card
paper. The example fully dressed use cases of ATM
withdraw functionality can be found as follows:

TABLE I. EXAMPLE FULLY DRESSED USE CASE

Use Use Summar Basic Alternati Busine

26 http://sites.google.com/site/ijcsis/
ISSN 1947-5500
(IJCSIS) International Journal of Computer Science and Information Security,
Vol. 8, No. 6, September 2010

Scenario-002 s
To allow bank' 1. Insert Card TS-002 s
To allow bank' 1. Insert Card
customers to 2. Input PIN customers to 2. Input PIN
withdraw money from 3. Select Inquiry withdraw money from 3. Select Inquiry
ATM machines 4. Select A/C Type ATM machines 4. Select A/C Type
anywhere in Thailand. 5. Check Balance anywhere in Thailand. 5. Check Balance
6. Select Withdraw 6. Select Withdraw
7. Select A/C Type 7. Select A/C Type
8. Input Balance 8. Input Balance
9. Get Money 9. Get Money
10. Get Card 10. Get Card
Scenario-003 To allow users to 1. Insert Card TS-003 To allow users to 1. Insert Card
transfer money to 2. Input PIN transfer money to 2. Input PIN
other banks in 3. Select Transfer other banks in 3. Select Transfer
Thailand from all 4. Select bank Thailand from all 4. Select bank
ATM machines 5. Select "To" ATM machines 5. Select "To"
account account
6. Select A/C Type 6. Select A/C Type
7. Input Amount 7. Input Amount
8. Get Receipt 8. Get Receipt
9. Get Card 9. Get Card
Scenario-004 To allow users to 1. Insert Card TS-004 To allow users to 1. Insert Card
transfer money to 2. Input PIN transfer money to 2. Input PIN
other banks in 3. Select Inquiry other banks in 3. Select Inquiry
Thailand from all 4. Select A/C Type Thailand from all 4. Select A/C Type
ATM machines 5. Check Balance ATM machines 5. Check Balance
6. Select Transfer 6. Select Transfer
7. Select bank 7. Select bank
8. Select "To" 8. Select "To"
account account
9. Select A/C Type 9. Select A/C Type
10. Input Amount 10. Input Amount
11. Get Receipt 11. Get Receipt
12. Get Card 12. Get Card

2. The second step is to automatically generate test 3. The next step is to prepare test data. This step allows to
scenarios from the previous use case scenarios [23]. manually prepare an input data for each scenario.
From the above table, we automatically generate the
following test scenarios: 4. The last step is to prepare other test elements, such as
expected output, actual output and pass / fail status

TABLE III. GENERATED TEST SCENARIOS
V. EVALUATION
Test Scenario Summary Basic Scenario The section describes the experiments design, measurement
Id metrics and results.
TS-001 s
To allow bank' 1. Insert Card
A. Experiments Design
customers to 2. Input PIN
withdraw money from 3. Select Withdraw 1. Prepare Experiment Data. Before evaluating the
ATM machines 4. Select A/C Type proposed methods and other methods, preparing
anywhere in Thailand. 5. Input Balance experiment data is required. In this step, 50
6. Get Money requirements and 50 use case scenarios are randomly
7. Get Card generated.
2. Generate Test Scenario and Test Case. A comparative
evaluation method has been made among the proposed
test scenario algorithm, Heumann’s technique Jim [23],
Ryser’s method [24], Nilawar’s algorithm [33] and the
proposed method presented in the previous section. It

27 http://sites.google.com/site/ijcsis/
ISSN 1947-5500
(IJCSIS) International Journal of Computer Science and Information Security,
Vol. 8, No. 6, September 2010

is included a prioritization requirement algorithm prior basic1, basic2, …, basicn.
to generate a set of test scenarios and test cases.
B. Measurement Metrics
3. Evaluate Results. In this step, the comparative
The section lists the measurement metrics used in the
generation methods are executed by using 50
experiment. This paper proposes to use three metrics, which
requirements and 50 use case scenarios. These methods
are: (a) size of test cases (b) total time and (c) percentage of
are also executed for 10 times in order to find out the
critical domain requirement coverage. The following describe
average percentage of critical domain requirement
the measurement in details.
coverage, a size of test cases and total generation time.
In total, there are 500 requirements and 500 use case 1. A Number of Test Cases: This is the total number of
scenarios executed in this experiment. generated test cases, expressed as a percentage, as follows:
The following tables present how to randomly generate data % Size = (# Size / # of Total Size)*100 (3)
for requirements and use case scenarios respectively. Where:
• % Size is a percentage of the number of test cases.
TABLE IV. GENERATE RANDOM REQUIREMENTS
• # of Size is a number of test cases.
Attribute Approach
• # of Total Size is the maximum number of test cases in the
Requirement Randomly generated from the following
experiment, which is assigned 1,000.
ID combination: Req + Sequence Number.
2. A Domain Specific Requirement Coverage: This is an
For example, Req1, Req2, Req3, …, indicator to identify the number of requirements covered in
ReqN. the system, particularly critical requirements, and critical
Description Randomly generated from the following domain requirements [5]. Due to the fact that one of the
combination: Des + Sequence Number goals of software testing is to verify and validate
same as Requirement ID. requirements covered by the system, this metric is a must.
Therefore, a high percentage of critical requirement
coverage is desirable.
For example, Des1, Des2, Des3, …,
DesN. It can be calculated using the following formula:
Type of Randomly selected from the following
% CRC = (# of Critical / # of Total)*100 (4)
Requirement values: Functional AND Non-Functional.
Where:
MoSCoW Randomly selected from the following
• % CRC is the percentage of critical requirement coverage.
Criteria values: Must Have (M), Should Have (S),
Could Have (C) and Won’t Have (W) • # of Critical is the number of critical requirements
Is it a critical Randomly selected from the following covered.
requirement values: True (Y) and False (N)
(Y/N)? • # of Total is the total number of requirements.
3. Total Time: This is the total number of times the
generation methods are run in the experiment. This metric
TABLE V. GENERATE RANDOM USE CASE SCENARIO is related to the time used during the testing development
phase (e.g. design test scenario and produce test case).
Attribute Approach Therefore, less time is desirable.
Use case ID Randomly generated from the
following combination: uCase + It can be calculated using the following formula:
Sequence Number. For example, Total = PTime + CTime + RTime (5)
uCase1, uCase2, …, uCasen. Where:
Purpose Randomly generated from the • Total is the total amount of times consumed by running
following combination: Pur + generation methods.
Sequence Number same as Use case
ID. For example, Pur1, Pur2, …, • PTime is the total amount of time consumed by
Purn. preparation before generating test cases.
Pre-condition Randomly generated from the • CTime is the time to compile source code / binary code in
following combination: pCon + order to execute the program.
Sequence Number same as Use case
ID. For example, pCon1, pCon2, …, • RTime is the total time to run the program under this
pConn. experiment.
Basic Scenario Randomly generated from the
following combination: uCase +
Sequence Number. For example,

28 http://sites.google.com/site/ijcsis/
ISSN 1947-5500
(IJCSIS) International Journal of Computer Science and Information Security,
Vol. 8, No. 6, September 2010

C. Results and Discussion For an ability to cover critical domain specific requirement,
This section discusses an evaluation result of the above the maximum and minimum percentage is 53.20% and 19%.
experiment. This section presents a graph that compares the The different value is 34.2%. The interval value is 6.84.
above proposed method to other three existing test case Therefore, it can be determined as follows: 5-Excellent (since
generation techniques, based on the following measurements: 46.36% to 53.2%), 4-Very good (between 39.52% and
(a) size of test cases (b) critical domain coverage and (c) total 46.36%), 3-Good (between 32.68% and 39.52%), 2-Normal
time. Those three techniques are: (a) Heumman’s method (b) (between 25.84% and 32.68%) and 1-Poor (from 19% to
Ryser’s work and (c) Nilawar’s approach. There are two 25.84%).
dimensions in the following graph: (a) horizontal and (b) For a total time, the maximum and minimum percentage is
vertical axis. The horizontal represents three measurements 31.82% and 30.20%. The different between maximum and
whereas the vertical axis represents the percentage value. minimum value is 1.62%. An interval value is equal to a result
of dividing the different values by 5. As a result, the interval
value is 0.324. Thus, it can be determined as follows: 5-
Excellent (since 30.2% to 30.524%), 4-Very good (between
30.524% and 30.848%), 3-Good (between 30.848% and
31.172%), 2-Normal (between 31.172% and 31.496%) and 1-
Poor (from 31.496% to 31.82%).
Therefore, the experiment result of those comparative
methods can be shown below:

TABLE VI. A COMPARISON OF TEST CASE REDUCTION METHODS
Algorithm A Cover Total
Number of Critical Time
Test Cases Domain
Specific Req.
Heumann’s 1 1 5
Method
Figure 6. An Evaluation Result of Test Generation Methods Ryser’s Method 1 1 1

The above graph shows that the above proposed method Nilawar’s 1 1 1
generates the smallest set of test cases. It is calculated as Method
80.80% where as the other techniques is computed over 97%.
Those techniques generated a bigger set of test cases, than a set Our Proposed 5 5 5
generated by the proposed method. The literature review Method
reveals that the smaller set of test cases is desirable. Also, the
graph shows that the proposed method consumes the least total
time during a generation process, comparing to other In the conclusion, the proposed method is the best to
techniques. It used only 30.20%, which is slightly less than generate the smallest size of test cases with the maximum of
others. Finally, the graph presents that the proposed method is critical domain coverage and the least time consumed in the
the best techniques to coverage critical domains. Its percentage generation process.
is much greater than other techniques’ percentage, over 30%.
From the above figure, this study determines and ranks the VI. CONCLUSION
above comparative methods into five ranking: 5-Excellent, 4- In this paper, we introduced a new test case generation
Very good, 3-Good, 2-Normal and 1-Poor. This study uses a method and process, with an additional requirement
maximum and minimum value to find an interval value for prioritization process. The approach inserts an additional
ranking those methods. process to ensure that all domain specific requirements are
captured during the test case generation. Also, the approach is
For a number of test cases, the maximum and minimum developed to minimize a number of test cases in order to be
percentage is 98% and 80.80%. The different between able to select suitable test cases for execution. Additionally, we
maximum and minimum value is 17.2%. An interval value is proposed an automated approach to generate test cases from
equal to a result of dividing the different values by 5. As a fully described UML use cases, version 2.0. Our generated
result, the interval value is approximately 3.4. Thus, it can be method can generate many test suites derived from UML Use
determined as follows: 5-Excellent (since 80.80% to 84.2%), 4- Case diagram, 2.0. Existing test case generation methods
Very good (between 84.2% and 87.6%), 3-Good (between generate only a large single test suite that contains a lot of
87.6% and 91%), 2-Normal (between 91% and 94.4%) and 1- numbers of test cases.
Poor (from 94.4% to 97.8%).
Furthermore, we conducted an evaluation experiment with
a random requirements and fully described use cases. Our

29 http://sites.google.com/site/ijcsis/
ISSN 1947-5500
(IJCSIS) International Journal of Computer Science and Information Security,
Vol. 8, No. 6, September 2010

evaluation result reveals that the proposed method is the most [15] David C. Kung, Chien-Hung Liu and Pei Hsia “An Object-
recommendation automated test case generation methods for Oriented Web Test Model for Testing Web Applications.” In
maximizing critical domain requirement coverage. Also, the Proceedings of the First Asia-Pacific Conference on Quality
result present that the proposed method is one of best methods Software (APAQS’00), page 111, Los Alamitos, CA, 2000.
to minimize a number of test cases. [16] Donald Firesmith “Prioritizing Requirements. Journal of Object
Technology”, Vol.3, No8, 2004.
The future research, we plan to enhance an ability to
prioritize requirements and conduct a large experiment for a [17] D. Harel “On visual formalisms.” Communications of the ACM,
large system development vol. 31, no. 5, pp. 514-530, 1988.
[18] D. Harel. “Statecharts: A Visual Formulation for Complex
REFERENCES System.” Sci.Comput. Program. 8(3):232-274, 1987.
[1] Ahl, V. “An Experimental Comparison of Five Prioritization [19] Flippo Ricca and Paolo Tonella “Analysis and Testing of Web
Methods” Master' Thesis, School of Engineering, Blekinge
s Applications.” Proc. of the 23rd International Conference on
Institute of Technology, Ronneby, Sweden, 2005. Software Engineering, Toronto, Ontario, Canada. pp.25-34,
2001.
[2] Alessandra Cavarra, Charles Crichton, Jim Davies, Alan
Hartman, Thierry Jeron and Laurent Mounier. “Using UML for [20] Harel, D. “Statecharts: a visual formalism for complex system.”
Automatic Test Generation.” Oxford University Computing Science of Computer Programming, v. 8, p. 231-274, 1987.
Laboratory, Tools and Algorithms for the Construction and [21] Hassan Reza, Kirk Ogaard and Amarnath Malge “A Model
Analysis of Systems, TACAS' 2000, 2000. Based Testing Technique to Test Web Applications Using
[3] Amaral. “A.S.M.S. Test case generation of systems specified in Statecharts.” Fifth International Conference on Information
Statecharts.” M.S. thesis – Laboratory of Computing and Technology, 2008.
Applied Mathematics, INPE, Brazil, 2006. [22] Ibrahim K. El-Far and James A. Whittaker “Model-based
[4] Annelises A. Andrews, Jeff Offutt and Roger T. Alexander. Software Testing”, 2000.
“Testing Web Applications. Software and Systems Modeling.”, [23] Jim Heumann “Generating Test Cases From Use Cases.”
2004. Rational Software, 2001.
[5] Avik Sinha, Ph.D and Dr. Carol S. Smidts. “Domain Specific [24] Johannes Ryser and Martin Glinz “SCENT: A Method
Test Case Generation Using Higher Ordered Typed Languages Employing Scenarios to Systematically Derive Test Cases for
fro Specification.” Ph. D. Dissertation, 2005. System Test”, 2000.
[6] A. Bertolino. “Software Testing Research and Practice.” 10th [25] Karl E. Wiegers “First Things First: Prioritizing Requirements.”
International Workshop on Abstract State Machines Published in Software Development, 1999.
(ASM' 2003), Taormina, Italy, 2003.
[26] Karlsson, J. “Software Requirements Prioritizing.” Proceedings
[7] A.Z. Javed, P.A. Strooper and G.N. Watson “Automated of the Second International Conference on Requirements
Generation of Test Cases Using Model-Driven Architecture.” Engineering (ICRE' 96). Colorado Springs, CO, April 15-18,
Second International Workshop on Automation of Software Test 1996. Los Alamitos, CA: IEEE Computer Society, p 110-116,
(AST’07), 2007. 1996.
[8] Beck, K. & Andres, C. “Extreme Programming Explained: [27] Karlsson, J. “Towards a Strategy for Software Requirements
Embrace Change", 2nd ed. Boston, MA: Addison-Wesley, 2004. Selection.” Licentiate. Thesis 513, Linköping University, 1995.
[9] Boehm, B. & Ross, R. “Theory-W Software Project [28] Karlsson, J. & Ryan, K. “A Cost-Value Approach for
Management: Principles and Examples.” IEEE Transactions on Prioritizing Requirements.” IEEE Software September/October,
Software Engineering 15, 4: 902-916, 1989. p67-75, 1997.
[10] B.M. Subraya, S.V. Subrahmanya “Object driven performance [29] Leffingwell, D. & Widrig, D. “Managing Software
testing in Web applications.” in: Proceedings of the First Asia- Requirements: A Use Case Approach”, 2nd ed. Boston, MA:
00),
Pacific Conference on Quality Software (APAQS' pp. 17-26, Addison-Wesley, 2003.
Hong Kong, China, 2000.
[30] Leslie M. Tierstein “Managing a Designer / 2000 Project.”
[11] Chien-Hung Liu, David C. Kung, Pei Hsia and Chih-Tung Hsu NYOUG Fall’97 Conference, 1997.
“Object-Based Data Flow Testing of Web Applications.”
[31] L. Brim, I. Cerna, P. Varekova, and B. Zimmerova
Proceedings of the First Asia-Pacific Conference on Quality
“Component-interaction automata as a verification oriented
00),
Software (APAQS' pp. 7-16, Hong Kong, China, 2000.
component-based system specification.” In: Proceedings
[12] C.H. Liu, D.C. Kung, P. Hsia, C.T. Hsu “Structural testing of 05),
(SAVCBS' pp. 31-38, Lisbon, Portugal, 2005.
Web applications.” in: Proceedings of 11th International
[32] Mahnaz Shams, Diwakar Krishnamurthy and Behrouz Far “A
Symposium on Software Reliability Engineering (ISSRE 2000),
Model-Based Approach for Testing the Performance of Web
pp. 84-96, 2000.
Applications.” Proceedings of the Third International Workshop
[13] Davis, A. “The Art of Requirements Triage.” IEEE Computer on Software Quality Assurance (SOQUA’06), 2006.
36, 3 p: 42-49, 2003.
[33] Manish Nilawar and Dr. Sergiu Dascalu “A UML-Based
[14] Davis, A. “Just Enough Requirements Management: Where Approach for Testing Web Applications.” Master of Science
Software Development Meets Marketing.” New York: Dorset with major in Computer Science, University of Nevada, Reno,
House (ISBN 0-932633-64-1), 2005. 2003.

30 http://sites.google.com/site/ijcsis/
ISSN 1947-5500
(IJCSIS) International Journal of Computer Science and Information Security,
Vol. 8, No. 6, September 2010

[34] Moisiadis, F. “Prioritising Scenario Evolution.” International International Transactions in Operational Research, 9, 321-336,
Conference on Requirements Engineering (ICRE 2000), 2000. 2002.
[35] Moisiadis, F. “A Requirements Prioritisation Tool.” 6th [45] Wiegers, K. “E. Software Requirements”, 2nd ed. Redmond,
Australian Workshop on Requirements Engineering (AWRE WA: Microsoft Press, 2003.
2001). Sydney, Australia, 2001. [46] Xiaoping Jia, Hongming Liu and Lizhang Qin “Formal
[36] M. Prasanna S.N. Sivanandam R.Venkatesan R.Sundarrajan “A Structured Specification for Web Application Testing”. Proc. of
Survey on Automatic Test Case Generation.” Academic Open the 2003 Midwest Software Engineering Conference
Internet Journal, 2005. 03).
(MSEC' Chicago, IL, USA. pp.88-97, 2003.
[37] Nancy R. Mead “Requirements Prioritization Introduction.” [47] Yang, J.T., Huang, J.L., Wang, F.J. and Chu, W.C.
Software Engineering Institute, Carnegie Mellon University, “Constructing an object-oriented architecture for Web
2008. application testing.” Journal of Information Science and
[38] Park, J.; Port, D.; & Boehm B. “Supporting Distributed Engineering 18, 59-84, 2002.
Collaborative Prioritization for Win-Win Requirements Capture [48] Ye Wu and Jeff Offutt “Modeling and Testing Web-based
and Negotiation.” Proceedings of the International Third World Applications”, 2002.
Multi-conference on Systemics, Cybernetics and Informatics [49] Ye Wu, Jeff Offutt and Xiaochen “Modeling and Testing of
99)
(SCI' Vol. 2. 578-584, Orlando, FL, July 31-August 4, 1999. Dynamic Aspects of Web Applications, Submitted for
Orlando, FL: International Institute of Informatics and Systemic publication.” Technical Report ISE-TR-04-01,
(IIIS), 1999. www.ise.gmu.edu/techreps/, 2004.
[39] Rajib “Software Test Metric.” QCON, 2006. [50] Zhu, H., Hall, P., May, J. “Software Unit Test Coverage and
[40] Robert Nilsson, Jeff Offutt and Jonas Mellin “Test Case Adequacy.” ACM Comp. Survey 29(4), pp 366~427, 1997.
Generation for Mutation-based Testing of Timeliness.”, 2006. [51] Kano Noriaki, Nobuhiku Seraku, Fumio Takahashi,Shinichi
[41] Saaty, T. L. “The Analytic Hierarchy Process.” New York, NY: Tsuji. “Attractive Quality and Must-Be Quality.” Journal of the
McGraw-Hill, 1980. Japanese Society for Quality Control. 14(2), pp 39~48, 1984.
[42] Shengbo Chen, Huaikou Miao, Zhongsheng Qian “Automatic [52] Cadotte, Ernest R., Turgeon, Normand “Dissatisfiers and
Generating Test Cases for Testing Web Applications.” Satisfiers: Suggestions from Consumer Complaints and
International Conference on Computational Intelligence and Compliments.” Journal of Consumer Satisfaction,
Security Workshops, 2007. Dissatisfactions and Complaining Behavior. 1, pp 74~79, 1988.
[43] Valdivino Santiago, Ana Silvia Martins do Amaral, N.L. [53] Brandt, D. Randall “How service marketers can identify value-
Vijaykumar, Maria de Fatima, Mattiello-Francisco, Eliane enhancing service elements.” Journal of Services Marketing.
Martins and Odnei Cuesta Lopes “A Practical Approach for 2(3), pp 35~41, 1988.
Automated Test Case Generation using Statecharts”, 2006. [54] Herzberg, Frederick, Mausner, B., Snyderman, B.B. “The
[44] Vijaykumar, N. L.; Carvalho, S. V.; Abdurahiman, V. “On motivation to work.” New York: Wiley, 2nd edition, 1959.
proposing Statecharts to specify performance models.”

31 http://sites.google.com/site/ijcsis/
ISSN 1947-5500