Docstoc

An introduction to the theory an

Document Sample
An introduction to the theory an Powered By Docstoc
					An introduction to the theory and
   practice of domain testing
             VISTACON
           HCMC, VIETNAM
           September, 2010


             Cem Kaner
         Sowmya Padmanabhan




                                    1
                                 Overview
Domain testing is the most widely taught (and perhaps the most widely used)
software testing technique.
Practitioners often study the simplest cases of domain testing under two other
names, "boundary testing" and "equivalence class analysis." That simplified
form applies
  – Only to tests of input variables
  – Only when tested at the system level
  – Only when tested one at a time
  – Only when tested in a very superficial way
In competent practice, and in the research literature, all of these limitations are
often set aside, but there is very little practical teaching of the more general
approaches.
Sowmya Padmanabhan did her M.Sc. thesis research on this. Sowmya, Doug
Hoffman and I are co-authoring the Domain Testing Workbook.
This talk will present a worksheet that we've developed for planning and
creating domain tests of individual variables or multiple variables that are
independent or linearly or nonlinearly related. I'll brush on some of the theory
underlying the approach, but mainly I want to present some of the lessons that
this work brought home to me about skilled (contrasted with inexperienced or
unskilled) domain testing.
                                                                                      2
    Understanding domain testing


       In domain testing,
          we partition a domain
            into sub-domains
              (equivalence classes)
                and then test using
                  values from each
                    sub-domain.



3
                  Notice how I format this
Variable       Valid case          Invalid case        Boundaries &    Notes
               equivalence class   equivalence class   special cases
X              0 – 100                                 0
                                                       100
                                   <0                  -1
                                   > 100               101


    – One line per test




                                                                               4
                             Domain Testing
Variable          Valid case          Invalid case        Boundaries &    Notes
                  equivalence class   equivalence class   special cases
X                 0 – 100                                 0
                                                          100
                                      <0                  -1
                                      > 100               101


    – Each test maps clearly to a description of the class that it represents
    – I start by focusing on classes directly tied to the variable's primary
      dimension.
      • I'll talk more about primary and secondary dimensions later, but for now
         note that
         – my first-draft chart works on X's number line,
         – There are no letters, no blank fields, no timeouts, no X-input-too-long
           overflows.
                                                                                  5
                             Domain Testing
Variable          Valid case          Invalid case        Boundaries &    Notes
                  equivalence class   equivalence class   special cases
X                                     <0                  -1
                  0 – 100                                 0
                                                          100
                                      > 100               101


    – I prefer "valid" values first, but some people prefer this order instead.




                                                                                  6
    Boundary table as a test plan component
    – Makes the reasoning obvious.
    – Makes the relationships between test cases fairly obvious.
    – Expected results are pretty obvious.
    – Several tests on one page.
    – Can delegate it and have tester check off what was done.
      Provides some limited opportunity for tracking.
    – Not much room for status.




7
     Building table like these (in practice)
    – Relatively few programs will come to you with all fields
      fully specified. Therefore, you should expect to learn what
      variables exist and their definitions over time.
    – To build an equivalence class analysis over time, put the
      information into a spreadsheet. Start by listing variables. Add
      information about them as you obtain it.
    – The table should eventually contain all variables. This means,
      all input variables, all output variables, and any intermediate
      variables that you can observe.
    – In practice, most tables that I’ve seen are incomplete. The
      best ones that I’ve seen list all the variables and add detail for
      critical variables.

8
           These tables are limited
– Most of these tables consider only input variables
  • People don't buy programs to enter data
  • They buy programs to analyze data and give results
  • Therefore, we need to test output variables
– These tables focus on individual variables
  • Individual variables are a start, but we need to test
    combinations of several variables together




                                                            9
                        Exercise 1
 – S is an input string. It will hold someone's name
 – You can enter letters, numbers or spaces into S
 – The program doesn't care what S is but it cannot be more
   than 30 characters
Do an equivalence class / boundary analysis on S.
 – Ignore secondary dimensions, for now

When you are done, please pass the chart to the person on your
left, and grade the chart of the person on your right.




                                                                 10
                        Exercise 2
–   A, B, and C are integers.
–   You can enter data into A and B, but not C
–   C is calculated from A and B
–   Do an equivalence class analysis on C, showing what values
    of A and B you need to test with, to test C.

– When you are done, please pass the chart to the person on
  your right, and grade the chart of the person on your left.




                                                                 11
                              Research
Cem Kaner & Sowmya Padmanabhan, "Practice and transfer of learning in the
teaching of software testing," Conference on Software Engineering Education
& Training, Dublin, July 2007.
  – http://www.kaner.com/pdfs/kanerPadmanabhanPractice.pdf
  – http://www.kaner.com/pdfs/CSEETdomain2007.pdf (slides)

Padmanabhan, S. (2004). Domain Testing: Divide & Conquer. Unpublished
M.Sc. Thesis., Florida Institute of Technology, Melbourne, FL.
http://www.testingeducation.org/a/DTD&C.pdf.

  – We tried teaching this technique via exemplars but had mixed results.
  – We ultimately concluded that we needed to embed the exemplars in a
    stronger cognitive structure.



                                                                            12
           Four central questions
– What domain are we testing?
– How do we determine how to group values of a
  variable(s) into equivalence classes?
– How do we determine which members of each class
  to test?
– How do we determine whether the program passed
  or failed the test?




                                                13
               Domains

  Input                         Output
 domain         F(x)            domain



          Functional testing:

We treat the program as a function that
     transforms inputs to outputs.
                (Howden)

                                          14
                     Domains
– Single-input tests check input filters:
  • Is this necessary at the system level?
  • Should we test the filter at the unit level?

– Output domain is more challenging
   K=I*J
   I, J, K are integers
   I, J are input variables. K is the output.
   Test K.
  We'll look at this example again later

                                                   15
           Four central questions
– What domain are we testing?
–How do we determine how to group
 values of a variable(s) into equivalence
 classes?
– How do we determine which members of each class
  to test?
– How do we determine whether the program passed
  or failed the test?


                                                16
                     Partitioning
How do we determine how to group variables into
equivalence classes?
 – Partitioning a set means splitting it into
   nonoverlapping subsets.
 – Disjunction (nonoverlapping) is important for some
   models, but practitioners often work with
   overlapping sets (Kaner, et al., 1993; Weyuker &
   Jeng, 1991). For our practical purposes, partitioning
   means dividing a set of possible values of a variable
   into subsets that don’t overlap (or not much).
 – Partitioning usually splits a set into equivalence
   classes.                                             17
                         Partitioning
Intuitive or subjective equivalence: two test values are
equivalent if they are so similar to each other that it seems
pointless to test both.
Intuitive definitions (e.g., Craig & Jaskiel, 2002; Kit, 1995)
appear obvious. For example,
  – “Basically we are identifying inputs that are treated the same
    way by the system and produce the same results” Craig &
    Jaskiel (2002, p. 162).
  – “If you expect the same result from two tests, you consider
    them equivalent. A group of tests forms an equivalence class
    if you believe they all test the same thing … Two people
    analyzing a program will come up with a different list of
    equivalence classes. This is a subjective process” Kaner
    (1988, p. 81).

                                                                 18
                        Partitioning
Specified equivalence: two test values are equivalent if the
specification says that the program handles them in the same way.
  – Burnstein, 2003; DeMillo, McCracken, Martin, &
    Passafiume, 1987; Myers, 1979.
  – Challenges:
    • Testers complain about missing specifications may spend
      enormous time writing specifications
    • Focus is on things that were specified, but there might be
      more bugs in the features that were un(der)specified




                                                               19
                      Partitioning
– Analysis of the code that defines or uses the variables: The
  intuitive and specified-equivalence approaches focus on the
  program’s variables, primarily from a black box perspective.
  Some authors (e.g., Howden, 1980) suggest ways to partition
  data or identify boundary values by reading the code.
– Same path: Descriptions that appear primarily in the research
  literature say that two values of a variable are equivalent if
  they cause the program to take the same branch or the same
  (sub)path. (Beizer, 1995; Clarke, et al., 1982; White, 1981)




                                                              20
                         Partitioning
Equivalent Paths: two test values are equivalent if they would
drive the program down the same path (e.g. execute the same
branch of an IF). (Beizer, 1995; Clarke, et al., 1982; White, 1981)
  – Tester should be a programmer
  – Tester should design tests from the code
  – Some authors claim that a complete domain test will yield a
    complete branch coverage.
  – No basis for picking one member of the class over another.
  – Two values might take program down same path but have
    very different subsequent effects (e.g. timeout or not timeout
    a subsequent program; or e.g. word processor's interpretation
    and output may be the same but may yield different
    interpretations / results from different printers.)

                                                                 21
                        Partitioning
Risk-based equivalence:
 – Several early authors pointed out that domain tests target
    specific types of errors (Binder, 2000; Boyer, 1975; Clarke,
    1976; White, 1981).
 – Weyuker & Ostrand (1980; see also Jeng & Weyuker, 1989,
    p. 38) proposed that the ideal partitioning should be
    revealing, meaning that it
      “divides the domain into disjoint subdomains with the
      property that within each subdomain, either the program
      produces the correct answer for every element or the
      program produces an incorrect answer for every element.”


                                                                   22
                      Partitioning
We adopt a subjective, risk-based definition of
equivalence:
 – Two tests are equivalent, relative to a potential error,
   if both should be error revealing (both could trigger
   the error) or neither should be revealing (Weyuker &
   Ostrand, 1980; Weyuker & Jeng, 1991 ).
 – The same test might be error-revealing relative to one
   potential error and not revealing relative to another.
 – Given the same variable under analysis, two testers
   are likely to imagine different lists of potential errors
   because of their different experience history.
                                                          23
           Four central questions
– What domain are we testing?
– How do we determine how to group values of a
  variable(s) into equivalence classes?
–How do we determine which members
 of each class to test?
– How do we determine whether the program passed
  or failed the test?




                                                   24
                                Selection
Suppose that our program design is:
   – INPUT < 10                              result: Error message
   – 10 <= INPUT < 25                        result: Print "hello"
   – 25 <=INPUT                              result: Error message
Some error types
   – Program doesn't like numbers
         • Any number will do
     –   Inequalities mis-specified (e.g. INPUT <= 25 instead of < 25)
         • Detect only at boundary
     –   Boundary value mistyped (e.g. INPUT < 52, transposition error)
         • Detect at boundary and any other value that will be
           handled incorrectly
Boundary values (here, test at 25) catch all 3 errors
Non-boundary values (consider 53) may catch only 1 of the 3 errors

25
                       Selection
How do we determine which members of each class to
test?
  – Boundaries are representatives of the equivalence
    classes we sample them from. They're more likely to
    expose an error than other class members, so they're
    better representatives.
  – “A best representative of an equivalence class is a
    value that is at least as likely as any other value in
    the class to expose an error.” (Kaner & Bach, 2003,
    p. 37)
  – Generalizing from boundaries to best
    representatives is useful for analyzing nonordered
    spaces, such as printers (Kaner, et al., 1993).
                                                         26
   Selection is a stratified sampling problem
Domain testing is an instance of stratified sampling
   In statistics, stratified sampling is a method of sampling from a
   population.
   When sub-populations vary considerably, it is advantageous to sample
   each subpopulation (stratum) independently. Stratification is the process
   of grouping members of the population into relatively homogeneous
   subgroups before sampling. The strata should be mutually exclusive:
   every element in the population must be assigned to only one stratum.
   The strata should also be collectively exhaustive: no population element
   can be excluded. Then random or systematic sampling is applied within
   each stratum. This often improves the representativeness of the sample
   by reducing sampling error.
                                                 Wikipedia: Stratified Sampling
 – Polling looks for typical representatives
 – Testing looks for “best” representatives (most likely to yield
                                                                              27
    failure or other new information)
           Four central questions
– What domain are we testing?
– How do we determine how to group values of a
  variable(s) into equivalence classes?
– How do we determine which members of each class
  to test?
–How do we determine whether the
 program passed or failed the test?


                                                28
                      Evaluation
How do we determine whether the program passed or
failed the test?
  – Stunningly little guidance on this in the domain
    testing literature
  – The answer is generally treated as obvious, but it is
    not
  – We did not buy the program in order to segregate
    100 from 101 for the variable X. We did not buy the
    program to obtain an input filter.


                                                        29
                    Evaluation
How do we determine whether the program passed or
failed the test?
  – We bought the program to obtain benefits.
  – We enter an X value to obtain benefits.
  – We test a value of X
    • Against the filter
    • Against any use of X later in the program.

      Failure to consider consequences with this
   technique is our students' most common failure.
       It is widespread among practitioners too.
                                                     30
                    Our structure for domain testing
Here is a list of several tasks that people often do as part of a domain test. We organized the book’s chapters around this list
because it seems to us to put the tasks into a logical order.
For any particular product or variable, you might skip several of these tasks. Or, you might do tasks in a different order than we list
here. This is an inventory, not a control structure.

A.   Identify the potentially interesting variables.
B.   Identify the variable(s) you can analyze now.
C.   Determine the variable’s primary dimension.
D.   Determine the variable’s type or scale.
E.   Determine whether the variable’s values can be ordered (smallest to largest)
F.   Partition (create equivalence classes):
     •    If the dimension is ordered, determine the sub-ranges and transition points.
     •    If the dimension is not ordered, determine what “similar” means for this variable, and base partitioning on that.
G.   Lay out the analysis in a table that shows the partitions and best representatives (or boundary cases) for each partition.
H.   Identify secondary dimensions and analyze each in the classical way.
I.   Generalize the analysis to multidimensional variables.
J.   Linearize the domain (if possible).
K.   Identify constraints among the variables
L.   Identify and test variables that hold results (output variables).
M.   Evaluate how the program uses the value of this variable.
N.   Identify additional potentially-related variables for combination testing.
O.   Create combination tests for multidimensional or related variables.
P.   Imagine risks that don’t necessarily map to an obvious dimension.
Q.   Identify and list unanalyzed variables. Gather information for later analysis.
R.   Summarize your analysis with a risk/equivalence table.
                                                                                                                                    31
        Our structure for domain testing
A. Identify the potentially interesting variables.
B. Identify the variable(s) you can analyze now.
 – Frequently, we notice more variables than we can analyze
    today.
 – Identify the ones you’re going to work with now.
 – Note the others and add details when you get them (or when
    you go looking for them).




                                                            32
                          Exercise 3
SunTrust issues Visa credit cards with credit limits in the range of
$400 to $40000. A customer is not to be approved for credit limits
outside this range. A customer can apply for the card using an
online application form in which one of the fields requires that
the customer type in his/her desired credit limit.
  – Identify the variables
  – Do the domain analysis on as many of these variables as you
    think are appropriate.
  – Explain why you restricted your analysis to the variable(s)
    that you did.




                                                                  33
          Our structure for domain testing
C. Determine the variable’s primary dimension.
 – If we’re trying to put a number into X between 0 and 100, the
    primary dimension is a number line that includes [0, 100].
 – To figure out the primary dimension of a variable, ask what
    the variable is for. What range of values will it contain if we
    only give it useful values? That range lies along the primary
    dimension.
 – Sometimes, this is the only dimension
    • If X is stored in a byte and is read by a function that will
      interpret the bit pattern in that byte as a signed integer, then
      no matter what we put into X, what gets read is an integer
      between -128 and 127.
 – Other times, there are possibilities outside of this dimension.
    We’ll talk about these as secondary dimensions.
                                                                    34
         Our structure for domain testing
D. Determine the variable’s type or scale.
   We will analyze
   • integers
       very differently from
   • strings.




                                             35
                        Exercise 4
What are the primary dimension and scale of:

    The page setup function of a text editor allows a user to
    set the width of the page in the range of 1 to 56 inches.
    The page width input field will accept (and remember) up
    to 30 places after the decimal point.



What are the boundaries between valid and invalid input?




                                                            36
                        Exercise 5
What are the primary dimension and scale of:

    An ATM allows withdrawals of cash in amounts of $20
    increments from $20 to $200 (inclusive).



What are the boundaries between valid and invalid input?




                                                           37
                        Exercise 6
What are the primary dimension and scale of:

    A StudentLastName field must start with an alphabetic
    character (upper or lower case). Subsequent characters
    must be letters, numbers, or spaces.



What are the boundaries between valid and invalid input?




                                                             38
                  Examples of ordered sets
     – ranges of numbers                – size of the sum of
     – character codes                    variables, or of some other
     – how many times                     computed value (think
       something is done                  binary and think digits)
       • (e.g. shareware limit on       – size of a number that you
         number of uses of a product)     enter (number of digits) or
       • (e.g. how many times you can     size of a character string
         do it before you run out of    – size of a concatenated
         memory)
                                          string
     – how many names in a              – size of a path specification
       mailing list, records in a
       database, variables in a         – size of a file name
       spreadsheet, bookmarks,          – size (in characters) of a
       abbreviations                      document


39
                  Examples of ordered sets
     – size of a file (note special     – amount of available memory
       values such as exactly 64K,        (> 128 meg, > 640K, etc.)
       exactly 512 bytes, etc.)         – visual resolution, size of
     – size of the document on the        screen, number of colors
       page (compared to page           – operating system version
       margins) (across different       – variations within a group of
       page margins, page sizes)          “compatible” printers, sound
                                          cards, modems, etc.
     – size of a document on a page,    – equivalent event times (when
       in terms of the memory             something happens)
       requirements for the page.       – timing: how long between
       This might just be in terms of     event A and event B (and in
       resolution x page size, but it     which order--races)
       may be more complex if we        – length of time after a timeout
       have compression.                  (from JUST before to way
     – equivalent output events           after) -- what events are
       (such as printing documents)       important?

40
                 Examples of ordered sets
     – speed of data entry (time     – transitions between
       between keystrokes,             algorithms (optimizations)
       menus, etc.)                    (different ways to compute
     – speed of input--handling of     a function)
       concurrent events             – most recent event, first
     – number of devices               event
       connected / active            – input or output intensity
     – system resources                (voltage)
       consumed / available (also,   – speed / extent of voltage
       handles, stack space, etc.)     transition (e.g. from very
     – date and time                   soft to very loud sound)



41
        Our structure for domain testing
E. Determine whether the variable’s values can be ordered
   (smallest to largest).
 – The classic examples are of ordered sets




                                                            42
                        Exercise 7
What is the proper ordering for this variable?:

     An ATM allows withdrawals of cash in amounts of $20
    increments from $20 to $200 (inclusive).



What are the boundaries between valid and invalid input?




                                                           43
                        Exercise 8
What is the proper ordering for this variable?:

    A StudentLastName field must start with an alphabetic
    character (upper or lower case). Subsequent characters
    must be letters, numbers, or spaces.



What are the boundaries between valid and invalid input?




                                                             44
         Our structure for domain testing
F. Partition (create equivalence classes):
    • If the dimension is ordered, determine the sub-ranges and
      transition points.
    • If the dimension is not ordered, determine what “similar”
      means for this variable, and base partitioning on that.
  – We partitioned the values of X into three “equivalence
    classes:”
    • Too small
    • Just right
    • Too big
  – For partitioning of non-orderable sets, see the Printers
    example in Kaner / Falk / Nguyen, Testing Computer
    Software

                                                                  45
          Examples of non-ordered sets
Here are examples of variables that don't fit the traditional mold
for equivalence classes but which have enough values that we
will have to sample from them. What are the boundary cases?
   – Membership in a common group
     • Such as employees vs. non-employees.
     • Such as workers who are full-time or part-time or contract.
   – Equivalent hardware
     • such as compatible modems, video cards, routers
   – Equivalent output events
     • perhaps any report will do to answer a simple the question:
       Will the program print reports?
   – Equivalent operating environments
46
     • such as French & English versions of Windows
           Our structure for domain testing
G. Lay out the analysis in a table that shows the partitions
and best representatives (or boundary cases) for each
partition.
Variable     Valid case          Invalid case        Boundaries &    Notes
             equivalence class   equivalence class   special cases
X            0 – 100                                 0
                                                     100
                                 <0                  -1
                                 > 100               101




Note that this shows only the primary dimension.

                                                                             47
                           Exercise 9
Partition this variable and lay out the analysis in a table that
shows the partitions and the best representatives for each
partition:

    The page setup function of a text editor allows a user to
    set the width of the page in the range of 1 to 56 inches.
    The page width input field will accept (and remember) up
    to 30 places after the decimal point.




                                                                   48
                       Exercise 10
Analyze this using steps A through G:

(a) FoodVan delivers groceries to customers who order food
    over the Net.
    – To decide whether to buy more vans, FoodVan tracks
        the number of customers who call for a van.
    – A clerk enters the number of calls into a database
        each day.
    – Based on previous experience, the database is set to
        challenge (ask, “Are you sure?”) any number greater
        than 400 calls.


49
                                 Exercise
A.       Identify the potentially interesting variables.
B.       Identify the variable(s) you can analyze now.
C.       Determine the variable’s primary dimension.
D.       Determine the variable’s type or scale.
E.       Determine whether the variable’s values can be ordered (smallest to
         largest)
F.       Partition (create equivalence classes):
     •    If the dimension is ordered, determine the sub-ranges and
          transition points.
     •    If the dimension is not ordered, determine what “similar” means
          for this variable, and base partitioning on that.
G.       Lay out the analysis in a table that shows the partitions and best
         representatives (or boundary cases) for each partition.
50
                     Exercise 10 (b)
Continued from Exercise 10(a).
Analyze this using steps A through G:

     FoodVan schedules drivers one day in advance. To be
     eligible for an assignment, a driver must have special
     permission or she must have driven within 30 days of the
     shift she will be assigned to.




51
                                 Exercise
A.       Identify the potentially interesting variables.
B.       Identify the variable(s) you can analyze now.
C.       Determine the variable’s primary dimension.
D.       Determine the variable’s type or scale.
E.       Determine whether the variable’s values can be ordered (smallest to
         largest)
F.       Partition (create equivalence classes):
     •    If the dimension is ordered, determine the sub-ranges and
          transition points.
     •    If the dimension is not ordered, determine what “similar” means
          for this variable, and base partitioning on that.
G.       Lay out the analysis in a table that shows the partitions and best
         representatives (or boundary cases) for each partition.
52
           Understanding domain testing
As you just saw in the last example, one of the underlying risks
addressed by domain testing is ambiguity. Interpretation of the
specification is often most difficult for the boundary cases.
This is one of the key reasons that we test equivalence classes at
their boundaries rather than at random “equivalent” points inside
the set. (Read Hamlet & Taylor, 1988; Ostrand & Balcer, 1988.)




53
          Our structure for domain testing
H. Identify secondary dimensions and analyze each in the classical
way.
 – This is where we consider other data types and other sources of
    variation that may affect the handling of this variable
    • Non-integer input
    • Non-numeric input
    • Buffer-attackingly large or small values
    • Time
 – Weinberg/Myers included these dimensions in their presentations
 – Many others (Binder, academics) ignored them completely (e.g.
    Binder’s treatment of Triangle, over 100 tests, all on primary
    dimension)
 – No other discussions that we’ve seen that distinguish primary from
    secondary (what the variable/feature is for versus what else might
    happen to it.)                                                     54
          Our structure for domain testing
I. Generalize the analysis to multidimensional variables.
  – Copeland cites this as the breakpoint between equivalence
    class / boundary analysis & domain analysis because so
    many academic treatments (I include Beizer in these) present
    domain testing as a multidimensional (two or three variable)
    technique
  – The sampling theory is the same: partition & select
  – If variables are independent, we can use combination
    techniques for independent variables (e.g. combinatorial or
    random)
  – When value of one variable constrains the available values
    for the other(s), combinatorial (e.g. all-pairs) fails. Instead,
    we have to map the multidimensional space as it is and
    sample from the actual boundaries.
                                                                   55
                       Exercise 11
What are the secondary dimensions of:

   The page setup function of a text editor allows a user to
   set the width of the page in the range of 1 to 56 inches.
   The page width input field will accept (and remember) up
   to 30 places after the decimal point.



What are the boundaries between valid and invalid input—along
these secondary dimensions?



                                                                56
                      Exercise 12
What are the secondary dimensions of:

    An ATM allows withdrawals of cash in amounts of $20
   increments from $20 to $200 (inclusive).



What are the boundaries between valid and invalid input—along
these secondary dimensions?




                                                                57
                      Exercise 13
What are the secondary dimensions of:

    You are testing a program that includes an Integer
   Square Root function. The function reads a 32-bit word
   that is stored in memory, interprets the contents as an
   unsigned integer and then computes the square root of
   the integer, as a floating point number.



What are the boundaries between valid and invalid input—along
these secondary dimensions?


                                                                58
       Our structure for domain testing
I. Independent                 5

multidimensional variables.
  – Combine
    • X = 0 or 6 (its
      boundaries)              0               6
    • Y = 0 or 5 (its
      boundaries)



                                          59
            Our structure for domain testing
I.     Non-independent multidimensional    5
       variables.
     I. Combinations of
       • X = 0 or 6 (its boundaries)
       • Y = 0 or 5 (its boundaries)
       are not useful:
                                           0   5
       • (0, 0) is on boundary
       • (6, 0) is not near any boundary
       • (5, 0) is on boundary
       • (0, 5) is not near any boundary


                                                   60
        Our structure for domain testing
J. Linearize the domain (if possible).
The goal is to describe the boundaries of the
domain in terms of simple linear
inequalities, like this:

    0        ≤     X      ≤     1
    X        ≤     Y      ≤     5+X




                                                61
        Our structure for domain testing
J. Linearize the domain (if possible).
Extensive academic writing about simple linear
inequalities:

    0        ≤      X      ≤      1
    X        ≤      Y      ≤      5+X

 – On points, off points, in points, out points
 – Special cases that have confused students
   for years
 – The heuristics fall apart for simple
   nonlinear cases, like:
   X2        +     Y2      ≤       10
                                                  62
       Our structure for domain testing
K. Identify constraints among the variables
 – 30 days in September but not August or October
    • (month, day)
 – How much is sales tax in this city?
    • (tax rate, city)
 – Oh, that’s against the law here?
    • (laws, country)




                                                    63
        Our structure for domain testing
L. Identify and test variables that hold results
   (output variables).
 – Simple example: Mailing labels
   • Inputs: Your first, middle and last names
   • Output: A label that your name can’t fit on




                                                   64
                           Exercise 14
     – I, J, and K are (signed) integers.
     – You can enter data into I and J. You cannot enter data directly
       into K.
     – The program calculates K = I * J.

a. What are the constraints among I, J, and K?
b. What invalid values are possible for K?
c. How would you create a table to show tests for the boundary
   cases of K?




65
        Our structure for domain testing
M. Evaluate how the program uses the value of this
variable.
 – Every variable that can be affected by this variable
   can be tested with boundaries (and other interesting
   values) of this variable
 – 1st through Nth order data flows
 – Program slicing addresses this in maintenance

Lecture example: resizing the width of PowerPoint
slides

                                                          66
         Our structure for domain testing
N. Identify additional potentially-related variables for
   combination testing.
O. Create combination tests for multidimensional or related
   variables.

 – Earlier, we considered an individual variable that was inherently
   multidimensional (a point)
 – Now consider testing a bunch of variables together that are not
   inherently dimensions of one larger variable.
 – We can reduce the enormous set of tests via stratified sampling
  • All-pairs is one stratified sampling approach (sample with a
     coverage criterion)
  • Cause-effect graphing is similar but for constraining variables
  • Other possibilities …
                                                                       67
        Our structure for domain testing
P. Imagine risks that don’t necessarily map to an
   obvious dimension.
 – Domain analysis is the start, not the end of test
    design




                                                       68
       Our structure for domain testing
Q. Identify and list unanalyzed variables. Gather
   information for later analysis.
 – You will always have more research to do
 – You will never have a complete test plan




                                                    69
R. Summarize your analysis with a risk/equivalence table
        For more examples, see workbook handout p. 62, 68, 135
          Our structure for domain testing
R. Summarize your analysis with a risk/equivalence table.
 – Classical domain analysis starts with the primary dimension,
    partitions it, and selects "best representative" tests for each
    partition.
      Underlying this is a theory of error: programmers may
      misclassify values of the variable, especially at boundaries
      between partitions.
 – Many authors extend this analysis to values of the variable that are
    "off" the primary dimension, such as letters in a number field or
    entering only two values into a 3-value dialog.
      These extensions to secondary dimensions are interesting
      when the user error (or other source of bad data) is plausible.
      In these cases, we pick a best representative of this class of
      error—an example of this type of error that is most likely to
      cause the program to fail.
                                                                     71
         Our structure for domain testing
R. Summarize your analysis with a risk/equivalence table.
 – Risk-based testing starts from a possible error and designs
    tests that should expose the error if it is there.
 – Domain testing is a type of risk-based testing that is focused
    on classification errors and their consequences.
 – Risk-based domain testing starts from the potential error:
    • We identify how the variable we are working with might be
      involved with that error
    • We partition values of the variable into classes
      – We ignore the classes that cannot expose the error
      – We test best representatives from classes that can expose
        the error if it is in the code.
                                                               72
           Our structure for domain testing
R. Summarize your analysis with a risk/equivalence table.
We prefer the classical table for simple, academic examples because the risk-
oriented table is so much more complex.
The weakness of the very simple examples is that they are divorced from real-
life software:
   – You analyze a variable, but you don’t know why a program needs it, what
      the program will do with it, what other variables will be used in
      conjunction with it.
   – As soon as you know the real-life information, many risks (should)
      become apparent, and these are very difficult to represent in the classical
      table because it is best used to highlight one or a few primary dimensions
      of a variable.
The risk-oriented table helps us organize the testing that is based on this
broader knowledge of the application.
   – Any time you are thinking beyond the basic “too big / too small” tests,
      this style of table might be more helpful than the classical one.

                                                                               73
          Notes on the FoodVan Exercise
Even these simple specifications are ambiguous:
  – Does “within 30 days” mean “less than 30” or “less than or equal
    to 30” ?
  – When does the special permission have to have been issued?
  – If you can work tomorrow morning on the basis of permission, can
    you work tomorrow afternoon on the basis of experience? Is
    tomorrow morning within 30 days of tomorrow afternoon?
  – Do we compute 30 days in days or hours (minutes / seconds)?
  – What result if the last day you worked was 28 days ago? 29 days
    ago? 30 days ago?
Even if you are clear on the answers to these, do you believe that the
programmer and the specification writer will come to the same
answers?

74
                        References
For a list of references, see our file on disk—excerpt from the
Domain Testing Workbook




                                                                  75

				
DOCUMENT INFO
Shared By:
Tags: domain, check
Stats:
views:14
posted:3/23/2011
language:English
pages:75