Automated Test Data Generation for Aspect-Oriented Programs

Document Sample
Automated Test Data Generation for Aspect-Oriented Programs Powered By Docstoc
					                Computer Science



Automated Test Data Generation for
    Aspect-Oriented Programs

    Mark Harman (King’s College London, UK)
    Fayezin Islam (T-Zero Processing Services, US)
    Tao Xie (North Carolina State University, US)
    Stefan Wappler (Berner & Mattner, Germany)
Background
Automated testing of aspect-oriented programs

• Testing aspectual composition behavior
  (pointcut behavior)
 [Ferrari et al. ICST 08, Anbalagan&Xie ISSRE 08, …]


• Testing aspectual behavior
  (advice behavior)
 [Xie&Zhao AOSD 06, Xie et al. ISSRE 06, ...]



             Computer Science
Testing Aspectual Behavior
                                                    [Xie&Zhao AOSD 05]
• Aspect weaving (e.g., with ajc, abc)
   – Aspect  Class (bytecode)
   – Advice  Method (bytecode)
• Straightforward unit testing: feed aspect classes to OO
  test generation tools based on bytecode
   – Issues: arguments can be thisJoinPoint or AroundClosure objects


• Aspectra: generate tests for woven classes but focus on
  aspectual behavior
   – Feed woven classes to OO test generation tools
   – Base classes == “test drivers”
• Leverage existing OO tools for testing AOP programs
                                      E.g., Parasoft Jtest
               Computer Science
                                      (random testing tool)
Example
                              Program
                             Under Test




            +




          Computer Science
New Contributions
• A new system of automated test data generation
  for AOP based on Search-Based Testing, i.e.,
  evolutionary testing
  – Input domain: receiver object, method parameters
  – E.g., account.debit(amount)

• Empirical studies to demonstrate the benefits of
  the system in AOP structural testing
  – Effectiveness: better than random testing
  – Efficiency: AOP domain reduction techniques
  – Efficiency: focusing test effort on aspectual branches to
    improve efficiency
             Computer Science
What is Search-Based Testing?


In search-based testing, we apply search
  techniques to search large input spaces,
  guided by a fitness function.

Fitness function measures how good/close
  one input is in reaching the goal
  e.g., covering true branch of


          Computer Science
Evolutionary Algorithms
                                                          chromosome
                         Insertion


      Mutation

                                     Fitness evaluation


    Recombination
                                        End?

                        Selection



          Computer Science
Evolutionary Testing

                         Insertion

                             Test cases
      Mutation

                                     Fitness evaluation


    Recombination
                                          End?

                        Selection



          Computer Science
Evolutionary Testing

                         Insertion

                             Test cases
      Mutation

                 Execution           Fitness evaluation


    Recombination
                                          End?

                        Selection



          Computer Science
Evolutionary Testing
                                                          chromosome
                         Insertion
                                                          Method seq
                             Test cases
      Mutation

                 Execution           Fitness evaluation


    Recombination            Monitoring
                                          End?

                        Selection



          Computer Science
Structural Evolutionary Testing
    Structural Evolutionary Testing




                                  Target:
                                  true branch




          Computer Science
Structural Evolutionary Testing
    Structural Evolutionary Testing
  Evaluation of predicate in a branching condition
  if (A < B)

     Fitness = (A - B) + 1
     101 =    (100 – 0) + 1        Lower fitness value, the better
                                   0 fitness value, reach the target
      51 =    (100 – 50) + 1
       0 =    (100 – 101) + 1
                                                            Target:
                                                            true branch




                Computer Science
AOP Domain Reduction - Motivation
• Input domain: receiver object, method parameters
  – E.g., account.debit(amount)
• Not all input variables are relevant to the coverage
  of the target branch inside aspects




                                    Target:
                                    false branch
          …
            Computer Science
Slicing-based Domain Reduction
 • Irrelevant-input-variable identification
    • Start with slicing criterion: predicates of target branches
    • Extract backward program slices (based on data and
      control dependence)
    • Identify relevant input variables: input variables showing
      up in the slices: account.debit(amount)
 • Domain reduction: searching for only relevant input vars




                                            Target:
             …                              false branch
               Computer Science
EvolutionaryAspectTester (EAT)
System Implementation        Indus slicer
                                      [Ranganath et al. 07]




           Aspectra
           [Xie&Zhao 06]




                              EvoUnit
                             [Wappler 08]



          Computer Science
Evaluation Benchmarks




 14 benchmarks from
 [Xie&Zhao 06, Rinard et al. 04, Dufour et al. 04, Hannemann&Kiczales 02]

               Computer Science
Study 1: Assessment of evo testing

RQ 1.1. Can evolutionary testing outperform random
testing for AOP testing?




           Computer Science
     RQ 1.1: Assessment of evo testing
                                                                    43%




 Coverage improvement of evolutionary testing over random testing

                       on 5/14 benchmarks
Better branch coverageComputer Science
     RQ 1.1: Assessment of evo testing cont.
                                                                 61%




  Effort reduction of evolutionary testing over random testing
                       Computer Science
Effort reduction on 9/14 benchmarks
Findings: Assessment of evo testing

RQ 1.1. Can evolutionary testing outperform random
testing for testing aspect-oriented programs?

•Better branch coverage on 5/14 benchmarks
           (0%~43%)
•Effort reduction on 9/14 benchmarks
            (0%~61%)




           Computer Science
Study 2: Impact of domain reduction
RQ 2.1. #branches that have irrelevant parameters
and %parameters that are irrelevant for each such
branch?

RQ 2.2. %effort reduction for each such branch?

RQ 2.3. %effort reduction for each program?




            Computer Science
     RQ 2.1: Impact of domain reduction




90/434 aspectual branches with irrelevant parameters
                      Computer Science
    RQ 2.1: Impact of domain reduction




  Input domain reduction for branches with non-0 reduction
                     Computer Science
Input domain reduction (25%~100%)
   RQ 2.2: Impact of domain reduction
                                                                    94%




       Easy/trivial
       branches




       -88%

 Effort reduction per branch of using domain reduction
                      Computer
Effort increase on 25%, sameScience and reduction on 69% branches
                               on 6%,
    RQ 2.3: Impact of domain reduction




  Effort reduction per program of using domain reduction
                     Computer
Effort reduction (17%~93%) Science
Findings: Impact of domain reduction
RQ 2.1. #branches that have irrelevant parameters
(99/434) and %parameters that are irrelevant for
each such branch (25%~100%)?

RQ 2.2. %effort reduction for each such branch
(-88%~94%)(69% branches get reduction)?

RQ 2.3. %effort reduction for each program
(17%~93%)?




            Computer Science
Study 3: Impact of focusing on testing
aspectual behavior
RQ 3.1. %effort reduction for test data generation if
aspectual behavior instead of all behavior is focused
on?




            Computer Science
      RQ 3.1: Impact of aspect focusing
                                                                          99.99%




              3%




   Effort reduction of focusing on aspectual behavior over all behavior

                         Computer Science
Effort reduction on all 14 benchmarks
     RQ 3.1: Impact of aspect focusing cont.


                                                                         62%




   Coverage improvement of focusing on aspectual behavior over all behavior

Coverage improvementComputer Science
                     on 6/14 benchmarks
Study 3: Impact of focusing on testing
aspectual behavior
RQ 3.1. %effort reduction for test data generation if
aspectual behavior instead of all behavior is focused
on?

•Effort reduction on all 14 benchmarks
          (3% ~ 99.99%)
•Coverage improvement on 6/14 benchmarks
          (0% ~ 62%)



            Computer Science
Conclusion
• A new system of automated test data generation
  for AOP based on Search-Based Testing
• Empirical studies to demonstrate the benefits of
  the system in AOP structural testing
  – Effectiveness: better than random testing
  – Efficiency: AOP domain reduction techniques
  – Efficiency: focusing test effort on aspectual branches to
    improve efficiency

• Future work on more advanced techniques (e.g.,
  symbolic execution), more testing objectives, larger
  AOP programs
             Computer Science
        Questions?




Computer Science
Structural Evolutionary Testing
    Structural Evolutionary Testing
   Level 4                   Approximation level

                               Identify relevant branching statements using control
    Level 3                       dependence, e.g.,(#expected/#actual – 1)


   Level 2                   Local distance

                              Evaluation of predicate in a branching condition
         Level 1              if (A < B)      Local_Distance = (A – B) + 1

         Target
                           Fitness = Approximation_Level + Local_Distance
                           101 = 0 + (100 – 0) + 1
                                                        Lower fitness value, the better
                             51 = 0 + (100 – 50) + 1   0 fitness value, reach the target
                             0 = 0 + (100 – 101) + 1



                   Computer Science
    RQ 2.4: Impact of domain reduction




  Co-lateral coverage improvement effect of domain reduction
                     Computer
9 branches have statistically Science
                              significant change in co-lateral coverage