Docstoc

Component integration testing10

Document Sample
Component integration testing10 Powered By Docstoc
					Integration Testing




    Testing of Large Systems   1
                        [Abrain and Moore, 2004]

Testing levels
– Unit testing
   • Verifies isolated functioning of seperately testable software
     pieces
– Integration testing
   • Verifying the interaction between software components
   • Architecture-driven
– System testing
   • Concerned with the behavior of the full system
   • Functional and ”non-functional”/quality properties of the
     system



                      Testing of Large Systems                       2
                                                 War Story I

In a project I was developing a component for
creating UML sequence diagrams
– The component worked perfectly well (presentation,
  editing, undo/redo, load/save, …) when tested by
  itself
When combined and tested with a class diagram
component it failed
– The class diagram component might delete/change
  packages related to sequence diagram
   • So if a class was created, sequence diagrams containing
     objects instantiated from that class created, and the class
     deleted this would lead to a failure
                      Testing of Large Systems                     3
                                              War Story II

I was writing a component that should interface
with a serial port temperature sensing device
(TH-03)
– It would read a temperature based on a protocol for
  TH-03, do some processing, and eventually give that
  temperature away
Worked well…
When connected to actual TH-03 device, nothing
happened
– Needed to set special serial port options to power
  device through serial cable…
                   Testing of Large Systems              4
                                       Integration defects

Binder list typical
interface defects…
– “2/3 of defects are in
  integration”


War stories are
covered by it 




                       Testing of Large Systems          5
                                       Integration testing
Integration testing deals with finding defects in the way
individual parts work together
We use composition at almost any level in software
engineering
– May argue (as does Binder [2000]) that integration testing occurs
  at all levels.




                      Testing of Large Systems                    6
                        Why integration testing?

Why test component interactions???

If we provide
– ‘sufficient’ unit testing
and
– ‘sufficient’ system testing
is it not OK then???




                     Testing of Large Systems   7
                            Why integration testing?
Binder relates an argument (on axiomatizing test case adequacy):

Antidecomposition axiom:
– There exists a program P and component Q [of P] such that T is
  adequate for P, T’ is the set of vectors of values that variables can
  assume on entrance to Q for some t of T, and T’ is not adequate for Q
– System scope coverage does not necessarily achieve component
  coverage
– Why? Examples?

Anticomposition axiom:
– There exist programs P and Q, and test set T, such that T is adequate
  for P, and the set of vectors of values that variables can assume on
  entrance to Q for inputs in T is adequate for Q, but T is not adequate for
  P;Q
– Adequate testing at component level is not equivalent to adequate
  testing of a system of components
– Why? Examples?

                         Testing of Large Systems                          8
             Premise for integration testing

At any given level:
System consisting of components

Testing interactions between components
require that these are stable
– Stable = sufficient dependability to allow integration
  testing
– Threshold reflects practical considerations


In case of unstable component – then what?
                   Testing of Large Systems                9
               Component stability/volatility

An important aspect is of course to preserve our
investment in testing
Spending staff hours to test a class that is later
fundamentally changed is a waste of time
– In opposition to our wish for early integration testing
– Postpone integration of volatile components!




                    Testing of Large Systems                10
               Integration and architecture

Integration testing is highly coupled to
architecture
Definition by [Bass et al., 2003]

The software architecture of a computing system
 is the structures of the system, which comprise
     software elements, the externally visible
      properties of those elements, and the
            relationships among them.


                 Testing of Large Systems     11
                   Integration and architecture

Thus the architecture defines the testing in terms
of what are

– the elements
   • (stable units that have already been unit tested adequately)


– and the relations
   • interactions that contain defects we wish to reveal


(Binder focuses on the module view of a
software architecture)
                      Testing of Large Systems                      12
                                                Dependencies

Components may interact in a large number of
ways using either explicit or implicit relations
–   composition and aggregation
–   delegation to objects / API calls
–   inheritance
–   global variables
–   instance variable access
–   objects as message parameters
–   RMI / socket communication



                     Testing of Large Systems              13
                                               Dependencies

Many are however implicit dependencies that
are much harder to spot
–   database access / persistent store access
–   initialization sequences
–   timing constraints
–   file formatting / coding of byte streams


Binder states:
– Integration testing does not focus on implicit
  dependencies
–?
                    Testing of Large Systems              14
                            Dependency analysis

(Explicit) dependencies often dictate the
sequence of testing
– I.e. in which order are component interactions tested?


Thus a very helpful tool is to do a dependency
analysis
– Similar to what build systems do to determine
  compilation order
– Hard for implicit dependencies



                  Testing of Large Systems            15
                                                        Coupling

A measure of the strength of dependencies between two
                       subsystems
Which is preferable from a testing perspective?
             high coupling               low coupling




Cf. testing tactics
                      Testing of Large Systems                 16
           Example from Binder




Testing of Large Systems     17
                                      Example from Binder
Remove cycles, do a                          Here arrows are uses relations
topological sort on the
directed, acyclical dependency
graph

Root level
 – unit not used by any other unit
   in cluster under test
 – often there are several roots
Leaf level
 – units that do not use any other
   units in cluster under test.
Cycles
 – either be tested as a “unit”
 – or stubs introduced to break
   cycle

                           Testing of Large Systems                      18
                                       Lessons learned

Key lessons from early 1970s is that
incremental integration and testing is most
effective

Integrating all components at the end of
development is problematic:
– Last-minute changes are necessary but no time for
  adequate testing
– Testing often not systematic


                  Testing of Large Systems            19
                                       Lessons learned

Advantages in incremental testing

– Interfaces systematically exercised and shown stable
  before new unproven interfaces are exercised

– Observed failures more likely to come from most
  recently added components making debugging more
  efficient.




                  Testing of Large Systems           20
                                            Stubs

So, the doctrine says integration test often but
how do we do integration with components that
are not stable or not developed at all?

Stub: partial or surrogate implementation of a
component




                 Testing of Large Systems          21
                                         Reasons for stubs

Stubs are powerful tools for many reasons

Make testing of certain conditions possible
– Hardware simulator that outputs data that are seldom occurring
  in practice
    • Example: specific patterns of temperature from temperature sensor
– Replace random behavior with predictable behavior
    • Example: testing backgammon move validation (calling
      getNextMove() or similar to get move to validate)


Make testing economic
– Stub for a database that is costly to reset/set up

                        Testing of Large Systems                      22
                                       Reasons for stubs

Often stubs are relevant to decouple interactions
– Stub for complex algorithm with ‘simple’ answer
   • Ensures that the defect found does not lie in the algorithm
     implementation
– Stub for a component in a cycle


There is no free lunch!
– Stubs can be costly to produce
– … and sometimes you chase defects in production
  code – that are actually embedded in the stubs 


Morale: Keeps stubs simple!
                      Testing of Large Systems                     23
                             Integration Strategies

Binder lists nine integration strategies,
documented in pattern form, for doing integration
testing

We focus on the classic strategies:
– Big Bang (actually an anti-pattern )
– Bottom-up
– Top-down




                   Testing of Large Systems      24
                                              Big Bang

Intent
– demonstrate stability by attempting to exercise an
  entire system with a few test runs
Context
– bring all components together all at once. All
  interfaces tested in one go.
– usually ends in ‘big-bang’ – system dies miserably…
Entry criteria
– all components have passed unit testing
Exit criteria
– Test suite passes
                   Testing of Large Systems             25
                                                  Big Bang

Consequences
– Fails – then what? Failure diagnosis is very difficult.
– Even if exit criteria met, many interface faults can still
  hide.
– On the plus side: if it works, no effort has been spent
  on test drivers and writing stubs.

– may be the course of action for
   • small systems with adequate unit testing
   • existing system with only minor modifications
   • system made from certified high quality reusable
     components
                     Testing of Large Systems              26
                                              Bottom Up

Intent
– Demonstrate system stability by adding components
  to the SUT in uses-dependency order, starting with
  component having the fewest dependencies


Context
– stepwise verification of tightly coupled components




                   Testing of Large Systems             27
                                             Bottom up

Strategy:
1. stage: leafs




                  Testing of Large Systems           28
                                        Bottom up

2. stage..




             Testing of Large Systems           29
                                          Bottom up

Last stage..




               Testing of Large Systems           30
                                            Bottom up

Entry criteria
– components pass unit tests
Exit criteria
– interface for each subcomponent has been exercised
  at least once
– complete when all root-level components pass test
  suites




                 Testing of Large Systems           31
                                                     Bottom up

Consequences
– Actually most unit tests in OO are partly integration tests
    • Bottom-up testing implicitly takes place in a bottom up development
      process
Disadvantages
– Driver development cost significant
– Fix in lower level component may require revisions and retest up
  the chain
– Interfaces only indirectly exercised – only the ones used by the
  component under test
– Upper levels testing may require stubs in the bottom to test
  special conditions
– High level testing very late in cycle, i.e., validation only possible
  late
                        Testing of Large Systems                       32
                                             Bottom up

Advantages
– Parallel implementation and testing possible
– Little need for stub writing




                  Testing of Large Systems           33
                                      Tools: JUnit

Stubs?

Drivers?




           Testing of Large Systems             34
                              Tools: Mock Objects

http://www.mockobjects.com/ /
http://www.jmock.org/
– Library for testing Java code using mock objects


Mock objects
– Given an interface create an advanced stub at
  runtime using reflection
– May define expected values on mock objects using
  constraints


Example...
                  Testing of Large Systems           35
               Tools: Using jMock




Testing of Large Systems        36
                                                    Top Down

Intent
– Demonstrate stability by
  adding components to the
  SUT in control hierarchy
  order, beginning with the
  top-level control objects
Strategy
– 1 stage:
    • test control objects




                         Testing of Large Systems          37
                                      Top Down

2. stage




           Testing of Large Systems          38
                                         Top Down

Final Stage




              Testing of Large Systems          39
                                             Top Down

Entry
– each component to be integrated passes unit test
Exit
– interface of each component has been exercised at
  least once
– complete when leaf-level components passes system
  scope test suite




                  Testing of Large Systems           40
                                             Top Down

Disadvantages
– Large number of stubs necessary which is costly
– Stubs are brittle
– Fix in lower level component may require revisions up
  the chain
– Difficult to get lower level components sufficiently
  exercised
– Lower level components are exercised late
Advantages
– Low driver development costs
– early demonstration of user-related behavior
                  Testing of Large Systems           41
                                               Variations

There are many variations of these strategies
– Sandwich testing
   • Moving from both top and bottom


Bottom line
– there is no substitute from being clever and utilize a
  combination of techniques that is most cost-efficient
  for the project at hand.




                    Testing of Large Systems               42
                   High-frequency integration

Binder also describes high-frequency integration
that is a strategy whose characteristics lies in
the timing, not the ordering, of testing
– It is an intrinsic part of the process pattern daily build
– At an extreme: continuous integration of eXtreme
  Programming fame


Intent
– Integrate new code with stabilized baseline frequently
  to prevent integration bugs from going undiscovered
  for a long time; and to prevent divergence from the
  baseline
                    Testing of Large Systems                   43
                    High-frequency integration

Context
– A stable baseline must be present; increments are the
  focus of high-frequency integration
– Increments size must match integration frequency
   • daily builds + integrations -> increments must be deliverable
     in a day
– Test code developed in parallel with code
– Testing must be automated…
– Software Configuration Management must be in place




                     Testing of Large Systems                    44
                      High-frequency integration
Procedure:

First
– Revise code + test code on private branch
– Desk check code and test
– When all component testing passes, check-in to the integration
  branch
Second
– Integration tester builds system of increments
– Testing using
    • smoke tests and as much additional as time permits


Any increments that break HFI are corrected
immediately

                       Testing of Large Systems                    45
                 High-frequency integration

Disadvantages
– Automated tests must be in place
– High commitment to maintaining code as well as tests
– Be aware of adequacy criteria – the suite that found
  the old bugs may not find the new ones
Advantages
– Focus on maintaining tests is an effective bug
  prevention strategy
– Defects found early; debugging easier
– Morale high as system works early and keeps doing it


                  Testing of Large Systems           46
             Integration test in the TMM (1/2)
Basic integration test appears in level 2 (as does all
basic test techniques/methods).
Planning of the test process (incl. integration test) is also
initiated at level 2.
At level 3, integration test (and all other test activities) is
integrated in the sw life cycle.
– In modern iterative/agile development methods, this happens
  more or less automatically
At level 4, the integration test process gets measured.
At level 5 the integration test process is continuously
optimized.



                      Testing of Large Systems                  47
              Integration test in the TMM (2/2)

At level 2,
Managers
– Should ensure that multilevel testing is part of policies,
  planned for and catered for via project schedules and
  resources
Developers
– Should help PM’s to make multilevel testing work, help
  develop use cases/acceptance criteria, implement tests at
  all levels
Users/clients
– Should help provide acceptance criteria, provide ample
  and due feedback, participate in tests (accept/beta)
                       Testing of Large Systems                48
                                                   Summary

Testing levels
– Unit testing
   • Verifies isolated functioning of seperately testable software
     pieces
– Integration testing
   • Verifying the interaction between software components
   • Architecture-driven
– System testing
   • Concerned with the behavior of the full system
   • Functional and ”non-functional”/quality properties of the
     system



                      Testing of Large Systems                       49