Component integration testing10 by HQprime


									Integration Testing

    Testing of Large Systems   1
                        [Abrain and Moore, 2004]

Testing levels
– Unit testing
   • Verifies isolated functioning of seperately testable software
– Integration testing
   • Verifying the interaction between software components
   • Architecture-driven
– System testing
   • Concerned with the behavior of the full system
   • Functional and ”non-functional”/quality properties of the

                      Testing of Large Systems                       2
                                                 War Story I

In a project I was developing a component for
creating UML sequence diagrams
– The component worked perfectly well (presentation,
  editing, undo/redo, load/save, …) when tested by
When combined and tested with a class diagram
component it failed
– The class diagram component might delete/change
  packages related to sequence diagram
   • So if a class was created, sequence diagrams containing
     objects instantiated from that class created, and the class
     deleted this would lead to a failure
                      Testing of Large Systems                     3
                                              War Story II

I was writing a component that should interface
with a serial port temperature sensing device
– It would read a temperature based on a protocol for
  TH-03, do some processing, and eventually give that
  temperature away
Worked well…
When connected to actual TH-03 device, nothing
– Needed to set special serial port options to power
  device through serial cable…
                   Testing of Large Systems              4
                                       Integration defects

Binder list typical
interface defects…
– “2/3 of defects are in

War stories are
covered by it 

                       Testing of Large Systems          5
                                       Integration testing
Integration testing deals with finding defects in the way
individual parts work together
We use composition at almost any level in software
– May argue (as does Binder [2000]) that integration testing occurs
  at all levels.

                      Testing of Large Systems                    6
                        Why integration testing?

Why test component interactions???

If we provide
– ‘sufficient’ unit testing
– ‘sufficient’ system testing
is it not OK then???

                     Testing of Large Systems   7
                            Why integration testing?
Binder relates an argument (on axiomatizing test case adequacy):

Antidecomposition axiom:
– There exists a program P and component Q [of P] such that T is
  adequate for P, T’ is the set of vectors of values that variables can
  assume on entrance to Q for some t of T, and T’ is not adequate for Q
– System scope coverage does not necessarily achieve component
– Why? Examples?

Anticomposition axiom:
– There exist programs P and Q, and test set T, such that T is adequate
  for P, and the set of vectors of values that variables can assume on
  entrance to Q for inputs in T is adequate for Q, but T is not adequate for
– Adequate testing at component level is not equivalent to adequate
  testing of a system of components
– Why? Examples?

                         Testing of Large Systems                          8
             Premise for integration testing

At any given level:
System consisting of components

Testing interactions between components
require that these are stable
– Stable = sufficient dependability to allow integration
– Threshold reflects practical considerations

In case of unstable component – then what?
                   Testing of Large Systems                9
               Component stability/volatility

An important aspect is of course to preserve our
investment in testing
Spending staff hours to test a class that is later
fundamentally changed is a waste of time
– In opposition to our wish for early integration testing
– Postpone integration of volatile components!

                    Testing of Large Systems                10
               Integration and architecture

Integration testing is highly coupled to
Definition by [Bass et al., 2003]

The software architecture of a computing system
 is the structures of the system, which comprise
     software elements, the externally visible
      properties of those elements, and the
            relationships among them.

                 Testing of Large Systems     11
                   Integration and architecture

Thus the architecture defines the testing in terms
of what are

– the elements
   • (stable units that have already been unit tested adequately)

– and the relations
   • interactions that contain defects we wish to reveal

(Binder focuses on the module view of a
software architecture)
                      Testing of Large Systems                      12

Components may interact in a large number of
ways using either explicit or implicit relations
–   composition and aggregation
–   delegation to objects / API calls
–   inheritance
–   global variables
–   instance variable access
–   objects as message parameters
–   RMI / socket communication

                     Testing of Large Systems              13

Many are however implicit dependencies that
are much harder to spot
–   database access / persistent store access
–   initialization sequences
–   timing constraints
–   file formatting / coding of byte streams

Binder states:
– Integration testing does not focus on implicit
                    Testing of Large Systems              14
                            Dependency analysis

(Explicit) dependencies often dictate the
sequence of testing
– I.e. in which order are component interactions tested?

Thus a very helpful tool is to do a dependency
– Similar to what build systems do to determine
  compilation order
– Hard for implicit dependencies

                  Testing of Large Systems            15

A measure of the strength of dependencies between two
Which is preferable from a testing perspective?
             high coupling               low coupling

Cf. testing tactics
                      Testing of Large Systems                 16
           Example from Binder

Testing of Large Systems     17
                                      Example from Binder
Remove cycles, do a                          Here arrows are uses relations
topological sort on the
directed, acyclical dependency

Root level
 – unit not used by any other unit
   in cluster under test
 – often there are several roots
Leaf level
 – units that do not use any other
   units in cluster under test.
 – either be tested as a “unit”
 – or stubs introduced to break

                           Testing of Large Systems                      18
                                       Lessons learned

Key lessons from early 1970s is that
incremental integration and testing is most

Integrating all components at the end of
development is problematic:
– Last-minute changes are necessary but no time for
  adequate testing
– Testing often not systematic

                  Testing of Large Systems            19
                                       Lessons learned

Advantages in incremental testing

– Interfaces systematically exercised and shown stable
  before new unproven interfaces are exercised

– Observed failures more likely to come from most
  recently added components making debugging more

                  Testing of Large Systems           20

So, the doctrine says integration test often but
how do we do integration with components that
are not stable or not developed at all?

Stub: partial or surrogate implementation of a

                 Testing of Large Systems          21
                                         Reasons for stubs

Stubs are powerful tools for many reasons

Make testing of certain conditions possible
– Hardware simulator that outputs data that are seldom occurring
  in practice
    • Example: specific patterns of temperature from temperature sensor
– Replace random behavior with predictable behavior
    • Example: testing backgammon move validation (calling
      getNextMove() or similar to get move to validate)

Make testing economic
– Stub for a database that is costly to reset/set up

                        Testing of Large Systems                      22
                                       Reasons for stubs

Often stubs are relevant to decouple interactions
– Stub for complex algorithm with ‘simple’ answer
   • Ensures that the defect found does not lie in the algorithm
– Stub for a component in a cycle

There is no free lunch!
– Stubs can be costly to produce
– … and sometimes you chase defects in production
  code – that are actually embedded in the stubs 

Morale: Keeps stubs simple!
                      Testing of Large Systems                     23
                             Integration Strategies

Binder lists nine integration strategies,
documented in pattern form, for doing integration

We focus on the classic strategies:
– Big Bang (actually an anti-pattern )
– Bottom-up
– Top-down

                   Testing of Large Systems      24
                                              Big Bang

– demonstrate stability by attempting to exercise an
  entire system with a few test runs
– bring all components together all at once. All
  interfaces tested in one go.
– usually ends in ‘big-bang’ – system dies miserably…
Entry criteria
– all components have passed unit testing
Exit criteria
– Test suite passes
                   Testing of Large Systems             25
                                                  Big Bang

– Fails – then what? Failure diagnosis is very difficult.
– Even if exit criteria met, many interface faults can still
– On the plus side: if it works, no effort has been spent
  on test drivers and writing stubs.

– may be the course of action for
   • small systems with adequate unit testing
   • existing system with only minor modifications
   • system made from certified high quality reusable
                     Testing of Large Systems              26
                                              Bottom Up

– Demonstrate system stability by adding components
  to the SUT in uses-dependency order, starting with
  component having the fewest dependencies

– stepwise verification of tightly coupled components

                   Testing of Large Systems             27
                                             Bottom up

1. stage: leafs

                  Testing of Large Systems           28
                                        Bottom up

2. stage..

             Testing of Large Systems           29
                                          Bottom up

Last stage..

               Testing of Large Systems           30
                                            Bottom up

Entry criteria
– components pass unit tests
Exit criteria
– interface for each subcomponent has been exercised
  at least once
– complete when all root-level components pass test

                 Testing of Large Systems           31
                                                     Bottom up

– Actually most unit tests in OO are partly integration tests
    • Bottom-up testing implicitly takes place in a bottom up development
– Driver development cost significant
– Fix in lower level component may require revisions and retest up
  the chain
– Interfaces only indirectly exercised – only the ones used by the
  component under test
– Upper levels testing may require stubs in the bottom to test
  special conditions
– High level testing very late in cycle, i.e., validation only possible
                        Testing of Large Systems                       32
                                             Bottom up

– Parallel implementation and testing possible
– Little need for stub writing

                  Testing of Large Systems           33
                                      Tools: JUnit



           Testing of Large Systems             34
                              Tools: Mock Objects /
– Library for testing Java code using mock objects

Mock objects
– Given an interface create an advanced stub at
  runtime using reflection
– May define expected values on mock objects using

                  Testing of Large Systems           35
               Tools: Using jMock

Testing of Large Systems        36
                                                    Top Down

– Demonstrate stability by
  adding components to the
  SUT in control hierarchy
  order, beginning with the
  top-level control objects
– 1 stage:
    • test control objects

                         Testing of Large Systems          37
                                      Top Down

2. stage

           Testing of Large Systems          38
                                         Top Down

Final Stage

              Testing of Large Systems          39
                                             Top Down

– each component to be integrated passes unit test
– interface of each component has been exercised at
  least once
– complete when leaf-level components passes system
  scope test suite

                  Testing of Large Systems           40
                                             Top Down

– Large number of stubs necessary which is costly
– Stubs are brittle
– Fix in lower level component may require revisions up
  the chain
– Difficult to get lower level components sufficiently
– Lower level components are exercised late
– Low driver development costs
– early demonstration of user-related behavior
                  Testing of Large Systems           41

There are many variations of these strategies
– Sandwich testing
   • Moving from both top and bottom

Bottom line
– there is no substitute from being clever and utilize a
  combination of techniques that is most cost-efficient
  for the project at hand.

                    Testing of Large Systems               42
                   High-frequency integration

Binder also describes high-frequency integration
that is a strategy whose characteristics lies in
the timing, not the ordering, of testing
– It is an intrinsic part of the process pattern daily build
– At an extreme: continuous integration of eXtreme
  Programming fame

– Integrate new code with stabilized baseline frequently
  to prevent integration bugs from going undiscovered
  for a long time; and to prevent divergence from the
                    Testing of Large Systems                   43
                    High-frequency integration

– A stable baseline must be present; increments are the
  focus of high-frequency integration
– Increments size must match integration frequency
   • daily builds + integrations -> increments must be deliverable
     in a day
– Test code developed in parallel with code
– Testing must be automated…
– Software Configuration Management must be in place

                     Testing of Large Systems                    44
                      High-frequency integration

– Revise code + test code on private branch
– Desk check code and test
– When all component testing passes, check-in to the integration
– Integration tester builds system of increments
– Testing using
    • smoke tests and as much additional as time permits

Any increments that break HFI are corrected

                       Testing of Large Systems                    45
                 High-frequency integration

– Automated tests must be in place
– High commitment to maintaining code as well as tests
– Be aware of adequacy criteria – the suite that found
  the old bugs may not find the new ones
– Focus on maintaining tests is an effective bug
  prevention strategy
– Defects found early; debugging easier
– Morale high as system works early and keeps doing it

                  Testing of Large Systems           46
             Integration test in the TMM (1/2)
Basic integration test appears in level 2 (as does all
basic test techniques/methods).
Planning of the test process (incl. integration test) is also
initiated at level 2.
At level 3, integration test (and all other test activities) is
integrated in the sw life cycle.
– In modern iterative/agile development methods, this happens
  more or less automatically
At level 4, the integration test process gets measured.
At level 5 the integration test process is continuously

                      Testing of Large Systems                  47
              Integration test in the TMM (2/2)

At level 2,
– Should ensure that multilevel testing is part of policies,
  planned for and catered for via project schedules and
– Should help PM’s to make multilevel testing work, help
  develop use cases/acceptance criteria, implement tests at
  all levels
– Should help provide acceptance criteria, provide ample
  and due feedback, participate in tests (accept/beta)
                       Testing of Large Systems                48

Testing levels
– Unit testing
   • Verifies isolated functioning of seperately testable software
– Integration testing
   • Verifying the interaction between software components
   • Architecture-driven
– System testing
   • Concerned with the behavior of the full system
   • Functional and ”non-functional”/quality properties of the

                      Testing of Large Systems                       49

To top