Software Testing Ch 13_ Integration - LORE

Document Sample
Software Testing Ch 13_ Integration - LORE Powered By Docstoc
					            Software Testing
                            13. Integration

Daniel Riegelhaupt - Thomas De Vylder - Gertjan Loots - Tim Saquet - Philip De Smedt
13. Integration

• Integration in Object-oriented
• Integration Patterns

                                                  13.1.1 Definitions

A System is composed out of Components who
are themselves systems of smaller components

Component                System                   Typical intercomponent interface
(Focus of integration)   (Scope of integration)   (Location of integration faults)
Method                   Class                    Instance variables
                                                  Intraclass messages
Class                    Cluster                  Interclass messages

Cluster                  Subsystem                Interclass messages
                                                  Interpackage messages
Subsystem                System                   Interprocces communication
                                                  Remote procedure call
                                                  ORB services
                                                  OS services

  13.1.2 Integration testing is essential

3 basic kinds of testing needed to show that components are
minimally interoperable:
   • Tests of individual components
   • Tests of the system resulting from the federation of components
   • Test of components interoperation

Integration testing = a search for component faults that
cause intercomponent failures

             Integration testing is essential

In contrast:
• Component scope testing = search for intracomponent
• System scope testing = search for faults that lead to a
   failure to meet a system scope responsibility

            Integration testing is essential

• Antidecomposition axiom:
  A test suite that achieves coverage at system scope will
  not necessarily result in the same coverage for its
  In other words: even if the program works as a whole this
  doesn’t mean we test every component fully (some code
  might never be reached)
• Anticomposition axiom:
  Adequate testing of components (in isolation) ≠ adequate
  testing of a system of components

             Integration testing is essential

Integrating most or all components in a system at the same
time is usually problematic : 
   - Debugging is difficult (bug can be in any interface)
   - Last minute patches may be necessary (=> low quality, not
     adequately tested )
   - No systematic testing can lead to undetected bugs

     Incremental testing !

             Integration testing is essential

• Advantages of incremental testing:
   - Interfaces are systematically exercised and proven to
     be stable before exercising unproven interfaces
   - Buggy interface code will be reached; if not triggered
   - Observe failures are most likely to come from most
     recently added component
• Incremental testing requires discipline and commitment:
      choose a sequence of components by analyzing
       dependencies (=> architecture must be known
       or planned!). Next manage plan and automate

• Integration testing takes place at all scopes and is repeated in each
 development increment

             Integration testing is essential

Integration test plan must answer:
   • what are the major and minor development increments ?
   • which components interfaces / subsystems will be the focus
     of integration ?
   • In what sequence will components and their interfaces be
     exercised ?
   • Which stubs and drivers must be developed ?
   • Which test design patterns should be used ?
   • When can integration testing be considered complete ?

            Integration testing is essential

Key considerations:
   • Scope of an increment and dependencies of components
   • Testability ( # components   testability  )
   • Component stability (postpone integration of volatile
     components, start with stable ones)
   • Test environment stability

                        13.1.3 Dependency Analysis
Class/cluster scope dependencies:
•   Composition and aggregation
•   Inheritance
•   Global variables …
•   pointers to object used as message parameters
•   …
Intercomponent dependencies:
• Explicit:
    • Message exchange
    • Remote procedure calls
    • Interprocess communication
• Implicit: Not considered in Integration testing

                                    Dependency Analysis

Explicit dependencies:
•   often dictate the sequence of testing
•   typically correspond to interfaces that should be exercised by an
    integration TS

                                    Dependency Analysis

Make a dependency tree:
•   by hand (if small)
•   Use topological search ( Unix: tsort; commercial tools)
•   Can have multiple roots, cyclical dependencies

Note that tools might have problems finding dependencies
over: Networks, VM , OS …

                                      13.1.4 Faults Model

• Configuration/version control problems
• Missing, overlapping or conflicting functions
• Problems with file or DB ( incorrect/ inconsistent
  representation; conflicting view /usage; data integrity ...)
• Wrong method called due to coding or binding error
• Client sends illegal message to server (violates pre
  condition or sequential constraints)
• Wrong object bound to message
• Wrong parameter or incorrect parameter value

                                           Faults Model
• Failures due to incorrect memory management
• Incorrect usage of VM , ORB or OS services
• Attempt by the IUT to use target environment services
  that are obsolete / not supported in the currant version/
• Intercomponent conflicts (e.g. thread X will crash when
  process Y is running)
• Resource contention: target env. can’t allocate resources
  needed for nominal load (e.g. Need to open 6 windows
  but IUT crashes when fifth is opened)

IUT = implementation under test

13. Integration

• Integration in Object-oriented Development
• Integration Patterns

                             Integration patterns

• Big Bang Integration. Attempt to demonstrate
  system stability by testing all components
  at the same time.
  - Useful in a few limited circumstances.

• Bottom-up Integration. Interleave component
  and integration testing by following usage
  - Works well for small to medium systems.

                            Integration patterns

• Top-down Integration. Interleave component
  and integration testing by following the
  application control hierarchy.
  - Works for almost any scope or architecture.

• Collaboration Integration. Choose an order of
  integration according to collaborations and
  their dependencies. Exercise interfaces by
  testing one collaboration at a time.
  - Works for almost any scope or architecture.

                             Integration patterns

• Backbone Integration. Combine Top-down
  Integration, Bottom-up integration, and
  Big Bang Integration to reach a stable
  system that will support iterative development.
  - Well suited to small to medium systems and
    especially useful for embedded applications.

• Layer Integration. Incrementally exercise
  interfaces and components in a layered
  - Applies to layered architectures.

                             Integration patterns

• Client/Server Integration. Exercise a loosely
  coupled network of components, all of
  which use a common server component.
  - Appropriate for client/server architectures.

• Distributed Services Integration. Exercise a
  loosely coupled network of peer-to-peer
  - Appropriate for decentralized networks containing
    peer-to-peer nodes.

                            Integration patterns

• High-frequency Integration. Develop and rerun
  an integration test suite test hourly, daily,
  or weekly.
  - Can be applied at almost any scope or

For complete scope specific considerations
See book 13.2.1

                           Big Bang Integration

• Intent
  - Demonstrate stability with a few test runs

• Context
  - SUT is stabilized and only a few components have
    been added since the last time
  - SUT is small and testable
  - Only feasible for monolithic systems
     • Tightly coupled system
     • Bad design

                         Big Bang Integration

• Fault Model
  - Does the system run? Demonstrate that system
    testing can begin

• Strategy
  - No incremental integration testing
  - Entire system is built and a test suite is applied

Big Bang Integration

                        Big Bang Integration

• Entry Criteria
  - All components have passed component scope
  - The VM to be used is stable
  - No anomalies found that interfere with integration

                            Big Bang Integration

• Exit Criteria
   - The test suite passes

• Consequences
   - Many interface bugs: where to look?
      • Debugging is HARD: every component is suspect
      • Even if SUT passes: many hidden interface faults?

                       Big Bang Integration

• Known Uses
  - Subsystem Big Bang Integration is used in
    Backbone Integration.

                     Bottom-Up Integration

• Intent
  - Add components to the SUT in uses-dependency
    order. Components with fewest dependencies

• Context
  - Stepwise verification between tightly coupled
  - Test each component and then integrate it with
    already-tested components

                      Bottom-Up Integration

• Context (2)
  - Useful when you can defer testing until all
    components have been developed
     • Components developed by teams
  - Early assessment of a component that must
    implement a critical requirement

• Fault Model
  - See the Integration Faults

                           Bottom-Up Integration

• Strategy
  - Test Model
     • Dependency tree
     • For each component: responsibility test suite
         - Do not only test responsibilities
             » Also exercise subcomponents

   - Test Procedure
     • Go from the leaves of the dep tree to the root
         - n levels -> n stages
     • Major steps (see next slide)

                    Bottom-Up Integration

1. Leaf-level components are coded. Test drivers
   are coded for the leaf-level components and
   the leaf level is tested by these drivers.

2. Components on the next higher level are
   coded. These components use object tested
   in a previous stage. Iterate for each level.

3. The entire system is exercised using the root-
   level component.

Bottom-Up Integration

Bottom-Up Integration

Bottom-Up Integration

                         Bottom-Up Integration

• Strategy
  - Automation
     • One driver for each component
     • Bottom-up driver corresponds to
        - Independent component
        - Set of components
     • Testing several components with a single driver may be
        - But you should avoid this
        - Reuse of drivers!
        - Reduce test driver maintenance

                       Bottom-Up Integration

• Entry Criteria
   - VM is stable
   - Components in each stage meet exit criteria
   - No anomalies interfere with integration testing

• Exit Criteria
   - Each component meets its exit criteria
   - Interface (per subcomponent) has been tested
     once or more
   - All root-level components pass their test suites

                         Bottom-Up Integration
• Disadvantages 
  - Driver development (huge code size)
     • Promote reuse of drivers and test components
  - Changes to a tested component
     • Revise test configuration
  - No direct testing of intercomponent interfaces
     • Limited to component that is root of the subtree under
  - Getting values or exceptions from lower-level
    components = difficult
  - Critical sections are tested last
     • Highest level of control in root -> tested last
     • Unstable component indications will come when
       component is exercised and not when it is integrated.

                         Bottom-Up Integration

• Advantages 
  - Start testing as soon as any leaf-level component
    is ready
  - Work in parallel
     • Using subtrees
  - Well suited to responsibility-based design
     • Only component interfaces are exercised.
        - Find mistakes in requirements

                         Bottom-Up Integration

• Known Uses
  - Used with clusters
  - Integration by Contract
     • Test design based on the relationship between client and
       server classes
  - Object Relations
     • “To test a class, what other classes must be tested?”

• Read “Testing Object-Oriented Systems” for
  more information

                               Top-down Integration

• Intent
  - Add components to the SUT in control hierarchy
    order, beginning with top-level control objects

• Context
  - Iterative and incremental approach
  - Control structure of the system
     • Modeled as a tree
     • Top-level components have control responsibilities
           - Control objects

                           Top-down Integration

• Context
  - Control objects
     • Essential control strategies: high risk
  - Focus on control object components first

  - Incremental development
     • Add layer under control layer on each increment
  - Concurrent hw/sw development
     • Develop hardware interfaces and control components at
       the same time
  - Concurrent sw development
     • Develop VM and control components at the same time

Top-down Integration

Top-down Integration

Top-down Integration

                         Top-down Integration

• Disadvantages 
  - Stub development
     • Large number of stubs coded
     • Unforeseen requirements in a lower-level component may
       change many top-level components, breaking (part of)
       the test suite
     • Stubs are implementation-specific

  - Interoperability of all components in the SUT is
    not tested until the last component replaces its
    stub and the test suite passes

                           Top-down Integration

• Advantages 
  - Early testing and integration
  - Reduced cost of driver development
     • Only one driver instead of a driver per subtree
  - Upper-level components are tested multiple times
     • Each time a lower-level component is added
     • Regression testing!
  - Parallel development
     • Different levels of components and stubs

                   Collaboration Integration

• Intent
  - Add sets of components to the SUT that are
    required to support a particular collaboration

• Context
  - Test interfaces between participants of a
  - Organize integration according to collaborations
  - Not every collaboration must necessarily be
     • Completed when all components and interfaces have
       been exercised

                     Collaboration Integration

• Context
  - Indicators (or when to use collaboration
     • Use it when SUT has clearly defined collaborations that
       can cover all components and interfaces in its scope
     • To demonstrate a working collaboration asap. High
       technical risk or critical collaboration.
     • Some system require more testing than just a pass
       through the top of the control hierarchy.
        - Example: search function with critical performance objectives

• Fault model
  - See Integration Faults

                    Collaboration Integration

• Test Procedure
  1. Develop a dependency tree
     • Map collaborations onto the tree
  2. Choose a sequence in which to apply the
     • Simplest collaborations first, complex later
     • Start with the collaboration that requires fewest stubs
     • Use a state machine that models sequential constraints on
     • Test in order of risk of disruption for system testing
  3. Develop the test suite for the first collaboration
  4. Run the test suite. Debug until the first
     collaboration passes. Revise

Collaboration Integration

Collaboration Integration

Collaboration Integration

                    Collaboration Integration

• Automation
  - Single driver coupled to the interface for the
  - Stubs required
     • For untested components that do not participate in a
       collaboration. Relevant stubs are replaced with each

• Entry Criteria
  - VM is stable
  - Components in collaboration are operable
  - No anomalies interfere with integration testing

                   Collaboration Integration

• Exit Criteria
   - Each tested collaboration meets the exit criteria
   - All component and intercomponent messages in
     the system have been exercised at least once

                   Collaboration Integration

• Disadvantages 
  - Intercollaboration dependencies may be subtle
  - Participants in a collaboration are not tested
     • Remember Big Bang Integration?
  - Requirement of stubs
  - Testing lower-level components may be difficult
  - Collaborations may be incomplete

                     Collaboration Integration

• Advantages 
  - Interface coverage in a few test runs
     • Possibly without testing all collaborations
  - Useful for system scope testing
     • Focus on end-to-end functionality
  - Minimal coupling due to a single test interface
  - Testing and integration may begin when
    components in the first collaboration are coded
     • Between top-down and bottom-up integration
  - Minimize driver development costs

                          Backbone Integration

• Intent
  - Combine elements of Top-down Integration,
    Bottom-up Integration and Big Bang
    Integration to verify interoperability among
    tightly coupled subsystems

• Context
  - Used e.g. by embedded system applications.
     • Backbone provides services essential for running tests
       and applications
     • Creating doubles for entire backbone would be impractical

                            Backbone Integration

• Fault model
  - See Integration Faults

• Strategy
  - Test model
     •   The top and possibly the second level are tested top-down
     •   Application subsystems are developed bottom-up
     •   Backbone components tested in isolation
     •   The tested components are then big banged.

                         Backbone Integration

- Test Procedure
   1. Each backbone component is tested first in isolation,
      bottom-up. Drivers and stubs are needed. (Fig 1)
   2. Application control is tested next, top-down. Stubs are
      needed. (Fig 2)
   3. Backbone components are big-banged together and
      tested under a driver. (Fig 3)
   4. Big Bang control components and backbone. (Fig 4)
   5. Use top-down on the remaining leaves. (Fig 5)

Backbone Integration

Figuur 1

Backbone Integration

Figuur 2

Backbone Integration

Figuur 3

Backbone Integration

Figuur 4

Backbone Integration

Figuur 5

                        Backbone Integration

• Automation
   - Stubs and drivers are required.

• Entry criteria
   - System architecture and component interfaces
     have been designed and passed careful review
   - Backbone and application of roughly same size and

                           Backbone Integration

• Exit criteria
   - All components meet exit criteria for applied test pattern
   - Each intercomponent interface exercised at least once.
   - Complete increment passed system scope test suite.

• Advantages 
   - Mitigates disadvantages of Top-Down and Bottom-Up
     (curtails their use at the point they lose effectiveness)

• Disadvantages 
   - Requires careful analysis of system structure and
   - Requires stubs and drivers

                          Backbone Integration

• Known uses
  - Development of Windows NT can be characterized
    as a Backbone Integration followed by High-
    Frequency Integration.

• Related patterns
  - Sandwich integration:
     • Combination of Top-Down and Bottom-Up
     • Final configuration meeting in the middle
     • But Backbone requires more testing of components in

                                 Layer Integration

• Intent
  - Uses an incremental approach to verify stability in
    a layered architecture.

• Context
  - Verifies interoperability in systems that can be
    modeled as an hierarchy that only allows
    interfaces between adjacent layers.
     • Top layer doesn’t communicate with bottom layer
     • Instead, top layer communicates with middle layer and the
       middle layer communicates with bottom layer

                                 Layer Integration

• Fault model
  - See integration faults
  - Propagation through layers must meet hard, real-
    time deadline  Inadequate performance is often
    a critical failure

• Strategy
  - Test Model
     • Top layer and possible second layer tested top-down.
     • Middle layer developed bottom-up.
     • Primary components bottom layer tested in isolation

                                     Layer Integration

  - Test procedure
     • May be Top-Down or Bottom-Up
        1. Test each layer in isolation using drivers where needed. (Fig 1)
        2. Perform Top-Down/Bottom-Up Integration of layers.
     • Figure 2 shows the test configurations for a Top-Down
       Integration, Bottom-Up would reverse the order.

• Automation
  - Requires stubs and drivers for each layer
  - Driver for top layer must be designed so it can
    exercise entire system

      Layer Integration

Figuur 1

      Layer Integration

Figuur 2

                               Layer Integration

• Entry criteria (same as Collaboration Integration)
   - VM is stable, else consider Backbone Integration
   - Components in collaboration are operable
   - No anomalies interfere with integration testing

• Exit criteria
   - Each driven component meets exit criteria for its
     test pattern
   - Each collaboration that traverses two or more
     layers has been exercised once.

                             Layer Integration

• Advantages & Disadvantages
  - Similar to those of Top-Down (for the top-down
    variant) and Bottom-Up (for the bottom-up variant)

• Known uses
  - Protocol stacks are a widely
    used layered application

                  Client/Server Integration

• Intent
  - Demonstrate stability of interactions among clients
    and servers.
  - Initially test clients and servers in isolation, then
    controlled increases in scope

• Context
  - Client/Server Integration achieves stepwise
    verification between client components that are
    loosely coupled to a single server component
  - Concurrent development/testing is possible

                     Client/Server Integration

• Fault model
  - See Integration Faults

• Strategy
  - Test model
     • Test plan must identify clients and servers
     • In two-tier star  basic integration strategy is client-server
     • In three-tier star  both client-server and server-server
       configurations, followed by client-server-server configuration

                    Client/Server Integration

- Test procedure
   • Each client is tested with stub from server
   • Server is tested with stubs from all client types
   • Next, pairs of clients tested with actual server (server may
     retain stubs for other clients)
   • Finally, all stubs are removed and individual use case tests
     are replayed.

   • Two-tier client/server star integration follows a three-step
     procedure for each client group:
       1. Representative client + server stub
       2. Server + client stub
       3. Representative client + server

Client/Server Integration

    Figuur 1

                    Client/Server Integration

• Automation
  - Test script for a stage should be designed to support
    two kinds of use
     • Should be repeatable to support regression testing upon
       integration of more components
     • Should be extensible so system scope tests may be added
       after final integration testing is completed
  - Debugging/Testing tools mostly platform-specific
     • Difficult to perform controlled testing and debugging across
       multiple platforms
  - Replicate an isomorphic test environment to
    replicate the structure of the architecture

                      Client/Server Integration

• Entry criteria
   -   VM is stable, else consider Backbone Integration
   -   Components in collaboration are operable
   -   No anomalies interfere with integration testing
   -   Multiplatform test suite installed and working
   -   Multiplatform version control system

• Exit criteria
   - Each driven component meets exit criteria for its test
   - Interface to each subcomponent exercised at least once
   - Test suite passes in test environment that is isomorphic
     with respect to the target environment

                  Client/Server Integration

• Advantages 
  - Avoids problems of Big Bang Integration
  - Order of client/server integration usually has few
    constraints  integration sequenced by priority/risk

• Disadvantages 
  - Cost of driver and stub development
  - Testers cannot exercise end-to-end use cases until
    midway or late in development cycle.

• Known uses
  - Read book for extensive example

                      Client/Server Integration

• Entry criteria
   -   VM is stable, else consider Backbone Integration
   -   Components in collaboration are operable
   -   No anomalies interfere with integration testing
   -   Multiplatform test suite installed and working
   -   Multiplatform version control system

• Exit criteria
   - Each driven component meets exit criteria for its test
   - Interface to each subcomponent exercised at least once
   - Test suite passes in test environment that is isomorphic
     with respect to the target environment

                        Integration Patterns

• Distributed Services Integration. Exercise a
  loosely coupled network of peer-to-peer
  components. Appropriate for decentralized
  networks containing peer-to-peer nodes.
  - See book for more info

• High-frequency Integration. Develop and rerun
  an integration test suite test hourly, daily, or
  weekly. Can be applied at almost any scope or
  - See book for more info


Shared By: