Docstoc

Integration and Component-based Software Testing Learning

Document Sample
Integration and Component-based Software Testing Learning Powered By Docstoc
					                                                                                                                   Learning objectives
                                                                                              •! Understand the purpose of integration testing
                                                                                                –! Distinguish typical integration faults from faults that
      Integration and Component-based                                                              should be eliminated in unit testing
                                                                                                –! Understand the nature of integration faults and how
               Software Testing                                                                    to prevent as well as detect them
                                                                                              •! Understand strategies for ordering construction
                                                                                                 and testing
                                                                                                –! Approaches to incremental assembly and testing to
                                                                                                   reduce effort and control risk
                                                                                              •! Understand special challenges and approaches
                                                                                                 for testing component-based systems
      (c) 2007 Mauro Pezzè & Michal Young                                    Ch 21, slide 1      (c) 2007 Mauro Pezzè & Michal Young               Ch 21, slide 2




              What is integration testing?                                                           Integration versus Unit Testing
                      Module test           Integration test          System test
                                                                                              •! Unit (module) testing is a necessary foundation
Specification:        Module                Interface specs, module Requirements                –! Unit level has maximum controllability and visibility
                      interface             breakdown               specification
                                                                                                –! Integration testing can never compensate for
Visible structure: Coding details           Modular structure         — none —                     inadequate unit testing
                                            (software architecture)
                                                                                              •! Integration testing may serve as a process check
                                                                                                –! If module faults are revealed in integration testing,
Scaffolding           Some                  Often extensive           Some                         they signal inadequate unit testing
required:
                                                                                                –! If integration faults occur in interfaces between
Looking for faults Modules                  Interactions,             System                       correctly implemented modules, the errors can be
in:                                         compatibility             functionality                traced to module breakdown and interface
                                                                                                   specifications

      (c) 2007 Mauro Pezzè & Michal Young                                    Ch 21, slide 3      (c) 2007 Mauro Pezzè & Michal Young               Ch 21, slide 4
                           Integration Faults
                                                                                                Example: A Memory Leak
•! Inconsistent interpretation of parameters or values
   –! Example: Mixed units (meters/yards) in Martian Lander                               Apache web server, version 2.0.48
•! Violations of value domains, capacity, or size limits                         Response to normal page request on secure (https) port
   –! Example: Buffer overflow
•! Side effects on parameters or resources
   –! Example: Conflict on (unspecified) temporary file
                                                                             static void ssl io filter disable(ap filter t *f)
•! Omitted or misunderstood functionality                                    { bio filter in ctx t *inctx = f->ctx;
   –! Example: Inconsistent interpretation of web hits
                                                                                                                        No obvious error, but
•! Nonfunctional properties                                                      inctx->ssl = NULL;       Apache leaked memory
   –! Example: Unanticipated performance issues
                                                                                 inctx->filter ctx->pssl =slowly (in normal use) or
                                                                                                           NULL;
•! Dynamic mismatches                                                                                                   quickly (if exploited for a
   –! Example: Incompatible polymorphic method calls                         }
                                                                                                                        DOS attack)

    (c) 2007 Mauro Pezzè & Michal Young                     Ch 21, slide 5        (c) 2007 Mauro Pezzè & Michal Young                        Ch 21, slide 6




                  Example: A Memory Leak                                                        Example: A Memory Leak
           Apache web server, version 2.0.48                                              Apache web server, version 2.0.48
  Response to normal page request on secure (https) port                         Response to normal page request on secure (https) port


static void ssl io filter disable(ap filter t *f)                            static void ssl io filter disable(ap filter t *f)
{ bio filter in ctx t *inctx = f->ctx;                                       { bio filter in ctx t *inctx = f->ctx;
   SSL_free(inctx -> ssl);                                                      SSL_free(inctx -> ssl);
                              The missing code is for a                                                    Almost impossible to find
   inctx->ssl = NULL;         structure defined and                             inctx->ssl = NULL;         with unit testing.
   inctx->filter ctx->pssl =created elsewhere,
                               NULL;                                            inctx->filter ctx->pssl =(Inspection and some
                                                                                                            NULL;
                              accessed through an                                                          dynamic techniques could
}                                                                            }
                                          opaque pointer.                                                               have found it.)

    (c) 2007 Mauro Pezzè & Michal Young                     Ch 21, slide 7        (c) 2007 Mauro Pezzè & Michal Young                        Ch 21, slide 8
                 Maybe you’ve heard ...                                                                      Translation...
•! Yes, I implemented                                                            •! Yes, I implemented   •! I didn’t think at all
   !module A", but I                                                                !module A", but I       about the strategy
   didn’t test it                                                                   didn’t test it          for testing. I didn’t
   thoroughly yet. It                                                               thoroughly yet. It      design !module A" for
   will be tested along                                                             will be tested along    testability and I
   with !module A" when                                                             with !module A" when    didn’t think about
   that’s ready.                                                                    that’s ready.           the best order to
                                                                                                            build and test
                                                                                                            modules !A" and !B".


    (c) 2007 Mauro Pezzè & Michal Young                        Ch 21, slide 10      (c) 2007 Mauro Pezzè & Michal Young                Ch 21, slide 11




            Integration Plan + Test Plan                                                          Big Bang Integration Test
                                                                                 An extreme and desperate approach:
                                             •! Integration test                 Test only after integrating all modules
     ...
                                                plan drives and is
                                                driven by the
     ...                                                                         +!Does not require scaffolding
                                                project “build
                                                                                        •!The only excuse, and a bad one
 Build Plan                      Test Plan      plan”
                                                                                 -! Minimum observability, diagnosability, efficacy,
                                                –! A key feature of the
     ...                                           system architecture              feedback
       System Architecture                         and project plan              -! High cost of repair
                                                                                    •! Recall: Cost of repairing a fault rises as a function of
                                                                                       time between error and repair

    (c) 2007 Mauro Pezzè & Michal Young                        Ch 21, slide 12      (c) 2007 Mauro Pezzè & Michal Young                Ch 21, slide 13
  Structural and Functional Strategies                                                                          Top down .
•! Structural orientation:                                                                                                 Top

   Modules constructed, integrated and tested
   based on a hierarchical project structure
  –! Top-down, Bottom-up, Sandwich, Backbone
                                                                                                    stub A
•! Functional orientation:                                                                                                stub B        stub C


   Modules integrated according to application
   characteristics or features                                                     Working from the top level (in terms of “use”
  –! Threads, Critical module                                                      or “include” relation) toward the bottom.
                                                                                   No drivers required if program tested from
                                                                                   top-level interface (e.g. GUI, CLI, web app,
                                                                                   etc.)

   (c) 2007 Mauro Pezzè & Michal Young                           Ch 21, slide 14    (c) 2007 Mauro Pezzè & Michal Young                          Ch 21, slide 15




                              Top down ..                                                                      Top down ...
                                          Top                                                                              Top




                     A                   stub B         stub C                                        A                     B             C




                                                  Write stubs of called or                                                         As modules replace
                                                  used modules at each                                                             stubs, more
                   stub X                stub Y                                                     stub X                stub Y
                                                  step in construction                                                             functionality is
                                                                                                                                   testable


   (c) 2007 Mauro Pezzè & Michal Young                           Ch 21, slide 16    (c) 2007 Mauro Pezzè & Michal Young                          Ch 21, slide 17
               Top down ... complete                                                                        Bottom Up .
                                      Top




                  A                                                                              Driver
                                                                                                                       Starting at the leaves of the
                                      B               C
                                                                                                                       “uses” hierarchy, we never
                                                                                                                       need stubs
                                               ... until the program is
                                               complete, and all
                  X                    Y       functionality can be                                X

                                               tested


(c) 2007 Mauro Pezzè & Michal Young                            Ch 21, slide 18   (c) 2007 Mauro Pezzè & Michal Young                         Ch 21, slide 19




                          Bottom Up ..                                                                    Bottom Up ...

                                                                                                 Driver




                Driver                Driver     ... but we must                                   A
                                                                                                                             ... an intermediate
                                                 construct drivers for                                                       module replaces a
                                                 each module (as in                                                          driver, and needs its
                                                 unit testing) ...                                                           own driver ...
                  X                     Y                                                          X                    Y




(c) 2007 Mauro Pezzè & Michal Young                            Ch 21, slide 20   (c) 2007 Mauro Pezzè & Michal Young                         Ch 21, slide 21
                         Bottom Up ....                                                                   Bottom Up .....
                Driver                Driver
                                                                                                 Driver                 Driver            Driver




                  A                     B
                                                                                                   A                      B                 C




                                                                                                                                    ... so we may have
                                                                                                                                    several working
                  X                     Y
                                                                                                   X                       Y        subsystems ...



(c) 2007 Mauro Pezzè & Michal Young                           Ch 21, slide 22   (c) 2007 Mauro Pezzè & Michal Young                                Ch 21, slide 23




               Bottom Up (complete)                                                                          Sandwich .
                                       Top                                                                            Top (parts)




                  A                     B             C                                          Stub                                       C




                                               ... that are eventually                                                              Working from the
                                               integrated into a                                                                    extremes (top and
                  X                     Y                                                                                  Y
                                               single system.                                                                       bottom) toward center,
                                                                                                                                    we may use fewer
                                                                                                                                    drivers and stubs

(c) 2007 Mauro Pezzè & Michal Young                           Ch 21, slide 24   (c) 2007 Mauro Pezzè & Michal Young                                Ch 21, slide 25
                            Sandwich ..                                                                           Thread ...
                                      Top (more)                                                                          Top




                  A                                       C                                           A                                C



                                                                                                                   A “thread” is a portion of several
                                                    Sandwich integration                                           modules that together provide a
                                                    is flexible and                                                user-visible program feature.
                  X                       Y                                                           X
                                                    adaptable, but
                                                    complex to plan


(c) 2007 Mauro Pezzè & Michal Young                               Ch 21, slide 26   (c) 2007 Mauro Pezzè & Michal Young                        Ch 21, slide 27




                              Thread ...                                                                          Thread ...
                                        Top                                                                               Top




                  A                       B               C                                           A                   B            C




                                                   Integrating one                                                              As in sandwich
                                                   thread, then another,                                                        integration testing, we
                  X                       Y
                                                   etc., we maximize                                  X                    Y
                                                                                                                                can minimize stubs
                                                   visibility for the user                                                      and drivers, but the
                                                                                                                                integration plan may
                                                                                                                                be complex
(c) 2007 Mauro Pezzè & Michal Young                               Ch 21, slide 28   (c) 2007 Mauro Pezzè & Michal Young                        Ch 21, slide 29
                          Critical Modules                                                           Choosing a Strategy

•! Strategy: Start with riskiest modules                                      •! Functional strategies require more planning
   –! Risk assessment is necessary first step                                   –! Structural strategies (bottom up, top down,
   –! May include technical risks (is X feasible?), process                        sandwich) are simpler
      risks (is schedule for X realistic?), other risks                         –! But thread and critical modules testing provide
                                                                                   better process visibility, especially in complex
•! May resemble thread or sandwich process in                                      systems
   tactics for flexible build order                                           •! Possible to combine
   –! E.g., constructing parts of one module to test                            –! Top-down, bottom-up, or sandwich are reasonable
      functionality in another                                                     for relatively small components and subsystems
•! Key point is risk-oriented process                                           –! Combinations of thread and critical modules
   –! Integration testing as a risk-reduction activity,                            integration testing are often preferred for larger
                                                                                   subsystems
      designed to deliver any bad news as early as possible
   (c) 2007 Mauro Pezzè & Michal Young                      Ch 21, slide 30      (c) 2007 Mauro Pezzè & Michal Young                   Ch 21, slide 31




    Working Definition of Component                                               Components — Related Concepts
•! Reusable unit of deployment and composition                                •! Framework
   –! Deployed and integrated multiple times                                         •!Skeleton or micro-architecture of an application
                                                                                     •!May be packaged and reused as a component, with “hooks”
   –! Integrated by different teams (usually)
                                                                                       or “slots” in the interface contract
       •!Component producer is distinct from component user
                                                                              •! Design patterns
•! Characterized by an interface or contract
                                                                                     •!Logical design fragments
       •!Describes access points, parameters, and all functional and
                                                                                     •!Frameworks often implement patterns, but patterns are not
         non-functional behavior and conditions for using the
                                                                                       frameworks. Frameworks are concrete, patterns are
         component
                                                                                       abstract
       •!No other access (e.g., source code) is usually available
                                                                              •! Component-based system
•! Often larger grain than objects or packages
                                                                                     •!A system composed primarily by assembling components,
   –! Example: A complete database system may be a                                     often “Commercial off-the-shelf” (COTS) components
      component                                                                      •!Usually includes application-specific “glue code”
   (c) 2007 Mauro Pezzè & Michal Young                      Ch 21, slide 33      (c) 2007 Mauro Pezzè & Michal Young                   Ch 21, slide 34
       Component Interface Contracts                                          Challenges in Testing Components
•! Application programming interface (API) is                              •! The component builder’s challenge:
   distinct from implementation                                               –! Impossible to know all the ways a component may be
  –! Example: DOM interface for XML is distinct from                             used
     many possible implementations, from different                            –! Difficult to recognize and specify all potentially
     sources                                                                     important properties and dependencies
•! Interface includes everything that must be
   known to use the component                                              •! The component user’s challenge:
  –! More than just method signatures, exceptions, etc                        –! No visibility “inside” the component
  –! May include non-functional characteristics like                          –! Often difficult to judge suitability for a particular
     performance, capacity, security                                             use and context
  –! May include dependence on other components
   (c) 2007 Mauro Pezzè & Michal Young                   Ch 21, slide 35      (c) 2007 Mauro Pezzè & Michal Young                   Ch 21, slide 36




 Testing a Component: Producer View                                             Testing a Component: User View
•! First: Thorough unit and subsystem testing                              •! Not primarily to find faults in the component
  –! Includes thorough functional testing based on                         •! Major question: Is the component suitable for
     application program interface (API)                                      this application?
  –! Rule of thumb: Reusable component requires at least                      –! Primary risk is not fitting the application context:
     twice the effort in design, implementation, and                              •!Unanticipated dependence or interactions with environment
     testing as a subsystem constructed for a single use                          •!Performance or capacity limits
     (often more)                                                                 •!Missing functionality, misunderstood API
•! Second: Thorough acceptance testing                                        –! Risk high when using component for first time
  –! Based on scenarios of expected use                                    •! Reducing risk: Trial integration early
  –! Includes stress and capacity testing                                     –! Often worthwhile to build driver to test model
       •!Find and document the limits of applicability
                                                                                 scenarios, long before actual integration
   (c) 2007 Mauro Pezzè & Michal Young                   Ch 21, slide 37      (c) 2007 Mauro Pezzè & Michal Young                   Ch 21, slide 38
 Adapting and Testing a Component                                                             Summary

                             Application                     •! Integration testing focuses on interactions
                                                               –! Must be built on foundation of thorough unit testing
                                                               –! Integration faults often traceable to incomplete or
                                Adaptor                           misunderstood interface specifications
                                                                    •!Prefer prevention to detection, and make detection easier
                                                                      by imposing design constraints

                             Component                       •! Strategies tied to project build order
                                                               –! Order construction, integration, and testing to
                                                                  reduce cost or risk
•! Applications often access components through              •! Reusable components require special care
   an adaptor, which can also be used by a test                –! For component builder, and for component user
   driver
 (c) 2007 Mauro Pezzè & Michal Young       Ch 21, slide 39      (c) 2007 Mauro Pezzè & Michal Young                     Ch 21, slide 40

				
DOCUMENT INFO