TIME BOXED TESTING

Document Sample
TIME BOXED TESTING Powered By Docstoc
					TIME BOXED TESTING

          BCS SIGIST

         13th July 1998

   Graham Thomas - OSI Group



                               1
Abstract
There is great pressure upon developers today to improve productivity and
   effectiveness. To achieve this there is a move away from the traditional
   structured methodologies towards more dynamic, iterative and RAD
   approaches.
This is being combined with Object and Component based techniques, and
   delivered with a new generation of IDE’s, to produce thin client, web
   based, voice and data products.

So how do we test it?
     Propose Time Boxed Testing
     Outline an approach
     Discuss the problems
     Give a flavour of the fun ?


                                                                              2
Agenda
 Development   Lifecycle
 Testing Lifecycle
 Testing Helix
 Testing Stage by Stage
 Traceability
 Testing Process
 How to sell time boxed testing
 Lessons Learnt


                                   3
The Project
 Telephony   based project
 2 types of proprietary hardware
 Client Server running Windows NT
 Interfaces to legacy systems
 Rational Rose case tool (using UML)
 Objectory process
 Visual Studio
 Was considering Java & thin clients
 Incremental/Iterative development
                                        4
Incompatible Models?

                        Rqmts.                         UAT

                          Analysis                Sys Test

                                 Design         Int Test

                                     Build   Unit Test
      Breadth

Iterative (RAD) Model                Classic V Model

                                                             5
Difficult to Represent Test Stages
                    Component
            unit    Integration



                   unit
                                  Component
                                  Integration
 Depth
                          unit


                                      Component
                                      Integration

                                       System
         Breadth
                                      Acceptance




                                                    6
Project Lifecycle

             

                    
                        




                            ...




                                  7
Testing Lifecycle

            

                    
                        




                            ...




                                  8
A dual spiral life cycle ?

                    

                                   
                                          




              B R
              D A       E P                   ...
   Rqmts                C D   Plan
   Analysis                   Design
   Design                     Construct
   Build                      Execute


                                                    9
Testing Lifecycle
 Unit   Testing
 Component    Integration Testing
 System Testing
 Acceptance Testing
 Year   2000 Millenium Compliance
 TestingProcesses and Activities
 Tools & Traceability
 Resources
 Planning
                                     1
Scope of Test Phases
Conical
Model


   Unit    Component
   Testing Integration System     User
           Testing     Testing    Acceptance
                                  Testing



   Iterative            Release

                                               1
Tuned Test Lifecycle
               Testing Activities Brought Forward




                            System              User
  Unit    Component         Testing          Acceptance
  Testing Integration   - Non functional
                          Requirements
                                              Testing
                                           - Scenario Testing
          Testing       - Performance      - Acceptance
                         Analysis            Criteria




   Iterative               Release

                                                                1
Unit Testing
 Iterative Time-Boxed                                     Sub Unit
  Activity                               Unit Testing      Testing


 Specific to delivered       Unit
                           Integration
  components                 Testing


 ‘White Box’ testing
  techniques employed     External
                          Interface                     Unit
                           Testing
 External Interfaces                                   Testing
  tested
                              Static
 Use of tools for           Analysis

  Static & Dynamic                         Dynamic
                                           Analysis         Quality
  analysis                                                 Assurance


                                                                       1
Unit Testing
 Sub
                                    Unit                                          Package
 Unit



                                                             Unit

                            Sub                    Sub
 Sub                        Unit                   Unit                 Package
                                    Unit                  Unit
 Unit                        Sub                   Sub
                             Unit                  Unit
                                                                 Unit

                                                            Sub
                                                            Unit

 Sub
 Unit                      Sub      Unit                                          Package
                           Unit



Sub Unit Testing                    Unit Testing                 Unit Integration Testing

           Previously tested & checked-in
           Stubs, Drivers, Harnesses
                                                                                            1
Component Integration Testing
 Iterative Time-Boxed                                     Package
                                            Component      Integration
  Activity                                  Integration


 Testing prioritised by    Performance
  Business                    Analysis


 End to End process                                      Component
  threads where            Non Functional
                                                          Integration
  possible                 Requirements                   Testing
 Contains elements of
  following test phases               Functional
                                    Requirements
                                    Regression Test
 Build regression test                                    Scenario
                                                          Regression
  packs                                                      Test

                                                                         1
Component Integration Testing
Step 1 - Package Integration


                           Component
           Package


     Package                           Package Integration
               Component               must precede
                                       Component Integration
     Package

                Package
                           Component




                                                               1
Component Integration Testing
Step 2 - Component Integration


                                        Component
                                  Component
                                             Component
    Component
            Component        Component
                                               Component


                             Component
  Component   Component                          Component

          Component
                                 Component
                                             Component
                                       Component




         Drop1                           Drop2               ...
                                                                   1
Component Integration Testing
Drop by Drop                                                         
                                                             Component
                                                                    Component
                                                          Component      Component
                                  Component
      Component              Component Component             Component
                                                      Component            Component
                                                                    Component
 Component Component
                            Component                                       Component
                                          Component
                                                                   Component
                                                              Component
                                                      Component
         Component
   Component                Component
                                                                          Component
                                         Component       Component
                                Component
                                     Component                       Component
                                                              Component


                                                                                        ...




                       Automated Regression Test Pack
                                                                                              1
System Testing
 One   off activity                 Non Functional
                                     Requirements
 Structured testing
  techniques
                          Functional
 End to End process    Requirements
                        Regression Test

  threads                                                System
                                                         Testing
 Ties together the
                           Performance
  development                Analysis


  iterations
 Regression test                           Design
                                          Verification

  already constructed
                                                                   1
Acceptance Testing
 One  off activity                         Scenario
                                            Regression

 Test that the computer                    Testing

                               Acceptance
  system integrates with        Criteria
                                Testing
  business processes
 Structured testing       Business
                                               Acceptance
                           Process              Testing
  techniques               Testing


 End to End process
                            Data Take On
  threads                   & Conversion
                               Testing

 Regression test                           Usability
                                             Testing
  already constructed
                                                            2
Y2000 - Millennium Compliance
                     6 Date Scenarios
              Across  Wholly
  This
              Century  next     Jan 1st   Feb 29th   Dec 31st
 Century
             Boundary Century



                                      Separate test runs
 2 Business Tests                     Carried out in
                                       sequence
 Financial    Fiscal
  Year ?     Periods ?                Some tests can be
                                       exercised post Y2000

                                                                2
Testing Lifecycle Traceability
      High Level                User
       Function                 Requirements


            Functional
                              Non FR
           Requirement



                   Use Case


    Development          Scenario           Test Script
    Repository

                                                    Test Planning
                                Test Case

                                                                    2
Testing Processes and Activities
Process

   Quality                          Problem                         Conf. Mgmt.
  Interface                          Mgmt                            Interface




                              Test Management
Activity


Entry      Test Planning         Test Execution    Test Reporting    Test Stage   Exit
            Test Design           Test Automation Monitor            Sign off
             Test Preparation      Regression Test   Measure
              Analysis of Change                      Manage


                                                                                         2
Processes
 Essential
      Configuration Management
      Impact Analysis using a Test Planning repository
      Quality Management
 Good     practice
      Fault Reporting - on the Intranet - Live
      Gathering metric information



                                                          2
Tools
   Full set of testing tools is required
        Static analysis
        Dynamic Analysis
        Interactive debugging
        Test Automation/Capture Replay essential
        Test Planning Repository
        Performance monitoring (stress/load)
        Fault Reporting system


                                                    2
Monitoring
                          Cumulative Error Curve
140


120


100


 80


 60


 40


 20


  0
      1   2   3   4   5    6   7   8   9   10                                      Fault
                                                11   12   13   14   15   16   17

                                                                                           2
Monitoring
                                     Outstanding
50


45


40


35


30


25


20


15


10


 5

 0
     1   2   3   4   5   6   7   8    9   10   11   12   13   14   15   16   17


                                                                                  2
Monitoring
                           Faults by Priority
 140


 120


 100


  80
                                                                    A
  60                                                                B
                                                                    C
  40


  20


   0
       1   2   3   4   5    6   7    8    9     10   11   12   13

                                                                        2
Monitoring
       Aged Defect Analysis by Priority
  30



  25



  20
                                          A
  15
                                          B
                                          C
  10



   5



   0
       W1       W2        W3       W4

                                              2
Test Resources

               Component                          User
   Unit                                 System
               Integration                        Acceptance
   Testing                              Testing
               Testing                            Testing



                            TEST
                            TEAM
                                      Business     Business
  Developers   Developers              Users        Users
                            Testers



                       Secondment
                                                               3
Test Planning
  Decision

                Test Approach
  Definition




                    Test Strategy

                         Testing Process
  Development




                                Test drop 
                                              Test drop 
                                                            Test drop 

                                Non Functional Testing

                                                                          System Test
  Delivery




                                                                                        Acceptance Test
                                                                                         Y2000 Test

                                           Time-boxed Testing
                                                                                                          3
Selling the Approach
 Target   the following;
     Test Team
     Systems Architect
     Development team
     Standards & Methods
     Business Representatives
     Project Sponsor
 Via
     Animated Presentation
     Test Strategy document
                                 3
So how easy was this?




                        3
So What went well?
 Systems   Architect liked it 
 The   presentation was well received 




                                           3
What was difficult?
 Initial cool response from other members of the
  test team
 Development      Team were uneasy
      Dev. Why test anyway, our code is wonderful and our
       test harnesses will do all the testing?
 Business Representatives      were unhappy about;
    Anything less than 100% testing
    Bringing their effort forward in the development
     lifecycle
    The difference between the ‘Use Case’ modeling they
     had done, and the test planning that they would need to
     carry out
                                                               3
What was difficult? (2)
 Standards    & Methods
   Saw nothing wrong with their approach, it was either;
      • structured in which case their approach worked fine,
               or
      • it was incremental, in which case their approach
         worked fine.
      • They did not do iterative development, but if they
         did their approach would still work.
   This took several meetings to gain approval.
 Project   Sponsor
      Cancelled the project. (Time Boxed Testing was not
       cited as the reason !!!)
                                                               3
So what fun did we have ?




                            3
What did I learn from this?
   Organisations change their development approach
    more quickly than their testing approach

   There is a reluctance to do anything less than 100%
    testing

   Testing is a key and integral part of any development
    lifecycle
   The more you think about it, the more sense Time
    Boxed Testing makes


                                                            3
     Contact Information
          Graham Thomas
               OSI
         the management support company


       e-mail: gthomas@osi.co.uk
         Phone : 0171 323 3353

Copies of slides including animated gifs at
              www.osi.co.uk
                                              3

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:8
posted:6/12/2012
language:
pages:39