Docstoc

Test Requirements

Document Sample
Test Requirements Powered By Docstoc
					    David Capocci, CQA, CSTE




                         Test Requirements:
                         The Basis of Testing
                         David Capocci, CQA, CSTE
                          Sr. QA Systems Analyst
                           SAFECO Corporation
                            davcap@safeco.com
1
    David Capocci, CQA, CSTE




                               Agenda
       • Defining Test Requirements (TR)
            – What, Why, Where
       • Fitting TR‟s into the testing picture
            – What‟s within our testing process
            – Generating TR‟s
       • Organizing & Decomposing TR‟s
       • Test Requirements Hierarchy Samples
       • Setting the stage for measuring test
         coverage
2
    David Capocci, CQA, CSTE


                                 Background
        • “Test Requirements Hierarchy” is a term coined
          from Rational‟s SQA Team Test software.
        • The principle of identifying, organizing, and
          measuring test requirements is universal to many
          test processes and methodologies
        • Much of this in-service is distilled from:
                  • Rational methodologies (we are an SQA Team Test
                    house after all)
                  • QAI Workbench model
                  • Seminar topics from
                        – 19th annual International Software Testing Conference
                        – Star „98 West Conference

3
                  • Ed Kit‟s “Software Testing in the Real World”
    David Capocci, CQA, CSTE




                           Test Requirements

        • What exactly is a Test Requirement?
        • Why identify Test Requirements?
        • Where does a Test Requirement come
          from?


4                                         Defining TR‟s: What, Why, Where
    David Capocci, CQA, CSTE

                        What exactly is a Test
                          Requirement?
        Identifies the “WHAT” of testing
                  What needs to be tested, AND
                  What are you going to validate about it
        Includes both normal and error conditions
        Covers business rules, functionality, non-
         functional standards
        Do NOT have case specific data values
         assigned to them yet (data appears in test
         cases, the “How” of testing) examples…
5                                                      Defining TR‟s: What, Why, Where
    David Capocci, CQA, CSTE


            Example 1: Testing the inserting of a
                     record to a table
      Test Requirements Identified (among others):
    “Validate that you can insert an entry”
    “Validate that insertion fails if entry already present”
    “Validate that insertion fails if table already full”
    “Validate that you can insert an entry to an empty table (initial)”
     • These are test requirements NOT tests because they
       do not describe the data element being inserted
     • The data is irrelevant at this level, it will appear in
       the test cases used to cover these test requirements
     • “Validate you can insert „John Doe‟” is a test case
6      not a test requirement
                                                       Defining TR‟s: What, Why, Where
    David Capocci, CQA, CSTE


        Why identify Test Requirements?
      It‟s what QC “owns” in our workbench:
       Requirements-based or Function-based testing
      It‟s the basis for establishing the completion of testing
      Helps determine the scale of the testing effort
      Governs the types of resources you will need
      Serves to identify automation strategies you can use
      Becomes a roadmap for your testing effort
      Can be a tool for leverage and dialog with developers
       and business analysts
      Dev Team can sign off on them (Verification!)

7
                                                 Defining TR‟s: What, Why, Where
    David Capocci, CQA, CSTE



             Where does a TR come from?
     Traditional: Business Requirements,
      functionality, internal logic…
          Marketing specs, Functional specs, Technical specs
     Reality:
          “Interview Analysis”, Non-Functional Checklists
           (standards & compliance), Use Cases (from
           business scenarios and users), Discovery during
           testing, any other deliverables from previous
           workbenches (diagrams, modeling, flowcharts,
           etc.)
8                                             Defining TR‟s: What, Why, Where
    David Capocci, CQA, CSTE

                How do Test Requirements
                 relate to the Test Plan?
      Traditionally, the Test Plan has represented “what”
       was going to be tested, even including test cases.
      Paradigm is shifting: Test Plan should relate what
       your testing process (and deliverables) will be for a
       particular project.
      A Test Plan should build confidence in the testing
       process for a project, making approaches clear.
      A Test Plan can include the Test Requirements
      However, if the Test Requirements are too lengthy,
       they should become their own document: A Test
9
       Requirements Hierarchy                  Defining TR‟s: What, Why, Where
     David Capocci, CQA, CSTE




                                Agenda
         • Defining TR‟s: What, Why, Where
         • Fitting TR‟s into the testing picture
              – What‟s within our testing process
              – Generating TR‟s
         • Organizing & Decomposing TR‟s
         • Test Requirements Hierarchy Samples
         • Setting the stage for measuring test
           coverage
10
     David Capocci, CQA, CSTE


                       Drilling down: Where test
                requirements fit into the picture
                Business
               Requirement
           1


                                   Test
           Generates    M       Requirement
                                1

                                                     Test
                                                  Scenarios/   M            Executes/Runs
                                Generates     M     Cases                                1

                                                                             Test
                                                                          Procedure/
                                                                            Script


11                                                                 Fitting TR‟s into the testing picture
     David Capocci, CQA, CSTE




                                  Drilling Down
                                                                 First, Let‟s look
             Business                                          at this relationship:
            Requirement                                         What‟s within our
                                                                  testing process

                                 Test
                              Requirement


                                               Test
                                            Scenarios/
            Then we‟ll look                   Cases
          at this relationship:                                   Test
           Gernerating TR‟s                                    Procedure/
         from what feeds into                                    Script
          our testing process
12                                                       Fitting TR‟s into the testing picture
     David Capocci, CQA, CSTE



          ATM Example: Practice Writing
               Test Requirements
      Business Requirements:       Group Exercise!
                                   1. Limit the scope to these
      - “ATM must do                  3 requirements.
         withdrawals”
                                   2. What will you validate
      - “Withdrawals are between
         $20-$300”
                                      (test for)?
      - “Withdrawals are in $20    3. Are there any implied
         multiples”                   requirements that may
                                      not be written out?

13                                          What‟s within our testing process
     David Capocci, CQA, CSTE

         Example 2: Testing Withdrawals on an ATM
     Test Requirements Identified (among others):
     “Validate that a withdrawal option is available”
     "Validate that a withdrawal of a multiple of $20, between $20-$300 can be done"
     "Validate that <$20 is not allowed"
     "Validate that >$300 is not allowed"
     "Validate that $20 multiples >$300 is not allowed"
     "Validate that non-$20 multiples between $20-$300 not allowed"
     "Validate strange numeric amounts/combinations not allowed (all zero's, all 9's,
       20.0000)"
     “Validate that the withdrawal received is equal to the amount requested”
     "Validate that a valid withdrawal amount must be below the account balance”
     • These are test requirements NOT tests because they do not
       describe the data element being used (like $20, $40, $60, $1)
     • The data is irrelevant at this level, it will appear in the test cases
       used to cover these test requirements
14
                                                             What‟s within our testing process
     David Capocci, CQA, CSTE



                                Test Scenarios/Cases for -
                  “Validate that a withdrawal of a multiple of $20,
                          between $20-$300 can be done”

           Case # P/F $ entered Expected                     Actual
                                Results                      Results
             WD01         Pass        20     $20 withdrawn
             WD02         Pass        40     $40 withdrawn
             WD03         Pass        60     $60 withdrawn
             WD04         Pass        80     $80 withdrawn
             WD05         Pass       100    $100 withdrawn
              :            :           :           :
             WD13         Pass       260    $260 withdrawn
             WD14         Pass       280    $280 withdrawn
             WD15         Pass       300    $300 withdrawn
15                                                      What‟s within our testing process
     David Capocci, CQA, CSTE



         Test Procedure & Script for previous example
          Procedure:                        Script: (in   pseudo-code )

       Step 1: Insert Card                Do until EOF „until end of data file
       Step 2: Enter PIN                     Input data record
       Step 3: Select Withdraw option        Senddata CARDINFO to “Cardfield”
       Step 4: Enter dollar amount           Senddata “Enter”
       Step 5: Validate amount received      Senddata PIN to “PINFfield”
                                             Senddata “Enter”
                                             Senddata “W” to “SelectionField”
                                             Senddata AMOUNT to “DollarField”
                                             Senddata “Enter”
                                             If ErrorMsg > 0 then print ErrorMsg
                                             Print “DollarAMTgiven”
                                          Loop
16                                                          What‟s within our testing process
     David Capocci, CQA, CSTE




                                Agenda
         • Defining TR‟s: What, Why, Where
         • Fitting TR‟s into the testing picture
              – What‟s within our testing process
              – Generating TR‟s
         • Organizing & Decomposing TR‟s
         • Test Requirements Hierarchy Samples
         • Setting the stage for measuring test
           coverage
17
     David Capocci, CQA, CSTE



                        The Workbench Concept
                                      Rework




              Product                            Check     Product
              Input             DO
                                                           Output



           Entrance                                      Exit
                                     Standards
            Criteria                                     Criteria
                                       Tools




           Our workbench is called “Generating Test Requirements”
18                                                              Generating TR‟s
     David Capocci, CQA, CSTE


          Entrance Criteria for Business Requirements
                to generate Test Requirements
              Visible ?
              Clear? (unambiguous)
              Complete?
              Consistent? (conflicting requirements must be prioritized)
              Reasonable? (achievable)
              Measurable? (quantifiable)
              Modifiable? (will it change or is it stable?)
              Traceable? (the source is known)
              Dependent requirements identified?
              Testable? (given current environment, resources, skills)

19                                                               Generating TR‟s
     David Capocci, CQA, CSTE


           Exit Criteria for Test Requirements
        Can another tester create test cases/scenarios from these?
        Does a Test Requirement specify what is being tested and what
         about it we are validating? (Clear?)
        Are the Test Requirements…
             -   Complete?
             -   Consistent? (conflicting requirements must be prioritized)
             -   Reasonable? (achievable)
             -   Measurable? (quantifiable for measuring test coverage)
             -   Modifiable? (will it change or is it stable?)
             -   Traceable? (the source is known)
             -   Testable? (given current environment, resources, skills)
        Do the test requirements cover the complete scope of the project?
        Are all the test requirements verified and signed off by the
         Development Team?
20                                                                            Generating TR‟s
 David Capocci, CQA, CSTE


  When creating Test Requirements (“Do”)...
• Use “action” verbs & words
         - “Validate that…”
         - “Check for…”
         - “Test that…”
• Trace them back to the source
• Remember that different applications arrange in
  different ways
         - Think of MF, batch, C/S, web, e-commerce, GUI, etc.
         - Use “testing considerations” checklists that generally cover what kinds of
           things should be considered when testing your specific situation

 • Make your Test Requirements document a “living
21 document”                              Generating TR‟s
     David Capocci, CQA, CSTE


         Also...
 • Maintain a level of balance between too much &
   too little
             - Too High level: won‟t be useful, vague, can‟t generate test cases from it.
             - Too low level: Over-process, over documentation, no productivity
             - General rule: 5-7 levels deep in an outline format

 • Organize them!
             -   Outline/Hierarchy format recommended
             -   Look at how the BA breaks down the project into areas
             -   Look at how the PA breaks down the project into areas
             -   Organize by functional areas
             -   Organize by “types” of testing (Function vs. System vs. Non-
                 Functional)
22                                                                           Generating TR‟s
     David Capocci, CQA, CSTE




                                Agenda
         • Defining TR‟s: What, Why, Where
         • Fitting TR‟s into the testing picture
              – What‟s within our testing process
              – Generating TR‟s
         • Organizing & Decomposing TR‟s
         • Test Requirements Hierarchy Samples
         • Setting the stage for measuring test
           coverage
23
     David Capocci, CQA, CSTE


             Distinguishing the types of testing….
         I. Function-Based Tests                  IX. Stress Tests (breaking point:
         II. User Interface Tests                    memory, resources)

         III. Security Tests                      X. Resource Usage Tests
         IV. Installation Tests                   XI. Documentation Tests
         V. Configuration Tests                   XII. Compatibility Tests
         VI. Performance Tests                    XIII. Recovery Tests
             (Response)                           XIV. Serviceability Tests
         VII. Load Tests (simultaneous
             users, lots of small transactions)   and others…
         VIII. Volume Tests (Big                  *III - XIV are all “Systems-
           transactions)                             based tests”

24                                                                        Organizing TR‟s
     David Capocci, CQA, CSTE



            Organizing by Functional areas….
         • Most testers perform function-based, or
           requirements-based tests
                   - Tests on functionality that the system will
                     provide
                   - Usually scenario/case based
                   - includes normal and alternate (invalid) cases




25                                                           Organizing TR‟s
     David Capocci, CQA, CSTE



            Organizing by Functional areas….
         • Most testers also perform User Interface
           Style Tests
                   - These are generally separate from the
                     functionality that the software will provide
                   - Usually encompass the architectural standards
                     & compliance (like Windows Design
                     Standards)
                   - Also includes tests of navigation, menus,
                     admin functions (like printing, saving)

26                                                         Organizing TR‟s
     David Capocci, CQA, CSTE




                                Agenda
         • Defining TR‟s: What, Why, Where
         • Fitting TR‟s into the testing picture
              – What‟s within our testing process
              – Generating TR‟s
         • Organizing & Decomposing TR‟s
         • Test Requirements Hierarchy Samples
         • Setting the stage for measuring test
           coverage
27
     David Capocci, CQA, CSTE




          Remember this?…Drilling down
             Business
            Requirement

                               Test
                            Requirement


                                             Test
                                          Scenarios/
                                            Cases
                                                                Test
                                                             Procedure/
                                                               Script


28                                                     Fitting TR‟s into the testing picture
     David Capocci, CQA, CSTE

             Decomposing: Drilling down within a
                    Test Requirement
                                                Business
                                                Function
                                                      Tasks within the
           Business                                      Function
                                                                Transactions to
          Requirement                                           perform a task

                                                                          Data Entry Types
                                                                           for transactions


                                                                                     Field Validation
                             Test
                          Requirement


         Keep the                          Test
      function-based                    Scenarios/
        perspective                       Cases
         in mind!                                                            Test
                                                                          Procedure/
                                                                            Script

29                                                                   Fitting TR‟s into the testing picture
     David Capocci, CQA, CSTE



         Test Requirement Decomposition
                                               The "what" that the software is intended to
         Business Function                        do, like "Rent a Car to a customer"

                        Tasks within the                   The tasks involved: 1) query car rates,
                           Function                       2) check customer policy coverages, ....

                                       Transactions to               The database transactions necessary
                                       perform a task                        to perform the task.


         The types of data entry the               Data Entry Types
       transaction can process: Add,                for transactions
          Change, Delete, Inquire


               The edits on the data included within
                                                                 Field Validation
                        the different types



30                                                                                       Decomposing TR‟s
     David Capocci, CQA, CSTE



         Test Requirement Decomposition
                Business                                High level Functional Areas:
                Function                                        usually from
                                                          “Functional Spec” type
                         Tasks within the
                                                              docs, or BA work
                            Function


                                   Transactions to
                                   perform a task
          Lower level
                                             Data Entry Types
       Functional Areas:
                                              for transactions
          usually from
     “Technical Spec” type
        docs regarding                                  Field Validation
         internal logic,
           or PA work
31                                                                            Decomposing TR‟s
     David Capocci, CQA, CSTE




         Test Requirement Decomposition
         • You can start on the high level functional
           areas early into the project & build in lower
           level areas as they become available
         • Any level can contain a test requirement
           which can also be made up of (or broken
           down into) lower level test requirements



32                                                Decomposing TR‟s
     David Capocci, CQA, CSTE



                     Business Function Level
         • Try to identify “groups” of functions or
           functions connected by similar themes
                    file management functions, printing functions, help
                       functions, car rental functions, reservation functions,
                       ticket purchase functions, claim reporting functions
         • Be sure all areas of the system are covered.
           If something is left out or doesn‟t fit into a
           group, it becomes its own group.
         • It may be easier to identify functional areas
           by “window” instead of by function.
                                                  Trans Data     Field
33       Function           Task    Transaction                             Decomposing TR‟s
                                                     Type      Validation
     David Capocci, CQA, CSTE



                     Business Function Level
        • At this level, the idea is seeing the
          connections, integration, and interactions of
          the system‟s functionality.
        • May not necessarily be identifying a test
          requirement at this level as much as just
          identifying the functional area.




                                                 Trans Data     Field
34        Function          Task   Transaction                             Decomposing TR‟s
                                                    Type      Validation
     David Capocci, CQA, CSTE




                                   Task Level
         • Break down each Function area into the
           tasks within the function
         • For complex tasks, it is possible to break
           them down further into sub-tasks
         • Some Business Functions can not
           decompose into further tasks (example:
           Check Writing function)

                                                 Trans Data     Field
35         Function         Task   Transaction                             Decomposing TR‟s
                                                    Type      Validation
     David Capocci, CQA, CSTE



                                Transaction Level
         • From this level down, we start to address
           the internal things that occur to make the
           functions and tasks happen
         • Identify any logical transactions that ties the
           task to the database or any other
           transactions necessary to perform the task.
         • Identify any data processing, calculation,
           data formatting type transactions
                   Note: A screen or window may cause the execution of
                    several different logical transactions
                                   Transaction   Trans Data     Field
36      Function         Task                                              Decomposing TR‟s
                                                    Type      Validation
     David Capocci, CQA, CSTE




               Transaction Data Type Level
         • Identify which of the four types the
           transaction can become: Add, Change,
           Delete, Inquire
         • It is entirely possible that a transaction can
           be multiple types.
         • If a transaction is only one type, you can
           wrap this level up into the higher level.
                                               Trans Data
                                                              Field
37      Function          Task   Transaction      Type                   Decomposing TR‟s
                                                            Validation
     David Capocci, CQA, CSTE



                       Field Validation Level
        • Most testers like to jump directly to this
          level. It‟s the most obvious at times.
        • Field validation covers all edits & cross
          edits on fields and data.
        • Be careful of the detail you document at this
          level. Remember the separation between
          test requirement and test case.


                                                 Trans Data     Field
           Function
                                                              Validation Decomposing TR‟s
38                          Task   Transaction
                                                    Type
     David Capocci, CQA, CSTE


                       Field Validation Level
      • Not all test requirements (especially at this
        level) fit well to be documented in this manner.
                 - Example: Testing all the stated properties of windows
                   objects
                 - Use “Validate that the properties of all the objects in
                   this window match the properties listed on the Object
                   Properties Reference Table in Appendix B upon
                   initialization of the window”
                 - Don‟t list each property check as a separate test
                   requirement if it can be summarize under one test
                   requirement
                 - This is a judgement call YOU make for your given
                   project.
                                                 Trans Data     Field
           Function
                                                              Validation Decomposing TR‟s
39                          Task   Transaction
                                                    Type
     David Capocci, CQA, CSTE



             Example 3: Rental Car Application
         1. Validate that a Rental can occur.
            1.1 Check Customer policy coverage
            1.2 Query Car availability
                                                           Let‟s look at
            1.3 Query Car rates                          the lower levels
            1.4 Open a Rental ticket                        for this one

                 1.4.1 Validate that a customer record can be entered
                 1.4.2 Validate that credit card approval is obtained
                 1.4.3 Validate that status on the car record is changed
                        from “waiting” to “rented”          Then we‟ll try it
         2. Billing Function                                   on this one

         3. Reservation Function
40                                                                   Decomposing TR‟s
     David Capocci, CQA, CSTE



             Example 3: Rental Car Application
     1. Validate that a Rental can occur.
        1.4 Open a Rental ticket
             1.4.1 Validate that a customer record can be entered
                     1.4.1.1 Validate that a new customer can be added to
                             the customer table
                        1.4.1.1.1 Validate that the first name is all alpha
                        1.4.1.1.2 Validate that the age is > 21.
                        1.4.1.1.3 Validate that the phone number is
                                     numeric
                        1.4.1.1.4 Validate area code is an existing area
                                     code number.
                     1.4.1.2 Validate changing an existing customer
41                                                              Decomposing TR‟s
     David Capocci, CQA, CSTE



             Example 3: Rental Car Application
     1. Validate that a Rental can occur.
        1.4 Open a Rental ticket
             1.4.2 Validate that credit card approval is obtained


                         …fill in the lower level test requirements!

                         First, Identify any sub-areas (further tasks, or even
                                 separate transactions within this)
                         Then, Identify the lowest level field validation test
                                 requirements (think about what is typically
                                 involved with credit card authorizations)
42                                                                   Decomposing TR‟s
     David Capocci, CQA, CSTE



                   What did you come up with?
     1. Validate that a Rental can occur.
        1.4 Open a Rental ticket
             1.4.2 Validate that credit card approval is obtained
                      _________________________________________
                       _________________________________________
                       _________________________________________
                       _________________________________________
                       _________________________________________
                       _________________________________________
                       _________________________________________
                       _________________________________________
43                                                      Decomposing TR‟s
     David Capocci, CQA, CSTE



                  Possible Test Requirements...
                                                  Function
     1. Validate that a Rental can occur.
        1.4 Open a Rental ticket           Task                   Transaction

             1.4.2 Validate that credit card approval is obtained
                         1.4.2.1 Validate the expiration date is a valid future date
                         1.4.2.2 Validate the expiration date is not within 1 month of
                                  expiring.
                         1.4.2.3 Validate that the CC# is 12 digits
                         1.4.2.4 Validate that the $ amount is <= credit balance available
                         1.4.2.5 Validate that an authorization # is received.




44                                                                           Decomposing TR‟s
     David Capocci, CQA, CSTE




                                Agenda
         • Defining TR‟s: What, Why, Where
         • Fitting TR‟s into the testing picture
              – What‟s within our testing process
              – Generating TR‟s
         • Organizing & Decomposing TR‟s
         • Test Requirements Hierarchy Samples
         • Setting the stage for measuring test
           coverage
45
     David Capocci, CQA, CSTE


                                TRH Samples
         Let‟s look at a few examples of how test
           requirements can be organized:
                    - Sample 1: Excerpt from a Personal Finance app
                    - Sample 2: actually organizing Function
                      requirements, but good representative of the concept
                    - Sample 3: QBS, from Rational‟s sample project
                      (adds types of testing into the hierarchy)
                    - Sample 4: another view of the Personal Finance app
                    - Sample 5: A mainframe batch system



46
     David Capocci, CQA, CSTE




                                Test Coverage Measures
         • Test Requirements are the “what” of testing
           & are the basis for establishing the
           completion of testing
         • TR‟s give us the point of measurement for
           test coverage
         • Each TR should receive a Priority, Risk,
           and Weight
         • Each TR should be tracked for Verification
           () and Validation (%)
47                                                       Test Coverage Measures
     David Capocci, CQA, CSTE




                            Summary & Recap
         • Defined: What, Why, Where
         • Described: How TR‟s “fit” into the big
           picture
         • Decomposed: Organizing TRs & Generate
           TR‟s into a measurable format
         • Demonstrated: Sampling some TRH
         • Determined: The Measurement Connection

48

				
DOCUMENT INFO