Software Testing

Document Sample
Software Testing Powered By Docstoc
					         Software Testing

  See Word file
“Software Testing”

                 8 historical definitions
• Establishing confidence that a program does what
  it is supposed to do (Hetzel, 1973).
• The process of executing a program or system
  with the intent of finding errors (Myers, 1979).
• Detecting specification errors and deviations from
  the specification.
• Any activity aimed at evaluating an attribute or
  capability of a program or system (Hetzel, 1983).

                8 historical definitions
• The measurement of software quality (Hetzel,
• The process of evaluating a program or system.
• Verifying that a system satisfies its specified
  requirements or identifying differences between
  expected and actual results.
• Confirming that a program performs its intended
  functions correctly.

                 IEEE/ANSI definitions
• The process of operating a system or component
  under specified conditions, observing or recording
  the results, and making an evaluation of some
  aspect of the system or component (IEEE/ANSI,
  1990 [Std 610.12 1990]).
• The process of analysing a software item to detect
  the difference between existing and required
  conditions (that is, bugs) and to evaluate the
  features of the software items (IEEE/ANSI, 1983
  [Std 829-1983]).
             Verification: Validation

 One kind of testing, verification, can be
  performed on any work product any
  intermediate product in the development
 The other kind of testing, validation, can
  only be performed by executing the code.

                            Validation axioms
• Testing can be used to show the presence of errors, but
  never their absence.
• One of the most difficult problems in testing is knowing
  when to stop.
• Avoid unplanned, non-reusable, throw-away test cases
  unless the program is truly a throw-away program.
• A necessary part of a test case is a definition of the
  expected output or result. Always carefully compare the
  actual versus the expected results of each test.

                            Validation axioms
• Test cases must be written for invalid and unexpected, as
  well as valid and expected, input conditions. "Invalid" is
  defined as a condition that is outside the set of valid
  conditions and should be diagnosed as such by the
  program being tested.
• Test cases must be written to generate desired output
  conditions. Less experienced testers tend to think only
  from the input perspective. Experienced testers determine
  the inputs required to generate a pre-designed set of

                           Validation axioms
• With the exception of unit and integration testing a
  program should not be tested by the person or organisation
  that developed it. Practical cost considerations usually
  require developers do unit and integration testing.
• The number of undiscovered errors is directly proportional
  to the number of errors already discovered.

                       Validation Testing

• Testing has been the basic form of defect
  removal since the software industry began.
• For a majority of software projects, it is the
  only form of defect removal utilised.
• It is surprising that so little quantitative data
  has been published on this topic.

                         Testing (Jones)

• "the dynamic execution of software and the
  comparison of the results of that execution
  against a set of known, predetermined
• i.e. validation


• The forms of testing are divided into three
  broad categories:
   general testing,
   specialised testing,
   testing that involves the users or clients

• The general forms of testing are concerned with any kind
  of software and seek to eliminate common kinds of bugs
  such as branching errors, looping errors, incorrect outputs,
  and the like.
• The specialised forms of testing are more narrow in focus
  and seek specific kinds of errors such as problems that
  only occur under full load, or problems that might slow
  down performance.
• The forms of testing involving users are aimed primarily at
  usability problems an ensuring that all requirements have
  been in fact implemented.
Approximate Distribution of Testing Methods for U.S. Software Projects

 Testing Stage                             Percentage of Projects
                                            Utilising Test Stage

 General Forms of Testing
 Subroutine testing                                 100
 Unit testing                                        99
 System testing of full application                  95
 New function testing                                90
 Regression testing                                  70
 Integration testing                                 50
 Specialised Forms of Testing
 Viral protection testing                            45
 Stress or capacity testing                          35
 Performance testing                                 30
 Security testing                                    15
 Platform testing                                    5
 Year 2000 testing                                   5
 Independent testing                                 3
 Forms of Testing Involving Users
 Customer acceptance testing                         35
 Field (Beta) testing                                30
 Usability testing                                   20
 Lab testing                                         1
 Clean-room statistical testing                      1
                            Black & White

• Black box testing: the inner structure or
  control flow of the application is not known
  or viewed as irrelevant for constructing test
• The application is tested against the external
  specifications and/or requirements in order
  to ensure that a specific set of input
  parameters will in fact yield the correct set
  of output values.                             14
                          Black & White

• White box testing: the test-case developer is
  privy to the inner structure of the
  application and knows the control flow
  through the application, or at least knows
  the control flow if the software works
• This form is also sometimes known as
  "glass box" testing
                           Black & White

• Black box testing is useful for ensuring that
  the software more or less is in concordance
  with the written specifications and written
• White box or glass box testing is useful for
  ensuring that all or at least most paths
  through the application have been executed
  in the course of testing.
Black Box, White Box, and Mixed Testing Approaches

    Subroutine testing                   White box
    Unit testing                         White box
    Viral protection testing             White box
    Stress or capacity testing           White box
    Performance testing                  White box
    Security testing                     White box
    Year 2000 testing                    White box

    System testing of full application   Black box
    New function testing                 Black box
    Lab testing                          Black box
    Usability testing                    Black box
    Customer acceptance testing          Black box
    Field (Beta) testing                 Black box
    Clean-room statistical testing       Black box

    Independent testing                  Mixed
    Regression testing                   Mixed
    Integration testing                  Mixed
    Platform testing                     Mixed
                  General: Subroutine

• Subroutine: a small collection of code that
  may constitute less than 10 statements or
  perhaps one-tenth of a function point.
• Testing consists of executing a just
  completed subroutine to see if it compiles
  properly and performs as expected.
• Subroutine testing is a "white box" form of
                          General: Unit

• Unit testing is the execution of a complete
  module or small program that will normally
  range from perhaps 100 to 1,000 source
  code statements, or roughly from 1 to
  perhaps 10 function points.
• May be performed informally, it is also the
  stage at which actual test planning and
  test-case construction begins.
                            General: Unit

• Usually performed by the programmers who
  write the module
• Unit testing is a "white box" form of testing.
• Unit testing is also often plagued by "bad
  test cases" which themselves contain errors.

                           General: New

• Often teamed with regression testing, and
  both forms are commonly found when
  existing applications are being updated or
• Aimed at validating new features that are
  being added to a software package.
• For entirely new projects, as opposed to
  enhancements, this form of testing is also
  known as "component testing”.                21
                          General: New

• Normally supported by formal test plans,
  planned test cases, and occurs on software
  that is under full configuration control.
• Defect reporting for new function testing is
  both common and reasonably accurate.
• Both "white box" and "black box" forms of
  new function testing have been noted,
  although black box testing is more common.
                  General: Regression

• The opposite of new function testing.
• In the context of testing regression means
  accidentally damaging an existing feature as
  an unintended by-product of adding a new
• Regression testing also checks to ensure
  that prior known bugs have not
  inadvertently stayed in the software after
  they should have been removed.             23
                  General: Regression

• After a few years of software evolution,
  regression testing becomes one of the most
  extensive forms of testing because the
  library of available test cases from prior
  releases tends to grow continuously.
• Regression testing involves the entire base
  code of the application which for major
  systems can exceed 10,000,000 lines of
  code or 100,000 function points.            24
                 General: Regression

• Both white box and black box forms of
  regression testing have been noted.
• Regression test libraries, though often
  extensive, are sometimes troublesome and
  have both redundant test cases and test
  cases which themselves contain errors.

                  General: Integration

• Testing on a number of modules or
  programs that have come together to
  comprise an integrated software package.
• Since integration testing may cover the
  work of dozens or even hundreds of
  programmers, it also deals with rather large
  numbers of test cases.

                  General: Integration

• Often occurs in “waves” as new builds of an
  evolving application are created.
• Applications undergoing integration testing
  are usually under formal configuration
• Integration testing normally make use of
  formal test plans, planned suites of test
  cases, and formal defect reporting
  procedures.                                27
                 General: Integration

• Both black box and white box forms of
  integration testing have been noted.
• Integration testing can be performed by
  developers themselves, by professional test
  personnel, or by software quality assurance.

                         General: System

• System testing of full application is usually
  the last form of internal testing before
  customers get involved with field testing.
• For large systems, a formal system test can
  take many months and can involve large
  teams of test personnel. Also, the entire set
  of development programmers may be
  needed in order to fix bugs that are found
  during this critical test stage.              29
                        General: System

• System testing demands formal
  configuration control and also deserves
  formal defect tracking support.
• Normally based on black box principles,
  although sometimes the white box testing
  form is used.
• System testing can be performed by
  developers, professional test personnel, or
  by quality assurance personnel.               30
                        General: System

• For software that controls physical devices
  (such as telephone switching systems) the
  phrase "system test" may include concurrent
  testing of hardware components.
• System testing may sometimes overlap a
  specialised form of testing, termed “1ab
  testing," where special laboratories are used
  to house complex new hardware/software
  products that will be tested by clients.    31
                         Special: Capacity
• Aimed at judging the ability of an
  application to function when nearing the
  boundaries of its capabilities in terms of the
  volume of information used.
• Usually a black box form of testing.
• Capacity testing may either be a separate test
  stage, or performed as a subset of integration or
  system test (it cannot be performed earlier, since
  the full application is necessary.)
                Special: Performance
• Aimed at judging whether or not an
  application can meet the performance goals
  set out for it.
• For many applications performance is only
  a minor issue, but for some kinds of
  applications it is critical.
• White box form of resting.
• Most performance testing is associated with
  integration and system testing.
                               Special: Viral
• This is rapidly moving from a specialised
  form of testing to a general one, although it
  still has been noted on less than half of our
  client's projects.
• Virus testing is a white box form of testing.
• Although commercial virus protection software
  can be run by anybody, major commercial
  developers of software also use special proprietary
  tools to ensure that master copies of software
  packages do not contain viruses.                  34
                               Special: Security
• Most common and most sophisticated for
  military software, followed by software that
  deals with very confidential information
  such as bank records, medical records, tax
  records, and the like.
• Security testing is a white box form of
  testing usually performed by highly trained
  specialised personnel.
• Some military projects use "penetration teams" who
  attempt to break the security of applications by various
  covert means …
                       Special: Platform

• Found among companies whose software
  operates on different hardware platforms
  under different operating systems.
• While the features and functions of the
  application may be identical on every
  platform, the mechanics of getting the
  software to work on various platforms
  requires separate versions and separate test
  stages for each platform.                    36
                      Special: Platform

• Usually a white box form of testing.
• Another aspect of platform testing is to
  ensure that the software package correctly
  interfaces with any other software packages
  that might be related to it. This is the stage
  where the interfaces between the two would
  be verified.
• Platform testing is also termed
  "compatibility testing" by some companies.37
                                 Special: Y2K
• The most recent form of specialised testing
  noted among SPR clients.
• By 1996 over 50 commercial tools for
  seeking out Year 2000 hits had entered the
  commercial market, and several hundred
  consulting and outsourcing companies had
  started Year 2000 services.
• Year 2000 testing was primarily a white box form
  of testing, although black box testing of inputs and
  outputs occurred too.                              38
                 Special: Independent

• Very common for military software - it was
  required by Department of Defence
• It can occur for commercial software, and
  indeed there are several commercial testing
  companies who do testing on a fee basis.
• Very rare for management information
  systems, civilian systems software projects,
  and outsource or contract software.        39
                   Special: Independent

• Performed by a separate company or at least
  a separate organisation from the one that
  built the application.
• Both white box and black box forms of
  independent testing are noted.
• A special form of independent testing may occur
  from time-to-time as part of litigation. The
  plaintiff or defendant, or both, may commission a
  third-party to test the software.
                         User: Usability

• Performed in usability laboratories.
• Usability testing involves actual clients who
  utilise the software under controlled and
  sometimes instrumented conditions so that
  their actions can be observed.
• Usability testing is common for commercial
  software produced by large companies such
  as IBM and Microsoft.
                         User: Usability

• Usability testing is a black box form of
  testing and usually occurs at about the same
  time as system test.
• Sometimes usability testing and Beta testing
  are concurrent, but it is more common for
  usability testing to precede Beta testing.

                                User: Beta

• Field (Beta) testing is a common testing
  technique for commercial software.
• An external test involving customers.
• Beta testing usually occurs after system
  testing, and is a black box form of testing.
• External Beta testing and internal usability
  testing may occur concurrently.

                                          User: Beta
• Microsoft has become famous by conducting the
  most massive external Beta tests, with more than
  10,000 customers participating.
• High-volume Beta testing with thousands of customers is
  very efficient in terms of defect removal efficiency levels
  and can exceed 85% removal efficiency if there are more
  than 1,000 Beta test participants.
• However, if Beta test participation comprises less than a
  dozen clients removal efficiency is around 35% to 50%.

                                User: Lab
• Found primarily with hybrid products that
  consist of complex physical devices that are
  controlled by software, such as telephone
  switching systems, weapons systems, and
  medical instruments.
• Normally a black box form of testing,
  although if certain kinds of problems are
  noted, such as capacity problems, white box
  testing can be part of the process, too.
       User: Customer Acceptance
• Commonly found for contract software and
  often found for management information
  systems, systems software, and military
• The only form of software where
  acceptance testing is rare or does not occur
  is that of high-volume commercial “shrink
  wrapped” software.
• Usually a black box form of testing.
                          User: Clean-room
• Found only in the context of clean-room
  development methods.
• The clean-room approach is unusual in that the
  developers do not perform unit tests, and the test
  cases themselves are based on statistical assertions of
  usage patterns.
• Clean-room testing is inextricably joined with formal
  specification methods and proofs of correctness.
• Black box and always performed by testing
  specialists or quality assurance personnel rather than
  developers themselves.                              47
Approximate Distribution of Testing Stages for U.S. Software Projects

             Number of          Percentage of Projects
             Testing Stages     Utilising Test Stages
                  1 stage                   2
                 2 stages                   8
                 3 stages                   12
                 4 stages                   14
                 5 stages                   16
                 6 stages                   18
                 7 stages                   5
                 8 stages                   5
                 9 stages                   7
                10 stages                   5
                11 stages                   3
                12 stages                   1
                13 stages                   1
                14 stages                   1
                15 stages                   1
                16 stages                   1
                17 stages                   0
                18 stages                   0
                   Total                  100 %
                             Survey Results
• Majority of software projects in the USA
  (70%) use six or fewer discrete testing
  stages, and the most common pattern of
  testing observed includes the following:
  –   Subroutine testing
  –   Unit testing
  –   New function testing
  –   Regression testing
  –   Integration testing
  –   System testing
                           Survey Results

• Below 1,000 function points (and especially
  below 100) sometimes only three testing
  stages are found, assuming the project in
  question is new and not an enhancement:
  – Subroutine testing
  – Unit testing
  – New function testing

Defect Removal and Testing Stages Noted During Litigation for Poor Quality

                            Reliable Software   Software Involved in
                                                Litigation for Poor Quality
Formal design inspections         Used          Not used
Formal code inspections           Used          Not used
Subroutine testing                Used          Used
Unit testing                      Used          Used
New function testing              Used          Rushed or omitted
Regression testing                Used          Rushed or omitted
Integration testing               Used          Used
System testing                    Used          Rushed or omitted
Performance testing               Used          Rushed or omitted
Capacity testing                  Used          Rushed or omitted

Shared By: