SPI-SW Testing

Document Sample
SPI-SW Testing Powered By Docstoc
					                   LCG-SPI: SW-Testing

              LCG AppArea internal review (20/10/03)
              LCG Software Process & Infrastructure

M Gallas
CERN EP-SFT               LCG-SPI: SW-Testing          1

    SPI SW-Testing

•   SW-Testing Overview
•   LCG-SPI SW-Testing
•   Test Frameworks
•   Test Documents
•   User support

       M Gallas
       CERN EP-SFT        LCG-SPI: SW-Testing   2
SW-Testing Overview

      GOAL (from RTAG2)
•Software testing should be an                                                 Nightly Building
integral part of the software                                                        System
development in the LCG App Area.
•All level of software testing should
be run as part of an automatic             SPI SW-Testing
process.                                                              Automated testing

           Software testing                   Test FrameWorks             Use in Exp.

   Acceptance test             LHC           SW- testing support           Examples

      System test                             SW-testing polices            System
                            Sw-testing                                       Tests
   Integration test                            SW-testing doc              Integration
                             Software                                         Tests
        Unit test                                                         Work Package
                                              CODE                            Test

           M Gallas                                  Code Documentation
           CERN EP-SFT                    LCG-SPI: SW-Testing                      3
LCG-SPI SW-Testing
                                • to produce test code suitable to be run in automatic way
Aim: to help developers:
                                • to document their tests
       to help projects:
                            • to plan their test phase
                            • to run all their test in automatic way
                            • to browse and scan easily the test results and failures
Requirements from the users :
• Work with different languages (C++, Python, …) and in different environments (platforms & compilers)
• Allow two ways of testing: (1) “check the output of the test-code”
                                (2) “check the test results inside the test-code”.
• Provide a easy way to integrate existing tests.
• Allow the creation of dependencies among tests.
• Easily establish a criteria if a test should pass or fail in a given release
• Organize the tests by components or packages / integration / system /acceptance .
• Regression testing and check respect to the previous sw-release.
• Easy integration with the Nightly Building System.
• Provide also a graphical interface to run the tests and examine the test results.

Our constraints:                                                  Our inputs:
• Avoid commercial software and                                   • Contacts within HEP-community.
  licensing problems.                                             • What is available as free open source code.
• Avoid “do it yourself solutions”
•Try to adopt commonly used
  open-source software.
             M Gallas
             CERN EP-SFT                              LCG-SPI: SW-Testing                               4
 Test Frameworks: Global picture

                                                                                     End User
                                                                Nightly             Installation
                                                                Building              checks

                                   Project Release
                                                                                     QA activity

Top layer                                                                    • Uses a GUI for creating and running tests (also in batch).
 - Integrates different ways                                      QMTest     • Can run tests in parallel, supports execution of a single
to test.                                                                       test or many at once (test-cases & test-suites)
 - Common environment to                                                     • Organizes tests hierarchically
run the tests and to access                                                  • Records dependencies among tests
the test results.

Bottom layer                                                                          Oval                                 Acceptance
                                         SW Product Examples                                                                 testing

- Adaptable to the
programming language
and developer
- Prepared to be run in              CppUnit           PyUnit               Old               Test
automatic way                                                              tests             Scripts
                                               JUnit            QtUnit                                                  Unit testing
                               X-Unit family

                  M Gallas
                  CERN EP-SFT                                   LCG-SPI: SW-Testing                                    5
Test Frameworks: X-Unit family
                                                     A simple test:
     PyUnit                        1.   Subclass the TestCase CppUnit or PyUnit class
                                   2.   Override the method runTest().
                                   3.   When you want to check a value, call the
                                        (CppUnit or PyUnit) ASSERT(bool) and pass in an
    QtUnit                              expresion that is true if the test succeeds

  X-Unit family

•Main Scope: Unit–testing
•The same “assertion style” in
different languages.
•Provides:- The name of the
test case that failed.
-The name of the source file
that contains the test.
-The line number where the
failure occurred
• Different platforms/compilers:

             M Gallas
             CERN EP-SFT                             LCG-SPI: SW-Testing                  6
Test Frameworks: Oval

                                                Oval build
Oval:                                Test
• validation and regression         Source                   Executable
• developed & used within            Code
      CMS experiment                                                      Oval run

•Can be use for Unit-test to                                                     Log
Validation tests.                                                                file
•Compare the output log file with      file
a given reference file (Smart                                      Oval
comparison of those lines which                                    diff
start with “[Oval]”)
• It is possible to set different
run environments.
• Can run external scripts and
external binaries.

            M Gallas
            CERN EP-SFT                  LCG-SPI: SW-Testing                         7
Test Frameworks: QMTest

•Uses a graphical
interface for creating and
running tests
•The configuration files
are in XML and can be
created from the GUI. We
provide also script to do it
• Runs tests in parallel
• Organizes tests
• Supports execution of a
single test or many at
• Records dependencies
among tests
• Can be run in batch
mode -> easy integration
with the Nightly-Building

              M Gallas
              CERN EP-SFT      LCG-SPI: SW-Testing   8
 Test Documents
• For all project is needed
a Test Plan in order to
have a clear idea of the
type of tests, schedule,
environment and
• We use a template

•Test Case template
collects information
about the existing
tests and how they
cover the expected
functionality of the

             M Gallas
             CERN EP-SFT      LCG-SPI: SW-Testing   9
User support: Web information




                                               Test Frameworks


                                                   Test doc

     M Gallas
     CERN EP-SFT         LCG-SPI: SW-Testing                           10
User support: How-To
                                                               *       *





   3    Follow sw-testing
       Automatic testing requires
         some standardization

         M Gallas
         CERN EP-SFT                LCG-SPI: SW-Testing   11
Final Remarks
                        • Software testing is a needed step in the software
                        development process.
    SPI SW-Testing
•   SW-Testing          • SPI provides test frameworks, templates and help to
    Overview            have a coherent automatic testing procedure with a
•   LCG-SPI SW-         common interface to run and browse the test results.
•   Test Frameworks     •The proposed testing tools were chosen following the
•   Test Documents      criteria of a simple approach, tools in use in the
•   User support        experiments and freely available.

                        • Following standards makes life easier to SPI and the users.
                        •Automatic testing is always needed and is a way to consolidate
                        the software.
Thanks to:              • Bug reports and typical use-cases from the users
• LCG-PI team            should be converted into tests
• LCG-POOL team
• LCG-SEAL team         • SPI SW-testing has to evolve and consolidate following
• LCH-experiments       the needs of the LCG projects.
                        •All SPI sw-testing tools, templates, documentation can be
                         used by any project.
          M Gallas
          CERN EP-SFT                LCG-SPI: SW-Testing                      12

Shared By: