Docstoc

Tools for Automating Verification and Validation

Document Sample
Tools for Automating Verification and Validation Powered By Docstoc
					        Tools for Automating Verification and
                      Validation
Validation: checking: Are we building the right system?

Verification: checking: Are we building the system right?

•   Model dependent
•   size dependent
•   platform dependent

Purpose: improve the S/W development process

•   less error-prone         - avoid delays, reworks
•   more standard
•   better defined
•   more predictable
•   faster (time-to-market)
•   better managed process
•   assist in better understanding product (helping to manage
    complexity of product)
             Automating V & V in the S/W
                 Development Process
Must be at SEI CMM Level 3 or higher.
(- process is defined)
[pre-condition for use of tools]



e.g.                                       BUT only automate
       Reqts.             High Level       where necessary
                          Design

                                         Detailed
                                         Design


 Release &
 Support
                                              Codes Inspections
                                              & Unit Test


  Regression
  Performance                Build
  ...                        & System
                             Test       Level 4 CMM
                                        - metrics to help in
                                        process management
 taking the process view for V & V tool assessment,

1) Where in the process does the tool apply?

2) What is the entry criteria to the use of the tool(set)?

3) What is the exit criteria (for stopping the use of the
   tool?)
                         Ed Kit’s Tool Classifications
 Tools for reviews and inspections
    - assist in reviews, walkthroughs and inspections of
    requirements, functional design, internal design, and code.
    - Complexity analysis
      - “20% of code causes 80%of the problems” (Pare to
    principle) [not good predicates of bad code, easy to collect $]
    - Code comprehension
      - help to understand unfamiliar code (take an unbiased look)
    - Syntax and semantic analysis
      - extensive error checking to find errors a compiler would
    miss
      - flag potential defects

 Tools for Test Planning
    - define the scope, approach, resources and schedule of testing
    activities
    - tools required are:
      - templates for test plan documentation
      - test schedule and staffing estimates
      - complexity analyzer


# of
issues
         R8 R3 R1 R2 R4 R10 R5 R6 R7 R9
               Kit’s Tool Classifications (2)
Tools for Test Design and Development
• detailing the overall test approach specified in the test plan for
   software features or combinations of features, and identifying
   and prioritizing the associated test cases.
• Not much tool support
• types of tools required:
     –   test data generator
     –   requirements-based test design tool
     –   capture/playback
     –   coverage analysis


Test Execution and Evaluation Tools
• process of executing test cases and evaluating the results
     – includes selecting test cases for execution, setting up the
       environment, running the selected tests, recording the execution
       activities, analyzing potential product failures, and measuring the
       effectiveness of the effort.
•   Types of tools required:
     –   capture/playback
     –   coverage analysis
     –   memory testing
     –   simulators and performance
     –   regression testing - necessary
               Kit’s Tool Classifications (3)

Software Testing Support Tools
• tools that lend overall support to the overall test process
• when these tools are of poor quality or non-existent the
   professional tester suffers
• types of tools required:
    – Problem management
          • defect tracking tools, etc., used to record, track, and
            assist with the management of defects and
            enhancements throughout the lifecycle
          • Configuration management
               – manages, controls and coordinates changes to
                 documents and other artifacts


Questions before tool acquisition
•   How do the tools fit into and support our test process?
•   Do we know how to plan and design tests?
•   Who will be responsible for making sure we get the proper
    training on our new tool?
•   Who will promote and support tool use within the organization
    on an ongoing basis?
            Beizer’s Test Tool Classifications
• Fundamental Tools
   –   coverage certifiers
   –   flowgraph generators and analyzers (e.g. MSC generator)
   –   metrics support
   –   instrumentation support (probe insertion & monitoring)
• Test Execution Automation
   – capture/replay
        •   test editor
        •   smart comparator
        •   local and distributed
        •   single and multi-thread
        •   exception reports
   – stubs and drivers
        •   test case initialization
        •   input simulation
        •   outcome comparison
        •   path instrumentation and verification
        •   reporting
        •   passage to next test
        •   debugging support
   – test environments
        • support for testing in a CASE environment
       Beizer’s Tool Classifications (2)



• Test Design Automation
   –   structural generators
   –   data-flow test generators
   –   functional generators
   –   FSM (EFXM) based generators
   –   random test data generators
   –   Test Result Analysis Automation
            Load and Performance Testing

Current Listings

Benchmark Factory
Chariot
CYRANO Test
ITF
Load Runner Product Family
Pegasus
PLATINUM Final Exam Internet Load Test
pre Vue-C/S
pre Vue-X
QALoad
SQA LoadTest
Teleprocessing Network Simulator
TestMaster
WebLoad
Silk Performer
   Non-GUI Drivers and Test Suite Managers


Current Listings

Acqua                        QES/EZ
AcquaNavio                    SNA Development Test Facility
AdaTEST                       SQA Manager
ANVL                          SQA Suite
ARTT                          STAR
BenchWorks                    STEPMaster
Cantata                       STW/Regression
CobRetest                     STW/Regression/MSW
CTB                           TALC2000
DejaGnu                       TBGEN
ESIM Testing Environment      TCS
  for Embedded Software       TestDirector
Internetworking Test Facility TestExpert
MYNAH                         TestMate
OTF - An Object Testing Framework TestPlan
Pilot                         TMS
Pre Vue                       TMX
QA Director                   TETware
QES/Architect                 VPRO-G
                   Test Evaluation Tools
Current Listings

Branch validator
C-Cover
CodeTEST
CoverTest
CTC++
DeepCover for Java
DeepCover for C/C++
GCT
Hindsight/TCA
Hindsight/TPA
Insure++
LDRA Testbed
Logiscope
McCabe Visual Testing Toolset
ObjectCoverage
Panorama-C/C++
pf77
PureCoverage
safec
SofInst
STW/Coverage
TCMON
TCAT for Java
TestCenter                       VisionSoft/TEST
                   GUI Test Drivers
Current Listings

ATF (Automated Test Facility)
B-Tree Validor Family
CAPBAK/X, CAPBAK/MSW
CYRANO TEST
Evaluator
Ferret
QARun
Panorama-2
PLATINUM’s Final Exam C/S-Test
PLATINUM Final Exam Internet Test
QA Partner
QC/Replay
QES/EZ for GUI
ReplayXt
RSW Web Analyst
SET (Solution Evaluation Tool)
Smalltalk Test Mentor               Vermont High Test Plus
SQA Robot                           Visual Test 4.0
STW/Web                             WinRunner
Test Edge                           Win Vue
Test Now               WITT for OS/2, WITT/Windows, X/WITT
TestRunner                          xrc - X Remote Control
TestWeb                             XRunner
                Marick’s Corner
            Site Hosted by ST Labs

http://www.stlabs.com/marick/faqs/t-design.htm



 CALIBER: www.tbi.com

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:5
posted:2/11/2012
language:English
pages:13