PPT - UVa Computer Science

Document Sample
PPT - UVa Computer Science Powered By Docstoc
					Defect Density Estimation through
   Verification and Validation


    Mark Sherriff and Laurie Williams
     North Carolina State University
               HCSS ’06
             April 19, 2006
                      Agenda
• Background
  – Motivation and Hypothesis
  – Software Certificates and Software Certificate
    Management Systems (SCMS)
• DevCOP (Defect Estimation with V&V Certificates
  on Programming)
  – Research Goal and Methodology
  – DevCOP Certificates
  – The DevCOP SCMS Eclipse plug-in
• Limitations
• Current Status
• Questions
                             2
                      Motivation
• Software Reliability
   – Often not estimated until development is complete
   – Actual reliability not known until system is shipped to
     customers
• Corrective action is more expensive later in the
  process
• If defect density could be estimated in-process…
   – Steps could be taken to address issues early
   – More economical, could improve development effort


                               3
                  Hypothesis
• Defect density estimation can be based upon the
  history of verification and validation techniques
  that have been performed on the project.

Questions that need to be answered:
• What is the best way to record V&V techniques?
• How do I build a model that can predict defect
  density with V&V information?


                          4
      Recording V&V Techniques
• Software Certificates
  – A record of a verification and validation (V&V) practice
    employed by developers and used to support
    traceability between code and evidence of the V&V
    technique used
• Software certificates could be any type of record:
  – Logs from test case runs
  – Reports from code inspections
  – Details on pair programming assignments
• However, these are all different forms of V&V
  information and maintaining this information could
  be expensive.

                              5
     Recording V&V Techniques
• Software Certificate Management Systems
  – Provides an interface and infrastructure to
    automatically create, maintain, and analyze
    software certificates
  – Benefits:
    • Software maintenance
    • Analysis of V&V technique effectiveness
    • Reference in future projects
• Current research:
  – OGI/PSU: Programatica, a SCMS for Haskell

                           6
          Parametric Modeling
• Method by which dependant variables are
  related to one or more independent
  variables with regard to previous data
• In Software Engineering…
  – Purpose is to provide an estimated answer to a
    software development question earlier in the
    development lifecycle
• Famous SE parametric models: COCOMO
  81 and COCOMO II
                         7
             Parametric Modeling
• Software Testing Reliability Early Warning
   – Java and Haskell versions
   – Uses a suite of metrics gathered on static code to
     provide a reliability estimate
   – The model is calibrated to a particular organization
     using a regression equation
   – STREW is a good option because using operational
     profiles for a similar prediction can be expensive to
     create and maintain
• If a reliability estimate can be created from testing
  and static metrics, could it be improved if we
  added other verification and validation
  information to the model?
                               8
                 DevCOP
• Research hypothesis: Defect density
  estimation can be based upon the history of
  V&V techniques that have been performed
  on the project.
• Methodology: Build a parametric model
  using software certificate information and
  previous project defect data to create a
  prediction model for future projects.

                      9
          DevCOP Certificates
• Records:
  – identifying information for the function it is
    associated with including its name, signature,
    class, and file location;
  – identifying information for the developer that
    created it;
  – the type of V&V technique used;
  – a hash of the function’s abstract syntax tree
    (AST); and
  – a significance weight.

                         10
             DevCOP Certificates
• Types of Certificates
  – Manual
     • Includes all manual techniques, such as pair programming and
       code inspections
  – Automated Static Analysis
     • Includes all techniques that can be run automatically on
       uncompiled code
  – Dynamic
     • Includes all techiques performed automatically at run time,
       such as black box testing
  – Formal
     • Includes all formal methods, such as lambda calculus and
       proofs

                                 11
  DevCOP SCMS Eclipse Plug-in
• Currently supports manual V&V techniques
  and jcoverage certificates
• Provides different methods for examining
  and managing certificate data
• Demo




                     12
Building the Model




         13
                  Limitations
• Granularity of Certificates
  – Method level, not class or line of code
• Composition of Certificates
  – Not much is known about how one V&V
    technique adds to another
• Defect Severity
  – All defects are treated equally



                          14
            Potential Side Effects
• Retrospective Causal Analysis
   – Once certificates are recorded on a project and bugs
     are reported, developers can use certificate and defect
     information to evaluate the efficacy of their V&V
     practices.
• Building certificate information in with compiled
  code base
   – If certificate information could travel with compiled
     code, it could be referenced at runtime so that other
     systems could evaluate whether it was to work with
     that system.

                              15
               Thank you!
• Questions? Queries? Quandries?

Contact Information:
Mark Sherriff
mark.sherriff@ncsu.edu
DevCOP Project:
 http://agile.csc.ncsu.edu/devcop/


                      16