VV_A by decree


									Modeling and Simulation
Verification, Validation,
   & Accreditation

                       Joe Hale

      Pre-decisional; all options are still under consideration
                                           A Motivation for VV&A

                                                                                                     Crater Simulation Analysis
                                                                            “... the debris assessment team used the Crater software developed by Boeing Co.
                                                                            engineers. Crater is normally intended for prelaunch predictions about how small
                                                                            debris, usually ice, could damage the shuttle's external tank. The software is also used
                                                                            postlaunch to analyze divots in the shuttle's exterior tiles.” R. Edwards, FCW.com,
                                                                            September 8, 2003

                                                                             (Boeing) “gave their findings to NASA on Jan. 23. Within the report, however, were
                                                                            uncertainties raised by the program: the foam could potentially cut a gouge deeper
                                                                            than the thickness of tile, though the report assured NASA that Crater was
                                                                            „conservative,‟ that is, it tended to overestimate damage. The report emphasized the
                                                                            view that the tile would survive, and the engineers suggested that a more dense layer at
                                                                            the base of each tile would further blunt the effect of the foam. The mission
                                                                            management team quickly accepted the analysis the next day and moved on.” J.
                                                                            Schwartz, New York Times, August 25, 2003
“Engineering analysis (involving the Crater simulation) conducted during
                                                                            “The use of Crater in this new and very different situation compromised NASA's
the flight concluded for NASA managers that although the foam might
                                                                            ability to accurately predict debris damage in ways that Debris Assessment Team
have caused some structural damage to the wing area, it would not have      engineers did not full comprehend.” CAIB Report, August 2003
been sufficient to cause a catastrophic event.” R. Dittemore, Columbia
mission manager, February 3, 2003

"We have found the                                                                                     Relevant CAIB Findings
smoking gun. The test we
                                                                            F6.3-11 Crater initially predicted tile damage deeper than the actual tile
conducted ... demonstrates
                                                                            depth, but engineers used their judgment to conclude that damage would not
that this (foam debris) is in
                                                                            penetrate the densified layer of tile. Similarly, RCC damage conclusions were
fact the most probable
                                                                            based primarily on judgment and experience rather than analysis.
cause creating the breach
that led to the accident of
the Columbia and the loss                                                   F6.3-18 After Program managers learned about the foam strike, their belief
of the crew and vehicle."                                                   that it would not be a problem was confirmed (early, and without analysis) by
S.    Hubbard,    Columbia                                                  a trusted expert who was readily accessible and spoke from "experience." No
Accident       Investigation                                                one in management questioned this conclusion.
Board (CAIB) member, July
7, 2003
  In the beginning, we thought that NASA had made a Type II error using an unvalidated simulation but found that they had made a Type I error instead.
                                           Pre-decisional; all options are still under consideration
           Basis for NASA Approach

NASA approach based on the Defense
 Modeling and Simulation Office (DMSO)
 Recommended Practices Guide (RPG) and
 the Joint Accreditation Support Activity
 (JASA) Accreditation Support Package (ASP)

JASA provides support for the verification,
  validation and accreditation (VV&A) of
  models and simulations used in acquisition
  programs across the Services

           Pre-decisional; all options are still under consideration
           Simulation Based Acquisition

                      VV&A Purpose
Establish a clear understanding of strengths and
  weaknesses of the M&S and the bounds within which it
  is credible.

                         VV&A Policy
• It is (Proposed) ESMD Policy that:
   Models and Simulations used to support Exploration
   Systems decision-making organizations and processes
   shall be accredited for that specific purpose by the M&S

               Pre-decisional; all options are still under consideration
                                       M&S VV&A
Definitions (DMSO RPG (additions, my emphasis in blue)

  Verification:   The process of determining that a model implementation and its associated
                  data accurately represent the developer’s conceptual description and
                                “Did I build the thing right?”
                           “Are the equations solved correctly?”

  Validation:     The process of determining the degree to which a model and its associated
                  data provide an accurate representation of the real world from the
                  perspective of the intended uses of the model. [Note: differs from “meeting
                  customer’s expectation”]
                               “Did I build the right thing?”
                             “Do I have the correct equations?”

  Accreditation: The official certification that a model, simulation, or federation of models
                  and simulations and its associated data is acceptable for use for a specific
                                “Should it be used?”
                                “Should it be trusted?”

                          Pre-decisional; all options are still under consideration

 How to Validate and Accredit M&S with
       little or no Real World data?

Proposed NASA Approach:
  - Adopt a multi-level Accreditation policy
  - Determine acceptable Accreditation Level based on
      - Risk
      - Point in Lifecycle Timeline

                Pre-decisional; all options are still under consideration
                            VV&A and Risk

                                         Magnitude of risk reduction
                                     correlated with level of Accreditation

Likelihood of an
inaccurate result

                                                Consequence of
                                                inaccurate result

                    Pre-decisional; all options are still under consideration
                               VV&A And Lifecycle Timeline
     04        05        06       07          08            09              10             11         12          13                14           15

                                                                                                                                         1st Crewed
                                                                                                                                         CEV Flight

                                                        (Program Initiation)                              IOC                      FOC
                Concept          Technology              System Development &              Production & Deployment              Operations &
    Joint      Refinement        Development                 Demonstration                                   Full-rate Prod
                                                                                          Low-rate Initial                        Support
Capabilities                                                                                                 & Deployment
 Integration                                        System Integration   System Demo        Production
      &                                                                                       (LRIP)            FRP Decision
                 Concept                                                 Design Readiness
Development                                                                                                       Review
               Decision (CD)                                              Review (DRR)
                    Pre-Systems Acquisition                                  Systems Acquisition                                    Sustainment

                                              PDR                                    CDR                                      ORR
                    A&B                                          C&D                                   E&F
    Notional VV&A Implementation Phases for Legacy M&S
    A. Govt M&S VVA Level 1
    B. Contractor M&S VV&A Level 0/1
    C. Govt M&S VV&A Level 3
    D. Contractor M&S VV&A Level 2/3
    E. Govt M&S VV&A Level 4
    F. Contractor M&S VV&A Level 3/4
                                       Pre-decisional; all options are still under consideration
            Simulation Based Acquisition

  – The DMSO/JASA experience-base confirms that the magnitude of the
    VV&A effort is significant, especially for legacy M&S! There is a very
    large number of M&S in NASA that has not been through a VV&A
  – Risks and uncertainty analysis associated with use of M&S are
    qualitative assessments. Quantitative assessment methodology is
    being developed by DMSO (IOC late FY 06).
  – Facilitates reuse of M&S in follow-on spirals.
  – Inclusion of V&V in well established simulation development adds little
  – Because VV&A consists of a managed set of processes, there is no
    necessity for an up-front, all-or-nothing, go/no-go decision.
  – Implementation of VV&A will force good Configuration Management
    practices in the development of M&S.
  – One aspect of the in-process NASA culture shift

                 Pre-decisional; all options are still under consideration

What should be NASA’s policy regarding
 Contractor M&S VV&A?
 • At CE&R Workshop #2, contractors said
   NASA should impose VV&A policy on their
   • Proposed filter: only M&S used to validate a
   • VV&A Working Group proposed to refine filter(s) and
     discuss who serves as Accreditation Authority

             Pre-decisional; all options are still under consideration

To top