ICME course verification and validation

Document Sample
ICME course verification and validation Powered By Docstoc
					         ICME V&V Guidelines and Recommended Best Practice
    Notice Distribution subject to restrictions on the "TITLE" worksheet of this document


                Recommended Best Practice and Supporting Tools
          Verification and Validation of ICME for Aerospace Applications
                                   B.A. Cowles and D.G. Backman
                                             15-Jun-11



                                      Appendix I
                             Recommended Best Practice
                        ICME Verification and Validation Process

Introduction: This document is intended to succinctly present a recommended best practice for
assessment and planning of Verification and Validation for ICME methods and models (V&V). It
was developed specifically for aerospace structural materials. It was intended to include a simple
listing of recommended steps in the best practice, an illustration of the recommended best practice in
flow-chart format, and the various support tools and references that have been developed to assist
practitioners. These support tools are in the form of Excel spreadsheet tables, and include an ICME
V&V System-Level Checklist, an ICME V&V Model Checklist, a Tool Maturity Level (TML)
Assessment Guide, and a Risk Matrix tool for assessment and mitigation of application risks.

Recommended Practice for ICME V&V Planning and Assessment: Rigorous,
systematic verification and validation is regarded critical to successful development and broad
implementation of ICME. Development of a comprehensive V&V plan and model assessment is
highly recommended whenever a new or derivative model is under development, and whenever a
new program or project application is planned. The recommended practice for V&V planning and
assessment can be simply stated in the following steps:

   1. Define and document the model and the intended application.
   2. Develop a comprehensive V&V plan for each model or sub-model.
   3. Assess the tool maturity level that is required and that will be attained at completion of model
      development and V&V efforts.
   4. Iterate the V&V plan and develop any risk mitigation plans that may be required for the
      application.
   5. Execute the development and V&V plans and assess the outcome.

The recommended process is conceptually illustrated in its simplest form by the flow chart in Figure
1-1. Note that the V&V Plan and Tool Maturity Level Assessment are related and are intended to be
interactive. Criteria in the TML Assessment Guide relate to V&V Checklist items, and an objective
assessment of the expected state of tool maturity is of course dependent upon the plans and
execution of the ICME V&V Checklist items.
                   Initiate V&V Planning and
                     Assessment Process
              (ICME development activity, or current model
                           assessment)




              1. Definition and Documentation
                 (Models, inputs, outputs, applications)




            2. Develop the ICME V&V Plan
              (System-Level and Model Checklists)




             3. Tool Maturity Level Assessment
           (ICME TML Assessment Guide, and risk mitigation if
                            required)



                                                                Execution of ICME V&V Plans
                       V&V Planning and                            • Verification (of code, models)
                         Assessment                                • UQ (Uncertainty Quantification)
                          Complete                                 • Validation (sub-models, system)
                                                                   • Risk mitigation plans if required.



       Figure 1-1: Flow Chart of Simplified ICME V&V Planning and Assessment Process.

The recommended practice is illustrated in more detail in Figure 1-2. The intent is to show all of the
key process steps, and indicate where the various support tools should be applied. The
recommended process can be described by the following steps:

   1. Initiate V&V Panning and Assessment Process: when a new or derivative ICME method
      or model is proposed or under development, or when an existing model is planned for use in
      a new application. This step represents a formal commitment to enter into V&V assessment
      and planning activity.
   2. Definition and Documentation: intended to formally document the model or method,
      including the intended output, the intended application, and any required inputs or
      supporting analyses or data that may be required. It is when the model developer(s) or
      researcher(s) must assess customer or project requirements and determine at a high level
      how the proposed or current ICME approach relates. The objectives of the ICME model or
      method development and V&V activity should be specified in this step.
   3. System and Application Assessment: the ICME V&V Systems Level Checklist should be
      addressed in this step. This step is intended to complete detailed agreement and
      documentation of the application requirements and ICME model objectives and approach
      between the model developers and the project or program team for the intended application.
 4. Assessment of Tool Maturity Level (TML) Requirements: The TML requirements for the
    intended application should be assessed in this step. This requires that the TRL, MRL, IPD,
    or other appropriate technology or product development requirements be assessed for the
    intended application, and a determination made as to what Tool Maturity Level is
    appropriate or required for the ICME method or model. This assessment is key to guiding
    the level and fidelity of the activities that are required for successful V&V. Trending or
    ranking applications may require limited V&V demonstration; use of ICME models to

         1. Initiate V&V Planning and
             Assessment Process
      (ICME development activity, or current model         7. Tool Maturity Level Assessment
                   assessment)                                   (ICME TML Assessment Guide)



      2. Definition and Documentation
         (Models, inputs, outputs, applications)
                                                                      8. V&V and TML
                                                                         Application
     3. System/Application Assessment                                   Assessment
          (ICME V&V System-Level Checklist)                            (Model judged fully
                                                                          adequate for
                                                                         application?)


        4. Assess TML Requirement                                                                          Yes
 (TML Assessment Guide – what TML is required?)                                    No
                                                                 9. Risk-Consequences
                                                                       Assessment
                                                             (Develop risk mitigation plans if required)


 5. ICME V&V Model Checklists
       (for each model and sub-model)
                                                                     10. V&V Planning
                                                                     and Assessment
                                                                         Complete



                    6. V&V Plan
No                     Review                        Yes     Execution of ICME V&V Plans
                   (Consistent with                               • Verification (of code, models)
                     ASME V&V                                     • UQ (Uncertainty Quantification)
                   Guide 10-2006?)                                • Validation (sub-models, system )
                                                                  • Risk mitigation plans if required.


     Figure 1-2: Flow Chart of Recommended ICME V&V Planning and Assessment Process.


    significantly guide or reduce experimental or other analytical efforts may require more
    detailed V&V activities, and use of ICME for key decisions or analyses with limited
    supporting data may require extensive V&V activity including benchmark case
    demonstrations.
 5. ICME V&V Model Checklists: V&V requirements, planning, and tracking the execution
    of the V&V plan should be addressed in this step by use of the ICME
       V&V Model Checklist. Individual checklists should be developed for each model or
       significant sub-model used in the process.
   6. V&V Plan review: should include an assessment and review of the overall plan for
       completeness and adequacy. It is highly recommended that the plan be reviewed for
       consistency with the guidelines presented in the ASME V&V Guide 10-2006. Any
       recognized deficiencies in the plan should be addressed or documented in this step. It is
       intended that this step represent a “gate review,” where passing the review constitutes
       program or project concurrence with the V&V assessments and plan for the ICME model
       and application.
   7. Tool Maturity Level Assessment (TML): This step represents the assessment of the
       expected TML that will result upon successful execution of the V&V plan, or the current
       state of TML assessed for existing models when being reviewed for a new or expanded
       application. It is intended to use the Tool Maturity Level (TML) Assessment Guide.
       Although steps 5 and 7 are shown sequentially, it is intended and expected that these steps
       be interactive and integrated, as was illustrated in Figure 1-1.
   8. V&V and Tool Maturity Level Application Assessment: This step is intended to
       determine whether the resulting model maturity will be fully adequate for the intended
       application, or if some form of risk analyses or risk mitigation should be investigated. As
       such, it is illustrated as a decision step – where the model developers in conjunction with
       program or project representatives, should determine whether additional risk mitigation is
       warranted.
   9. Risk-Consequences Assessment: This step is recommended when results of the assessment
       performed in Step 8 indicate that some form of risk mitigation is appropriate for use of the
       ICME methods or models for the intended specific application. It is highly recommended
       that this process be utilized whenever significant program or project decisions will be based
       on the ICME methods or models under development or review. The recommended project
       risk management approach is based on the NASA “Likelihood vs. Consequences 5X5 Risk
       Matrix” [8]. The matrix modified for ICME V&V application is recommended.
   10. V&V Planning and Assessment Complete: this represents that the plans and assessments
       are complete and agreed upon between program or project management and the ICME
       model developers.

Completion of the preceding process ensures that the execution phase for ICME V&V can be
initiated with confidence, including all verification, uncertainty quantification, and validation
efforts. It is certainly expected that periodic review and reassessment would be conducted, and any
appropriate changes implemented.

Closing Comments: It was the intention of the authors to provide a useful, rational, and
structured process to facilitate and standardize ICME V&V activities. Associated support tools were
developed, documented, and integrated to aid in practitioner implementation. The intent was to
develop a process that would be broadly applicable and flexible, but not overly prescriptive. In the
end, successful use is dependent upon the practitioners who employ the process and those who
require its use.
                                        ICME V&V Guidelines and Recommended Best Practice
                                   Notice: Distribution subject to restrictions on the "TITLE" worksheet of this document



                         Verification and Validation ICME System Level Checklist
Project:                                                                   Date:                           Version #
        Planning




                                                                                                                          See Status Key and note at the
        Status




                                                                                                                             bottom of the checklist
Item #                 Definition of Customer’s ICME Needs and Business Case Established
A.1          * • ICME opportunity identified and high level goals, approach and payoff established
A.2          * • Specific customer(s) has been identified who could benefit from proposed ICME output to support their decision-making
A.2.1        *    – Decision(s) affected by proposed ICME model predictions and associated TRL/MRL levels identified and understood
A.2.2        *    – Current decision-making process and associated data/methods and accuracy requirements identified and assessed
A.2.3             – Technical, cost, and schedule strengths, weaknesses, and issues identified for current decision-making process
A.3            • Definition of ICME opportunity refined including simulation scope and bounds (reality of interest)
A.3.1             – Customer input applied to define range of applicability and accuracy requirements for the ICME modeling system
A.4            • Business case established for applying ICME to modify and strengthen traditional decision-making approach
A.5            • Discussions held with customer to review business case, address ICME requirements, and obtain approval
A.5.1             – Customer feedback obtained and addressed relative to range of applicability and accuracy requirements
A.5.2             – Customer feedback obtained and addressed relative to risks associated with use of ICME to modify decision-making process
A.5.3             – Approval requirements established for application of ICME to modify decision-making process
A.5.4             – Customer agreement on business case and support for modified decision-making process, if customer requirements are met
                       ICME Modeling System Formulation
B.1          * •       Participating organization(s) roles & responsibilities defined and agreed-upon detailed work plan established
B.2            •       Overall modeling system architecture established
B.2.1        *          – Agreement achieved within the team on appropriate approach for individual models within the system (stat, phenom, phys)
B.2.2        *          – Modeling system structure diagrammed showing model information flow (inputs/output) and data exchange formats established
B2.3                    – Provisional, estimated system uncertainty assessment carried out and uncertainty goals established for each modeling module
B.3                •   Software platform(s) identified for individual modeling modules and software suppliers integrated into program as required
B.3.1                   – Feedback obtained from possible software suppliers regarding model accuracy requirements, integration issues, and other risks
B.3.2        * •       Model integration strategy established and required integration software/approach identified
B.4            •       Computational issues have been assessed, including execution time, computer resources, and operator training requirements
B.5            •       Implementation issues identified by entire team (module owners and possible software suppliers)
                       ICME Modeling System Risks
C.1          * • Project risk management plan established (format, update frequency, roles & responsibilities)
C.2            • Risk assessment performed for individual models, the modeling system, implementation, and customer acceptance
C.1.1            – Risks identified and rated relative to both probability of occurrence and impact for technical, cost, and schedule risks
C.1.2            – Abatement option(s) identified for highest risks and abatement plan established
                       ICME Modeling System Uncertainty Quantification Methods & Tools
D.1          * • Technical uncertainty quantification expert resource identified and engaged
D.2            • Project uncertainty quantification methods, tools, and application approach established and communicated within the team
D.2.1             – Team-wide approach established for modeling sensitivity studies and identify key input variables and internal modeling parameters
D.2.2             – Statistical methods established to quantify and compare modeling and experimental uncertainty in support of model validation efforts
D.2.3             – Methods established to calibrate models (optimization) & quantify the effect of model calibration on error and range of applicability
D.3            • System wide error/uncertainty propagation analysis strategy and methods formulated and high-level notional plan established
D.3.3             – Roles and responsibilities for execution of system level error/uncertainty propagation analysis determined
                       ICME Modeling System Level Verification and Validation
E.1          * • Hierarchical validation strategy formulated and high-level notional validation plan established
E.1.1             – Roles and responsibilities for execution of system level V&V distributed among subsidiary modeling teams
E.2            • Model system level integration tested and verified
E.3            • System level validation modeling and experimental design completed
E.3.1             – Validation experiments listed with description of approach, methods, measurements, and pedigree data
E.3.2             – Validation modeling trials, model outputs, and formats defined
E.3.3             – Validation uncertainty quantification approach, methods, and tools determined

Check Box Color Code                                              Activities relevant to early ICME project Planning are delineated with an
                   Not Relevant                                   asterisk in the "Planning" column
                   Completed successfully
                   In-process, OK
                   In-process, with issues
                   Not started

                                                       Notice: Distribution subject to restrictions on title page of this document.
                                       ICME V&V Guidelines and Recommended Best Practice
                                  Notice: Distribution subject to restrictions on the "TITLE" worksheet of this document

         Verification and Validation ICME System Level Checklist and TML Integration
Project:                                                                 Date:                            Version #



                                                                                                                                                                         Recommended
                                                                                                                                                                              TML
        Planning




                                                                                                                            See Status Key and note at the                Integration
        Status




                                                                                                                               bottom of the checklist                    Levels* (at
Item #              Definition of Customer’s ICME Needs and Business Case Established                                                                                     Completion)

A.1          * • ICME opportunity identified and high level goals, approach and payoff established                                                                             1
A.2          * • Specific customer(s) has been identified who could benefit from proposed ICME output to support their decision-making                                         1
A.2.1        *    – Decision(s) affected by proposed ICME model predictions and associated TRL/MRL levels identified and understood                                           2,3
A.2.2        *    – Current decision-making process and associated data/methods and accuracy requirements identified and assessed                                             2,3
A.2.3             – Technical, cost, and schedule strengths, weaknesses, and issues identified for current decision-making process                                            2,3
A.3            • Definition of ICME opportunity refined including simulation scope and bounds (reality of interest)                                                            2
A.3.1             – Customer input applied to define range of applicability and accuracy requirements for the ICME modeling system                                             2
A.4            • Business case established for applying ICME to modify and strengthen traditional decision-making approach                                                     2
A.5            • Discussions held with customer to review business case, address ICME requirements, and obtain approval                                                        3
A.5.1             – Customer feedback obtained and addressed relative to range of applicability and accuracy requirements                                                      3
A.5.2             – Customer feedback obtained and addressed relative to risks associated with use of ICME to modify decision-making process                                   3
A.5.3             – Approval requirements established for application of ICME to modify decision-making process                                                                3
A.5.4             – Customer agreement on business case and support for modified decision-making process, if customer requirements are met                                     3
                    ICME Modeling System Formulation
B.1          * • Participating organization(s) roles & responsibilities defined and agreed-upon detailed work plan established                                                 2
B.2            • Overall modeling system architecture established                                                                                                              1
B.2.1        *    – Agreement achieved within the team on appropriate approach for individual models within the system (stat, phenom, phys)                                   1,2
B.2.2        *    – Modeling system structure diagrammed showing model information flow (inputs/output) and data exchange formats established                                 1,2
B2.3              – Provisional, estimated system uncertainty assessment carried out and uncertainty goals established for each modeling module                                2
B.3            • Software platform(s) identified for individual modeling modules and software suppliers integrated into program as required                                    2
B.3.1             – Feedback obtained from possible software suppliers regarding model accuracy requirements, integration issues, and other risks                             2,3
B.3.2        * • Model integration strategy established and required integration software/approach identified                                                                 2,3
B.4            • Computational issues have been assessed, including execution time, computer resources, and operator training requirements                                     3
B.5            • Implementation issues identified by entire team (module owners and possible software suppliers)                                                               3
                    ICME Modeling System Risks
C.1          * • Project risk management plan established (format, update frequency, roles & responsibilities)                                                                  2
C.2            • Risk assessment performed for individual models, the modeling system, implementation, and customer acceptance                                                  3
C.1.1            – Risks identified and rated relative to both probability of occurrence and impact for technical, cost, and schedule risks                                   3,4,5
C.1.2            – Abatement option(s) identified for highest risks and abatement plan established                                                                            3,4,5
                    ICME Modeling System Uncertainty Quantification Methods & Tools
D.1          * • Technical uncertainty quantification expert resource identified and engaged                                                                                   1,2
D.2            • Project uncertainty quantification methods, tools, and application approach established and communicated within the team                                     1,2,3
D.2.1             – Team-wide approach established for modeling sensitivity studies and identify key input variables and internal modeling parameters                           2
D.2.2             – Statistical methods established to quantify and compare modeling and experimental uncertainty in support of model validation efforts                        2
D.2.3             – Methods established to calibrate models (optimization) & quantify the effect of model calibration on error and range of applicability                       2
D.3            • System wide error/uncertainty propagation analysis strategy and methods formulated and high-level notional plan established                                    3
D.3.3             – Roles and responsibilities for execution of system level error/uncertainty propagation analysis determined                                                  3
                    ICME Modeling System Level Verification and Validation
E.1          * • Hierarchical validation strategy formulated and high-level notional validation plan established                                                               3
E.1.1             – Roles and responsibilities for execution of system level V&V distributed among subsidiary modeling teams                                                   3
E.2            • Model system level integration tested and verified                                                                                                            3
E.3            • System level validation modeling and experimental design completed                                                                                            4
E.3.1             – Validation experiments listed with description of approach, methods, measurements, and pedigree data                                                       4
E.3.2             – Validation modeling trials, model outputs, and formats defined                                                                                             4
E.3.3             – Validation uncertainty quantification approach, methods, and tools determined                                                                              4

Check Box Color Code                                           Activities relevant to early ICME project Planning are delineated with an asterisk
                   Not Relevant                                in the "Planning" column
                   Completed successfully                                                          *Notes for TML Integration:
                   In-process, OK                                                                  1. Intent is to provide initial guidance for relating TML and Checklists
                   In-process, with issues                                                         2. Suggested levels are preliminary recommendations
                   Not started                                                                     3. Multiple levels indicate varying levels to be addressed




                                                                 Notice: Distribution subject to restrictions on title page of this document.
                                           ICME V&V Guidelines and Recommended Best Practice
                                     Notice: Distribution subject to restrictions on the "TITLE" worksheet of this document



                              ICME Model Verification and Validation Checklist
         Project:                                                                           Date:
Model/Sub-Model:                                                                        Version:
         Planning
         Status




                                                                                                          See Status Key at the bottom of the checklist
Item #                  ICME Model Development
W.1                 • Modeling team has reviewed, revised, and agreed with ICME modeling system formulation (incl accuracy and range of applicability req'ts)
W.2                 • Participating organization(s) roles & responsibilities defined for both modeling and experimental efforts
W.3                 • Detailed model development approach established
W.3.1                 – Prior research/literature reviewed to identify key material science & engineering physics and related models (for physically based models)
W.3.2                 – Prior research/literature reviewed to identify phenomenological, functional forms, & constraints (data-driven models)
W.3.3                 – Conceptual model established including identification of key mechanisms, boundary and initial conditions, & model parameters
W.3.4                 – Modeling assumptions listed and technical rationale provided that support the use of the assumptions
W.3.5                 – Rationale established for the development approach; addressing prior experience and/or alternative modeling approaches
W.4                 • Model flow diagram (showing submodels, BC & IC relationships, & inputs/outputs) established
W.5                 • Mathematical model developed and checked for errors
W.5.1                 – Model input variables and internal parameters listed and initial values identified or estimated
W.6                 • Initial computational model developed using initial values for inputs and internal parameters and model submitted for provisional verification
W.7                 • Initial sensitivity study conducted (per system QA methods) to rank key inputs, internal parameters using provisionally verified model
W.7.1                 – Sensitivity results used to select key parameters requiring experimental measurement and those likely requiring calibration
W.7.2                 – Impact of calibrated parameters assessed using system level UQ methods
W.8                 • Final sensitivity study conducted (per system QA methods) to verify key inputs, internal parameters using verified model & exp'tal data
W.9                 • Uncertainty analysis conducted to determine model output uncertainty based on uncertainty of model inputs and internal parameters
W.10                • Support provided to carryout system level uncertainty propagation per assigned responsibilities
                        Experimental Support for ICME Model Development & Uncertainty Assessment
X.11                • Detailed experimental approach established to support model development and uncertainty assessment
X.12                • Experimental methods, plan, and source identified to determine measureable internal parameters for the model and BC/IC conditions
X.12.1                – Experiments conducted and UQ methods applied to determine uncertainty of internal parameters and BC/IC conditions
X.13                • Experimental methods, plan, and source identified to determine measureable model input variables and outputs
X.13.1                – Experiments conducted and UQ methods applied to determine uncertainty of associated input variables & output values
X.14                • System level validation data requirements assessed and experimental measurements evaluated relative to measurement uncertainty
X.14.1                – Experiments methods and measurement techniques assessed up-front, as feasible
                        Model Verification
Y.1                 • Model verification plan established and participating organization(s) roles & responsibilities defined
Y.2                 • Prior research/literature reviewed to identify analytical models, model output, data, and/or trend info suitable for verification benchmarking
Y.3                 • Initial computational model code checked & executed to identify initial coding errors; unexpected model output investigated & errors fixed
Y.4                 • Initial computational model exercised to verify model results against identified verification benchmarks
Y.4.1                  – Model deficiencies identified during benchmarking diagnosed and model repairs/alternations implemented
Y.5                 • Final computational model exercised to verify model results against identified verification benchmarks
Y.5.1                  – Model deficiencies identified during benchmarking diagnosed and model repairs/alternations implemented
                        Model Validation & System Level Validation Support
Z.1                 •   Model Validation strategy and approach established and participating organization(s) roles & responsibilities defined
Z.2                 •   Detailed validation experimental plan defined that addresses the targeted range of applicability & accuracy/uncertainty goals
Z.3                 •   Validation experimental plan executed and results analyzed to assess data uncertainty via system level UQ methods
Z.4                 •   Validation modeling plan executed and results analyzed to assess uncertainty via system level UQ methods
Z.5                 •   UQ methods applied to experimental and modeling results to determine model accuracy, range of applicability
Z.6                 •   System Level Model validation support completed
Z.6.1                   – Detailed validation plan established to support system validation
Z.6.2                   – Supporting modeling runs executed and associated results and uncertainty information transmitted to system validation lead
Z.6.3                   – Support provided to the system validation lead in interpreting validation results and identifying modeling system improvements

Check Box Color Code
                    Not Relevant
                    Completed successfully
                    In-process, OK
                    In-process, with issues
                    Not started                           Notice: Distribution subject to restrictions on title page of this document.
                                           ICME V&V Guidelines and Recommended Best Practice
                                      Notice: Distribution subject to restrictions on the "TITLE" worksheet of this document

            ICME Model Verification and Validation Checklist and TML Integration
         Project:                                                                           Date:
Model/Sub-Model:                                                                        Version:




                                                                                                                                                                                Recommended
         Planning
         Status




                                                                                                          See Status Key at the bottom of the checklist                        TML Integration
                                                                                                                                                                                 Levels* (at
Item #                  ICME Model Development
                                                                                                                                                                                 Completion)
W.1                 • Modeling team has reviewed, revised, and agreed with ICME modeling system formulation (incl accuracy and range of applicability req'ts)                         1
W.2                 • Participating organization(s) roles & responsibilities defined for both modeling and experimental efforts                                                       1
W.3                 • Detailed model development approach established                                                                                                                 1
W.3.1                  – Prior research/literature reviewed to identify key material science & engineering physics and related models (for physically based models)                   1
W.3.2                  – Prior research/literature reviewed to identify phenomenological, functional forms, & constraints (data-driven models)                                        1
W.3.3                  – Conceptual model established including identification of key mechanisms, boundary and initial conditions, & model parameters                                 1
W.3.4                  – Modeling assumptions listed and technical rationale provided that support the use of the assumptions                                                         1
W.3.5                  – Rationale established for the development approach; addressing prior experience and/or alternative modeling approaches                                       1
W.4                 • Model flow diagram (showing submodels, BC & IC relationships, & inputs/outputs) established                                                                     1
W.5                 • Mathematical model developed and checked for errors                                                                                                             2
W.5.1                  – Model input variables and internal parameters listed and initial values identified or estimated                                                              2
W.6                 • Initial computational model developed using initial values for inputs and internal parameters and model submitted for provisional verification                  2
W.7                 • Initial sensitivity study conducted (per system QA methods) to rank key inputs, internal parameters using provisionally verified model                          2
W.7.1                  – Sensitivity results used to select key parameters requiring experimental measurement and those likely requiring calibration                                  2
W.7.2                  – Impact of calibrated parameters assessed using system level UQ methods                                                                                       3
W.8                 • Final sensitivity study conducted (per system QA methods) to verify key inputs, internal parameters using verified model & exp'tal data                         3
W.9                 • Uncertainty analysis conducted to determine model output uncertainty based on uncertainty of model inputs and internal parameters                               4
W.10                • Support provided to carryout system level uncertainty propagation per assigned responsibilities                                                                 4
                        Experimental Support for ICME Model Development & Uncertainty Assessment
X.11                • Detailed experimental approach established to support model development and uncertainty assessment                                                              1
X.12                • Experimental methods, plan, and source identified to determine measureable internal parameters for the model and BC/IC conditions                               2
X.12.1                 – Experiments conducted and UQ methods applied to determine uncertainty of internal parameters and BC/IC conditions                                            2
X.13                • Experimental methods, plan, and source identified to determine measureable model input variables and outputs                                                    3
X.13.1                 – Experiments conducted and UQ methods applied to determine uncertainty of associated input variables & output values                                          3
X.14                • System level validation data requirements assessed and experimental measurements evaluated relative to measurement uncertainty                                  4
X.14.1                 – Experiments methods and measurement techniques assessed up-front, as feasible                                                                                4
                        Model Verification
Y.1                 • Model verification plan established and participating organization(s) roles & responsibilities defined                                                          1
Y.2                 • Prior research/literature reviewed to identify analytical models, model output, data, and/or trend info suitable for verification benchmarking                  2
Y.3                 • Initial computational model code checked & executed to identify initial coding errors; unexpected model output investigated & errors fixed                      2
Y.4                 • Initial computational model exercised to verify model results against identified verification benchmarks                                                        3
Y.4.1                  – Model deficiencies identified during benchmarking diagnosed and model repairs/alternations implemented                                                       3
Y.5                 • Final computational model exercised to verify model results against identified verification benchmarks                                                          4
Y.5.1                  – Model deficiencies identified during benchmarking diagnosed and model repairs/alternations implemented                                                       4
                        Model Validation & System Level Validation Support
Z.1                 •   Model Validation strategy and approach established and participating organization(s) roles & responsibilities defined                                         1
Z.2                 •   Detailed validation experimental plan defined that addresses the targeted range of applicability & accuracy/uncertainty goals                                 2
Z.3                 •   Validation experimental plan executed and results analyzed to assess data uncertainty via system level UQ methods                                             2
Z.4                 •   Validation modeling plan executed and results analyzed to assess uncertainty via system level UQ methods                                                     3,4
Z.5                 •   UQ methods applied to experimental and modeling results to determine model accuracy, range of applicability                                                   4
Z.6                 •   System Level Model validation support completed                                                                                                              4,5
Z.6.1                    – Detailed validation plan established to support system validation                                                                                         3,4
Z.6.2                    – Supporting modeling runs executed and associated results and uncertainty information transmitted to system validation lead                                 4
Z.6.3                    – Support provided to the system validation lead in interpreting validation results and identifying modeling system improvements                             4

Check Box Color Code                                                                                *Notes for TML Integration:
                    Not Relevant                                                                    1. Intent is to provide initial guidance for relating TML and Checklists
                    Completed successfully                                                          2. Suggested levels are preliminary recommendations
                    In-process, OK                                                                  3. Multiple levels indicate varying levels to be addressed.
                    In-process, with issues
                    Not started




                                                                   Notice: Distribution subject to restrictions on title page of this document.
                                                                                                        ICME V&V Guidelines and Recommended Best Practice
                                                                                                  Notice: Distribution subject to restrictions on the "TITLE" worksheet of this document




                                                                                                                  ICME V&V Tool Maturity Level Assessment Guide
                                                                                                                                                    V3-5/30/2011




                                                                        Model Basis and                        Complexity and                                                                                                   Range of Applicability and
TML          ICME Tool Maturity Description                                                                                                              Supporting Data                      Model Verification                                                                 Validation
                                                                          Definition                           Documentation                                                                                                    Uncertainty Quantification



      Model has been defined, including inputs, outputs, and        Model basis defined.                 Model flow diagram generated.         Supporting data if empirical basis.        Code and model verification plan     UQ plan developed for model
      application intent. Analytical process is exploratory in      Application intent defined.          Overall V&V Checklist                                                            developed.                           parameter sensitivity and model output
 1    nature. Fidelity of predictions are largely unproven.         Model inputs defined.                completed.                                                                                                            uncertainty analysis.
      Provides some physical insight, but cannot reduce             Model outputs defined.
      development testing.



                                                                    Significant submodels identified.    Checklists for submodels completed.   Adequate supporting data available.        Computer code verified.              Range of applicability defined.          Validation plan developed.
                                                                    All sub-model inputs and             Dependence upon sequential or         Supporting data documented.                Model computations verified          All input parameter ranges defined.      Initial Risk-Consequence
      Proven capability for comparative assessment, ranking or      outputs defined.                     interdependent computations           Supporting data archived.                  against supporting data.             UQ sensitivity analysis performed for    assessment for application.
 2    trending. Experimental validation is still necessary. Can     Ranges for all inputs and            defined.                                                                                                              model parameters.
      drive development or assessment plan and test matrix.         outputs defined.                     User guide initiated.                                                                                                 Demonstrate trends or qualitative
                                                                    Business case generated for                                                                                                                                results consistent with known results
                                                                    application.                                                                                                                                                or expectations.

                                                                                                         Dependence upon input from            Supporting data represents full range of   Model computations verified over     Model ouput exercised and assessed       Sub-models validated.
      Material or process can be developed or assessed with                                              other analytical tools defined and    application.                               range of use.                        over full range of applicability.        Selected validation cases
      significantly reduced testing. Expectation that development                                        assessed.                             All supporting data documented and         Version control implemented and      Limitations embedded to control use or   completed.
 3    iterations will be reduced or eliminated. Accuracy and                                             User guide developed and              archived                                   documented.                          warn user if beyond range.               Risk-Consequences assessed
      uncertainty effects must be quantified. Range of                                                   documented.                                                                      Version control includes archiving                                            for application.
      applicability well defined.                                                                                                                                                         and preservation.



                                                                                                         User guide updated to reference       Supporting data adequate to determine      Code and model verification          UQ analysis performed for model          Model validated over range of
      Material or process performance and impact on system or
                                                                                                         supporting data, UQ analysis,         accuracy of the model.                     complete per ASME V&V Guide          output.                                  interest.
      application is understood. Accuracy and uncertainty effects
                                                                                                         and validation cases.                 Supporting data adequate to determine                                                                                    Model accuracy validated.
 4    have been assessed and verified. Additional data may be
      required when applied to new materials or processes, or to
                                                                                                                                               variation of the model.                                                                                                  Risk-Consequences assessed
      extend range of application.                                                                                                             All supporting data documented and                                                                                       for application.
                                                                                                                                               archived

                                                                                                         User guide updated.
                                                                                                         Vaildation cases documented                                                                                           Benchmark cases including UQ             Model fully validated over range
      All material and process performance and system                                                    and archived.                         All supporting data documented and                                              established.                             of interest.
      interaction effects are understood within defined range of                                                                               archived                                                                        Model performance validated              Model uncertainty validated.
 5    application. Analytical process can be applied without                                                                                                                                                                   equivalent or better than experimental   Benchmark validation case(s)
      testing.                                                                                                                                                                                                                 capability.                              established.
                                                                                                                                                                                                                                                                        Risk-Consequences assessed
                                                                                                                                                                                                                                                                        for application.




      version 3 - 5/30/2011




                                                                                                                      Notice: Distribution subject to restrictions on title page of this document.
 ICME V&V Guidelines and Recommended Best Practice
    Notice: Distribution subject to restrictions on the "TITLE" worksheet of this document




             Figure 1: Consequences v.s Likelihood 5X5 Risk Matrix*
                                              Adapted for ICME Applications

                                       5
          Likelihood of Occurrence




                                     (>70%)

                                       4
                                     (>50%)

                                       3
                                     (>30%)

                                       2
                                     (>5%)

                                       1
                                     (<5%)

                                                  1          2            3            4           5
                                                 Minor      Low       Moderate     Significant   Severe


                                                   Consequences (Increasing Severity >>)


      Low risk: Monitor to ensure risk remains low.
      Medium Risk: Aggressively manage. Consider alternatives or risk mitigation actions.
      High Risk: Develop and implement alternative or concurrent approaches.

*Adapted from: NASA, “Guidelines for Preparing Project Risk Management Plans”, New Millennium
Program, NMP-GL-07, 6/7/2005.


                                                                                                                         Figure 2: Example of Risk Identification, Impact, and Mitigation Actions
                                                                                                                 ICME for Prediction and Use of Residual Stresses Resulting from Processing and Machining


                                                                                                          Item                                        Risk                          Likelihood        Consequence Rating

                                                                                                                             Thermal process model does not predict                  3 (> 30%,
                                                                                                                 Risk:                                                                                 3 (Moderate)     Y
                                                                                                                             temperature transients accurately.                      Moderate)
                                                                                                           1     Impact:     Degrades all downstream analyses.
                                                                                                                             Add measurements to improve heat transfer
                                                                                                                 Mitigation:                                                              2                  3          G
                                                                                                                             predictions.

                                                                                                                               Stress relief predictions do not match measured
                                                                                                                 Risk:                                                             2 (>5%, Low)        3 (Moderate)     G
                                                                                                                               forging data after heat treatment
                                                                                                           2                 Some effect on residual stress magnitude
                                                                                                                 Impact:
                                                                                                                             predictions.
                                                                                                                 Mitigation: None required: monitor.

                                                                                                                               Range of microstructure and grain size                3 (> 30%,
                                                                                                                 Risk:                                                                                2 (Some impact)   Y
                                                                                                                               predictions exceeds desired control limits            Moderate)
                                                                                                           3
                                                                                                                 Impact:       May affect life prediction with residual stresses
                                                                                                                 Mitigation: Extend control limits; monitor production.            2 (>5%, Low)       2 (Some impact)   G

                                                                                                                               Machining distortion requires cold-straightening
                                                                                                                 Risk:                                                             2 (>5%, Low)        3 (Moderate)     G
                                                                                                                               and stress relief to meet final envelop
                                                                                                           4
                                                                                                                 Impact:       Extra process & modeling step, but predictable
                                                                                                                 Mitigation: None required: monitor.

                                                                                                                             Uncertainty quantification indicates RS values        4 (>50%, More       4 (Significant
                                                                                                                 Risk:                                                                                                  R
                                                                                                                             have insufficent fidelity for life prediction         likely than not)       impact)
                                                                                                           5                 Cannot include residual stress predictions
                                                                                                                 Impact:
                                                                                                                             directly in life predictions.
                                                                                                                             Consider alternate process or use bounded               3 (>30%,
                                                                                                                 Mitigation:                                                                           3 (Moderate)     Y
                                                                                                                             rather than explicit values.                            Moderate)




    Figure 3: Example Illustration of Risk Matrix and Mitigation Actions

                                       5
          Likelihood of Occurrence




                                     (>70%)

                                       4
                                     (>50%)

                                       3
                                     (>30%)

                                       2
                                     (>5%)

                                       1
                                     (<5%)

                                                  1          2            3            4           5
                                                 Minor      Low       Moderate     Significant   Severe


                                                   Consequences (Increasing Severity >>)


      Low risk: Monitor to ensure risk remains low.
      Medium Risk: Aggressively manage. Consider alternatives or risk mitigation actions.
      High Risk: Develop and implement alternative or concurrent approaches.

      Risk Item: from table                              Effect of risk mitigation action.

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:14
posted:9/25/2012
language:English
pages:13