Validating Simulink Models Using by niusheng


									           Model-Validation in
         Model-Based Development
      Kurt Woodham               Ajitha Rajan, Mats Heimdahl
      L-3 Communications                University of Minnesota

                OSMA SAS ’08      September 8-12

SAS_08_Model_Val_Tech_Heimdahl                         MAC-T IVV-08-152
 Problem: Model Validation
•    Model-Based Development (MBD) is here to stay
         Use of MBD is accelerating
              Estimate 50% of NASA development projects using some form of
              Many advantages: model-checking, code generation, desktop testing,
               closed-loop simulation
         Enhances early detection of requirement, design, or
          implementation defects
              “Executable Specifications” enable evaluation of behavior that might
               otherwise be relegated to Inspections and Testing
•    How do we know the models are “right”?
         Manually develop black-box tests
•    When have we validated enough?
         Measure test coverage on an implementation/model

SAS_08_Model_Val_Tech_Heimdahl         2         MAC-T IVV-08-152
 Problem : Current Practice
• Measure black-box test coverage over the model
       Indirect measure
             Defects of omission in model not exposed.
                   Weak                                      Model
                  Test set

       Executable artifact is necessary
             Adequacy can only be determined late in the
              development process

SAS_08_Model_Val_Tech_Heimdahl   3      MAC-T IVV-08-152
 Goals of Project
 • Define metrics for objective,
     implementation-independent measure of
     adequacy of a black-box test suite

 • Develop tools to measure validation
     adequacy based on the defined metrics

 • Provide capability for autogeneration of
     black-box test suites
SAS_08_Model_Val_Tech_Heimdahl   4   MAC-T IVV-08-152
  Testing – What does it mean?
In General                                                   Based Testing
                                              Does it        (ABT) to
                                              implement?     Validate Model
              Does it

                                      Model                  Model-Based
                                              Does it
                                                             Testing (MBT)
                                              implement?     to Verify Code

                                     Source Code

Our contribution is in providing novel ABT capabilities

SAS_08_Model_Val_Tech_Heimdahl   5        MAC-T IVV-08-152
   What are Assertions?

                                                     Properties/ Formal

                                 Defined                 Can also be over
                                 over                     components,

      in1                                                                 out1
     ink                                                                  outm

SAS_08_Model_Val_Tech_Heimdahl             6    MAC-T IVV-08-152
   Contributions - ABT

            Assertions                          Black-Box Tests

                                                                       Measure    1
      3     Assess Model                             Model             Adequacy
            and Assertion
We provide the following contributions in the Assertion-Based testing domain
  (indicated by     in the above figure):
1. Objective, implementation-independent measure of adequacy of a black-
   box test suite
2. Auto-generation of black-box validation tests directly from assertions
3. Objective assessment of completeness of model as well as
 SAS_08_Model_Val_Tech_Heimdahl         7           MAC-T IVV-08-152
The Idea
     Write assertions in a formal notation…

      G (FD_On -> Cues_On);                1

                                                                                                OFF       {steps_remaining=   [start &&
                                                          AND   NOT                                                                                                             /steps_remaining--;
                                                                                                entry:    steps_to_cook;}      steps_to_cook>0]

      G((¬ Onside_FD_On Λ ¬
                                                                                                mode=1;                                                                                          3

                                           2                                                                                                                                    FAILED
                                                                                                                                                                                entry: mode=2;
                                 Right_Independent_Mode                                                                                                                     2                          [door_closed]

       Is_AP_Engaged) →
                                                                      OR           1
                                                                                                                         [steps_remaining<=0]                                                                       1
                                                                           Property_Satisfied                                                                                                    1
                                           3                                                                                                                    [start && ...                        [clear_off || ... 2
                                                                                                                                                                door_closed]                         !door_closed]

         X(Is_AP_Engaged →
                                                                                                                                                                                entry: mode=3;
                                                                                                                              [clear_off]                                   2
           Onside_FD_On))           Right_FGS_Active

    Temporal Logic                                               Synchronous                                                               Pri nted 14-Jul- 2006 12:51:47


     …then define structural coverage metrics to
     directly and objectively describe coverage of
SAS_08_Model_Val_Tech_Heimdahl      8                     MAC-T IVV-08-152
LTL Temporal Operators
      Operator         Notation                          Meaning
     Globally A          G(A)     Formula A is true in all states
     Future A             F(A)    Formula A is true in some future state
     A until B           AU B     Formula A is true in every state until B
                                  becomes true. B must eventually become true
                                  for the property to be true.

     Next A              X(A)     Formula A is true in the next state

         A               A           A              A                      A

         S0             S1          S2              S3                     Si


SAS_08_Model_Val_Tech_Heimdahl       9         MAC-T IVV-08-152
 Formalizing Assertions
 “If the onside FD cues are off, the onside FD cues shall
   be displayed when the AP is engaged”
 G((¬ Onside_FD_On  ¬ Is_AP_Engaged) →
   X(Is_AP_Engaged → Onside_FD_On))
 • Possible Coverage Metrics
          Assertion coverage: single test case that
           demonstrates that assertion is satisfied
               Prone to “dumb” tests, e.g., execution in which AP is
                never engaged.
          More rigorous metric is necessary

SAS_08_Model_Val_Tech_Heimdahl   10      MAC-T IVV-08-152
 Task - 1
 • Define a collection of assertion coverage criteria
 • Formalize the assertion coverage obligations

SAS_08_Model_Val_Tech_Heimdahl   11   MAC-T IVV-08-152
Antecedent Coverage

•    Many of the assertions in the FGS are of the
     form :
             Globally if „A‟ occurs then „B‟ will occur
              G (A → B)
             Two ways of satisfying (A → B)
              – A is false                                         What if:
              – A is true and B is true                          ACD → B

•    Antecedent Coverage – test cases will
     exercise the antecedent.
                     S0           S1                    Sn

                   Not A         Not A                 A, B
SAS_08_Model_Val_Tech_Heimdahl           12   MAC-T IVV-08-152
 Modified Condition/Decision
 Coverage (MC/DC)
• To satisfy MC/DC
       Every point of entry and exit in the model should be invoked
        at least once,
       Every basic condition in a decision in the model should take
        on all possible outcomes at least once, and
       Each basic condition should be shown to independently
        affect the decision’s outcome

                                 A   B    A or B
   Basic Conditions
                                 F   F       F
                                 T   F       T               effect of A
            effect of B
                                 F   T       T

SAS_08_Model_Val_Tech_Heimdahl       13       MAC-T IVV-08-152
Unique First Cause (UFC) Coverage

  “System shall eventually generate an Ack (A) or a Time
     Out (B)”
  Req. LTL property - F(A  B) .
 ¬A, ¬B         ¬A, ¬B           A, ¬ B      ¬A, ¬B                ¬A, B

    S0              S1            S2               S3                Si

Path satisfies UFC obligation for A but not B.
                                            ¬A, ¬B       ¬A, ¬B            ¬A, B
To show independence of B,
                                              S0          S1               Si

Formal UFC obligation for A : ¬(A  B) U (A  ¬B)
                                  for B : ¬(A  B) U (B  ¬A)

SAS_08_Model_Val_Tech_Heimdahl         14       MAC-T IVV-08-152
 UFC Coverage
 •   G(A)+ = {A U (a  G(A)) | a є A+}
     G(A)- = {A U a | a є A-}              Michael Whalen, Ajitha Rajan,
                                           Mats Heimdahl and Steven Miller.
 •   F(A)+ = {¬A U a | a є A+}
          - = {¬A U (a  G(¬A))| a є A-}
                                           Coverage Metrics for
                                           Requirements-Based Testing. In
     F(A)                                  Proceedings of ISSTA 2006.

 •   (A U B)+ =
         {(A  ¬B) U ((a  ¬B)  (A U B)) | a є A+} 
         {(A  ¬B) U b | b є B+}
     (A U B)- =
         {(A  ¬B) U (a  ¬B) | a є A-} 
         {(A  ¬B) U (b  ¬(A U B)) | b є B-}
 •   X(A)+ = {X(a) | a є A+}
     X(A)- = { X(a) | a є A-}

SAS_08_Model_Val_Tech_Heimdahl   15      MAC-T IVV-08-152
Task 2 – Validation Adequacy
         Measurement Tool
             Formal Assertions
                 (Eg. LTL
                                            Assertion. Cov.                    Validation
                                          Obligations/formulas                 Test Suite

               Assertion Cov.                                Run Test Suite

                                                             Evaluate and
We currently support the following                          Check formulas

coverage metrics:
• Assertion Coverage
                                                            Calculate ratio
• Assertion Antecedent Coverage                             of formulas that
                                                                were true

• Assertion UFC Coverage
                                                          Assertion Coverage
                                                           Achieved by Test

SAS_08_Model_Val_Tech_Heimdahl                   16              MAC-T IVV-08-152
 Task - 3
 • Automatically generate requirements-based
     tests from …
        Formal assertions
        Abstract model called Assertion Model created

         using assertions and environmental constraints
         (specified as invariants)
     … to provide the defined assertion coverage.

SAS_08_Model_Val_Tech_Heimdahl   17   MAC-T IVV-08-152
 Automatically Generating
 Requirements-Based Tests
              Inputs, Ouputs,               Formal Assertions               Assertion Cov.
               Environmental               (Eg. LTL properties)                Criteria
                constraints                                                   (eg. UFC)
                                                                                             Common with
                                                                                             the Adequacy
Assertions and                                                                               Measurement
environmental                                                          Generate                   Tool
 specified as
                                                                    Trap Properties
  invariants                Assertion Model                         (for Cov. Oblig.)

                                               Model Checker

                                           (Assertion-based test

SAS_08_Model_Val_Tech_Heimdahl                   18               MAC-T IVV-08-152
 What Are Model Checkers?
 •   Breakthrough technology of the 1990’s
 •   Widely used in hardware verification
          (Intel, Motorola, IBM, …)
 •   Several different types of model checkers
          Explicit, Symbolic, Bounded, Infinite Bounded, …
 •   Exhaustive search of the global state space
          Consider all combinations of inputs and states
          Equivalent to exhaustive testing of the model
          Produces a counter example if a property is not true
 •   Easy to use
          “Push button” formal methods
          Very little human effort unless you’re at the tool’s limits
 •   Limitations
          State space explosion

SAS_08_Model_Val_Tech_Heimdahl       19        MAC-T IVV-08-152
 Preliminary Evaluation
 Interested in determining:
 • Feasibility of generating assertion-based tests from a set of
          Generated assertion-based tests to provide UFC coverage over the
 •   Effectiveness of these test sets in validating the system
          Measured MC/DC achieved by the test sets over the system model

 Used three realistic sized examples:
 • Flight Guidance System (FGS),
 • and two models related to the Display Window Manager
   system (DWM1 and DWM2)

SAS_08_Model_Val_Tech_Heimdahl     20       MAC-T IVV-08-152
                                  Number of Assertion-Based Tests Generated



                                   FGS       DWM1       DWM2
                                                                              MC/DC Achieved by Assertion-Based Tests

                           Ajitha Rajan, Michael Whalen, and Mats             80
                           Heimdahl. Model Validation using                                                       MC/DC
                           Automatically Generated Requirements-                                                  Achieved
                           Based Tests. In Proceedings of 10th IEEE           40
                           High Assurance Systems Engineering                 20
                           Symposium, Nov 2007.
                                                                                     FGS   DWM1    DWM2

 Results and Analysis
• UFC test suites achieved high MC/DC coverage over
    DWM models – well defined set of assertions
•   Test-suite generated for UFC achieved very low
    MC/DC over the FGS model
     “When the FGS is in independent mode, it shall be active”.
     G(m_Independent_Mode_Condition.result →
       X(Is_This_Side_Active = 1))
                                       RSML–e Macro
 Structure of Independent_Mode_Condition is not captured in the
Independent_Mode_Condition = ((Is_LAPPR_Active & Is_VAPPR_Active
        & IS_Offside_LAPPR_Active & Is_Offside_VAPPR_Active) |
        ( Is_VGA_Active & Is_Offside_VGA_Active))

SAS_08_Model_Val_Tech_Heimdahl   22   MAC-T IVV-08-152
 Benefits of ABT
 • Saves time and effort in generating
     validation test suites from assertions
 •   Effective method for generating model
     validation tests when the assertions are well
 •   Helps in identifying missing assertions and
     over constrained models

SAS_08_Model_Val_Tech_Heimdahl   23   MAC-T IVV-08-152
 Bonus Task –
 Adequacy of Conformance Testing
                    Measure                             Measure
                    Adequacy     Conformance            Adequacy
     Model                               Run
1. Direct assessment of
   how well tests exercise
   the assertions                                                 Useful ??

2. Will expose defects of
3. Assertion coverage               Code
   could necessitate
   longer test cases than
   for model coverage
SAS_08_Model_Val_Tech_Heimdahl      24         MAC-T IVV-08-152
Assertion Coverage as an Adequacy
Measure for Conformance Testing
Hypothesis 1(H1): Conformance tests providing
assertion UFC coverage are more effective than
conformance tests providing MC/DC over the

Hypothesis 2(H2): Conformance tests providing
assertion UFC coverage in addition to MC/DC
over the model are more effective than
conformance tests providing only MC/DC over the

SAS_08_Model_Val_Tech_Heimdahl   25   MAC-T IVV-08-152
 • Used four industrial systems :
        Two models from the display window manager
        Two models representing the mode logic of a

         flight guidance system
 • Assessed effectiveness of test suites in terms
     of their fault finding ability
           Ajitha Rajan, Michael Whalen, Matt Staats, and Mats Heimdahl.
           Requirements Coverage as an Adequacy Measure for Conformance
           Testing. To Appear in Proceedings of 10th International Conference on
           Formal Engineering Methods, Oct 2008.

SAS_08_Model_Val_Tech_Heimdahl           26         MAC-T IVV-08-152
 Results – Hypothesis 1
                                                MC/DC vs UFC


     % Fault Finding

                        60                                                            Avg. MC/DC

                        40                                                            Avg. UFC


                             DWM1           DWM2           Latctl           Vertmax

                             Hypothesis 1 rejected at 5% statistical significance
                             on all but the Latctl system

SAS_08_Model_Val_Tech_Heimdahl                  27       MAC-T IVV-08-152
 Analysis – Hypothesis 1
 • Model coverage better than assertion coverage for
     measuring adequacy of conformance test suites
 •   Assertion UFC coverage is heavily dependent on
     the nature and completeness of the assertions
 •   Rigor and robustness of assertion coverage metric
     used is important
          UFC metric gets cheated when assertions are structured
           to hide the complexity of conditions

SAS_08_Model_Val_Tech_Heimdahl   28   MAC-T IVV-08-152
 Results – Hypothesis 2

                                         MC/DC vs MC/DC + UFC

    % Fault Finding

                                                                               Best MC/DC
                                                                               Avg. Combined
                           DWM1        DWM2          Latctl          Vertmax

                            Hypothesis 2 accepted at 5% statistical significance
                            on all but the DWM2 system

SAS_08_Model_Val_Tech_Heimdahl                 29        MAC-T IVV-08-152
 Analysis – Hypothesis 2
Does UFC really help in revealing additional faults?

                     UFC Achieved by   Achievable UFC        Rel. Diff
                     MC/DC suites
      DWM1           28.3%             96.9%                 70.8%

      DWM2           59.7%             64%                   6.7%

      Latctl         94.7%             99.5%                 4.8%

      Vertmax        97.4%             99%                   1.6%

SAS_08_Model_Val_Tech_Heimdahl    30      MAC-T IVV-08-152
 Summary – Bonus Task
 • UFC > MC/DC                   FALSE
          3 of the 4 case examples at 5% statistical significance
 • UFC + MC/DC > MC/DC                       TRUE
          3 of the 4 case examples at 5% statistical significance
 • Combine rigorous metrics for assertion coverage
     and model coverage to measure adequacy of
     conformance test suites
 •   UFC metric is sensitive to structure of assertions
          Need assertion coverage metrics that are robust to
           structure of assertions

SAS_08_Model_Val_Tech_Heimdahl     31    MAC-T IVV-08-152
Technology Readiness Level
                                 •    “Requirements-Based Test
                                      Generation Tool” TRL = 6
                                             System/subsystem model or prototype
                                              demonstration in a relevant
                                 •    “Validation Adequacy
                                      Measurement Tool” TRL = 6
                                             System/subsystem model or prototype
                                              demonstration in a relevant

SAS_08_Model_Val_Tech_Heimdahl       32          MAC-T IVV-08-152
 Relevance to NASA
 •   MBD is here - estimate one-half of all NASA missions in
     development or on the books will use model-based
     subsystem development
          Extensive use in avionics industry
 •   How do we know the models are right?
          Model validation problem
 •   We provide the capability to
          Objectively measure the “quality” of assertion-based black-box
           validation tests
          Objectively assess the completeness of a model
               Does the model address all assertions?
          Objectively assess the adequacy of a set of assertions
               Are there enough assertions to adequately describe the model?
          Automatically generate truly assertion-based tests

SAS_08_Model_Val_Tech_Heimdahl         33        MAC-T IVV-08-152
 Achievements to Date
 •   Formal assertion notation identified
          Most work with LTL
          Extended to work with Live Sequence Charts (LSC)
 •   Objective validation metrics defined
          Requirements, Antecedent, Unique First Cause, and Unique Cause
 •   Test case generation tool developed
          Developed tool generating tests from LTL
               Capable of generating tests to all metrics defined
          Prototype tool working on LSC developed
 •   Developed test-adequacy measurement tool for the defined
     validation metrics
 •   Evaluation of metrics and tool
 •   12 papers and one PhD dissertation (Ajitha Rajan)

SAS_08_Model_Val_Tech_Heimdahl          34        MAC-T IVV-08-152
 Next Steps
 • Investigate alternative requirements
     notations to LTL
 •   Complete empirical evaluation of the
     effectiveness in model validation
        Flight Guidance Sensor (FGS) evaluation
        Display Manager (DM) evaluation in work

        Coordinate evaluation on NASA IV&V project

 • Coordinate technology transfer
SAS_08_Model_Val_Tech_Heimdahl   35   MAC-T IVV-08-152

SAS_08_Model_Val_Tech_Heimdahl   36   MAC-T IVV-08-152

To top