Validating Simulink Models Using

Document Sample
Validating Simulink Models Using Powered By Docstoc
					           Model-Validation in
         Model-Based Development
      Kurt Woodham               Ajitha Rajan, Mats Heimdahl
      L-3 Communications                University of Minnesota

                OSMA SAS ’08      September 8-12



SAS_08_Model_Val_Tech_Heimdahl                         MAC-T IVV-08-152
 Problem: Model Validation
•    Model-Based Development (MBD) is here to stay
         Use of MBD is accelerating
              Estimate 50% of NASA development projects using some form of
               MBD
              Many advantages: model-checking, code generation, desktop testing,
               closed-loop simulation
         Enhances early detection of requirement, design, or
          implementation defects
              “Executable Specifications” enable evaluation of behavior that might
               otherwise be relegated to Inspections and Testing
•    How do we know the models are “right”?
         Manually develop black-box tests
•    When have we validated enough?
         Measure test coverage on an implementation/model

SAS_08_Model_Val_Tech_Heimdahl         2         MAC-T IVV-08-152
 Problem : Current Practice
• Measure black-box test coverage over the model
       Indirect measure
             Defects of omission in model not exposed.
                                                           Incomplete
                   Weak                                      Model
                 Black-Box
                  Test set

       Executable artifact is necessary
             Adequacy can only be determined late in the
              development process



SAS_08_Model_Val_Tech_Heimdahl   3      MAC-T IVV-08-152
 Goals of Project
 • Define metrics for objective,
     implementation-independent measure of
     adequacy of a black-box test suite

 • Develop tools to measure validation
     adequacy based on the defined metrics

 • Provide capability for autogeneration of
     black-box test suites
SAS_08_Model_Val_Tech_Heimdahl   4   MAC-T IVV-08-152
  Testing – What does it mean?
                                                             Assertion
                                     Assertions
In General                                                   Based Testing
                                              Does it        (ABT) to
                                              implement?     Validate Model
    Specification
                                       Model
              Does it
              implement?

   Implementation
                                      Model                  Model-Based
                                              Does it
                                                             Testing (MBT)
                                              implement?     to Verify Code

                                     Source Code


Our contribution is in providing novel ABT capabilities

SAS_08_Model_Val_Tech_Heimdahl   5        MAC-T IVV-08-152
   What are Assertions?

                                                     Properties/ Formal
              Assertions
                                                        Assertions



                                 Defined                 Can also be over
                                 over                     components,
                                                           interfaces,...

      in1                                                                 out1
                                       System
     ink                                                                  outm




SAS_08_Model_Val_Tech_Heimdahl             6    MAC-T IVV-08-152
   Contributions - ABT
                                   2
                              Auto-generate

            Assertions                          Black-Box Tests

                                              Validate
                                                                       Measure    1
      3     Assess Model                             Model             Adequacy
            and Assertion
            Completeness
We provide the following contributions in the Assertion-Based testing domain
  (indicated by     in the above figure):
1. Objective, implementation-independent measure of adequacy of a black-
   box test suite
2. Auto-generation of black-box validation tests directly from assertions
3. Objective assessment of completeness of model as well as
   assertions
 SAS_08_Model_Val_Tech_Heimdahl         7           MAC-T IVV-08-152
The Idea
     Write assertions in a formal notation…
                                                                                                                              microwave_library_temp/mode_logic



      G (FD_On -> Cues_On);                1
                                 Left_Independent_Mode
                                                                                                                                                               ON

                                                                                                                                                                                [steps_remaining>0]
                                                                                                OFF       {steps_remaining=   [start &&
                                                          AND   NOT                                                                                                             /steps_remaining--;
                                                                                                entry:    steps_to_cook;}      steps_to_cook>0]


      G((¬ Onside_FD_On Λ ¬
                                                                                                mode=1;                                                                                          3

                                           2                                                                                                                                    FAILED
                                                                                                                                                                                entry: mode=2;
                                 Right_Independent_Mode                                                                                                                     2                          [door_closed]


       Is_AP_Engaged) →
                                                                      OR           1
                                                                                                                         [steps_remaining<=0]                                                                       1
                                                                           Property_Satisfied                                                                                                    1
                                           3                                                                                                                    [start && ...                        [clear_off || ... 2
                                                                                                                                                                door_closed]                         !door_closed]


         X(Is_AP_Engaged →
                                    Left_FGS_Active
                                                                                                                                                                                      1
                                                          AND
                                                                                                                                                                                Ok
                                                                                                                                                                                entry: mode=3;
                                                                                                                              [clear_off]                                   2
                                           4
           Onside_FD_On))           Right_FGS_Active
                                                                                                                              /steps_remaining=0;




    Temporal Logic                                               Synchronous                                                               Pri nted 14-Jul- 2006 12:51:47




                                                                  Observers

     …then define structural coverage metrics to
     directly and objectively describe coverage of
                      assertions
SAS_08_Model_Val_Tech_Heimdahl      8                     MAC-T IVV-08-152
LTL Temporal Operators
      Operator         Notation                          Meaning
     Globally A          G(A)     Formula A is true in all states
     Future A             F(A)    Formula A is true in some future state
     A until B           AU B     Formula A is true in every state until B
                                  becomes true. B must eventually become true
                                  for the property to be true.

     Next A              X(A)     Formula A is true in the next state


         A               A           A              A                      A

         S0             S1          S2              S3                     Si

                                                     B

SAS_08_Model_Val_Tech_Heimdahl       9         MAC-T IVV-08-152
 Formalizing Assertions
 “If the onside FD cues are off, the onside FD cues shall
   be displayed when the AP is engaged”
 G((¬ Onside_FD_On  ¬ Is_AP_Engaged) →
   X(Is_AP_Engaged → Onside_FD_On))
 • Possible Coverage Metrics
          Assertion coverage: single test case that
           demonstrates that assertion is satisfied
               Prone to “dumb” tests, e.g., execution in which AP is
                never engaged.
          More rigorous metric is necessary

SAS_08_Model_Val_Tech_Heimdahl   10      MAC-T IVV-08-152
 Task - 1
 • Define a collection of assertion coverage criteria
 • Formalize the assertion coverage obligations




SAS_08_Model_Val_Tech_Heimdahl   11   MAC-T IVV-08-152
Antecedent Coverage

•    Many of the assertions in the FGS are of the
     form :
             Globally if „A‟ occurs then „B‟ will occur
              G (A → B)
             Two ways of satisfying (A → B)
              – A is false                                         What if:
              – A is true and B is true                          ACD → B

•    Antecedent Coverage – test cases will
     exercise the antecedent.
                     S0           S1                    Sn

                   Not A         Not A                 A, B
SAS_08_Model_Val_Tech_Heimdahl           12   MAC-T IVV-08-152
 Modified Condition/Decision
 Coverage (MC/DC)
• To satisfy MC/DC
       Every point of entry and exit in the model should be invoked
        at least once,
       Every basic condition in a decision in the model should take
        on all possible outcomes at least once, and
       Each basic condition should be shown to independently
        affect the decision’s outcome

                                 A   B    A or B
   Basic Conditions
                                 F   F       F
                                                             Independent
            Independent
                                 T   F       T               effect of A
            effect of B
                                 F   T       T

SAS_08_Model_Val_Tech_Heimdahl       13       MAC-T IVV-08-152
Unique First Cause (UFC) Coverage

  “System shall eventually generate an Ack (A) or a Time
     Out (B)”
  Req. LTL property - F(A  B) .
 ¬A, ¬B         ¬A, ¬B           A, ¬ B      ¬A, ¬B                ¬A, B

    S0              S1            S2               S3                Si

Path satisfies UFC obligation for A but not B.
                                            ¬A, ¬B       ¬A, ¬B            ¬A, B
To show independence of B,
                                              S0          S1               Si

Formal UFC obligation for A : ¬(A  B) U (A  ¬B)
                                  for B : ¬(A  B) U (B  ¬A)

SAS_08_Model_Val_Tech_Heimdahl         14       MAC-T IVV-08-152
 UFC Coverage
 •   G(A)+ = {A U (a  G(A)) | a є A+}
     G(A)- = {A U a | a є A-}              Michael Whalen, Ajitha Rajan,
                                           Mats Heimdahl and Steven Miller.
 •   F(A)+ = {¬A U a | a є A+}
          - = {¬A U (a  G(¬A))| a є A-}
                                           Coverage Metrics for
                                           Requirements-Based Testing. In
     F(A)                                  Proceedings of ISSTA 2006.

 •   (A U B)+ =
         {(A  ¬B) U ((a  ¬B)  (A U B)) | a є A+} 
         {(A  ¬B) U b | b є B+}
     (A U B)- =
         {(A  ¬B) U (a  ¬B) | a є A-} 
         {(A  ¬B) U (b  ¬(A U B)) | b є B-}
 •   X(A)+ = {X(a) | a є A+}
     X(A)- = { X(a) | a є A-}


SAS_08_Model_Val_Tech_Heimdahl   15      MAC-T IVV-08-152
Task 2 – Validation Adequacy
         Measurement Tool
             Formal Assertions
                 (Eg. LTL
                Properties)
                                 Derive
                                            Assertion. Cov.                    Validation
                                          Obligations/formulas                 Test Suite


               Assertion Cov.                                Run Test Suite
                  Criteria



                                                             Evaluate and
We currently support the following                          Check formulas

coverage metrics:
• Assertion Coverage
                                                            Calculate ratio
• Assertion Antecedent Coverage                             of formulas that
                                                                were true

• Assertion UFC Coverage
                                                          Assertion Coverage
                                                           Achieved by Test
                                                                 Suite



SAS_08_Model_Val_Tech_Heimdahl                   16              MAC-T IVV-08-152
 Task - 3
 • Automatically generate requirements-based
     tests from …
        Formal assertions
        Abstract model called Assertion Model created

         using assertions and environmental constraints
         (specified as invariants)
     … to provide the defined assertion coverage.



SAS_08_Model_Val_Tech_Heimdahl   17   MAC-T IVV-08-152
 Automatically Generating
 Requirements-Based Tests
              Inputs, Ouputs,               Formal Assertions               Assertion Cov.
               Environmental               (Eg. LTL properties)                Criteria
                constraints                                                   (eg. UFC)
                                                                                             Common with
                                                                                             the Adequacy
Assertions and                                                                               Measurement
environmental                                                          Generate                   Tool
                                Generate
 constraints
 specified as
                                                                    Trap Properties
  invariants                Assertion Model                         (for Cov. Oblig.)




                                               Model Checker




                                            Counter-examples
                                           (Assertion-based test
                                                  cases)

SAS_08_Model_Val_Tech_Heimdahl                   18               MAC-T IVV-08-152
 What Are Model Checkers?
 •   Breakthrough technology of the 1990’s
 •   Widely used in hardware verification
          (Intel, Motorola, IBM, …)
 •   Several different types of model checkers
          Explicit, Symbolic, Bounded, Infinite Bounded, …
 •   Exhaustive search of the global state space
          Consider all combinations of inputs and states
          Equivalent to exhaustive testing of the model
          Produces a counter example if a property is not true
 •   Easy to use
          “Push button” formal methods
          Very little human effort unless you’re at the tool’s limits
 •   Limitations
          State space explosion

SAS_08_Model_Val_Tech_Heimdahl       19        MAC-T IVV-08-152
 Preliminary Evaluation
 Interested in determining:
 • Feasibility of generating assertion-based tests from a set of
    assertions
          Generated assertion-based tests to provide UFC coverage over the
           assertions
 •   Effectiveness of these test sets in validating the system
     model
          Measured MC/DC achieved by the test sets over the system model

 Used three realistic sized examples:
 • Flight Guidance System (FGS),
 • and two models related to the Display Window Manager
   system (DWM1 and DWM2)

SAS_08_Model_Val_Tech_Heimdahl     20       MAC-T IVV-08-152
                                                                  Results
                                  Number of Assertion-Based Tests Generated

                           1000
                                                                           #Obligs
                            800
                                                                           #Tests
                            600
                                                                           Generated
                            400
http://www.umsec.umn.edu




                            200

                             0
                                   FGS       DWM1       DWM2
                                                                              MC/DC Achieved by Assertion-Based Tests

                                                                            100
                           Ajitha Rajan, Michael Whalen, and Mats             80
                           Heimdahl. Model Validation using                                                       MC/DC
                                                                              60
                           Automatically Generated Requirements-                                                  Achieved
                           Based Tests. In Proceedings of 10th IEEE           40
                           High Assurance Systems Engineering                 20
                           Symposium, Nov 2007.
                                                                               0
                                                                                     FGS   DWM1    DWM2

                                                                      21
 Results and Analysis
• UFC test suites achieved high MC/DC coverage over
    DWM models – well defined set of assertions
•   Test-suite generated for UFC achieved very low
    MC/DC over the FGS model
     “When the FGS is in independent mode, it shall be active”.
     G(m_Independent_Mode_Condition.result →
       X(Is_This_Side_Active = 1))
                                       RSML–e Macro
 Structure of Independent_Mode_Condition is not captured in the
 property
Independent_Mode_Condition = ((Is_LAPPR_Active & Is_VAPPR_Active
        & IS_Offside_LAPPR_Active & Is_Offside_VAPPR_Active) |
        ( Is_VGA_Active & Is_Offside_VGA_Active))

SAS_08_Model_Val_Tech_Heimdahl   22   MAC-T IVV-08-152
 Benefits of ABT
 • Saves time and effort in generating
     validation test suites from assertions
 •   Effective method for generating model
     validation tests when the assertions are well
     defined
 •   Helps in identifying missing assertions and
     over constrained models



SAS_08_Model_Val_Tech_Heimdahl   23   MAC-T IVV-08-152
 Bonus Task –
 Adequacy of Conformance Testing
                    Measure                             Measure
                    Adequacy     Conformance            Adequacy
                                    Tests
                                                                     Assertions
     Model                               Run
1. Direct assessment of
   how well tests exercise
   the assertions                                                 Useful ??

2. Will expose defects of
   omission
3. Assertion coverage               Code
   could necessitate
   longer test cases than
   for model coverage
SAS_08_Model_Val_Tech_Heimdahl      24         MAC-T IVV-08-152
Assertion Coverage as an Adequacy
Measure for Conformance Testing
Hypothesis 1(H1): Conformance tests providing
assertion UFC coverage are more effective than
conformance tests providing MC/DC over the
model

Hypothesis 2(H2): Conformance tests providing
assertion UFC coverage in addition to MC/DC
over the model are more effective than
conformance tests providing only MC/DC over the
model

SAS_08_Model_Val_Tech_Heimdahl   25   MAC-T IVV-08-152
 Experiment
 • Used four industrial systems :
        Two models from the display window manager
        Two models representing the mode logic of a

         flight guidance system
 • Assessed effectiveness of test suites in terms
     of their fault finding ability
           Ajitha Rajan, Michael Whalen, Matt Staats, and Mats Heimdahl.
           Requirements Coverage as an Adequacy Measure for Conformance
           Testing. To Appear in Proceedings of 10th International Conference on
           Formal Engineering Methods, Oct 2008.




SAS_08_Model_Val_Tech_Heimdahl           26         MAC-T IVV-08-152
 Results – Hypothesis 1
                                                MC/DC vs UFC

                       100

                        80
     % Fault Finding




                        60                                                            Avg. MC/DC

                        40                                                            Avg. UFC

                        20

                         0
                             DWM1           DWM2           Latctl           Vertmax




                             Hypothesis 1 rejected at 5% statistical significance
                             on all but the Latctl system


SAS_08_Model_Val_Tech_Heimdahl                  27       MAC-T IVV-08-152
 Analysis – Hypothesis 1
 • Model coverage better than assertion coverage for
     measuring adequacy of conformance test suites
 •   Assertion UFC coverage is heavily dependent on
     the nature and completeness of the assertions
 •   Rigor and robustness of assertion coverage metric
     used is important
          UFC metric gets cheated when assertions are structured
           to hide the complexity of conditions




SAS_08_Model_Val_Tech_Heimdahl   28   MAC-T IVV-08-152
 Results – Hypothesis 2

                                         MC/DC vs MC/DC + UFC

                      96
                      94
    % Fault Finding




                      92
                      90
                                                                               Best MC/DC
                      88
                                                                               Avg. Combined
                      86
                      84
                      82
                      80
                           DWM1        DWM2          Latctl          Vertmax




                            Hypothesis 2 accepted at 5% statistical significance
                            on all but the DWM2 system


SAS_08_Model_Val_Tech_Heimdahl                 29        MAC-T IVV-08-152
 Analysis – Hypothesis 2
Does UFC really help in revealing additional faults?

                     UFC Achieved by   Achievable UFC        Rel. Diff
                     MC/DC suites
      DWM1           28.3%             96.9%                 70.8%

      DWM2           59.7%             64%                   6.7%

      Latctl         94.7%             99.5%                 4.8%

      Vertmax        97.4%             99%                   1.6%




SAS_08_Model_Val_Tech_Heimdahl    30      MAC-T IVV-08-152
 Summary – Bonus Task
 • UFC > MC/DC                   FALSE
          3 of the 4 case examples at 5% statistical significance
 • UFC + MC/DC > MC/DC                       TRUE
          3 of the 4 case examples at 5% statistical significance
 • Combine rigorous metrics for assertion coverage
     and model coverage to measure adequacy of
     conformance test suites
 •   UFC metric is sensitive to structure of assertions
          Need assertion coverage metrics that are robust to
           structure of assertions


SAS_08_Model_Val_Tech_Heimdahl     31    MAC-T IVV-08-152
Technology Readiness Level
                                 •    “Requirements-Based Test
                                      Generation Tool” TRL = 6
                                             System/subsystem model or prototype
                                              demonstration in a relevant
                                              environment
                                 •    “Validation Adequacy
                                      Measurement Tool” TRL = 6
                                             System/subsystem model or prototype
                                              demonstration in a relevant
                                              environment




SAS_08_Model_Val_Tech_Heimdahl       32          MAC-T IVV-08-152
 Relevance to NASA
 •   MBD is here - estimate one-half of all NASA missions in
     development or on the books will use model-based
     subsystem development
          Extensive use in avionics industry
 •   How do we know the models are right?
          Model validation problem
 •   We provide the capability to
          Objectively measure the “quality” of assertion-based black-box
           validation tests
          Objectively assess the completeness of a model
               Does the model address all assertions?
          Objectively assess the adequacy of a set of assertions
               Are there enough assertions to adequately describe the model?
          Automatically generate truly assertion-based tests


SAS_08_Model_Val_Tech_Heimdahl         33        MAC-T IVV-08-152
 Achievements to Date
 •   Formal assertion notation identified
          Most work with LTL
          Extended to work with Live Sequence Charts (LSC)
 •   Objective validation metrics defined
          Requirements, Antecedent, Unique First Cause, and Unique Cause
 •   Test case generation tool developed
          Developed tool generating tests from LTL
               Capable of generating tests to all metrics defined
          Prototype tool working on LSC developed
 •   Developed test-adequacy measurement tool for the defined
     validation metrics
 •   Evaluation of metrics and tool
 •   12 papers and one PhD dissertation (Ajitha Rajan)

SAS_08_Model_Val_Tech_Heimdahl          34        MAC-T IVV-08-152
 Next Steps
 • Investigate alternative requirements
     notations to LTL
 •   Complete empirical evaluation of the
     effectiveness in model validation
        Flight Guidance Sensor (FGS) evaluation
         complete
        Display Manager (DM) evaluation in work

        Coordinate evaluation on NASA IV&V project

 • Coordinate technology transfer
SAS_08_Model_Val_Tech_Heimdahl   35   MAC-T IVV-08-152
 Discussion




SAS_08_Model_Val_Tech_Heimdahl   36   MAC-T IVV-08-152

				
DOCUMENT INFO