Document Sample

Model-Validation in Model-Based Development Kurt Woodham Ajitha Rajan, Mats Heimdahl L-3 Communications University of Minnesota OSMA SAS ’08 September 8-12 SAS_08_Model_Val_Tech_Heimdahl MAC-T IVV-08-152 Problem: Model Validation • Model-Based Development (MBD) is here to stay Use of MBD is accelerating Estimate 50% of NASA development projects using some form of MBD Many advantages: model-checking, code generation, desktop testing, closed-loop simulation Enhances early detection of requirement, design, or implementation defects “Executable Specifications” enable evaluation of behavior that might otherwise be relegated to Inspections and Testing • How do we know the models are “right”? Manually develop black-box tests • When have we validated enough? Measure test coverage on an implementation/model SAS_08_Model_Val_Tech_Heimdahl 2 MAC-T IVV-08-152 Problem : Current Practice • Measure black-box test coverage over the model Indirect measure Defects of omission in model not exposed. Incomplete Weak Model Black-Box Test set Executable artifact is necessary Adequacy can only be determined late in the development process SAS_08_Model_Val_Tech_Heimdahl 3 MAC-T IVV-08-152 Goals of Project • Define metrics for objective, implementation-independent measure of adequacy of a black-box test suite • Develop tools to measure validation adequacy based on the defined metrics • Provide capability for autogeneration of black-box test suites SAS_08_Model_Val_Tech_Heimdahl 4 MAC-T IVV-08-152 Testing – What does it mean? Assertion Assertions In General Based Testing Does it (ABT) to implement? Validate Model Specification Model Does it implement? Implementation Model Model-Based Does it Testing (MBT) implement? to Verify Code Source Code Our contribution is in providing novel ABT capabilities SAS_08_Model_Val_Tech_Heimdahl 5 MAC-T IVV-08-152 What are Assertions? Properties/ Formal Assertions Assertions Defined Can also be over over components, interfaces,... in1 out1 System ink outm SAS_08_Model_Val_Tech_Heimdahl 6 MAC-T IVV-08-152 Contributions - ABT 2 Auto-generate Assertions Black-Box Tests Validate Measure 1 3 Assess Model Model Adequacy and Assertion Completeness We provide the following contributions in the Assertion-Based testing domain (indicated by in the above figure): 1. Objective, implementation-independent measure of adequacy of a black- box test suite 2. Auto-generation of black-box validation tests directly from assertions 3. Objective assessment of completeness of model as well as assertions SAS_08_Model_Val_Tech_Heimdahl 7 MAC-T IVV-08-152 The Idea Write assertions in a formal notation… microwave_library_temp/mode_logic G (FD_On -> Cues_On); 1 Left_Independent_Mode ON [steps_remaining>0] OFF {steps_remaining= [start && AND NOT /steps_remaining--; entry: steps_to_cook;} steps_to_cook>0] G((¬ Onside_FD_On Λ ¬ mode=1; 3 2 FAILED entry: mode=2; Right_Independent_Mode 2 [door_closed] Is_AP_Engaged) → OR 1 [steps_remaining<=0] 1 Property_Satisfied 1 3 [start && ... [clear_off || ... 2 door_closed] !door_closed] X(Is_AP_Engaged → Left_FGS_Active 1 AND Ok entry: mode=3; [clear_off] 2 4 Onside_FD_On)) Right_FGS_Active /steps_remaining=0; Temporal Logic Synchronous Pri nted 14-Jul- 2006 12:51:47 Observers …then define structural coverage metrics to directly and objectively describe coverage of assertions SAS_08_Model_Val_Tech_Heimdahl 8 MAC-T IVV-08-152 LTL Temporal Operators Operator Notation Meaning Globally A G(A) Formula A is true in all states Future A F(A) Formula A is true in some future state A until B AU B Formula A is true in every state until B becomes true. B must eventually become true for the property to be true. Next A X(A) Formula A is true in the next state A A A A A S0 S1 S2 S3 Si B SAS_08_Model_Val_Tech_Heimdahl 9 MAC-T IVV-08-152 Formalizing Assertions “If the onside FD cues are off, the onside FD cues shall be displayed when the AP is engaged” G((¬ Onside_FD_On ¬ Is_AP_Engaged) → X(Is_AP_Engaged → Onside_FD_On)) • Possible Coverage Metrics Assertion coverage: single test case that demonstrates that assertion is satisfied Prone to “dumb” tests, e.g., execution in which AP is never engaged. More rigorous metric is necessary SAS_08_Model_Val_Tech_Heimdahl 10 MAC-T IVV-08-152 Task - 1 • Define a collection of assertion coverage criteria • Formalize the assertion coverage obligations SAS_08_Model_Val_Tech_Heimdahl 11 MAC-T IVV-08-152 Antecedent Coverage • Many of the assertions in the FGS are of the form : Globally if „A‟ occurs then „B‟ will occur G (A → B) Two ways of satisfying (A → B) – A is false What if: – A is true and B is true ACD → B • Antecedent Coverage – test cases will exercise the antecedent. S0 S1 Sn Not A Not A A, B SAS_08_Model_Val_Tech_Heimdahl 12 MAC-T IVV-08-152 Modified Condition/Decision Coverage (MC/DC) • To satisfy MC/DC Every point of entry and exit in the model should be invoked at least once, Every basic condition in a decision in the model should take on all possible outcomes at least once, and Each basic condition should be shown to independently affect the decision’s outcome A B A or B Basic Conditions F F F Independent Independent T F T effect of A effect of B F T T SAS_08_Model_Val_Tech_Heimdahl 13 MAC-T IVV-08-152 Unique First Cause (UFC) Coverage “System shall eventually generate an Ack (A) or a Time Out (B)” Req. LTL property - F(A B) . ¬A, ¬B ¬A, ¬B A, ¬ B ¬A, ¬B ¬A, B S0 S1 S2 S3 Si Path satisfies UFC obligation for A but not B. ¬A, ¬B ¬A, ¬B ¬A, B To show independence of B, S0 S1 Si Formal UFC obligation for A : ¬(A B) U (A ¬B) for B : ¬(A B) U (B ¬A) SAS_08_Model_Val_Tech_Heimdahl 14 MAC-T IVV-08-152 UFC Coverage • G(A)+ = {A U (a G(A)) | a є A+} G(A)- = {A U a | a є A-} Michael Whalen, Ajitha Rajan, Mats Heimdahl and Steven Miller. • F(A)+ = {¬A U a | a є A+} - = {¬A U (a G(¬A))| a є A-} Coverage Metrics for Requirements-Based Testing. In F(A) Proceedings of ISSTA 2006. • (A U B)+ = {(A ¬B) U ((a ¬B) (A U B)) | a є A+} {(A ¬B) U b | b є B+} (A U B)- = {(A ¬B) U (a ¬B) | a є A-} {(A ¬B) U (b ¬(A U B)) | b є B-} • X(A)+ = {X(a) | a є A+} X(A)- = { X(a) | a є A-} SAS_08_Model_Val_Tech_Heimdahl 15 MAC-T IVV-08-152 Task 2 – Validation Adequacy Measurement Tool Formal Assertions (Eg. LTL Properties) Derive Assertion. Cov. Validation Obligations/formulas Test Suite Assertion Cov. Run Test Suite Criteria Evaluate and We currently support the following Check formulas coverage metrics: • Assertion Coverage Calculate ratio • Assertion Antecedent Coverage of formulas that were true • Assertion UFC Coverage Assertion Coverage Achieved by Test Suite SAS_08_Model_Val_Tech_Heimdahl 16 MAC-T IVV-08-152 Task - 3 • Automatically generate requirements-based tests from … Formal assertions Abstract model called Assertion Model created using assertions and environmental constraints (specified as invariants) … to provide the defined assertion coverage. SAS_08_Model_Val_Tech_Heimdahl 17 MAC-T IVV-08-152 Automatically Generating Requirements-Based Tests Inputs, Ouputs, Formal Assertions Assertion Cov. Environmental (Eg. LTL properties) Criteria constraints (eg. UFC) Common with the Adequacy Assertions and Measurement environmental Generate Tool Generate constraints specified as Trap Properties invariants Assertion Model (for Cov. Oblig.) Model Checker Counter-examples (Assertion-based test cases) SAS_08_Model_Val_Tech_Heimdahl 18 MAC-T IVV-08-152 What Are Model Checkers? • Breakthrough technology of the 1990’s • Widely used in hardware verification (Intel, Motorola, IBM, …) • Several different types of model checkers Explicit, Symbolic, Bounded, Infinite Bounded, … • Exhaustive search of the global state space Consider all combinations of inputs and states Equivalent to exhaustive testing of the model Produces a counter example if a property is not true • Easy to use “Push button” formal methods Very little human effort unless you’re at the tool’s limits • Limitations State space explosion SAS_08_Model_Val_Tech_Heimdahl 19 MAC-T IVV-08-152 Preliminary Evaluation Interested in determining: • Feasibility of generating assertion-based tests from a set of assertions Generated assertion-based tests to provide UFC coverage over the assertions • Effectiveness of these test sets in validating the system model Measured MC/DC achieved by the test sets over the system model Used three realistic sized examples: • Flight Guidance System (FGS), • and two models related to the Display Window Manager system (DWM1 and DWM2) SAS_08_Model_Val_Tech_Heimdahl 20 MAC-T IVV-08-152 Results Number of Assertion-Based Tests Generated 1000 #Obligs 800 #Tests 600 Generated 400 http://www.umsec.umn.edu 200 0 FGS DWM1 DWM2 MC/DC Achieved by Assertion-Based Tests 100 Ajitha Rajan, Michael Whalen, and Mats 80 Heimdahl. Model Validation using MC/DC 60 Automatically Generated Requirements- Achieved Based Tests. In Proceedings of 10th IEEE 40 High Assurance Systems Engineering 20 Symposium, Nov 2007. 0 FGS DWM1 DWM2 21 Results and Analysis • UFC test suites achieved high MC/DC coverage over DWM models – well defined set of assertions • Test-suite generated for UFC achieved very low MC/DC over the FGS model “When the FGS is in independent mode, it shall be active”. G(m_Independent_Mode_Condition.result → X(Is_This_Side_Active = 1)) RSML–e Macro Structure of Independent_Mode_Condition is not captured in the property Independent_Mode_Condition = ((Is_LAPPR_Active & Is_VAPPR_Active & IS_Offside_LAPPR_Active & Is_Offside_VAPPR_Active) | ( Is_VGA_Active & Is_Offside_VGA_Active)) SAS_08_Model_Val_Tech_Heimdahl 22 MAC-T IVV-08-152 Benefits of ABT • Saves time and effort in generating validation test suites from assertions • Effective method for generating model validation tests when the assertions are well defined • Helps in identifying missing assertions and over constrained models SAS_08_Model_Val_Tech_Heimdahl 23 MAC-T IVV-08-152 Bonus Task – Adequacy of Conformance Testing Measure Measure Adequacy Conformance Adequacy Tests Assertions Model Run 1. Direct assessment of how well tests exercise the assertions Useful ?? 2. Will expose defects of omission 3. Assertion coverage Code could necessitate longer test cases than for model coverage SAS_08_Model_Val_Tech_Heimdahl 24 MAC-T IVV-08-152 Assertion Coverage as an Adequacy Measure for Conformance Testing Hypothesis 1(H1): Conformance tests providing assertion UFC coverage are more effective than conformance tests providing MC/DC over the model Hypothesis 2(H2): Conformance tests providing assertion UFC coverage in addition to MC/DC over the model are more effective than conformance tests providing only MC/DC over the model SAS_08_Model_Val_Tech_Heimdahl 25 MAC-T IVV-08-152 Experiment • Used four industrial systems : Two models from the display window manager Two models representing the mode logic of a flight guidance system • Assessed effectiveness of test suites in terms of their fault finding ability Ajitha Rajan, Michael Whalen, Matt Staats, and Mats Heimdahl. Requirements Coverage as an Adequacy Measure for Conformance Testing. To Appear in Proceedings of 10th International Conference on Formal Engineering Methods, Oct 2008. SAS_08_Model_Val_Tech_Heimdahl 26 MAC-T IVV-08-152 Results – Hypothesis 1 MC/DC vs UFC 100 80 % Fault Finding 60 Avg. MC/DC 40 Avg. UFC 20 0 DWM1 DWM2 Latctl Vertmax Hypothesis 1 rejected at 5% statistical significance on all but the Latctl system SAS_08_Model_Val_Tech_Heimdahl 27 MAC-T IVV-08-152 Analysis – Hypothesis 1 • Model coverage better than assertion coverage for measuring adequacy of conformance test suites • Assertion UFC coverage is heavily dependent on the nature and completeness of the assertions • Rigor and robustness of assertion coverage metric used is important UFC metric gets cheated when assertions are structured to hide the complexity of conditions SAS_08_Model_Val_Tech_Heimdahl 28 MAC-T IVV-08-152 Results – Hypothesis 2 MC/DC vs MC/DC + UFC 96 94 % Fault Finding 92 90 Best MC/DC 88 Avg. Combined 86 84 82 80 DWM1 DWM2 Latctl Vertmax Hypothesis 2 accepted at 5% statistical significance on all but the DWM2 system SAS_08_Model_Val_Tech_Heimdahl 29 MAC-T IVV-08-152 Analysis – Hypothesis 2 Does UFC really help in revealing additional faults? UFC Achieved by Achievable UFC Rel. Diff MC/DC suites DWM1 28.3% 96.9% 70.8% DWM2 59.7% 64% 6.7% Latctl 94.7% 99.5% 4.8% Vertmax 97.4% 99% 1.6% SAS_08_Model_Val_Tech_Heimdahl 30 MAC-T IVV-08-152 Summary – Bonus Task • UFC > MC/DC FALSE 3 of the 4 case examples at 5% statistical significance • UFC + MC/DC > MC/DC TRUE 3 of the 4 case examples at 5% statistical significance • Combine rigorous metrics for assertion coverage and model coverage to measure adequacy of conformance test suites • UFC metric is sensitive to structure of assertions Need assertion coverage metrics that are robust to structure of assertions SAS_08_Model_Val_Tech_Heimdahl 31 MAC-T IVV-08-152 Technology Readiness Level • “Requirements-Based Test Generation Tool” TRL = 6 System/subsystem model or prototype demonstration in a relevant environment • “Validation Adequacy Measurement Tool” TRL = 6 System/subsystem model or prototype demonstration in a relevant environment SAS_08_Model_Val_Tech_Heimdahl 32 MAC-T IVV-08-152 Relevance to NASA • MBD is here - estimate one-half of all NASA missions in development or on the books will use model-based subsystem development Extensive use in avionics industry • How do we know the models are right? Model validation problem • We provide the capability to Objectively measure the “quality” of assertion-based black-box validation tests Objectively assess the completeness of a model Does the model address all assertions? Objectively assess the adequacy of a set of assertions Are there enough assertions to adequately describe the model? Automatically generate truly assertion-based tests SAS_08_Model_Val_Tech_Heimdahl 33 MAC-T IVV-08-152 Achievements to Date • Formal assertion notation identified Most work with LTL Extended to work with Live Sequence Charts (LSC) • Objective validation metrics defined Requirements, Antecedent, Unique First Cause, and Unique Cause • Test case generation tool developed Developed tool generating tests from LTL Capable of generating tests to all metrics defined Prototype tool working on LSC developed • Developed test-adequacy measurement tool for the defined validation metrics • Evaluation of metrics and tool • 12 papers and one PhD dissertation (Ajitha Rajan) SAS_08_Model_Val_Tech_Heimdahl 34 MAC-T IVV-08-152 Next Steps • Investigate alternative requirements notations to LTL • Complete empirical evaluation of the effectiveness in model validation Flight Guidance Sensor (FGS) evaluation complete Display Manager (DM) evaluation in work Coordinate evaluation on NASA IV&V project • Coordinate technology transfer SAS_08_Model_Val_Tech_Heimdahl 35 MAC-T IVV-08-152 Discussion SAS_08_Model_Val_Tech_Heimdahl 36 MAC-T IVV-08-152

DOCUMENT INFO

Shared By:

Categories:

Tags:
Simulink models, Simulink library, Simulink model, Transformation Method, The MathWorks, Model Transformation, Simulink blocks, library blocks, Research Fellow, Symposium Chairs

Stats:

views: | 29 |

posted: | 2/21/2010 |

language: | English |

pages: | 36 |

OTHER DOCS BY niusheng

Docstoc is the premier online destination to start and grow small businesses. It hosts the best quality and widest selection of professional documents (over 20 million) and resources including expert videos, articles and productivity tools to make every small business better.

Search or Browse for any specific document or resource you need for your business. Or explore our curated resources for Starting a Business, Growing a Business or for Professional Development.

Feel free to Contact Us with any questions you might have.