Assurance Case Frameworks

Document Sample
Assurance Case Frameworks Powered By Docstoc
					Assurance Case Frameworks
            Part of
 High Confidence Software MSR


           T. Scott Ankrum
   MITRE — Software Engineering Center
            March 11, 2004
                      Credits


    • Part of the “High-Confidence Software
      Initiative” research project
    • Supported by the MITRE Sponsored
      Research program
    • Supporting cast                  Working for almost
         – Chuck Howell                   two years.
         – Alfred Kromholz
         – Jim Moore

March 11, 2004      T. Scott Ankrum — MITRE            2
                    Agenda


    •   What is an Assurance Case?
    •   Structuring an Assurance Case
    •   Problems With Assurance Cases
    •   Choosing a Tool
    •   Structuring Selected Standards
    •   Conclusions and Follow-on


March 11, 2004     T. Scott Ankrum — MITRE   3
      What Is an
   Assurance Case?



March 11, 2004   T. Scott Ankrum — MITRE   4
          History of Assurance Cases


• Originally Only Safety Cases
   –   Aerospace
   –   Railways, automated passenger
   –   Nuclear power
   –   Off-shore oil
   –   Defense
• Security Cases
   – Use compliance rules more than an assurance case
• Cases for Business Critical Systems


  March 11, 2004        T. Scott Ankrum — MITRE         5
           Definition of Safety Case


• From Adelard’s ASCE manual:
   “A documented body of evidence that provides a
   convincing and valid argument that a system is
   adequately safe for a given application in a given
   environment.”




March 11, 2004      T. Scott Ankrum — MITRE             6
     Definition of Assurance Case


• Generalizing that definition
    A documented body of evidence that provides a
   convincing and valid argument that a specified set
   of critical claims regarding a system’s properties
   are adequately justified for a given application in a
   given environment.




March 11, 2004      T. Scott Ankrum — MITRE           7
Where is an Assurance Case Used?


   – Critical systems under regulation or acquisition constraints
   – Third-party certification, approval, licensing, etc.
   – Documented body of evidence required
   – Need a compelling case that the system satisfies certain
     critical properties for specific contexts
   – Examples: DO-178B, Common Criteria, MIL-STD-882D
   – “safety case”, “certification evidence”, “security case”…

   Collectively we’ll refer to them as “assurance cases”

March 11, 2004          T. Scott Ankrum — MITRE                 8
       Structuring an
      Assurance Case



March 11, 2004   T. Scott Ankrum — MITRE   9
    Elements of an Assurance Case

•   Claims
•   Arguments
•   Evidence
•   Other elements, depending on notation




March 11, 2004    T. Scott Ankrum — MITRE   10
         Claims in Assurance Cases


• Assertion of compliance with key requirements and properties
• Must be in a specific context
     –   Environment
     –   Services or behavior
     –   Threats
     –   “Is this brick safe?” illustrates why…
• Sub-claims may be analogous to “lemmas” in a proof
     – separation of concerns
     – workflow
     – makes overall case more manageable

March 11, 2004               T. Scott Ankrum — MITRE     11
   Arguments in Assurance Cases


• Link evidence to claims via inference rules
    – Deterministic: defined rules => true/false assertion
    – Probabilistic: quantitative, statistical, numerical threshold
      (MTTF)
    – Qualitative: rules with an indirect link to desired properties
      (standards, process guides)
• No such thing as perfection:
       “It is quite possible to follow a faulty analytical process and
       write a clear and persuasive argument in support of an
       erroneous judgment.” – R. Heuer, The Psychology of Intelligence Analysis

March 11, 2004             T. Scott Ankrum — MITRE                       12
     Evidence in Assurance Cases


•   Process and people used to develop the system
•   Systematic testing
•   Product review and analyses
•   Mathematical proofs

None of these alone provides adequate evidence


March 11, 2004    T. Scott Ankrum — MITRE    13
    Problems With
   Assurance Cases



March 11, 2004   T. Scott Ankrum — MITRE   14
  Problems with Assurance Cases


• There are problems in every aspect of assurance cases
     – Building them
     – Reviewing them
     – Maintaining them
     – Reusing them
• Problems result from:
     – volume of material
     – little structuring support
     – ad hoc “rules of evidence”

March 11, 2004         T. Scott Ankrum — MITRE    15
 Building the Assurance Case – 1


  • Most guidance is:
       – strong on excruciating detail for format
       – weak on gathering, merging, and reviewing evidence

  • Guidance often uses the “cast a wide net” tactic
       – Assurance costs time and money
       – “Squandered diagnostic resources”
       – Some work on a “portfolio management” approach



March 11, 2004         T. Scott Ankrum — MITRE                16
 Building the Assurance Case – 2


  • With free format text and no tool support:
       – coordination is hard
       – tracking is hard
       – workflow management is hard


  • Imagine building a 500 page project plan by
    hand, on paper



March 11, 2004        T. Scott Ankrum — MITRE     17
Reviewing the Assurance Case – 1

• Stacks of free-format text makes review tedious
   – Hard to see linkages or patterns
   – Hides key results in sheer volume

• Weak guidance on review of arguments and evidence
  often results in ad hoc criteria
  (be very nice to your reviewer!)

• Rarely is there explicit guidance for weighing
  conflicting or inconsistent evidence

March 11, 2004        T. Scott Ankrum — MITRE       18
Reviewing the Assurance Case – 2


   “Often viewed as irrefutable, evidence is, in fact,
   an interpretive science, refracted through the
   varying perspectives of different disciplines. ...
   [Judging evidence requires] reasoning based on
   evidence that is incomplete, inconclusive, and
   often imprecise.”

The Evidential Foundations of Probabilistic Reasoning, David Schum




March 11, 2004           T. Scott Ankrum — MITRE                     19
Maintaining the Assurance Case – 1


  • The one thing more brittle than software is –
     the associated assurance case

  • It is difficult to understand impact of a change on
    assurance structure because:
       – volume of information is immense
       – impact of a change on assurance structure is complex




March 11, 2004          T. Scott Ankrum — MITRE                 20
Maintaining the Assurance Case – 2


 • Reasons for change
      – The claims and/or evidence have changed
      – Arguments no longer valid or new ones needed
      – Evidence is irrelevant or new evidence needed
      – “Weak link effect” of discrete systems compounds problem

 • Revalidation costs are a major burden

 • “Breakage” of successive dependencies

March 11, 2004        T. Scott Ankrum — MITRE             21
  Reusing the Assurance Case – 1


  • Assurance case frameworks are rarely the
    subject of study per se

  • More attention for these would be useful
       – tool support
       – idioms and templates
       – extracting patterns for future use


March 11, 2004        T. Scott Ankrum — MITRE   22
  Reusing the Assurance Case – 2


  • Relationship among claims, arguments,
    and evidence
       – not often explicit
       – hard to distinguish the reusable from the
         project specific portions of assurance case

  • Compare this with building a deck with
    the help of a project planning tool

March 11, 2004        T. Scott Ankrum — MITRE          23
    Choosing a Tool



March 11, 2004   T. Scott Ankrum — MITRE   24
What Should a Tool Provide? – 1


• Simple management of complexity and volume
     – MS Project-like planning and tracking of complexities
     – Checking simple structural properties
     – Browsing and report generation
• Support for multiple, geographically dispersed users
     – with data integrity
     – concurrently or asynchronously
• Useable for any domain
     – not specific to any one industry
     – not specifically for safety cases or security cases

March 11, 2004          T. Scott Ankrum — MITRE                25
What Should a Tool Provide? – 2


• Replanning as things change:
          (“No plan survives contact with the enemy.”)
• Templates and tailoring to
     – capture lessons learned
     – reduce wheel reinvention
• Uses and/or exchanges consistent notation for:
     – claims, evidence, and arguments
• Widely executable
     – runs under Windows 2000 or Windows XP
     – or has a Windows based GUI
March 11, 2004          T. Scott Ankrum — MITRE          26
                 Notations Considered


• Toulmin Structures
    – Stephen Toulmin, The Uses of Argument, 1958
• Goal Structuring Notation
    – Described in Tim Kelly’s dissertation, York, 1998
• ASCAD (Claims-Arguments-Data):
    – ESPRIT SHIP project headed by Adelard
• Proprietary

March 11, 2004        T. Scott Ankrum — MITRE       27
                 Selected Tool: ASCE


•   Established Notations: GSN & ASCAD
•   Not Industry or Safety Specific
•   Extensible through a Schema
•   Case is exportable to project documents
•   Stable, no failures during evaluation



March 11, 2004        T. Scott Ankrum — MITRE   28
                                   ASCAD Notation
                                                                         CLAIM
                                                                    Software Has Few
                                                                         Defects




                                                 Is a subclaim of    Is a subclaim of     Is a subclaim of



                             CLAIM                                      CLAIM                                          CLAIM
                       Requirements Are                              System Built to                            System Was Well Tested
                      Correct, Complete and                           Requirements
                           Consistent




                                                                        Supports                                Supports      Supports
                 Supports        Supports



     ARGUMENT                                                         ARGUMENT                     ARGUMENT
 A formal specification                ARGUMENT                                                                                      ARGUMENT
                                   Fagan's two famous               Requirements are             Peer reviews are a
adds rigor and exposes                                                                                                            Test scenarios were
                                   papers make a good               only useful if they         good way to increas
     defects early                                                                                                                  marked with the
                                       argument for                    are used to              quality and the only               completion date to
                                  inspections, and they                design the                available way for               document that all tests
                                  are a proven practice.                 system.                     test plans.                    were eventually
                                                                                                                                      successful.




    Is evidence for
                                                                     Is evidence for               Is evidence for
                                     Is evidence for
                                                                                                                                     Is evidence for




     EVIDENCE                          EVIDENCE                        EVIDENCE                                                         EVIDENCE
                                                                                                    EVIDENCE
     Written in Z                   Fagan Inspection                  Requirements                                                  Filled in test Plan
                                                                                                   Test Plan Peer
                                     of each section                Traced to Design
                                                                                                  Review statistics
                                                                      Components
Structuring Selected
     Standards



March 11, 2004   T. Scott Ankrum — MITRE   30
                  Hypotheses


• Assurance is Assurance is Assurance
   – All assurance cases are similar enough in structure
     that a distinct tool for each domain is not required
• Assurance Standard                Assurance Case
   – There is a relationship between the actual or
     implied structure of an assurance standard and the
     structure of an assurance case instantiated from
     that standard


March 11, 2004      T. Scott Ankrum — MITRE           31
   Mapping Standards into ASCE

• Computer Security:
     – Common Criteria — Evaluation Assurance Level 4
• Aviation Safety: DO-178B
     – Software Considerations in Airborne Systems
• Medical Device Safety:
     – Discussing with FDA
       Center for Devices & Radiological Health


March 11, 2004      T. Scott Ankrum — MITRE          32
                                                                                                                                                                                                           t?
                                                                                                                                                                                                        ien
                                                                                                                                                                                                    ffic
                                                                                                                                                                                                                         Argument




                                                                                                                                                                                                  su




                                                                                                                                                                                                                                       Ne
                                                                                                                                                                                                d
                                                                                                                                                                                                                          for Sub-




                                                                                                                                                                                              an
       Start




                                                                                                                                                                                                                                         w
                                                                                                                                                                                                                           claims




                                                                                                                                                                                           ry




                                                                                                                                                                                                                                           iss
                                                                                                                                                                                         sa


                                                                                                                                                                                        im




                                                                                                                                                                                                                                              ue
                                                                                                                                                                                       es


                                                                                                                                                                                     cla




                                                                                                                                                                                                                                                 re
                                                                                                                                                                                    ec
    Top-level




                                                                                                                                                                                   b-




                                                                                                                                                                                                                                                   ve
                                                                                                                                                                                 sn
                                           Needs supporting




                                                                                                                                                                                 su
     Claim




                                                                                                                                                                                                                                                     ale
                                                                                                                                                                               im
                                               claims




                                                                                                                                                                              er




                                                                                                                                                                                                                                                        d
                                                                                                                                                                            cla



                                                                                                                                                                            th
                                                                                                                                                                         no
                                                                                                                                                                         b-
                                                                                                                                                                       su



                                                                                                                                                                      sa
                                                                                                                                                       e



                                                                                                                                                                    ed
                                                                                                                                                     Ar


                                                                                                                                                                  Ne
                                                                                                                                                                                                                                   d                 Parent
                                           Cla im ne eds jus tification                                                                                                                                                e rev ea le
                                                                                                                                                                                                            N e w issu                               Claim


   Argument                                                                                                                                           Supporting
     under                                                                                                                                              Claim
                                                                                                                                                                                                                Nee ds sup porting claim
    Parent                                                               cla im
                                                             p po rtin g
                                                 N e ed s su
                                                                                                                                                                                                Ne
                                                                                                                                                                                                     ed




                                                                                                                                                                       Needs
                                                                                                                                                                                                        ss




                                                                                                              How
                                                                                                          l                                                                                          c la u pp
                                                                                                                                                                                                         i m o r ti
                                                                                               l   e ve                                                                                                     (s )    ng
                                                                                           est




                                                                                                                                                                             sibling
                                                                                                                  is claim
                                                                                      ow
                                                                               is l
                                                                      im
                                                               C la



                                                                                                                                                               rted

                                                                                                                                                                                     claim(s
 Evidence
to support                                                                                                                 suppo

                                                                                                                                                      ims suppo
  a claim




                                                                                                                                                                                            )
                                                                                                                                rte

                                Ar
                                      gu
                                                                                                                                   d by e



                                           me
                                                nt
                                                     ne
                                                                                                                                           Not all cla


                                                               ed
                                                                                                                                         viden




                                                                    sm
       Ho                                                                  o re
            w                                                                         ev
                do                                                                         ide
                                                                                                                                              ce?




                     es                                                                          nc
                          ev                                                                          e
                               id e
                                      nc
                                           es
                                                up
                                                     po
                                                          rt
                                                               cl a
                                                                      im                              Argument
                                                                           ?                                                                                                                                         Finished
                                                                                                         for
                                                                                                      Evidence
                                                                                                                                                                                                     te d
                                                                                                                                                                    s                     su pp or
                                                                                                                                                         A ll claim
                 Process Mechanics


• ASCAD notation:
     – Claims
     – Arguments
     – Evidence
• We used arguments between claims
     – This is a deviation from the notation
• Tried to capture all of the standard

March 11, 2004       T. Scott Ankrum — MITRE   34
             Advantages of the Tool


• Carries both graphic structure and text
• Hyperlinks from node to a web page or file
• Enforces structure rules
     – Rules can be temporarily suspended
     – User-supplied rules can be added
• Can export for inclusion in a document
• User views can show parts of the structure

March 11, 2004      T. Scott Ankrum — MITRE    35
   Mapping the Common Criteria


• Most hierarchical of the standards
     – Classes, Families, Components, Requirements
     – Components are atomic and cumulative
• Nearly mechanical process of mapping
• Most of the structure consists of Arguments
     – No sub-claims, only a top-level claim
     – Requirements are place-holders for evidence
• Objectives paragraphs became arguments

March 11, 2004      T. Scott Ankrum — MITRE          36
                                                                                                                                CLAIM
                                                                                                                                EAL4
                                                                                                                       [Confidence in Security                                                                         ARGUMENT
                                ARGUMENT
                                                                                                                      because the product has                                                                              AVA
                                   ACM                                   Supports                                                                                                Supports
                                                                                                                    been] methodically designed,                                                                       Vulnerability
                                Configuration
                                                                                                                        tested, and reviewed                                                                           Assessment
                                Management



                                                                                                                                                                                                                Supports                  Supports
                Supports
                                          Supports
                                                                                                                                                                                                                                                      ARGUMENT
 ARGUMENT                                                                                                                                                                                               ARGUMENT             Supports
                            Supports                                                                                                                                                                                                                   AVA_VLA
  ACM_AUT                                       ARGUMENT                                                                                                                                                 AVA_MSU                                      Vulnerability
CM Automation                                    ACM_SCP                                                                                                                                                  Misuse
                                                                                                                               Supports                                                                                                                Analysis
                                                 CM Scope
                                                                      Supports                                                                                                 Supports

                      ARGUMENT                                                                                                                                                                                                  ARGUMENT
                       ACM_CAP                                                                                                                                                                                                   AVA_SOF
                     CM Capabilities                                                                                                                                                                                          Strength of TOE
                                                                                                                                                                                                                             Security Functions
                                                                                                  Supports
                                                                                                                                                         Supports
                                                                                                                            ARGUMENT
                                                                                                                               AGD
                                                                                                                        Guidance Documents
                 ARGUMENT                                                                                                                                                                                                  ARGUMENT
                     ADO                                                                                                                                                                                                      ATE
                 Delivery and                                                                                                                                                                                                Tests
                  Operation                                                                                            Supports       Supports
                                                                                                                                                                                                                Supports                  Supports
                                                                                                             ARGUMENT
        Supports           Supports                                                                                                        ARGUMENT                                                 ARGUMENT
                                                                                                              AGD_ADM
                                                                                                                                            AGD_USR                                                  ATE_COV                                          ARGUMENT
                                                                                                             Administrator                                                                                            Supports    Supports
                                                                                                                                          User Guidance                                              Coverage                                          ATE_IND
                                 ARGUMENT                                                                     Guidance
ARGUMENT                                                                                                                                                                                                                                          Independent Testing
                                  ADO_IGS                             ARGUMENT
 ADO_DEL
                                 Installation,                            ADV
  Delivery
                                Generation and                        Development                                                                                             ARGUMENT
                                   Start-up                                                                                                                                       ALC                           ARGUMENT                  ARGUMENT
                                                                                            Supports                                                                       Life Cycle Support                    ATE_DPT                   ATE_FUN
                                                     Supports                                                                                                                                                     Depth                 Functional Tests
                                                                                                              ARGUMENT
                                                                                                               ADV_SPM
                                ARGUMENT                                                                     Security Policy
                                                                                       Supports                                                                     Supports                 Supports
                                 ADV_FSP                  Supports                                             Modeling
                                Development
                                                                                                                                                                                                        ARGUMENT
                                                                                                                                                       ARGUMENT
                                                                Supports         Supports                                                                                                                ALC_TAT
                                                                                                                                                        ALC_DVS                 Supports
                                                                                                                                                                                                         Tools and
                                                                                                     ARGUMENT                                      Development Security
                                                                                                                                                                                                        Techniques
                                         ARGUMENT                                                     ADV_RCR
                                          ADV_HLD                                                   Representation
                                      High-Level Design                                            Coorespondence


                                                                                                                                                                               ARGUMENT
                                                       ARGUMENT                       ARGUMENT                                                                                  ALC_LCD
                                                        ADV_IMP                        ADV_LLD                                                                             Life Cycle Definition
                                                     Implementation                 Low-Level Design
                                                     Representation
                                                                      ARGUMENT
                                                                          ATE
                                                                         Tests
                                                                       (50 words)
                                         Supports                                           Supports

                  ARGUMENT                                                                                   ARGUMENT
                   ATE_COV                                     Supports    Supports                            ATE_IND
                   Coverage                                                                              Independent Testing
                                                                                                              (53 words)

                                                ARGUMENT                       ARGUMENT
            Supports   Supports
                                                 ATE_DPT                         ATE_FUN
                                                                                                            Supports     Supports
                                                  Depth                      Functional Tests
 ARGUMENT               ARGUMENT                                                (92 words)
 ATE_COV.1              ATE_COV.2
   Coverage                                         Supports                                           ARGUMENT                     ARGUMENT
                         Analysis of                                             Supports
  (31 words)                                                                                            ATE_IND.1                    ATE_IND.2
                          Coverage
                                                                                                       Independent                  Independent
                         (35 words)             ARGUMENT                       ARGUMENT                   Testing                      Testing
                                                ATE_DPT.1                       ATE_FUN.1               (15 words)
Is evidence for                              Testing: High-Level             Functional Testing
                       Is evidence for             Design                       (27 words)
                                                 (43 words)                                            Is evidence for          Is evidence for
                        EVIDENCE
EVIDENCE                                                                       Is evidence for
                       ATE_COV.2.2             Is evidence for
ATE_COV.1.                                                                                              EVIDENCE                     EVIDENCE
                        Analysis of                                                                     ATE_IND.1.                   ATE_IND.2.
  Coverage
                         Coverage              EVIDENCE                          EVIDENCE              Independent                  Independent
 (53 words)
                        (27 words)             ATE_DPT.1.                       ATE_FUN.1.                Testing                      Testing
                                            Testing: High-Level               Functional Testing        (51 words)                   (41 words)
                                                  Design                         (87 words)
                 Mapping DO-178B


• Less structured, its title begins:
     – “Software Considerations …”
• Focused on system/software product lifecycle
     – Other standards are not time-structured
     – Claims, sub-claims, and evidence are laid out in
       approximately their chronological order
     – No linkages between the generation of one
       artifact and its later use

March 11, 2004       T. Scott Ankrum — MITRE         39
                                             CLAIM                                                                                   CLAIM                                                             CLAIM
                                               7.0                                                                                     8.0                                                               9.0
                                        SCM process is                                                                          SQA process is                                                       Certification
                                            properly                                                                                properly                                                     Liaison Process is
                                        established and                                                                         established and                                                   properly defined




                                                                                                                                    Supports                                                          Supports


                                                                                                                                                                                                    ARGUMENT
                                          Supports                                                                             ARGUMENT                                                               not explicit
                                                                                                                                not explicit                                                         Satisfactory
                                                                                                                             Satisfactory SQA                                                    Certification Liaison
                                                                                                                             process requires                                                    Process comprises
                                                                                                                           three characteristics                                                    three factors
                                                                                                                                (23 words)

                                       ARGUMENT                                                                                                                                                   Is a
                                                                                                                                                                                      Is a subclaim ofsubclaimaof
                                                                                                                                                                                                            Is subclaim of
                                        not explicit                                                                       Is
                                                                                                             Is a subclaim of a subclaim ofIs a subclaim of
                                     satisfactory SCM
                                   process includes six
                                         elements                                                CLAIM                                CLAIM                        CLAIM          CLAIM                  CLAIM               CLAIM
                                        (37 words)                                                8.1a                                  8.1b                      8.1c, 8.3        9.0                     9.1                 9.2
                                                                                                Software                        Transition criteria               Software       Relevant               Means of          Compliance
                                                                                             development &                         for software                  conformity   communication           compliance         substantiation
                                                                                           integral processes                        lifecycle                    review is         &                 is agreed to        is provided
                                                                                               comply with                       processes are


                                                                                                                                                      Supports                         Supports Supports                    Supports
                                                                                                                     Supports       Supports
                                    Is
                      Is a subclaim of a subclaim ofIs a subclaim of
                                                                                                                                                                                                                           ARGUMENT
                                                                                                                                                                                               ARGUMENT
                                                                                                                                ARGUMENT                                                                                       11.20
                                                                                                                                                                                                not explicit
                                                                                                                                 not explicit                                                                            Substantiation of
                                                                                                                                                                                               Plan for SW
                                                                                                                           SQA Records contain                                                                            Compliance is
                            Is a subclaim of                   of
                                                 Is a subclaim Is a subclaim of                                                                                                                 Aspects of
                                                                                                                           material for three areas                                                                       provided by two
                                                                                                                                                                                               Certification

                                                                                                                                  Is evidence for
           CLAIM                           CLAIM                              CLAIM                                                 EVIDENCE                                                                             Is evidence for
                                                                                                                                                                                              Is evidence for
            7.2.1                       7.2.3 - 7.2.6                         7.2.8                                                   11.19
        Configuration                    Problem &                           Software                                                  SQA
         items are                        Change                           load control                                              Records                                                   EVIDENCE                    EVIDENCE
         identified                    Management is                            is                                                                                                                11.1                       11.20
                                                                                                                                                                                               Plan for SW                    SW
                                                                                                                                                                                               Aspects of                Accomplishment
                                                                                                                                                                                               Certification                Summary
                      CLAIM                                      CLAIM                       CLAIM
                      7.2.2                                       7.2.7                       7.2.9
                  Baselines &                                   Archive,                  Software life
                   traceability                              retrieval and                    cycle
                       are                                    release are                 environment
                                      Supports                                                                           Is evidence for
                                                     Supports        Supports



                              Is evidence for
                                        Supports
                                                                Supports       Supports



        Is evidence for                                                                                   Supports
                                                              ARGUMENT
                                                               not explicit
                                                          SCM Records contain
                                                        evidence for the six areas
                                                               (42 words)

                                                             Is evidence for
 EVIDENCE                 EVIDENCE                                                                                 EVIDENCE
    11.16                   11.17                             EVIDENCE                                               11.15
   Software                Problem                              11.18                                           Software Lifecycle
Configuration              Reports                               SCM                                              Environment
    Index                 (0 words)                            Records                                            Configuration
  (0 words)                                                   (0 words)
                 Mapping ISO 14971


• Accompanying amendment is essential for
  mapping into ASCE
• No structural relation between the document
  and the assurance case
• Claims, arguments, and evidence identified
  by analyzing words and phrases
• Very few arguments for evidence
• For Each Identified Hazard…

March 11, 2004       T. Scott Ankrum — MITRE   41
                                                                                                                ARGUMENT
                                                                                                                    H.4
                                                                                                           Rationale for clause 4,
                                                                                                               Risk analysis



                                                                                                              Is a subclaim of



                                                                                                                  CLAIM
                                                                                                                    4
                                                                                                           Risk analysis (Steps
                                                                                                           1, 2 and 3 of Figure
                                                                                                                    2)



                                                                                                                   Supports


                                                                                                               ARGUMENT
                                                                                                                   H.4.1                                                                                    ARGUMENT
                                                                                                               Risk analysis                                                                                    H.4.4
    CLAIM                                                                                                       procedure                                        Supports
                                                          Supports                                                                                                                                       Estimation of the
      4.1                                                                                                                                                                                                 risk(s) for each
 Risk analysis                                                                                                                         Supports                                                                hazard                                                CLAIM
                                                                                                                                                                                                                                   Is a subclaim of
  procedure                                                                                                                                                    ARGUMENT                                                                                               4.4
                                                                                                Is a subclaim of
                                                                                                                                                                   H.4.3                                                                                      Estimation of the
                                                                                                                                                        Identification of known or                                                                         risk(s) for each hazard
                                                                                        CLAIM                                                             foreseeable hazards                                                                                       (Step 3)
    Supports
                                                                                           4.2
                                                                                Intended use/intended                                                                                                                                        Supports                                   Supports
                                                                                     purpose and                                                                                                                      ARGUMENT
    ARGUMENT
                                                                                    identification of                                                       Is a subclaim of                                              H.4.4                                                                       ARGUMENT
         4.1
   Compliance is                                                                                                                                                                                                   Estimation of the                                                                    NOTE 4
checked by inspection                                                                                                                                                                                               risk(s) for each
                                                                                                                                                                                                                                                      Supports
     of the risk                                                                          Supports                                                                 CLAIM                                                 hazard
                                                                  Supports                                                                                           4.3
    Is evidence for                                                                                                                                     Identification of known or
                                                                                            ARGUMENT                                                                                                                                                             Supports
                                                                                                                                                          foreseeable hazards
                                                                                                 4.2
     EVIDENCE                                                                                                                                                     (Step 2)
                                                                                      Compliance is checked by
         3.6                                                                                                                                                                                                                           ARGUMENT
                                                ARGUMENT                                inspection of the risk                                                                                                                                                                                           Is evidence for
  Risk Management                                                                                                                                                                                                                        NOTE 3                                   Supports
                                                   H.4.2                                  management file.
        File                                                                                                                                 Is a subclaim of
                                          Intended use/intended                                                                                                                          Supports
                                               purpose and                                                                                                                Supports                                                                                                                 Supports
                                              identification of                                                                       CLAIM
                                                                                            Is evidence for
                                                                                                                                       4.3
                                                                                                                                   Foreseeable                                                                ARGUMENT
                                                                                                                                                                                                                                                         ARGUMENT
                                                                                               EVIDENCE                        sequences of events                                                              NOTE 2                                                                                          EVIDENCE
                                                                                                                                                            Supports            ARGUMENT                                                                    4.4
                                                                                                  3.6                           that may result in a                                                            & H.4.3                                                                                            3.6
                                                                                                                                                                                  NOTE1
                               Supports                                                           Risk                                                                                                                                                                                                             Risk
                                                 Supports            Supports                                                                                                                                                Is evidence for
                                                                                              Management                                                                                                                                                                                                      Management File
                                                                                                  File                                                                                                                                                                      Supports
                                                                                                                                              Supports
                                                                                                                                                                                                                 Is evidence for
                                                                                                                                                                                     Supports                                                                                               ARGUMENT
                                                                                                                                                                                                                                                                                                 4.4
         ARGUMENT                                ARGUMENT                             ARGUMENT
                                                                                                                                                        ARGUMENT                                                                                         Is evidence for              Compliance is checked by
           NOTE 1                                  NOTE 2                               NOTE 3
                                                                                                                                                            4.3                                                         EVIDENCE                                                        inspection of the risk
                                                                                                                                                  Compliance is checked                  EVIDENCE                         Annex F                                                         management file.
                                                                                                                                                  by inspection of the risk               Annex D                   Information on risk
                                                Is evidence for                      Is evidence for                                                 management file.                   Examples of                analysis techniques
       Is evidence for
                                                                                                                                                                                          possible                                                                          Is evidence for
                                                                                                                                                                                        hazards and
        EVIDENCE                                   EVIDENCE                             EVIDENCE                                                       Is evidence for                   contributing                                                      EVIDENCE
          Annex A                                    Annex B                              Annec C                                                                                          factors                                                             3.6
   Questions that can be                   Guidance on risk analysis                 Guidance on risk                                                   EVIDENCE                       associated with                                                  Risk Management
  used to identify medical                   for in vitro diagnostic              analysis procedure for                                                   3.6                             medical                                                            File
 device characteristics that                   medical devices                     toxicological hazards                                           Risk Management File                    devices
   could impact on safety
           Validating Our Mappings


• Domain experts reviewed our mappings
   – Common Criteria
        • System security experts within MITRE
   – DO-178B
        • Evaluator (FAA Designated Engineering Representative)
   – ISO 14971
        • FDA CDRH
• Varying conclusions from validations

 March 11, 2004        T. Scott Ankrum — MITRE             43
    Conclusions and
      Follow-on



March 11, 2004   T. Scott Ankrum — MITRE   44
                 Ada Lovelace




March 11, 2004    T. Scott Ankrum — MITRE   45
                 Hypotheses Revisited


• Assurance Standard                    Assurance Case
     – There does not seem to be much of a
       relationship between the two structures
     – Experience with actual assurance supports this
• Assurance is Assurance is Assurance
     – Negation of the above hypothesis prevents us
       from coming to any conclusion on this one


March 11, 2004        T. Scott Ankrum — MITRE            46
                 Standards Templates


• Mappings might be used as templates
     – Could be a side benefit of the study
     – Without structural relation, possibility looks bad
• Advantages of consistency may help drive
  assurance-requirements standardization
     – Currently, hard to “compare apples and oranges”
     – Evaluation of assurance claims easier if
       requirements are consistent

March 11, 2004        T. Scott Ankrum — MITRE          47
                 Extensions to Tool


• Extend ASCE features to be more helpful
• Make ACSE more generic
• Enhance possibilities for user customization




March 11, 2004       T. Scott Ankrum — MITRE   48
                 Shadow a Real Project


• Activities
     – Document a real process
     – Identify where and how to incorporate technique
• Advantages
     – Learning opportunity for us
     – Minimal impact on the project
     – Not in the project’s critical path

March 11, 2004         T. Scott Ankrum — MITRE      49
                 Develop Training


• How to use the notation and notation options
• How to develop a structured assurance case
• How changes affect the assurance case
     – Software, hardware
     – Operation, environment
• How to write a structured assurance standard


March 11, 2004      T. Scott Ankrum — MITRE   50
                 Use on a Real Project


• Apply methdology within a project’s schedule
• Gain experience with maintenance of
  assurance cases
• Update process with lessons learned
• Propagate this knowledge to other projects



March 11, 2004         T. Scott Ankrum — MITRE   51
                 Discussion




March 11, 2004   T. Scott Ankrum — MITRE   52

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:4
posted:8/28/2012
language:Latin
pages:52