Design Integrity Concepts by u53UCYt3

VIEWS: 25 PAGES: 142

									Design Integrity Concepts




2005 MAPLD   1       Design Integrity Concepts
                    Unit Agenda
• Consistent terminology, consistent results –
       Introduction and definitions
• What does it have to do? –
       Specifying the design
• Letting constraints work for you –
       Proportional design
• More than the sum of its parts –
       Partitioning a design
• You mean we’re still working on it? –
       Sustaining a design
• What’s the exit strategy? –
       Verifying a design
• What do you mean, it doesn’t do what we thought? –
       Validating a design

       2005 MAPLD           2                Design Integrity Concepts
                   Who is this guy?
• John Stone
  – Chief Engineer (Department of Space
    Systems – SwRI®)
  – 18 years experience (Exclusively space-flight)
     •   C&DH design (hardware and software)
     •   Instrument electronics design
     •   Box-level integration and test
     •   Hardware project management
     •   Product assurance (parts, radiation, reliability)
     •   Process improvement
  – Broad interest in improving what we do


      2005 MAPLD                    3                        Design Integrity Concepts
What do we want to accomplish?
• Discuss techniques that may help us to do
  our jobs better
  – Based on general principles
  – Broad applicability (logic, board, box, system)
• Foster necessary discussion / brainstorming
  – To extent possible (as time allows)
  – No one person has all of the answers
  – Suggestions appreciated


     2005 MAPLD          4              Design Integrity Concepts
Consistent Terminology,
  Consistent Results
      Introduction and Definitions




2005 MAPLD          5            Design Integrity Concepts
                Agenda –
      Introductions and Definitions
• Design integrity – A working definition
• Why is this important?
   – A tale of two designs
   – Has anything changed?
   – Why should I care?
• The miracle cloud
• The alternative
   – Overview
   – Implications
   – Additional Definitions
• Summary



       2005 MAPLD             6             Design Integrity Concepts
              Definition and Goal
• Design – The invention and disposition of the
  forms, parts, or details of something according to a
  plan. (AH dictionary online)
• Integrity – The state of being unimpaired;
  Soundness; The quality or condition of being
  whole or undivided; completeness (AH)
• This seminar is intended to talk about techniques
  and issues that ensure the soundness and
  completeness of both the end product and the
  means used to produce it.

      2005 MAPLD          7               Design Integrity Concepts
Design 1 – Radarsat 1 ACP




 2005 MAPLD   8      Design Integrity Concepts
     Radarsat 1 ACP Overview
• Program dates: late 1990 – late 1992
• Specifications
   – Processor: 8 MHz 80386/80387
   – Memory:128 k x 8 SRAM, 192kx8 EEPROM, 16k x 8
     PROM
   – Interfaces: A/D (16), D/A (4-12), Synchronous serial (3
     input, 3 output), RS-232 GSE
   – Function: Attitude control processor for the
     RADARSAT1 satellite
   – Logic Implementation MSI logic + PALs (16L8, 16R6,
     16R8)
• Additional functions (cross-strap, control) external

       2005 MAPLD            9                Design Integrity Concepts
             PAL Reminder




2005 MAPLD        10        Design Integrity Concepts
Design 2 - Command Telemetry
         Board (CTB)




 2005 MAPLD   11       Design Integrity Concepts
                    CTB Overview
• Program Dates: early 2001 – late 2003
• Specifications:
   – Processor: RTX2010 (16 MHz)
   – Memory: 4M x 8 (random), various FIFOs (16k x8) as necessary
   – Interfaces
       •   MIL-STD-1553B
       •   Synchronous Serial (command / telemetry)
       •   Analog input
       •   High-level discrete (output)
       •   Low-level discrete (input / output)
   – Functionality: S/C command/telemetry (level 0 and active)
   – Logic Implementation: 4 54SX32S FPGAs
• Additional resources (in same box): RAD 750, Mass
  Memory, Instrument Interface Card

       2005 MAPLD                    12               Design Integrity Concepts
                    What’s Changed?
• Capability / Complexity
• Logic Density
• Specificity
   – RADARSAT (1 small specification with interface focus)
   – CTB (1 large specification with interface, s/w, operations focus)
• Software Centricity
• Initial Errors
   – RADARSAT: 3 jumpers; 1 PAL design change
   – CTB: 14 FPGA revisions
       • 2 spec change
       • 5-6 mistakes
       • 6-7 data dependency




       2005 MAPLD                  13                   Design Integrity Concepts
           What’s Not Changed?
•   Overall program schedules
•   Proportional budget
•   Expectation of correctness
•   Pain from mistakes




       2005 MAPLD       14       Design Integrity Concepts
             What Explains the (error)
                   Difference?
• Engineers aren’t as capable? – Insulting!
• Everything is just more complex? – Copout!
• Methodology?
   – Methodology hasn’t changed
       • Always inadequate, we just got lucky
       • Adequate for old designs, no longer adequate
   – Methodology has changed
       • Used to be adequate, but we lost the recipe
• Design philosophy of systems has changed?
   – Predicated on maximum flexibility
   – Expectation of extreme complexity
   – Over-specification – almost impossible to meet



          2005 MAPLD                    15              Design Integrity Concepts
What Do These Examples Illustrate?
• The incidence of initial correctness for designs seems to be
  decreasing
   – Design changes seem to be more common
   – Problems late in the verification/validation cycle seem to be more
     frequent
• Perhaps a combination of the factors presented explains
  this, but …
   – Desired complexity is not going to decrease
   – Budgets are not going to get bigger
   – The expectation of excellence isn’t going to go away
• The only solution is to develop and improve a consistent
  methodology for implementing robustly designed products
   – Based on basic principles
   – Applicable to a variety of conditions


       2005 MAPLD                  16                   Design Integrity Concepts
              Why Should I Care?
• Why do we work?
   – Self-actualization (fun, monetary reward, interest)
• Why do people want us to work for them?
   – They need what we produce
• What do people want engineers (especially in
  Aerospace) to produce?
   – A quality product that satisfies the customer’s needs
• How do they want us to produce such a product?
   – Consistently and efficiently



       2005 MAPLD             17                Design Integrity Concepts
       The Layman’s View –
        The Miracle Cloud




2005 MAPLD      18       Design Integrity Concepts
     The Miracle Cloud Method
• Note that too many engineering schools teach this
  approach without meaning to
• Advantages to the miracle cloud method
   – Total creative freedom
• Disadvantages to the miracle cloud method
   – Product quality is variable
       • Team makeup dependent
       • Team mood/morale dependent (Monday morning car)
       • Luck dependent
   – Product is not produced in a repeatable manner
   – Product is not produced in an efficient manner
• Result
   – Quality low
   – Customer Satisfaction Low


       2005 MAPLD                  19                 Design Integrity Concepts
         How Do We Replace the
            Miracle Cloud?
• Provide structure to the development effort
• Evaluate the effort and the product
  produced
• Improve the effort and the product
• Repeat



     2005 MAPLD        20           Design Integrity Concepts
     Definitions of Importance
• From Q9000-2000
• Process – A set of interrelated and
  interacting activities which transforms
  inputs to outputs [in our case ideas to
  devices]
• Product – The result of a process



     2005 MAPLD        21           Design Integrity Concepts
Implications From These Definitions
• If we want a consistent product, we must have a
  consistent process
• If we want to improve a product, we must improve
  the process
• If our company has no (or inadequate) process and
  we must produce a quality product, then we must
  establish a process [personal responsibility]
• Developing, imposing, and improving a process is
  not an end (in and of itself) it is only a means to
  an end


      2005 MAPLD          22             Design Integrity Concepts
     A Model for Discussing
       the Design Process




2005 MAPLD     23        Design Integrity Concepts
                 Notes on the Model
• Feedback / iteration are not shown for clarity
• Model may be recursive
   – Board development process includes FPGA requirement definition,
     FPGA development, instantiation, etc.
   – Board verification process includes the FPGA validation product
• Successes and failures are cumulative
   – Good requirements + successful development => successful
     instantiation
   – Bad requirements + failed development => failed instantiation
• Complexity multiplies
   – Complex requirements increase design complexity which, in turn,
     increases verification complexity
• Processes are absolute gates to the next stage of development


         2005 MAPLD                 24                  Design Integrity Concepts
    Implications From the Model
• All processes must be addressed at all levels of design
  [there are no shortcuts!]
   – Does not imply same formality at all levels
   – Does imply same rigor at all levels
• Up front work on requirements is essential!
   – Must provide adequate time and money
   – Must gain team buy-in to the process*
   – Benefits compound throughout the rest of the activities
• Simplicity is an essential virtue!
   – Complexity inevitably multiplies
• A product is not qualified until both verification and
  validation are complete


        2005 MAPLD                 25                  Design Integrity Concepts
       Additional Useful Definitions
        (courtesy of Q9000-2000)
• Specification – A document* stating requirements, needs, or
  expectations that are obligatory
• Quality – The degree to which a set of inherent characteristics fulfill
  requirements
• Customer satisfaction – Customer’s perception of the degree to which the
  customer’s requirements have been fulfilled
• Verification – Confirmation, through the provision of objective evidence,
  that specified requirements have been fulfilled
• Validation – Confirmation, through the provision of objective evidence,
  that the requirements for a specific intended use or application have been
  fulfilled
• Objective evidence – Data supporting the existence or verity of something
• Continual Improvement – recurring activity to increase the ability to
  fulfill requirements
• Note the importance of Requirements

         2005 MAPLD                 26                  Design Integrity Concepts
                         Summary
• I have no assurance that my product will have consistent
  quality without:
   – Well-defined requirements
   – A well planned approach to implementing the requirements
   – A clearly defined plan for verification and validation of the
     requirements
   – The ability to improve the process that produces the product
• Without quality product, customer satisfaction is
  impossible
• Without customer satisfaction, I won’t work!
• Therefore, I must care about ensuring design integrity


       2005 MAPLD                  27                   Design Integrity Concepts
What Does It Have to Do?
       Specifying the design




2005 MAPLD       28            Design Integrity Concepts
    Agenda – Specifying the Design
•   Basic concepts and implications
•   What should we include in a specification?
•   What should we avoid in a specification?
•   The role of the design engineer
•   Summary
•   FPGA specifications



       2005 MAPLD       29           Design Integrity Concepts
   Basic Specification Concepts
• #1 A specification has a limited set of purposes:
   – It is intended:
      • To capture product requirements that are essential to meeting a
        need – functionally and interfacially
      • To ensure traceability of the requirements from higher levels
      • To provide a fixed framework for verifying product
        conformance
      • To provide a framework for derived requirements
   – It is not intended:
      •   To impose a detailed implementation for a design
      •   To provide unnecessary theory of operation statements
      •   To become a user’s manual
      •   As a box to check on the schedule checklist


       2005 MAPLD                  30                  Design Integrity Concepts
Basic Specification Concepts (cont.)
• #2 Specification language must meet certain criteria
   – Structure
       • Similar requirements must be grouped together
       • All requirements must stand on their own
   – Verbs must have a consistent specific meaning
       • Shall, must -> impose imperatives
       • Should, might -> define preferences
       • Will -> defines objective information
   – Phraseology
       • Phrases must be unambiguous
       • Phrases must define one (and only one) requirement
       • Phrases must be short and understandable
   – All must agree on the meaning of what is written



       2005 MAPLD                    31                       Design Integrity Concepts
Basic Specification Concepts (cont.)
• #3 Specifications are inherently intrusive
   – Requirements exclude options in a design
       • The number and specificity of requirements determine the level of
         exclusion
   – Specifications impose non-negotiable verification requirements
       • Every requirement must be verified
       • The narrower the requirement, the more specific the verification
       • The more requirements, the more work needs to be performed
   – A well designed specification assists the design and verification
     process
   – A poorly designed specification
       • Unnecessarily shuts down trade space
       • Increases verification time and effort
       • Makes for an unhappy engineer



       2005 MAPLD                    32                      Design Integrity Concepts
What is Included in a Specification?
                      • Interfaces
                         – Number
                         – Type
                         – Timing / Handshaking
                      • Interrelationships
                         – Temporary storage
                         – Data Flow
                         – Software Support
                           (careful)
                      • System impact
                         – Quality and Reliability
                         – Parts, materials, and
                           processes



    2005 MAPLD   33               Design Integrity Concepts
   Examples of Things to Include
• Interfaces
    – The unit shall provide 16 low-level discrete interfaces
    – The unit shall configure 8 low-level discrete interfaces as pulsed interface
    – Pulse interfaces shall generate pulses between 2 and 20 ms long
• Interrelationships
    – The unit shall provide on-board temporary storage sufficient for 2 packets
      from the XYZ interface
    – The unit shall forward data from the XYZ to QRS interfaces on a packet
      by packet basis
    – The unit shall provide an interrupt to the S/W upon receipt of a complete
      package from the XYZ interface
• System Integration
    – The unit shall have a .75 probability of success as calculated per the parts
      count method of MIL-HDBK-217
    – The Unit shall use only EEE parts that meet the requirements of EEE-
      INST-002




         2005 MAPLD                     34                      Design Integrity Concepts
What Should Not Be Included in the
         Specification?
• Implementation details (unless essential to safety or
  reliability)
   – The unit shall implement standard pyrotechnic fire circuit #2 as
     found in …(marginally acceptable)
   – The unit shall use 4 7206 FIFOs made by IDT corporation
   – 370 ns after the last bit of a packet is received the unit shall raise
     bit 3 of interrupt control register 21
• Requirements based on inherent capability of devices used
   – The unit shall provide a 14-bit A/D converter for interface X
     (when based upon the part being used)
   – Note: Decisions on precision or accuracy etc. should be made for
     interface characteristics (physical measurement range and
     accuracy)


        2005 MAPLD                   35                     Design Integrity Concepts
    What Shouldn’t Be Included in the
          Specification (cont)?
• Ambiguous, unclear, or interpretive requirements
   – The unit shall be constructed using techniques found in generally
     accepted space hardware guidelines
   – The unit shall interface to other components per figure 2 (picture
     based requirements)
   – The unit shall provide the best performance possible
• Overly complex requirements
   – The unit reset shall be immediately triggered upon expiration of
     the watchdog timer unless the watchdog timer has timed-out five
     times (8 times if in mode 2) in which case the circuitry shall look
     at discrete bit 2 to determine whether to delay 3 s (discrete bit 2 =
     1) before timing out
• Extraneous information
   – The unit shall interface with the visible CCD used in
     spectrographic mode
   – The unit shall replace the functionality provided by p/n 276401-01

        2005 MAPLD                  36                    Design Integrity Concepts
 Why Shouldn’t These Be Included?
• This type of requirement increases cost [time, money,
  emotional] to develop a product
   – Increase design effort (less trade space, inefficient design, overly
     complex design)
   – Increase verification effort [planning, implementing, configuring]
     due to design complexity
   – Increase probability of re-design or re-build [validation
     uncertainty]
• This type of requirement reduces the overall quality of the
  product developed
   – Implemented design not per intention [interpreted requirements]
   – Implemented design is not 100% verifiable to specification
     [unclear requirements]
   – Implemented design requires significant number of waivers and
     deviations


        2005 MAPLD                  37                    Design Integrity Concepts
The Role of the Design Engineer
• The design engineering team must “buy in” to the
  specification development process
   – They have to deal with the consequences if it is done
     poorly
   – Lack of “buy in” produces a non-controlled process
• “Buy in” requires early review and participation
  by the design team
• Therefore, design engineers have a significant role
  to play in the specification development process
• What is this role?


       2005 MAPLD             38               Design Integrity Concepts
 Role of the Design Engineer (cont.)
• Know how to write a specification
   – Allows constructive review of the imposed specification
   – Improves qualities of derived requirements
• Try to understand the “why” behind the various
  requirements
   – Improves efficiency of design trade studies
   – Allows intelligent suggestion of alternatives
• Suggest alternatives when necessary
   – Expose more efficient ways of producing equivalent functions
   – Support trade offs between software and hardware functionality




       2005 MAPLD                  39                 Design Integrity Concepts
 Role of the Design Engineer (cont.)
• Think forward
   – Take the lead in defining derived requirements
   – Ask:
       • What implications does this have for the design?
       • What implications does this have for testability?
       • Will this let me sleep at night?
• Push back
   – Seek clarification of ambiguous requirements
   – Inform others of impractical or costly driving requirements
   – Ask for relief from impossible requirements
• Remain engaged in the process
   – Be thorough in review
   – Don’t be passive with suggestions



       2005 MAPLD                     40                     Design Integrity Concepts
                   Summary
• Specifications are critically important to the
  design engineer and must not be ignored
• Design teams must be intimately involved
  in the specification development process
• Detailed design and implementation will
  succeed or fail largely based on successful
  application of the specification process


      2005 MAPLD        41           Design Integrity Concepts
 FPGA Specifications - Rationale
• FPGAs are a major part of modern day spaceflight designs
   – Primary control functionality
   – Equivalent of multiple boards of traditional circuitry
   – Major schedule driver
• FPGAs are impossible to verify completely from external
  signals
   – Too many buried modes and functions
   – Too much dependence on simulation
• Correcting FPGAs is conceptually simple
   – Temptation to sloppiness
• “First time right” is an essential virtue
   – The FPGA specification is a means to achieve “first time right”



        2005 MAPLD                  42                   Design Integrity Concepts
  FPGA Specification Prototype
• Interface Section
   – Specific pinout requirements
   – I/O type definition (Names, Direction, Logic levels)
   – Imposed Software requirements (addressing, etc.)
        • Do not include non-imposed addressing / register mapping / software
          interfaces
• Functionality Section
   –   Interface interaction requirements
   –   Data flow requirements
   –   Exclusion requirements
   –   Do not include
        • Implementation details
        • Theory of operations
        • Usage instructions




         2005 MAPLD                   43                    Design Integrity Concepts
FPGA Specification Prototype (cont.)
• Fine timing section
  – All timing imposed by board peripheral circuits
  – Include
     • Min and max delay between I/Os
     • Set-up and hold for controlled peripherals
• Fine timing requirements should be an input
  to the FPGA development effort, not
  outputs of the concluded design


     2005 MAPLD             44                Design Integrity Concepts
     Why Include Fine Timing?
• Reduces influence of changes external to FPGA
  (encapsulation)
   – Board implementation can change significantly
   – FPGA implementation can change significantly
   – Changes ok as long as interfaces remain stable - but fine timing
     must be a controlled item
• Simplifies verification
   – Verification between implemented FPGA performance and fixed
     requirements defined at the board level can be easily accomplished
   – Reduces reliance on complex back-annotated simulations at the
     board level for FPGA specific verification
   – Increases reliability of static timing analysis
• Note – this only works if a structured design approach is
  used such that valid requirements can be imposed on the
  FPGA early in the process.


       2005 MAPLD                  45                   Design Integrity Concepts
      Stand-Alone or Integral?
• When should an FPGA specification stand alone?
  – When the FPGA design engineer is not the board
    design engineer
  – When there are multiple interrelated FPGAs on a board
  – When the FPGA design will be used in multiple board
    applications
  – When the FPGA will become an ASIC
  – When the development schedule between the board and
    FPGA are disjoint
  – When the complexity of the FPGA is greater than that
    of the board support circuitry


      2005 MAPLD           46               Design Integrity Concepts
 Stand Alone or Integral (cont.)?
• When should an FPGA be specified as part of the
  board specification?
   – When the FPGA is so intertwined with the peripheral
     circuitry that writing a stand-alone specification is not
     practical
   – When the FPGA functionality is easily expressed by
     simple gate level schematics
   – When the FPGA and board are designed by the same
     person
   – When heritage design with adequate specifications exist



       2005 MAPLD             47                Design Integrity Concepts
Letting Constraints Work For You

                Proportional Design




   2005 MAPLD            48           Design Integrity Concepts
    Agenda – Proportional Design
•   Conceptual background
•   Types of constraints
•   Examples
•   The proportional design mindset
•   Summary




       2005 MAPLD       49            Design Integrity Concepts
       Conceptual Background
• Three parts to solving a problem:
  – Need, solution set, constraints
• All parts have a role to play in the solution
• Ignoring any of them will lead to problems




      2005 MAPLD          50          Design Integrity Concepts
  Conceptual Background (cont.)
• Example
   – Need:          means of conveyance to work
   – Solution set:  Skateboard, bicycle, bus, jogging shoes, mid-size
                    sedan, luxury car, helicopter
   – Constraints: Distance (6 miles), $, not on bus route, $, not in
                    very good shape, $
   – Solution:      1992 Honda Accord (120 kmiles, 4 k$)
   – The constraints guide selection of the solution from the solution set
• The particular solution is not necessarily -
   – The cheapest (roller skates)
   – The most desired (Lexus LS400)
   – What is perceived as best for society (bus)
• But … the best overall fit to the needs


        2005 MAPLD                  51                   Design Integrity Concepts
  Conceptual Background (cont.)
• Definitions
   – Constraint: the state of being checked, restricted, or compelled to
     avoid or perform some action (AH)
   – Proportional: corresponding in some degree or intensity (AH)
• Proportional design is design that results in a product
  “sized” appropriately to the needs and restrictions of the
  specification
• The concept of proportional design:
   – Accepts the reality of constraints
   – Attempts to optimize the solution given the constraints
   – Accepts that the constraints provide benefits (more later)
       •   More efficient designs
       •   More thorough designs
       •   More correct designs
       •   Caveat – All other things being equal


        2005 MAPLD                     52                Design Integrity Concepts
            Types of Constraints
• External (mass, power, cost, quality)
• Internal
   – Derived (packaging, architecture, component
     availability, maximum clock speed)
   – Self-imposed
      • Design rules/guidelines (free space, clock use, logic structure,
        HDL language)
      • Documentation style (pre-design, post design)
      • Component acceptability (maturity of part, limited use of
        various features



      2005 MAPLD                  53                    Design Integrity Concepts
                   Examples (1)
• Problem: provide decoding logic for memory map
   – 0-3FFF = SRAM; 4000-4FFF = Peripheral; E000-FFFF = PROM
• Constraint: use minimum amount of logic




• But what about …
   – Unused addresses, future expansion, etc.
• Doesn’t matter – given the constraints

      2005 MAPLD             54                 Design Integrity Concepts
                     Examples (2)
• Problem: provide all combinational / sequential logic for
  the RADARSAT ACP
• Constraint: Only low density high speed logic available
  (16X8 PALs, MSI/SSI logic)
• What was forced by the constraint?
   – Careful mapping of peripherals into available address space
   – Careful partitioning between:
       • Programmable logic and MSI/SSI
       • MSI/SSI functionality
   – Efficient data bus partitioning (tri-state enable issues)
   – Special attention to component delays at the gate level



        2005 MAPLD                  55                    Design Integrity Concepts
The Proportional Design Mindset
• Constraints inevitably foster attention to detail (creativity
  “inside the box”)
   – With respect to methodology
   – With respect to level of planning
   – With respect to implementation
• Attention to detail is of inherent value because it produces
  carefully structured, well-thought out designs
   – Improved up-front correctness
   – Decreased design post-processing time (simulation, verification,
     validation, lab time)
   – Efficient designs that meet the stated requirements
   – Increased reliability
• Therefore, constraints are welcomed, whether externally
  imposed or self-imposed


        2005 MAPLD                 56                  Design Integrity Concepts
          The Proportional Design
              Mindset (cont.)
• Examples of self-imposed constraint
  – Ignoring achievable flexibility (when not
    necessary)
  – Removing non-specified capability
  – Avoiding gratuitous cleverness (especially with
    abstract design techniques)
  – Rejecting brute force solutions without analysis



     2005 MAPLD          57             Design Integrity Concepts
           The Proportional Design
               Mindset (cont.)
• Characteristics of the right mind set
   – Planning before starting
   – Reviewing before finalizing
   – Simplifying ruthlessly
   – Making the design do only what it must
   – Viewing resources as precious commodities to be used
     only to the extent needed
   – Understanding the implication of the design’s level of
     abstraction
   – Being satisfied with the result


       2005 MAPLD            58               Design Integrity Concepts
            The Proportional Design
                Mindset (cont.)
• Why aren’t self-imposed constraints more common?
   – They aren’t absolutely essential because we have:
       • Lots of logic space [FPGAs, ASICs]
       • Lots of memory space [DOS file systems, complicated operating
         systems]
       • Lots of bandwidth [fast data busses, general purpose communications
         protocols]
   – They don’t match the current paradigm
       •   Flexibility is all-important [re-use, re-configure, adapt]
       •   Specifications are malleable late in the game
       •   Software changes, why can’t hardware?
       •   We can catch problems in simulation and reprogram the part
   – They aren’t fun
   – We don’t train people to value constraints and work within them
• This is unfortunate because constraints can make our job
  easier without degrading the end product

       2005 MAPLD                     59                     Design Integrity Concepts
                  Summary
• The proportional design mindset is
  important because it:
  – Focuses on fulfilling needs, not wants
    [specification orientation]
  – Deepens understanding of the final design
    [ownership oriented]
  – Avoids unnecessary effort [efficiency oriented]
  – Fosters simplicity that aids verification and
    validation [quality oriented]


     2005 MAPLD          60             Design Integrity Concepts
More Than the Sum of Its Parts

               Design Partitioning




  2005 MAPLD            61           Design Integrity Concepts
  Agenda – Design Partitioning
• Definitions and examples
• Why partition an electronic design?
• Guidelines for partitioning an electronic
  design
• Why isn’t it done more often?
• Summary



      2005 MAPLD       62            Design Integrity Concepts
      Definitions and Examples
• Partition – to divide into parts or shares
   – Examples: budgeting, outlining, WBS development
• Design partitioning refers to the deconstruction of
  a design into various sub-designs in an ordered
  and logical manner
• Goal – to simplify the whole by optimizing the
  parts and thus increase:
   – Efficiency
   – Reliability
   – Maintainability


       2005 MAPLD          63              Design Integrity Concepts
         Why Partition (General)?
• Complexity interferes with ready comprehension
    – Comprehension of a complex system depends on ability to impose order
      upon it
    – But … a given mind has a finite capability to impose order which depends
      on the quantity and structure of the data related to the system
    – The more complex the system, the more difficult its comprehension
• Partitioning a design introduces a piece-wise reduction in complexity
    – Reduces quantity and complexity of design to manageable “chunks”
    – Improves comprehension of the various parts of the design
    – Increased comprehension of the parts leads to better comprehension of the
      whole
• Better comprehension of the whole increases the ability to
    – Verify correctness of the design
    – Correct errors in the design
    – Update the design when necessary



         2005 MAPLD                    64                    Design Integrity Concepts
         Why Partition (cont.)?
• Programmatic advantages
  – Refines scope of work
     • Identifies unexpected effort
     • Identifies reuse possibilities
     • Identifies staffing requirements
  – Identifies schedule dependencies
     • Improves allocation of resources
     • Identifies parallelization and schedule enhancement
       opportunities
  – Promotes management visibility


     2005 MAPLD                  65                  Design Integrity Concepts
        Why Partition? - Design
• Improved interface organization / more formal structure
   – Interfaces between functions have more predictable characteristics
   – Expansion by addition of functions is more controlled
• Enhanced functional encapsulation
   – Individual functions have predictable results when invoked
   – Functional design enhancements have limited side-effects when
     installed
   – Effects of faults are more easily predicted and mitigated
• Efficient design implementation
   – Functional (fewer functions and types of functions)
   – Data flow (fewer, more sensible data busses)
   – Control flow (simpler address decoding and state machines)
• All are side effects of additional thought put into “How?”


       2005 MAPLD                  66                  Design Integrity Concepts
    Why Partition? - Correctness
• Simpler inspection
   – Functionality may be “obvious at a glance”
   – Error space is more limited
       • Within the function or within the interconnect
• Simpler Qualification
   – Verification can begin with encapsulated modules or circuit
     subsets
   – Overall functionality correctness becomes less of a “late” concern
     because subset functionality is proven correct “early” in the
     process
• Simpler / more thorough review
   – Structure provides orientation to peer reviewers
   – Encapsulation allows easier review
   – Peer input more likely to be useful



       2005 MAPLD                    67                   Design Integrity Concepts
Why Partition? – Maintainability
• Clearer documentation
   – Documentation has “smaller” parts to focus on
   – Structure of documentation grows from the design structure
• Simpler maintenance
   – Changes affect only enhanced area
   – Interactions between changed area of design and remainder of
     original design is controlled by a formal structure
• Enhanced reuse
   – Sub-circuits / functions usable in other applications as long as the
     interface structure is observed
• Inherent capability of design better understood



        2005 MAPLD                  68                    Design Integrity Concepts
     Guidelines for Partitioning
• Take advantage of organizational strengths
   – Expertise (analog, digital, software, etc.) is seldom the
     same across organizations
   – Partition the design in accordance with organizational
     strengths according to primary functions
   – Divide auxiliary functions (those that can be assigned
     to multiple organizations) so that
      • Interfaces are simplified
      • Workload is equalized
      • Functions are easily tested without requiring all of the
        hardware



       2005 MAPLD                  69                   Design Integrity Concepts
  Guidelines for Partitioning (cont.)
• Example: Alice UV spectrograph electronics
  (Rosetta)
• Core expertise
   – Sensor Sciences: Detector and front end electronics
   – Mobius Systems: Embedded software
   – SwRI® space systems: Digital, low speed data
     acquisition
   – SwRI® space science: LV and HV power supplies
• Primary partition per core expertise

       2005 MAPLD            70                Design Integrity Concepts
  Guidelines for Partitioning (cont.)
• Alice example (work breakdown chosen)
  – Sensor Sciences: detector electronics through parallel
    digital output – simplified interface
  – Mobius: instrument control / protocol servicing (some
    error decoding partitioned to H/W)
  – Space systems: microcontroller; s/c interface; heater,
    motor, door digital control; analog high-voltage control
  – Space sciences: LVPS; HVPS; motor, door, and heater
    drive circuitry



      2005 MAPLD             71               Design Integrity Concepts
  Guidelines for Partitioning (cont.)
• Alice example – partitioning decisions
  – Auxiliary expertise split along sensible
    interfaces
     • C&DH to detector – analog or digital? (noise, ease
       of test) - digital
     • C&DH to HVPS – analog or digital? (space, noise
       susceptibility, ease of test) - analog
     • C&DH to S/W – protocol decoding? (performance
       margin, logic capability) – hardware error
       decoding, s/w protocol encoding


     2005 MAPLD             72               Design Integrity Concepts
  Guidelines for Partitioning (cont.)
• Use specific functionalities
   – Combine functionality with very similar characteristics
      • Low-level analog / discrete bi-level (comparator is difference)
      • Example: Spacelab flex interfaces
   – Cluster related functionalities
      • Shared data-flow direction, shared logical control
      • Examples
            – Constant level discretes / pulsed discretes
            – Low-level discretes / high-level discretes




       2005 MAPLD                     73                    Design Integrity Concepts
  Guidelines for Partitioning (cont.)
• Use specific functionalities (cont.)
   – Isolate related functional sub-groups from other items
      • Ex. – analog data acquisition group (multiplexers, converters,
        signal processing, control logic)
      • Isolate
            – Through appropriate data/address buffering
            – Through separate programmable logic sub-sets
   – Exploit directionality
      • Write functions; read functions; read/write functions –
        appropriate buffering
      • Examples
            – Write: Analog output, digital output, telemetry
            – Read: Analog input, digital input, command
            – R/W: Memory, bi-directional discretes, GSE




       2005 MAPLD                     74                        Design Integrity Concepts
  Guidelines for Partitioning (cont.)
• Exploit operational considerations
   – Operational considerations often determine the specific
     configuration of a set of common components
   – Example: memory system
      • Components: memory, write control, read control, sequencing,
        buffering
      • Application: telemetry system
            –   Packetized / unpacketized?
            –   Asynchronous timing / TDM ?
            –   Science data / engineering data?
            –   Pushed / pulled ?
      • The type of telemetry system determines the partitioning


       2005 MAPLD                      75             Design Integrity Concepts
   Example – Science Data Storage and Readback
                     System




• How should the logic be partitioned?
    – Is write logic part of science data process or memory process?
    – Is read logic part of telemetry system or memory process?
    – How complicated is the arbitration?
• How it is partitioned depends on specific operational requirements



         2005 MAPLD                    76                     Design Integrity Concepts
    Example - New Horizons Alice Science Data




•   Alice Operation
     –   Slow source accumulation relative to output speed
     –   Push interface initiated by instrument
•   Alice Implementation (others likely in different circumstances)
     –   Block 1: handshake and latch data
     –   Block 2: Ingest data, process, write to memory
     –   Block 3: Read, serialize, send, blank memory
     –   Arbitration simplified to a switched double buffered memory access (no real-time arbitration)




              2005 MAPLD                             77                            Design Integrity Concepts
    Guidelines for Partitioning (cont.)
• Ensure encapsulation is reflected in the form of the
  engineering documents
   – Functions contain many types of operations
   – Example: Telecommand interface
       • De-serialization, decoding, error determination, re-packetization,
         temporary storage
       • Real time functionality [level 0], forwarding to software
       • Storage access / arbitration, status log maintenance, data-bus handshaking
   – Partitioning should ensure that all reasonable aspects of a function are
     in one locality (we are ordering data for understanding)
       • One (or a few) pages in a schematic
       • One module in HDL
       • One object in software
   – Benefits: readability, error determination, testability, maintainability



          2005 MAPLD                    78                     Design Integrity Concepts
 Ordering Example – Bus Structure
• A logical bus structure
   –   Simplifies data flow
   –   Eases expansion / enhancement
   –   Identifies bottlenecks / opportunities for efficiency
   –   Ensures signal compatibility
   –   Reduces timing uncertainty (capacitive loading)
   –   Reduces power
   –   Simplifies control logic / arbitration
   –   Simplifies analysis


        2005 MAPLD              79                 Design Integrity Concepts
 Ordering Ex. – Bus Structure (cont.)
Before:




After:




          2005 MAPLD   80    Design Integrity Concepts
Ordering Ex. – Bus Structure (cont.)
• Directional data flow is clustered
• High-speed access devices (SRAM) not
  buffered
• Exclusive functions clustered (PROM/
  EEPROM)
• Simplification of
  – Timing
  – Loading
  – Control

     2005 MAPLD     81          Design Integrity Concepts
 Why is Partitioning Not a Priority?
• It is – at the S/C, sub-system, and box level (sometimes)
• Why not at the board level?
   – Requires potentially significant planning effort (schedule / cost)
   – The tool syndrome (CAD / CAE)
       • Crush creativity by forcing an early start to design
       • Primarily a way to communicate between software packages rather
         than humans (schematic => PWB)
   – Too much junk being placed in too little space (simplification may
     not always be space efficient)
   – Lack of emphasis from senior engineers
   – Boards aren’t where the action is (FPGAs) – less effort placed on
     them



        2005 MAPLD                  82                    Design Integrity Concepts
Why is Partitioning Not a Priority? (cont.)
  • At the FPGA level
    – Lack of solid design methodology
       • Methodology must be tailored to a tool
       • VHDL / Verilog are functional descriptions
       • VHDL / Verilog don’t inherently enable data flow
         visualization or block oriented deconstruction
       • The synthesizer can understand non-partitioned design
    – Perception of expansive resources (no / few constraints)
    – The tool syndrome (“I must start coding”)
    – Lack of effective design quality gates prior to start of
      detailed design


        2005 MAPLD                83                  Design Integrity Concepts
                             Summary
• A design is only as solid as its weakest part
• Proper planning and partitioning of a design:
    –   Ensures individual functions are logical and complete
    –   Ensures interconnects are ordered and efficient
    –   Provides for improved reliability / verifiability
    –   Allows easier modification and enhancement
    –   Enhances detailed understanding of how things work
• Partitioning requires ordering the design by considering
    –   The capabilities of the team
    –   The functionality of the modular pieces
    –   How the design will operate
    –   The individual components of a particular function
• Partitioning a design ensures that all parts are solid – resulting in a
  solid whole



          2005 MAPLD                    84                      Design Integrity Concepts
You Mean We’re Still Working
          On It?
               Sustaining a Design




  2005 MAPLD            85           Design Integrity Concepts
 Agenda – Design Sustainability
• Definitions
  – Sustainable – Capable of being supported (AH)
  – Sustainability – The characteristic of an item
    that allows it to be supported
• Why is this important?
• Suggestions for sustainability
• Summary


     2005 MAPLD         86             Design Integrity Concepts
             Why Sustainability?
• Missions may be multiple decades long
  – Flight systems may develop anomalous behavior
  – Ground equivalents may need repair
  – An understanding of the design is necessary to ensure
    mission success
  – Original designers will not be available for debugging
     • Other critical assignments
     • Working in telecon
     • Cruising the Pacific
  – Sustainable designs allow analysis and correction
    without the access to the original designers


      2005 MAPLD                87            Design Integrity Concepts
       Why Sustainability? (cont.)
• Many designs are derivative
   – Reuse of unmodified circuits essential for similar performance in
     modified designs
   – Acceptable modification depends on creative incorporation of what IS
• Derivation may take many years
   – Example – Alice UVS
       •   Design 1 – Rosetta (1997-2001)
       •   Design 2 – New Horizons (2002-2005)
       •   Design 3 – LRO (2005-2008)
       •   Design 4 – Juno (2006 - ???)
   – Staffing will not be constant
   – Human memory will not be precise
• Sustainability ensures an ability to efficiently build on past
  successes


           2005 MAPLD                 88               Design Integrity Concepts
      Why Sustainability (cont.)
• You may not be the person who has to make it work
   – Staffing is dynamic
       • You may quit
       • You may get re-assigned
       • Somebody with more clout may be needed to satisfy the customer
   – Teams produce a product and share debugging
       • Test technician
       • S/W designer
       • I&T team
   – Self-interest and common courtesy
       • You don’t want 18 questions per day
       • Ethics – Do unto others
   – Example (ICB)*



       2005 MAPLD                   89                    Design Integrity Concepts
   Suggestions for Sustainability
• Remember the dual nature of design input
   – The CAD perspective
       • Schematic => PCB layout package => circuit board
       • HDL => Synthesizer => Fuse file => Programmed FPGA
   – The human perspective
       • Schematic
            – Interrelationships (time, space, connection)
            – Debugging tool
            – Functionality description
       • HDL
            – Describes functions and interaction
            – Renders constraints understandable
   – Ensure Readability!



       2005 MAPLD                        90                  Design Integrity Concepts
Suggestions for Sustainability (cont.)
• Record the design process
   – Keep a notebook (type not vital)
   – Describe everything of importance
      • Why?
           – Is this bus used for this function?
           – Is this function implemented like this?
           – Etc.
      • How?
           –   Do these things talk to one another?
           –   Does this sequential logic work (state diagrams)?
           –   Is the address map decoded?
           –   Are errors handled?
           –   Etc.



      2005 MAPLD                      91                     Design Integrity Concepts
Suggestions for Sustainability (cont.)
• Record the Design Process (cont.)
   – Describe (cont.)
      • What?
            –   Signals are needed to perform this function?
            –   Do the waveforms look like?
            –   Timing do I expect to observe?
            –   Changes have I made?
            –   Etc.
      • When? – Record chronology
   – Provide a way to reproduce what was done
   – Make part of permanent project record
   – Example – Radarsat 1 Notebook



       2005 MAPLD                      92                      Design Integrity Concepts
Suggestions for Sustainability (cont.)
• Schematics
  – Provide an overview of the design
       • Schematic table of contents
       • Block diagram (hierarchical design if available)
  – Provide consistent naming scheme
       • Descriptive of signal direction / function / polarity
       • Consistent across logic gates and within various blocks
  –   Cluster sub-circuits on contiguous pages
  –   Make connections between components explicit
  –   Add comments where necessary for clarification
  –   Remove unused circuitry (for FPGA schematics)



       2005 MAPLD                  93                   Design Integrity Concepts
Suggestions for Sustainability (cont.)
• Don’t:




      2005 MAPLD   94        Design Integrity Concepts
Suggestions for Sustainability (cont.)
• Do:




        2005 MAPLD   95      Design Integrity Concepts
Suggestions for Sustainability (cont.)
• Help!




          2005 MAPLD   96    Design Integrity Concepts
Suggestions for Sustainability (cont.)
• HDL (Must be done from beginning!)
   – Provide overall orientation to design
   – Provide top-level comments on
       • Level of use (top, intermediate, etc.)
       • Overall purpose of function / block / module
       • Signal function and origination (external and internal)
   – Provide operational comments on
       •   State machine purpose and configuration (how? why?)
       •   Transition logic (theory and reasoning)
       •   Function of particular sequences
       •   State to control signal translation
   – Clarify obscure references
       • Remove superseded code (don’t comment out) and explain uncommon
         structures
   – Improve readability
       • Create logical file names
       • Minimize file, logic block, function sizes
       • Include related functions together (error generation, data interface, basic
         function, etc.



        2005 MAPLD                         97                        Design Integrity Concepts
Suggestions for Sustainability (cont.)



                 Comments?




    2005 MAPLD               98   Design Integrity Concepts
       Suggestions for Sustainability (cont.)
Header from same design!
(After the fact documentation)




                  2005 MAPLD     99   Design Integrity Concepts
Suggestions for Sustainability (cont.)
• Post process documentation
  – Theory of Operation / Users Manual
     • Generate One
     • Include
           –   Design concept / features
           –   Operational Constraints
           –   Appropriate Uses
           –   Complete Engineering Documentation
     • Update, release and correct as necessary
  – Create design archive
     • Self-consistent and complete
     • Place under revision control
     • Control changes


      2005 MAPLD                   100              Design Integrity Concepts
                    Summary
• Useful designs will be corrected, modified and
  evaluated
   – FOR A LONG TIME!
   – By people besides you
• Sustainability measures must be implemented to
  make this happen efficiently
• Sustainability requires
   – Adequate conceptual documentation and records
   – Clear and readable implementation records
   – Finalized and controlled configuration records
• Ensuring sustainability will preserve your legacy


      2005 MAPLD             101             Design Integrity Concepts
What’s the Exit Strategy?
             Verifying a Design




2005 MAPLD           102          Design Integrity Concepts
    Agenda – Design Verification
•   Concepts and implications
•   The specification connection
•   Verification before test
•   Test thoroughness
•   Tiered verification
•   Concluding the process
•   Summary

       2005 MAPLD       103        Design Integrity Concepts
           Verification Concepts
• Verification – Confirmation, through the provision of
  objective evidence, that the specified requirements have
  been fulfilled (Q9000-2000)
• Confirmation – the act of ratifying or corroborating (AH)




       2005 MAPLD            104               Design Integrity Concepts
        Verification Implications
• The verification process is not intended to be a “craps
  shoot”
   – Verification is primarily intended to highlight correctness
   – Verification is NOT primarily intended to reveal incorrectness
• The mindset is important!
   – Doubt that a design will work expresses a lack of confidence in the
     designer and the design process
   – Waiting until “verification” to guarantee correctness is an excuse
     for being sloppy
• We should expect our designs to work! Verification simply
  puts the seal of confirmation on the expectation



       2005 MAPLD                  105                  Design Integrity Concepts
 Verification Implications (cont.)
• Verification addresses two processes
   – Design
      • Primarily a one-time, iterative process in which the developed
        concept is proven sound
      • Fundamental correctness can be proven analytically
           – Inspection confirms logical correctness in all circumstances
               » Peer reviews are a manual form of inspection
               » Functional simulation is an automated form of inspection
               » Inspections must be tailored to design
           – Analysis verifies correctness in the presence of variability
               » Reliability (WC, Derating, FMEA, etc.)
               » Timing over environments and age




      2005 MAPLD                    106                   Design Integrity Concepts
 Verification Implications (cont.)
• Verification addresses two processes (cont.)
   – Instantiation
      • Particular instance must be sound
      • Particular correctness must be demonstrated experimentally
            – Inspection confirms that instructions are followed
                » Correct components installed
                » Correct configuration selected
                » Correct processes performed
            – Test and demonstration confirms that operation matches
              expectations
                » Functionally and parametrically
                » Predicted by nominal analytical predictions




       2005 MAPLD                    107                   Design Integrity Concepts
 Verification Implications (cont.)
• Verification “after the fact” is too late
   – Analysis and test are NOT equivalent
   – Design analysis and inspection must proceed in lockstep with
     design processes
   – Implementation tests and inspections should simply confirm what
     is known
• Design efforts fail because they occur in a vacuum
   – Creativity takes primacy over correctness
   – Functionality comes first with operability being “shoehorned” after
     the fact
       • The “monster” simulation syndrome
• Conclusion: Verify as you proceed


        2005 MAPLD                 108                 Design Integrity Concepts
 Verification Implications (cont.)
• Correctness is a matter of personal integrity
  for the designer
  – Verification is not simply externally imposed
    task
  – Verification is something that the designer must
    want to do (a part of his/her ethos)
  – Verification is confirmation that the designer is
    “worth his/her salt”


      2005 MAPLD         109             Design Integrity Concepts
   The Specification Connection
• “Requirements” are confirmed during verification
   – The designer does not have a free hand with respect to how a
     design is confirmed
   – Specification provides the constraint set under which verification is
     prosecuted
   – Therefore, verification must be defined prior to start of design
       • Not simply in a general manner
       • Specifically
   – Since specification is a customer and vendor document
       • Both customer and vendor must be involved in the verification
         process
       • “Trust me” is not a reasonable phrase when it comes to verification




        2005 MAPLD                   110                     Design Integrity Concepts
The Specification Connection (cont.)
• Specification contains a complete [ideally] set of requirements for the
  design
    – Some requirements are very specific
         • “All discrete outputs shall have a source impedance of 2 kOhms.
         • An adequate test is fairly obvious
    – Some requirements are not very specific
         • “The unit shall provide 512 kbytes of SRAM”
         • Note the implied requirement – The SRAM shall function correctly
         • What is an adequate functional test for this requirement? (walking 1’s, 0’s
           addressing, etc?)
    – Some requirements lend themselves to quantitative analysis – “The unit
      shall provide a .1° C accuracy at end of life”
    – Some requirements lend themselves to simple inspection – “The unit shall
      provide 4 analog output channels”
    – When do we decide adequacy of the method of verification and the
      individual verification cases?
        • – Before we start designing


         2005 MAPLD                         111                       Design Integrity Concepts
The Specification Connection (cont.)
• Therefore, verification planning must begin with
  specification creation
   – Each requirement must be assigned a method or
     methods
   – Each requirement must be assigned a test or analysis
     case
      • Must answer question – What is an adequate verification?
      • Must establish answer to question – When are we done?
   – Each requirement must be verified at one or more levels
     [FPGA, board, box, sub-system, system]
   – Customer must concur with decision
• Note that modern verification databases make this
  relatively straightforward

       2005 MAPLD                112                 Design Integrity Concepts
The Specification Connection (cont.)
• Advantages to this type of verification planning
   – More verifiable / verified designs
      • Up-front focus on programmatically difficult verification can
        be instituted
            – Earlier analysis
            – Simplification of overly complex circuit designs
      • Guidance for lower-level verification efforts can be established
            – Sub-module vs. module level
            – Module vs. unit level
      • Integral self-test provisions can be included in design
      • Earlier simulation vector definition (FPGAs / logic)



       2005 MAPLD                    113                    Design Integrity Concepts
The Specification Connection (cont.)
• Advantages (cont.)
   – More robust test sets
      • Early knowledge regarding end-item function communicated
      • Capability of test set matched to need
            – Precision (timing, voltage, current, etc.)
            – Speed
            – Test level (component, unit, system)
      • Test flow design considerations included in test set design
   – Knowing verification requirements early makes the
     entire development process more efficient



       2005 MAPLD                      114                 Design Integrity Concepts
The Specification Connection (cont.)
• Defining requirements and test cases at the same
  time as the specification clarifies and simplifies
  the FPGA verification process
   – Allows early start to simulation vector design
   – Creates early knowledge of additional functional
     models (SRAM, peripherals, etc.) needed
   – Clarifies decision regarding what can be functionally
     simulated and what must be simulated with back-
     annotation
   – Defines levels at which simulations must be run



       2005 MAPLD             115              Design Integrity Concepts
         Verification Before Test
• It is a relatively bad thing to discover a design flaw during
  test (better than nothing)
   – Late in the development cycle
   – Correction is more complicated / expensive
   – Personal stress is higher
• Yet … a surprising lack of verification occurs early (before
  test)
• Types of pre-test verification
   – Inspection - conformity evaluation by observation and judgment
     accompanied, as appropriate by measurement or testing (ISO)
       • Personal self-check
       • Peer Review
       • Basic functional verification (Signal audits, functional simulation,
         etc.)




        2005 MAPLD                    116                     Design Integrity Concepts
 Verification Before Test (cont.)
• Types of pre-test verification (cont.)
   – Analysis – Conformity evaluation of the predictions
     produced by realistic models of the system
      • Worst case analysis (voltage, current, frequency, time) using
        models that incorporate real world degradation of function
      • Derating analysis using models that reduce stress
      • Other
• Note that pre-test verification is fundamentally
  conceptual
   – Not tool dependent
   – Not design dependent
   – Can be accomplished many ways


       2005 MAPLD                 117                  Design Integrity Concepts
  Verification Before Test (cont.)
• Two approaches to pre-test verification
   – Verify full design at once (one big peer review, code walkthrough,
     analysis, simulation)
       • Benefits
            –   Complete design reduces need to develop assumptions
            –   When proved correct, does not need to be repeated
            –   When design is solid, less work
            –   Reduces final documentation for verification
       • Drawbacks
            – Allows small design flaws to propagate for a long time
            – Complexity reduces visibility (verification more prone to mistakes and
              oversights)
            – When design is flawed, it increases work
            – Leads to corrections that are global (hard to test effectively and predict
              side effects)
            – May take longer to execute than an aggregate of smaller verification
              activities


       2005 MAPLD                         118                        Design Integrity Concepts
 Verification Before Test (cont.)
• Two approaches to pre-test verification (cont.)
   – Verify incremental designs
      • Benefits
           –   Supports development of “known good” sub-designs
           –   Meshes with a partitioned design approach
           –   Facilitates visibility (fewer mistakes, clearer goals)
           –   Allows small flaws to be fixed quickly (minimizes downstream
               impact)
      • Drawbacks
           – Is more work in the ideal case (no/few flaws)
           – Increases documentation if formality is vital
• Either approach can be made to work, but one or
  the other must be chosen and implemented


      2005 MAPLD                     119                     Design Integrity Concepts
                  Test Thoroughness
• Principles for test thoroughness
    – Every “shall” in the specification that meets the following criteria must be
      tested
         • Items that may be affected by the instantiation process (subject to fabrication
           error)
         • Items for which the only extant verification is model based
         • Items for which the testing does not pose unacceptable risk (radiation,
           overstress, etc.)
    – Every instance of a “shall” must be tested
         • 18 interfaces require 18 specific tests
    – Requirements that have an exclusive character
         • Must be tested for correct operation
         • Should be tested at some level for abnormal operation
         • Example: If you write a 5Ah to something to set it on, you should verify that
           writing other values does not set it on
    – Requirements must be tested over a representative range of conditions




         2005 MAPLD                          120                       Design Integrity Concepts
     Test Thoroughness (cont.)
• A thorough test is complicated and time
  consuming
  – Requires some level of automation to be
    comprehensive
  – Requires some level of compromise to be
    practical
  – Requires team agreement to be acceptable



     2005 MAPLD         121           Design Integrity Concepts
       Test Thoroughness (cont.)
• Ensuring thorough, yet practical, tests
   – Define which tests require manual interaction and which can be
     automated
       • Output impedance or analog fine precision may need to be manual
       • Signal level or analog coarse precision may be automatable
   – Define which tests must be performed over environmental
     conditions and which can be performed at room temperature only
       • Make decision based on character of measurement
       • Do not decide based purely on practicality
   – Define level of complexity for all measurement sets
       • Example: 24 analog inputs (3 eight channel muxes)
       • Test full range of values for 24, or limited range for 7 of 8 mux inputs
         and full range for remaining
   – Define need for repetition of measurement at higher level of
     integration (tiered verification)
       • Example: voltage level / signal quality / clock jitter at board level
       • Repeat at box level?


       2005 MAPLD                      122                      Design Integrity Concepts
        Test Thoroughness (cont.)
• Test thoroughness issues affect everything
   – Customer relationship
   – Test set design (board, box, system)
   – Schedule / cost
• An adequate test program is not trivial
   – Cannot wait until system level
   – Must incorporate lowest level of design
   – Must be a constant background activity during design process
• All programs must balance perfect and good enough
• A good test program
   –   Will ensure that the specification is met
   –   Will verify everything that can be affected by fabrication
   –   Will build on the analysis and inspection effort
   –   Will maximize automation without sacrificing test accuracy


         2005 MAPLD                123                  Design Integrity Concepts
                  Tiered Verification
• Tiered verification is the process of matching the correct type of
  verification to the appropriate level of integration
• Tiered verification incorporates all types of verification (before and
  during testing)
• Tiered verification applied to the test regime attempts to have
  thoroughness with practicality
    – Test a requirement at the lowest conceivable level
    – Determine which tests must be repeated at higher levels by
         • Establishing the purpose of a test at a particular level
         • Determining whether the next level has the potential to change previous
           measured quantities
         • Determining what test set capabilities are
• Tiered verification cannot be retrofitted into the process
    – Must be planned up front
    – Must be implemented throughout the development process



         2005 MAPLD                        124                      Design Integrity Concepts
   Tiered Verification - Example
• Need: A set of 16 high-level pulsed discretes to control
  latching relays used to change the spacecraft power
  configuration
• Basic Requirements
   –   Quantity:               16
   –   Level:                  +30V +/- 2 V
   –   Load:                   2 kOhm, 4 mH
   –   Duration of pulse:      50 ms
   –   Rise / Fall time max:   1 ms
   –   Command capability:     Unique bit pattern, only one at a time,
                               arm first
   – S/W requirements:         Various (beyond scope of example)


         2005 MAPLD             125                  Design Integrity Concepts
Tiered Verification – Example (cont.)
• Tier 1 – Basic Verification (functional simulation)
   – Verify each channel fires for a specific bit pattern /
     address
   – Verify cannot be fired without an arm command
   – Verify that other complementary patterns don’t
     accidentally fire the pulse
   – Analyze drive capability of FPGA and input / output
     requirement / capability of translation chip
   – Evaluate glitch hardness of logic selected



       2005 MAPLD             126               Design Integrity Concepts
Tiered Verification – Example (cont.)
• Tier 2 – Board level
  – Test to confirm signal level between FPGA and
    driver chips is compatible
  – Test rise, fall, duration, level with dummy load
  – Repeat test for all 16 outputs
  – Replace dummy load with relay, verify that the
    relay switches for all 16 outputs
  – Repeat over temperature if necessary


     2005 MAPLD          127            Design Integrity Concepts
Tiered Verification – Example (cont.)
• Tier 3 – Box level
   – Attach GSE (verifies with relays)
   – Functionally pulse all outputs
      • Verify relay switches
      • Verify gross drive duration
• Tier 4 – Box and operational software level
   – Verify functionality commanded through software
   – GSE verifies correct relay switch
• Tier 5 – System
   – Verify gross functionality as needed for system
     operation


       2005 MAPLD                128           Design Integrity Concepts
       Tiered Verification - Notes
• Most complex and detailed functionality is verified at
  lower levels
    – Performance
    – Error cases
    – Predictive of future operation
•   Higher levels focus on functionality / operation
•   Lower level verification requires more manual touch
•   Higher level verification is more automated
•   STE and test procedure developed as appropriate
    – Purpose is maximum test completeness
    – As well as “bang for the buck”



        2005 MAPLD                     129      Design Integrity Concepts
         Concluding the Process
• (“… through the provision of objective evidence…”)
• The importance of traceability
   – Something is only verified (formally) when it is documented
   – The tie between verification item and specification must be made
     (verification matrix or database)
• The importance of planning
   – Establish the links between requirement, method, level, test case
     early
   – Use the established structure to develop verification items
• The importance of buy in
   – All responsible must complete verification and report results
   – Systems engineers must track and close
   – People must keep up with the process
• Only personal commitment from all involved will ensure
  that the process is successful

       2005 MAPLD                  130                  Design Integrity Concepts
                             Summary
• Verification is to prove success, not avoid failure
• Verification planning is an integral part of the project lifecycle
    – Must be initiated with specification (including customer buy-in)
    – Must be included in design effort
    – Must be used to develop documentation and appropriate test hardware
• Verification is more than final test
    – Begins at lowest possible level
    – Incorporates analysis and inspection throughout design process
    – Develops proof of compliance at all levels of operation
• A thorough test is complicated
    – Requires extensive work
    – Must trade perfect for practical (a tiered test approach can help)
• Verification processes must be tracked and documented
    – The whole team must buy in



         2005 MAPLD                     131                     Design Integrity Concepts
What Do You Mean It Doesn’t
  Do What We Thought?
               Validating a Design




  2005 MAPLD            132          Design Integrity Concepts
     Agenda – Design Validation
•   Concepts and implications
•   Specification mitigations
•   Design mitigations
•   Test set mitigation
•   Summary




       2005 MAPLD      133      Design Integrity Concepts
                      Concepts
• Validation – Confirmation, through the provision of
  objective evidence, that the requirements for a specified
  intended use or application have been fulfilled




       2005 MAPLD              134              Design Integrity Concepts
                         Implications
• The issues relating to validation encompass those of verification
• Additional concerns with validation
    – Caused by the need to match the application to the product
    – Desired application has been translated to specification through the
      requirements process
    – Requirements process is by nature imperfect
    – Sometimes the specification does not satisfy the needs of the application
• Result – a verified product might be invalid
    – May require significant rework to the product
    – May require accepting reduced functionality (waiver)
• A goal of the development process is to minimize validation failures
    – Begins in review of the requirements process (hopefully primary point)
    – Mitigate by design activities
    – Reduce by robust test set design




         2005 MAPLD                    135                     Design Integrity Concepts
      The Implication Illustrated
• Alice electronics (detector component and C&DH
  component)
• Joint requirement: process > 10 k sustained events per
  second
• Individual requirements defined for detector and C&DH
  processing
   – Both met individual requirements for processing
   – When combined only 6-7 k sustained events per second
• Verification of individual units led to invalid system
• What went wrong?
   – The overall requirements were not broken down correctly
   – The C&DH and DE test sets were not high fidelity
       • Verified functionality, not performance
• We got lucky that a waiver was acceptable

        2005 MAPLD                   136             Design Integrity Concepts
       Specification Mitigation
• Only list requirements, not desirements
• State unambiguous performance requirements
• Build in adequate margin
   – Not open-ended enhancement, but
   – Enough to accommodate performance shortfalls
• Ruthlessly remove TBDs
• Insist on definite test methods for mitigation
• Remember – Unless application needs can be
  unambiguously specified, they won’t be met!


      2005 MAPLD           137             Design Integrity Concepts
                     Design Mitigation
• Implement specification exactly
• Isolate various sub-sections
   – Minimizes “corner cases” and negative interactions
   – Allows correction with minimal impact when things don’t work
     right
• Verify complex functions early, thoroughly, and
  completely
   – Allows early look at potential problems
   – Analysis / simulation / what ifs should be as realistic as possible
• Insist on end-user review of implementation
   – Allows user community to comment
   – Minimizes misunderstandings upon delivery
• Develop test plans that have high fidelity to the end
  application


        2005 MAPLD                  138                   Design Integrity Concepts
                 Test Set Mitigation
• Ensure interfaces are maximally flight-like
   – Precludes misunderstandings of characteristics
   – Provides early indication of problems
   – Don’t emulate only one characteristic of interface (R vs. R+L)
• Make test set reasonably sophisticated
   – Sufficient complexity to reproduce operational timing
   – Adequate functionality for stress testing
       • Run all interfaces at maximum speed with margin
• Don’t let the same group build the tested unit (design) and
  the unit tester (test bench)
   – Identical assumptions might go into both ends of an interface
   – Faithful reproduction is dependent on familiarity (if possible, test
     bench should be provided by end user)



        2005 MAPLD                  139                    Design Integrity Concepts
      Test Set Mitigation (cont.)
• Make the control interface as application like as
  possible
   – Forces correct command structures / types
   – Allows all test scripts to be reproduced at higher levels
• If at all possible, incorporate early interface tests
  of real engineering hardware
• Keep the test (or simulation) environment constant
  unless the flight system changes
   – Don’t change test equipment hardware configurations
   – Apples to apples comparisons during tests vital
   – Ensure that flight changes are reflected in test set
     design


       2005 MAPLD              140               Design Integrity Concepts
       Test Set Mitigation (cont.)
• Use the same controls for test set development as for flight
  unit development
   – Configuration management
   – Software development
   – Peer reviews
• Build in diagnostics so that anomalies can be traced to test
  equipment or unit under test
• Ensure that test results mean something
   –   Pass / fail criteria clear
   –   Allowable flight parameter variations included
   –   Reasonable displays (with significant information clearly shown)
   –   Ensure that test set accommodates calibration



         2005 MAPLD                 141                  Design Integrity Concepts
                      Summary
• Successful verification does not always guarantee
  successful validation
• Techniques can be incorporated that improve the
  likelihood that validation will succeed
   – Careful specification development
   – Thorough and cautious design techniques
   – Extensive test set fidelity to flight requirements
• Effective techniques for validation are extra effort
   – More time consuming
   – More expensive
   – But, definitely worth it.


       2005 MAPLD                142             Design Integrity Concepts

								
To top