Docstoc

2009-9_Hutto-DOE

Document Sample
2009-9_Hutto-DOE Powered By Docstoc
					              Process                                                     Plan                                                       Produce   Ponder
                                    Cause-
Measurements Manpower   Materials    Effect
                                    (CNX)                                         In Front            In Back
                                    Diagram                               Face East Face West Face East Face West
                                      Response   Eyes Open Left Hand         0.43         0.58   0.52         0.40
               Causes                    to                  Right Hand      0.62         0.29   0.28         0.36
                                       Effect    Eyes Closed Left Hand       0.62         0.57   0.47         0.40
                                                             Right Hand      0.42         0.26   0.42         0.47
             Causes
   Milieu
              Methods   Machines
(Environment)




         Raising the Bar: Equipping Systems
             Engineers to Excel with DOE
                                                                                                                     presented to:
                                                                                                INCOSE Luncheon
                                                                                                 September 2009

                                                                                    Greg Hutto
                                                                          Wing Ops Analyst, 46th Test Wing
                                                                             gregory.hutto@eglin.af.mil

                                                                                                                                                        1
                 Bottom Line Up Front

   Test design is not an art…it is a science
       Talented scientists in T&E Enterprise however…limited
        knowledge in test design…alpha, beta, sigma, delta, p, & n
   Our decisions are too important to be left to
    professional opinion alone…our decisions should be
    based on mathematical fact
   53d Wg, AFOTEC, AFFTC, and 46 TW/AAC experience
       Teaching DOE as a sound test strategy not enough
       Leadership from senior executives (SPO & Test) is key
   Purpose: DOD adopts experimental design as the
    default approach to test, wherever it makes sense
       Exceptions include demos, lack of trained testers, no control

                                                                        2
         Background -- Greg Hutto
   B.S. US Naval Academy, Engineering - Operations Analysis
   M.S. Stanford University, Operations Research
   USAF Officer -- TAWC Green Flag, AFOTEC Lead Analyst
   Consultant -- Booz Allen & Hamilton, Sverdrup Technology
   Mathematics Chief Scientist -- Sverdrup Technology
   Wing OA and DOE Champion – 53rd Wing, now 46 Test Wing
   USAF Reserves – Special Assistant for Test Methods (AFFTC/CT)
    and Master Instructor in DOE for USAF TPS
            Practitioner, Design of Experiments -- 19 Years
            Selected T&E Project Experience – 18+ Years
   Green Flag EW Exercises „79      SCUD Hunting „91
   F-16C IOT&E „83                  AGM-65 IIR „93
   AMRAAM, JTIDS „ 84               MK-82 Ballistics „94
   NEXRAD, CSOC, Enforcer „85       Contact Lens Mfr „95
   Peacekeeper „86                  Penetrator Design „96
   B-1B, SRAM, „87                  30mm Ammo „97
   MILSTAR „88                      60 ESM/ECM projects „98--‟00
   MSOW, CCM ‟89                    B-1B SA OUE, Maverick IR+, F-15E
   Joint CCD T&E „90                 Suite 4E+,100‟s more ‟01-‟06       3
          Systems Engineering Experience

   1988-90 - Modular Standoff Weapon (MSOW) – Multinational NATO
    efforts for next gen weapons. Died deserved death …
   1990-91 – Next Generation AGM-130 yields surprising result – no
    next gen AGM-130; instead JDAM, JSOW, JAASM
   1991-92 – Clear Airfield Mines & Buried Unexploded Ordnance
     requirements – >98% P(clearance); <5m UXO location error
     engineering solutions - rollers, seismic tomography, explosive foams
   1994-1996 - Bare Base Study led to reduction of 50% in weight,
    cost, and improved sustainability through COTS solutions
       $20,000, 60 BTU-hr (5 ton) crash-survivable, miniaturized air
        conditioner replaced by 1 Window A/C - $600
                           Google Specs: Frigidaire FAM18EQ2A Window Mounted Heavy
                           Duty Room Air Conditioner, 18,000/17,800 BTU Cool, 16,000
                           BTU (Heat), 1,110 Approximately Cool Area Sq. Ft, 9.7 EER, 11"
                           Max. Wall Thickness (FAM18EQ2 FAM18EQ FAM18E FAM18
                           FAM-18EQ2A) $640.60
                                                                                            4
                 Overview

   Four Challenges – The 80% JPADS
   3 DOE Fables for Systems Engineers
   Policy & Deployment
   Summary




                                         5
                                                         Robust
                                                         Product
                                                         Design               Process Flow
                      AoA -                                                     Analysis
                    Feasibility
                     Studies
                                                               Product &
                                                            Process Design
 Requirements -       Concept                                                          Designed
 Quality Function                                                                    Experiments –
Deployment (QFD)           Historical /
                                                                                       Improve
                          Empirical Data
                                                                                     Performance,
                            Analysis
                                        Systems                                         Reduce
                                                                                       Variance
                                                                                               SPC – Defect
Failure Modes &
Effects Analysis
                                       Engineering                                            SourcesSupply
                                                                                                   Chain
     (FMEA)
                                       Challenges                                              Management
                                                                                                     Lean
     Decommission/                                                               Production
                                                                                                 Manufacturing
        End-use
                                                                                         Statistical
                                                                                      Process Control
                                                                                       (Acceptance
                                                                                           Test)
                                                  Simulation

                                                Operations
                                  Reliability                        Serviceability

                                       Maintainability         Availability
                  Systems Engineering
                  Simulations of Reality
                                             Simulation of Reality
          Acq Phase                 M&S         Hardware           System/Flight Test
Reqt'ts Dev
               AoA
Concepts
                                                                         Prototype
               Risk Reduction Warfare
                                      Physics
EMD                                           HWIL/SIL Captive Subsystem
                                                                          Prod Rep
               Prod & Mfr
Sustain                                                                  Production

   At each stage of development, we conduct experiments
       Ultimately – how will this device function in service (combat)?
       Simulations of combat differ in fidelity and cost
       Differing goals (screen, optimize, characterize, reduce variance,
        robust design, trouble-shoot)
       Same problems – distinguish truth from fiction: What matters? What
        doesn‟t?
                                                                                        7
        Industry Statistical Methods:
              General Electric

 Fortune 500 ranked #5 in 2005 - Revenues
 Global Most Admired Company - Fortune 2005
 America‟s Most Admired #2, World‟s Most Respected
  #1 (7 years running), Forbes 2000 list #2
 2004 Revenues $152 B
 Products and Services
    Aircraft engines, appliances, financial services,
     aircraft leasing, equity, credit services, global
     exchange services, NBC, industrial systems,
     lighting, medical systems, mortgage insurance,
     plastics
                    GE‟s (Re)volution In
                 Improved Product/Process

 Began in late 1980‟s facing foreign competition
 1998, Six Sigma Quality becomes one of three
  company-wide initiatives
 „98 Invest $0.5B in training; Reap $1.5B in benefits!
 “Six Sigma is embedding quality thinking - process
  thinking - across every level and in every operation of
  our Company around the globe”1
 “Six Sigma is now the way we work – in everything
  we do and in every product we design” 1

1 Jack Welch - General Electric website at ge.com
            What are Statistically
           Designed Experiments?
                                   weather, training, TLE,
                                   launch conditions

                    INPUTS                                         OUTPUTS
                   (Factors)                                      (Responses)
                 Altitude               PROCESS:
                                                             Miss Distance
                 Delivery Mode

                 Impact Velocity       Air-to-Ground
                                                             Impact Angle Delta
                                         Munitions
                 Impact Angle

                 Weapon type                                 Impact Velocity Delta



                                                                      Noise

   Purposeful, systematic changes in the inputs in order to observe
    corresponding changes in the outputs
   Results in a mathematical model that predicts system responses
    for specified factor settings
                    Responses  f Factors  
        Why DOE? Scientific Answers to
       Four Fundamental Test Challenges
Four Challenges
  1.   How many? Depth of Test – effect of test size on
       uncertainty
  2.   Which Points? Breadth of Testing – searching the
       vast employment battlespace
  3.   How Execute? Order of Testing – insurance against
       “unknown-unknowns”
  4.   What Conclusions? Test Analysis – drawing
       objective, supported conclusions
                                Noise
DOE effectively addresses
  all these challenges!         Inputs
                                         PROCESS
                                                   Outputs
                                 (X’s)              (Y’s)

                                                   Noise
              Today‟s Example –
           Precision Air Drop System
                                    The dilemma for airdropping supplies has always been a stark one.
                                    High-altitude airdrops often go badly astray and become useless or
                                    even counter-productive. Low-level paradrops face significant dangers
                                    from enemy fire, and reduce delivery range. Can this dilemma be
                                    broken?
                                    A new advanced concept technology demonstration shows promise,
                                    and is being pursued by U.S. Joint Forces Command (USJFCOM),
                                    the U.S. Army Soldier Systems Center at Natick, the U.S. Air Force
                                    Air Mobility Command (USAF AMC), the U.S. Army Project Manager
                                    Force Sustainment and Support, and industry. The idea? Use the
                                    same GPS-guidance that enables precision strikes from JDAM
                                    bombs, coupled with software that acts as a flight control system for
                                    parachutes. JPADS (the Joint Precision Air-Drop System) has been
                                    combat-tested successfully in Iraq and Afghanistan, and appears to
                                    be moving beyond the test stage in the USA… and elsewhere.

                                    Capability:
                                            Assured SOF re-supply of material
                                     Requirements:
                                            Probability of Arrival
   Just when you think of a good           Unit Cost $XXXX
    class example – they are                Damage to payload
    already building it!                    Payload
                                            Accuracy
   46 TS – 46 TW Testing JPADS             Time on target
                                            Reliability …
                                                                                                     12
            A beer and a blemish …




   1906 – W.T. Gossett, a         1998 – Mike Kelly, an engineer at
    Guinness chemist                contact lens company
   Draw a yeast culture           Draw sample from 15K lot
    sample
                                   How many defective lenses?
   Yeast in this culture?
                                   Guess too little – mad
   Guess too little –              customers; too much -- destroy
    incomplete fermentation;        good product
    too much -- bitter beer
                                   He wanted to get it right
   He wanted to get it right


                                                                   13
          The central test challenge …
   In all our testing – we reach into
    the bowl (reality) and draw a
    sample of JPADS performance
   Consider an “80% JPADS”
      Suppose a required 80%
       P(Arrival)
                                         The central
      Is the Concept version             challenge of

       acceptable?
    We don‟t know in advance which
                                          test – what’s
    bowl God hands us …                   in the bowl?
      The one where the system
       works or,
      The one where the system
       doesn’t



                                                          14
     Start -- Blank Sheet of Paper

 Let‟s draw a sample of _n_ drops
 How many is enough to get it right?
     3 – because that‟s how much $/time we have
     8 – because I‟m an 8-guy
     10 – because I‟m challenged by fractions
     30 – because something good happens at 30!


   Let‟s start with 10 and see …

                     => Switch to Excel File – JPADS Pancake.xls


                                                              15
            A false positive – declaring JPADS is
                 degraded (when it‟s not) -- a

   Suppose we fail JPADS when it                           In this bowl – JPADS
    has 4 or more misses                                  performance is acceptable
   We‟ll be wrong (on average)
    about 10% of the time
                                                      Maverick OK -- 80% We Should
                                                       JPADS
   We can tighten the criteria (fail                             Field
    on 7) by failing to field more
    good systems                                    350
                                                    300       Wrong
   We can loosen the criteria (fail

                                        Frequency
                                                    250
                                                              ~10% of
    on 5) by missing real                           200
                                                              time
                                                    150
    degradations                                    100
   Let‟s see how often we miss                      50
                                                      0
    such degradations …                                   3     4   5   6          7   8   9   10
                                                                            Hits




                                                                                                    16
               A false negative – we field JPADS
                    (when it‟s degraded) -- b
                                                                In this bowl – JPADS P(A)
   Use the failure criteria from the                            decreased 10% -- it is
    previous slide                                                       degraded
   If we field JPADS with 6 or
    fewer hits, we fail to detect the                     Maverick Poor -- 70% Pk -- We
                                                           JPADS
    degradation                                                   Should Fail
   If JPADS has degraded, with
                                                    300
    n=10 shots, we‟re wrong about                   250                                      Wrong
                                                                                             65% of
                                        Frequency
    65% of the time                                 200
                                                    150                                      time
   We can, again, tighten or                       100
    loosen our criteria, but at the                  50
    cost of increasing the other                      0
                                                            3    4   5    6          7   8    9   10
    error
                                                                              Hits




                                                                                                   17
             We seek to balance our
                chance of errors
                                                        JPADS
                                                       Maverick OK -- 80% We Should Field
 Combining, we can trade                             350
  one error for other (a for b                       300
                                                             Wrong
                                                      250




                                        Frequency
 We can also increase                                200    10% of
                                                             time
                                                      150
  sample size to decrease our                         100
                                                       50
  risks in testing                                      0
                                                              3   4   5   6          7   8   9   10
 These statements not                                                        Hits

                                                            JPADS                P(A)
                                                            Maverick Poor -- 70% Pk -- We
  opinion –mathematical fact                                        Should Fail    Wrong
  and an inescapable                                  300
                                                                                         65% of
                                                                                         time
  challenge in testing                                250



                                          Frequency
                                                      200
                                                      150
 There are two other ways                            100
                                                       50
  out … factorial designs and                           0
                                                              3   4   5   6          7   8   9   10
  real-valued MOPs                                                            Hits



    Enough to Get It Right: Confidence in stating results; Power to
                          find small differences                                                      18
            A Drum Roll, Please …

   For a = b = 10%, d = 10% degradation in PA

                      N=120!

   But if we measure miss distance for same
    confidence and power

                        N=8


                                                 19
            Recap – First Challenge

   Challenge 1: effect of sample size on errors – Depth
    of Test

   So -- it matters how many we do and it matters what
    we measure

   Now for the 2nd challenge – Breadth of testing –
    selecting points to search the employment
    battlespace




                                                           20
       Challenge 2: Breadth -- How Do
      Designed Experiments Solve This?
                  Designed Experiment (n). Purposeful control
                  of the inputs (factors) in such a way as to
                  deduce their relationships (if any) with the
                  output (responses).

JPADS Concept A B C …
                                                             RMS Trajectory Dev
Tgt Sensor (TP, Radar)
                               Test JPADS                    Hits/misses
Payload Type
Platform (C-130, C-117)
                              Payload Arrival                P(payload damage)
                                                             Miss distance (m)
Inputs (Conditions)                                          Outputs (MOPs)

                          Statistician G.E.P Box said …
               “All math models are false …but some are useful.”
                 “All experiments are designed … most, poorly.”

                                                                                  21
                   Battlespace Conditions for
                          JPADS Case
                                                                               Type              Measure of Performance
                                                                                    Objective    Target acquisition range
                                                                                                 Target Standoff (altitude)
                                                                                                 launch range
   Systems Engineering Question: Does JPADS                                                     mean radial arrival distance
    perform at required capability level across the                                              probability of damage
                                                                                                 reliability
    planned battlespace?                                                            Subjective   Interoperability
                                                                                                 human factors
                                                                                                 tech data
    Conditions                                Settings                                # Levels   support equipment
JPADS Variant:     A, B, C, D                                                            4       tactics
Launch Platform:   C-130, C-17, C-5                                                      3
Launch Opening     Ramp, Door                                                            2
Target:            Plains, Mountain                                                      2
Time of Day:       Dawn/Dusk, Mid-Day                                                    3           12 Dimensions
Environment:       Forest, Desert, Snow                                                  3            - Obviously a
Weather:           Clear (+7nm), Haze (3-7nm), Low Ceiling/Visibility (<3000/3nm)        3              large test
Humidity:          Low (<30%), Medium (31-79%), High (>80%)                              3             envelope …
Attack Azimuth:    Sun at back, Sun at beam, Sun on nose                                 3
                                                                                                     how to search
Attack Altitude:   Low (<5000‟), High (>5000‟)                                           2
Attack Airspeed:   Low (Mach .5), Medium (Mach .72), High (Mach .8)                      3
                                                                                                            it?
JPADS Mode:        Autonomous, Laser Guidance                                            2
                   Combinations                                                        139,968

                                                                                                                          22
      Populating the Space - Traditional
Altitude                 Altitude



            OFAT                            Cases



                Mach                           Mach
              Altitude



                                    Change variables together



                                    Mach
           Populating the Space - DOE
Altitude                      Altitude

                                                      Response
                  Factorial
                                                       Surface




                  Mach                              Mach

                Altitude


                                                       single point
                                          Optimal
                                                       replicate




                                         Mach
                   More Variables - DOE
Altitude                                   Altitude




                                                                  3-D
                   2-D               Factorials
                                                          Range
                            Mach                                        Mach




                                     4-D

Altitude   Range

           Mach    Weapon – type A           Weapon – type B
                Even More Variables

                            F
    C              –                  +
        B
            –           +
        A         D


–



E




+
            Efficiencies in Test - Fractions

                               F
    C               –                     +
        B
             –            +
        A           D


–



E




+
       We have a wide menu of design
             choices with DOE
Optimal Designs
                                          Response Surface
                     Full Factorials




                                                Space Filling
 Fractional
 Factorials




                  JMP Software DOE Menu
Problem context guides choice of designs

                                                Space-
                                                 Filling
                                                Designs
                                              Optimal
                                              Designs
Number of Factors




                                 Fractional Response
                                 Factorial   Surface
                                  Designs    Method
                    Classical                Designs
                    Factorials




                                                           29
           Challenge 2: Choose Points to
          Search the Relevant Battlespace
      4 reps 1 var
                                       2 reps 2 vars   JPADS A JPADS B
              JPADS A JPADS B
                 4       4               Ammo                2       2
                                         Food                2       2
   Factorial (crossed) designs
                                     1 reps 3 vars          JPADS A JPADS B
    let us learn more from the
                                                 Ammo          1       1
    same number of assets           Eglin (Low)
                                                 Food          1       1
   We can also use Factorials                   Ammo          1       1
                                   Nellis (High)
    to reduce assets while                       Food          1       1
    maintaining confidence and
    power                         ½ rep 4 vars                   JPADS A JPADS B
                                                          Ammo      1
   Or we can combine the two                Eglin (Low)
                                  Dawn (low               Food              1
                                    light)                Ammo              1
                                            Nellis (High)
                                                          Food      1
                                                          Ammo              1
All four Designs share the same    Midday
                                            Eglin (Low)
                                                          Food      1
 power and confidence              (bright)
                                            Nellis (High)
                                                          Ammo
                                                          Food
                                                                    1
                                                                            1

                                                                            30
       Challenge 3: What Order? Guarding
         against “Unknown-Unknowns”
                   Learning


                                        Task performance (unrandomized)



Good    Task performance (randomized)



                                                     run sequence

                                    time
                                                                          easy runs
                                                                          hard runs
   Randomizing runs protects from unknown background
    changes within an experimental period (due to Fisher)
          Blocks Protect Against Day-to-
                  Day Variation
               Day 1                                     Day 2
                           visibility


                  detection (no blocks)
                                    detection (blocks)
Good
                                                         time



                                  run sequence



                                                                large target
                                                                small target
   Blocking designs protects from unknown
    background changes between experimental periods
    (also due to Fisher)
         Challenge 4: What Conclusions -
              Traditional “Analysis”
   Table of Cases or Scenario settings and findings

              Sortie Alt Mach MDS Range Tgt Aspect OBA Tgt Velocity Target Type Result
                1   10K 0.7 F-16     4       0       0       0        truck      Hit
                1   10K 0.9 F-16     7      180      0       0         bldg      Hit
                2   20K 1.1 F-15     3      180      0      10         tank      Miss


   Graphical Cases summary
                                                                     P(hit)
     Errors (false positive/negative)
                                       1
     Linking cause & effect: Why?
                                     0.8
                                           0.6
                                           0.4
                                           0.2
                                             0
                                                   1 2 3 4 5 6 7 8 9 10 11 12 13
            How Factorial Matrices Work -- a peek
              at the linear algebra under hood
   We set the X settings, observe Y

     Simple 2-level, 2 X-factor design       y  Xb  e
                                y(1)  1  1  1 1  y  e(1) 
       Where y is the grand mean,
                                y  1 1  1  1    e 
     A,B are effects of variables alone
                                a
and AB measures variables working together
                                                     A    a 
                                yb  1  1 1 1  B   eb 
                                                    
                                yab  1 1 1 1  AB eab 
 Solve for b s.t. the error (e) is minimized
             e  y  Xb
           ee  (y  Xb)(y  Xb)
       d (ee) d ((y  Xb)(y  Xb))
                                    0
         db             db
    How Factorial Matrices Work II

d ((y  Xb)(y  Xb))
                      0
         db
d ((y  Xb)(y  Xb))
                       2X(y  Xb)  0
          db
 2X(y  Xb)  0
Xy  XXb  0        To solve for the unknown b’s, X
                    must be invertible
XXb  Xy           Factorial design matrices are
                    generally orthogonal, and
b  ( XX) Xy
          1
                    therefore invertible by design
         With DOE, we fit an empirical
          (or physics-based) model

y  b0   bi xi   bii xi 2   bij xi x j   biii xi 3     i 1,2,..., k
           i         i              ij           i


   Very simple to fit models of the form above (among
    others) with polynomial terms and interactions
   Models fit with ANOVA or multiple regression software
   Models are easily interpretable in terms of the physics
    (magnitude and direction of effects)
   Models very suitable for “predict-confirm” challenges
    in regions of unexpected or very nonlinear behavior
   Run to run noise can be explicitly captured and
    examined for structure
                      Analysis using DOE:
                        CV-22 TF Flight


                         Gross Weight
                       Radar Measurement
  INPUTS                                       OUTPUTS
 (Factors)                                    (Responses)
       Airspeed          PROCESS:
                                           Set Clx Plane Deviation
     Turn Rate

Set Clearance Plane                           Crossing Angle
    Ride Mode
                        TF / TA Radar
      Nacelle           Performance             Pilot Rating

   Terrain Type

                                                  Noise

                                                               DOE I S0-37
         Analysis Using DOE
Gross             Turn                     SCP     Pilot
Weight   SCP      Rate   Ride   Airspeed   Dev    Ratings
 55      300.00   0.00   Medium  160.00    5.6     4.5
 47.5    500.00   0.00   Medium  160.00    0.5     4.8
 47.5    300.00   4.00   Medium  160.00    7.5     4.2
 55      500.00   4.00   Medium  160.00    2.3     4.8
 47.5    300.00   0.00   Hard    160.00    5.2     4.2
 55      500.00   0.00   Hard    160.00    1.2     4.6
 55      300.00   4.00   Hard    160.00    12.0    3.2
 47.5    500.00   4.00   Hard    160.00    6.7     3.4
 47.5    300.00   0.00   Medium  230.00    4.0     4.8
 55      500.00   0.00   Medium  230.00    0.2     5.4
 55      300.00   4.00   Medium  230.00    15.0    2.8
 47.5    500.00   4.00   Medium  230.00    8.3     3.2
 55      300.00   0.00   Hard    230.00    5.8     4.5
 47.5    500.00   0.00   Hard    230.00    1.9     5.0
 47.5    300.00   4.00   Hard    230.00    16.0    2.0
 55      500.00   4.00   Hard    230.00    12.0    2.5
 47.5    400.00   2.00   Medium  195.00    4.0     4.2
 47.5    400.00   2.00   Hard    195.00    7.2     3.7
 55      400.00   2.00   Medium  195.00    6.6     4.4
 55      400.00   2.00   Hard    195.00    7.7     3.8
                     Interpreting Deviation from SCP:
                        speed matters when turning

 Design-Expert® Software                                                         Interaction
 Deviation from SCP                                                               D: Airspeed
                                                            18.0

    Design Points

    D- 160.000
                              D e v ia tio n fr o m S C P
    D+ 230.000                                              13.5


 X1 = B: Turn Rate
 X2 = D: Airspeed

 Actual Factors                                              9.0

 A: SCP = 400.00
 C: Ride = Medium

                                                             4.5


Note that we quantify our degree
of uncertainty – the whiskers
                                                             0.0
represent 95% confidence
intervals around our performance                                   0.00   1.00        2.00      3.00   4.00
estimates.
                                                                                 B: Turn Rate
An objective analysis – not
opinion
       Pilot Ratings Response Surface
                     Plot
Actual Factors:                       5.3
X = Airspeed
                                       4.8
Y = Turn Rate
                                       4.3

Actual Constants:    Pilot Ratings     3.8
SCP = 500.00
                                       3.3
Ride = Medium




                                     160.00

                                     177.50

                    Airspeed          195.00

                                       212.50
                                                                         3.00   4.00
                                                      1.00     2.00
                                        230.00 0.00

                                                             Turn Rate
Radar Performance Results
         SCP Deviation Estimating Equation




Prediction Model - Coded Units (Low=-1, High=+1)


         Deviation from SCP        =
                                      +6.51
                                -2.38      * SCP
                              +3.46     * Turn Rate
                                +1.08      * Ride
                              +1.39      * Airspeed
                             +0.61     * Turn * Ride
                           +1.46     * Turn * Airspeed
       Performance Predictions


     Name           Setting     Low Level    High Level
      SCP            460.00      300.00       500.00
       Turn Rate       2.80       0.00          4.00
       Ride           Hard        Medium        Hard
      Airspeed       180.00      160.00       230.00



                   Prediction   95% PI low   95% PI high
Deviation from SCP    6.96         4.93         8.98
Pilot Ratings         3.62         3.34         3.90
                         Design of Experiments Test Process
                                   is Well-Defined
      Planning: Factors                                            Desired Factors
                                                                   and Responses             Design Points
    Desirable and Nuisance
    Start



                   Yes
   Decision


            No

   Process Step


    Output




   Test Matrix                                     Results and Analysis      Model Build   Discovery, Prediction
A-o-A        Sideslip    St abilize r LE X Ty pe
  2             0             5           -1
 10             0            -5           1
 10             8             5           -1
  2             8             5           -1
  2             8            -5           -1
  2             0            -5           -1
 10             8            -5           1
  2             0             5           1
  2             8             5           1
 10             8             5           1
 10             8            -5           -1
 10             0             5           -1
 10             0            -5           -1
  2             8            -5           1
 10             0             5           1
  2             0            -5           1
         Caveat – we need good science!

 We understand
  operations,
  aero,
  mechanics,
  materials,
  physics, electro-
  magnetics …
 To our good
  science, DOE
  introduces the
  Science of Test
    Bonus: Match faces to names – Ohm, Oppenheimer, Einstein, Maxwell, Pascal, Fisher, Kelvin
           It applies to our tests: DOE in
           50+ operations over 20 years
   IR Sensor Predictions                        Characterizing Seek Eagle Ejector Racks
   Ballistics 6 DOF Initial Conditions          SFW altimeter false alarm trouble-shoot
   Wind Tunnel fuze characteristics             TMD safety lanyard flight envelope
   Camouflaged Target JT&E ($30M)               Penetrator & reactive frag design
   AC-130 40/105mm gunfire CEP evals            F-15C/F-15E Suite 4 + Suite 5 OFPs
   AMRAAM HWIL test facility validation         PLAID Performance Characterization
   60+ ECM development + RWR tests              JDAM, LGB weapons accuracy testing
   GWEF Maverick sensor upgrades                Best Autonomous seeker algorithm
   30mm Ammo over-age LAT testing               SAM Validation versus Flight Test
   Contact lens plastic injection molding       ECM development ground mounts (10‟s)
   30mm gun DU/HEI accuracy (A-10C)             AGM-130 Improved Data Link HF Test
   GWEF ManPad Hit-point prediction             TPS A-G WiFi characterization
   AIM-9X Simulation Validation                 MC/EC-130 flare decoy characterization
   Link 16 and VHF/UHF/HF Comm tests            SAM simulation validation vs. live-fly
   TF radar flight control system gain opt      Targeting Pod TLE estimates
   New FCS software to cut C-17 PIO             Chem CCA process characterization
   AIM-9X+JHMCS Tactics Development             Medical Oxy Concentration T&E
   MAU 169/209 LGB fly-off and eval             Multi-MDS Link 16 and Rover video test
                                                                                       45
                                    Three DOE Stories for T&E
              Process                                                     Plan                                       Produce   Ponder
                                    Cause-
Measurements Manpower   Materials    Effect
                                    (CNX)                                         In Front            In Back
                                    Diagram                               Face East Face West Face East Face West
                                      Response   Eyes Open Left Hand         0.43         0.58   0.52         0.40
               Causes                    to                  Right Hand      0.62         0.29   0.28         0.36
                                       Effect    Eyes Closed Left Hand       0.62         0.57   0.47         0.40
                                                             Right Hand      0.42         0.26   0.42         0.47
             Causes
   Milieu
              Methods   Machines
(Environment)




      Requirements: SDB II Build-up SDD Shot Design
      Acquisition: F-15E Suite 4E+ OFP Qualification
      Test: Combining Digital-SIL-Live Simulations


    We’ve selected these from 1000’s to show T&E Transformation

                                                                                                                                        46
            Testing to Diverse Requirements:
                   SDB II Shot Design
                                                                 Test Objective:
                                                                  SPO requests help – 46 shots right N?
                                                                  Power analysis – what can we learn?
                                                                  Consider Integrated Test with AFOTEC
                                                                  What are the variables? We do not
                                                                    know yet …
                                                                  How can we plan?
                                                                  What “management reserve”




DOE Approach:                            Perf shift            Results:                          Performance Shift from KPP in Units of 1 Standard Deviation

                                                                 Binary Pacq
                                                                                  Shots    0.25          0.5     0.75         1          1.25       1.5         2
  Partition performance                                                             4    0.176        0.387     0.65       0.855       0.958      0.992      0.999
                              Power 




                                                                                     6    0.219         0.52    0.815       0.959       0.995      0.999      0.999
   questions: Pacq/Rel +                 46 shots too few to       N=200+            8
                                                                                    10
                                                                                          0.259
                                                                                          0.298
                                                                                                       0.628
                                                                                                       0.714
                                                                                                                0.905
                                                                                                                0.952
                                                                                                                            0.989
                                                                                                                            0.997
                                                                                                                                        0.999
                                                                                                                                        0.999
                                                                                                                                                   0.999
                                                                                                                                                   0.999
                                                                                                                                                              0.999
                                                                                                                                                                1
   laser + coords + “normal                 check binary                            12    0.335        0.783    0.977       0.999       0.999      0.999        1
                                         values +/- 15-20%       Demo laser        14    0.371        0.837    0.989       0.999       0.999      0.999        1
   mode”                                                                            16    0.406        0.878    0.994       0.999       0.999        1          1

                                                         Goal      + coords         18
                                                                                    20
                                                                                           0.44
                                                                                          0.472
                                                                                                       0.909
                                                                                                       0.933
                                                                                                                0.997
                                                                                                                0.998
                                                                                                                            0.999
                                                                                                                            0.999
                                                                                                                                        0.999
                                                                                                                                        0.999
                                                                                                                                                     1
                                                                                                                                                     1
                                                                                                                                                                1
                                                                                                                                                                1
  Consider total test pgm:                                                         24    0.532        0.964    0.999       0.999         1          1          1

   HWIL+Captive+Live
                                                                   N=4 ea           28    0.587        0.981    0.999       0.999         1          1          1
                                                                                    32    0.636         0.99    0.999       0.999         1          1          1

                                                                 Prove
                                                                                    36    0.681        0.995    0.999         1           1          1          1

  Build 3x custom, “right-                                                         40
                                                                                    50
                                                                                          0.721
                                                                                          0.802
                                                                                                       0.997
                                                                                                       0.999
                                                                                                                0.999
                                                                                                                0.999
                                                                                                                              1
                                                                                                                              1
                                                                                                                                          1
                                                                                                                                          1
                                                                                                                                                     1
                                                                                                                                                     1
                                                                                                                                                                1
                                                                                                                                                                1
                                                                                                                             32-shot factorial
   size” designs to meet                                           normal           60
                                                                                    70
                                                                                          0.862
                                                                                          0.904
                                                                                                       0.999
                                                                                                       0.999
                                                                                                                    1
                                                                                                                    1
                                                                                                                              1
                                                                                                                              1
                                                                                                                                          1
                                                                                                                                screens 4-8
                                                                                                                                          1
                                                                                                                                                     1
                                                                                                                                                     1
                                                                                                                                                                1
                                                                                                                                                                1

   objectives/risks                                                mode N=32        80
                                                                                   100
                                                                                          0.934
                                                                                           0.97
                                                                                                       0.999
                                                                                                       0.999
                                                                                                                    1
                                                                                                                    1
                                                                                                                              1
                                                                                                                            variables to 0.5 std
                                                                                                                              1
                                                                                                                                          1
                                                                                                                                          1
                                                                                                                                                     1
                                                                                                                                                     1
                                                                                                                                                                1
                                                                                                                                                                1
                                                                                                                            dev shift from KPP
                                                                                                         47
                                                                 • Integrate 20 AFOTEC shots for “Mgt Reserve”
                    Acquisition: F-15E Strike Eagle
                      Suite 4E+ (circa 2001-02)
                                                                                   Test Objectives:
                                                                                         Qualify new OFP Suite for Strikes
                                                                                          with new radar modes, smart
                                                                                          weapons, link 16, etc.
                                                                                         Test must address dumb weapons,
                                                                                          smart weapons, comm, sensors,
                                                                                          nav, air-to-air, CAS, Interdiction,
                                                                                          Strike, ferry, refueling…
                                                                                         Suite 3 test required 600+ sorties
DOE Approach:                                                                      Results:
 Build multiple designs spanning:                                                  Vast majority of capabilities passed
      EW and survivability                                                         Wrung out sensors and weapons
      BVR and WVR air to air                                                         deliveries
       engagements                                                                  Dramatic reductions in usual trials
      Smart weapons captive and live                                                 while spanning many more test
      Dumb weapons regression                                                        points
      Sensor performance (SAR and                                                  Moderate success with teaming with
       TP)                                                                            Boeing on design points (maturing)
       Source: F-15E Secure SATCOM Test, Ms. Cynthia Zessin, Gregory Hutto, 2007 F-15 OFP CTF 53d Wing / 46 Test Wing, Eglin AFB, Florida
                                                                                                                                            48
         Strike Weapons Delivery a Scenario
              Design Improved with DOE


                    Cases Design   Same N -    -50% N          Comparison
                                   Factorial   Factorial
     Cases                6           32          16       2.5 to 5x more (530%)
        n                10            2           2
       N                 60           64          32       -50% to +6.7% more
Sensitivity (d/s)         1            1           1
       a                5%            5%          5%               Same
        b               20%          2.5%        20%              th
                                                             1/10 error rate
  Power (1-b)           80%        97.50%        80%       detect shifts equal or
                                                                     ++




                                                                             49
              Case: Integration of Sim-
            HWIL-Captive-Live Fire Events
                                                            Test Objective:
                                     $ - Credibility
        1000’s        Predict                                Most test programs face this – AIM-9X,
                                              15-20 factors    AMRAAM, JSF, SDB II, etc…
    Digital Mod/Sim
                                                             Multiple simulations of reality with

                          100’s Predict
                                                               increasing credibility but increasing cost
                         HWIL or              8-12 factors  Multiple test conditions to screen for
                         captive                               most vital to performance
                 Validate
                                                             How to strap together these simulations
                                     10’s   +
                                     Live      3-5 factors     with prediction and validation?
                          Validate   Shot
                     DOE Approach:                                                Results:
•     In digital sims screen 15-20 variables with          •   Approach successfully used in 53d Wing EW
      fractional factorials and predict performance            Group
•     In HWIL, confirm digital prediction (validate        •   SIL labs at Eglin/PRIMES > HWIL on MSTE
      model) and further screen 8-12 factors; predict          Ground Mounts > live fly (MSTE/NTTR) for
•     In live fly, confirm prediction (validate) and           jammers and receivers
      test 3-5 most vital variables                        •   Trimmed live fly sorties from 40-60 to 10-20
•     Prediction Discrepancies offer chance to                 (typical) today
      improve sims                                         •   AIM-9X, AMRAAM, ATIRCM: 90% sim
                                                               reduction
             A Strategy to be the Best …
            Using Design of Experiments
   Inform Leadership of Statistical Thinking
    for Test
   Adopt most powerful test strategy (DOE)
   Train & mentor total team
   Combo of AFIT, Center, & University
   Revise AF Acq policy, procedures
   Share these test improvements
      Adopting DOE            Targets
     53d Wing & AFOTEC         HQ AFMC
     18 FTS (AFSOC)
     RAF AWC                 ASC   & ESC
     DoD OTA: DOT&E,           Service DT&E
    AFOTEC, ATEC,
    OPTEVFOR & MCOTEA
     AFFTC & TPS
     46 TW & AAC
     AEDC
                                                51
     53d Wing Policy Model: Test Deeply
     & Broadly with Power & Confidence

   From 53d Wing Test Manager‟s Handbook*:
    “While this [list of test strategies] is not an all-
     inclusive list, these are well suited to operational
     testing. The test design policy in the 53d Wing
     supplement to AFI 99-103 mandates that we
     achieve confidence and power across a broad
     range of combat conditions. After a thorough
     examination of alternatives, the DOE methodology
     using factorial designs should be used whenever
     possible to meet the intent of this policy.”



                        * Original Wing Commander Policy April 2002 52
          March 2009: OTA Commanders
          Endorse DOE for both OT & DT
“Experimental design further provides a
valuable tool to identify and mitigate risk
in all test activities. It offers a framework
from which test agencies may make well-
informed decisions on resource
allocation and scope of testing required
for an adequate test. A DOE-based test
approach will not necessarily reduce the
scope of resources for adequate testing.
Successful use of DOE will require a
cadre of personnel within each OTA
organization with the professional
knowledge and expertise in applying
these methodologies to military test
activities. Utilizing the discipline of DOE
in all phases of program testing from
initial developmental efforts through
initial and follow-on operational test
endeavors affords the opportunity for
rigorous systematic improvement in test
processes.”
                                                53
       Nov 2008: AAC Endorses DOE
      for RDT&E Systems Engineering




   AAC Standard Systems Engineering Processes and Practices

                                                               54
July 2009: 46 TW Adopts DOE as
      default method of test
          We Train the Total Test Team
           … but first, our Leaders!
                           Leadership Series
          DOE Orientation (1 hour)
          DOE for Leaders (half day)
          Introduction to Designed Experiments (PMs- 2 days)

           OA/TE Practitioner Series
        10 sessions and 1 week each
        Reading--Lecture--Seatwork
   Basic Statistics Review (1 week)
      Random Variables and Distributions
      Descriptive & Inferential Statistics
      Thorough treatment of t Test             Journeymen Testers
   Applied DOE I and II (1 week each)
      Advanced Undergraduate treatment
      Graduates know both how and why

                                                                56
        Ongoing Monday Continuation
                 Training
                                Operations Analyst Forum/DOE Continuation Training for 04 May 09:
   Weekly seminars online
                                  Location/Date/Time: B1, CR220, Monday, 04 May 09, 1400-1500 (CST)

   Topics wide ranging           Purpose: OPS ANALYST FORUM, DOE CONTINUATION TRAINING, USAF T&E COLLABORATION
                                  Live: Defense Connect Online: https://connect.dco.dod.mil/eglindoe
       New methods               Dial-In: (VoIP not available yet) 850-882-6003/DSN 872-6003
                                  Data:
       New applications             USAF DOE CoP
                                     https://afkm.wpafb.af.mil/SiteConsentB anner.aspx ?ReturnUrl=%2fASPs%2fCoP%2fOpenCoP.as
       Problem decomposition         p%3fFilter%3dOO-TE-MC-79& Filter=OO-TE-MC-79 (please let me know if this link works for
                                      you)
       Analysis challenges
       Reviewing the basics
       Case studies
       Advanced techniques
   DoD web conferencing
   Q&A session following
   Monday 1400-1500 CR 22 Bldg 1
    or via DCO at desk
                 DOE Initiative Status

   Gain AAC Commander endorsement and policy announcement
   Train and align leadership to support initiative
   Commence training technical testers
   Launch multiple quick-win pilot projects to show applicability
   Communicate the change at every opportunity
   Gain AAC leadership endorsement and align client SPOs
   Influence hardware contractors with demonstrations and
    suitable contract language
   Keep pushing the wheel to build momentum
   Confront nay-sayers and murmurers
   Institutionalize with promotions, policies, Wing structures and
    practices
   Roll out to USAF at large and our Army & Navy brethren

                                                                      58
        But Why Should Systems Engineers
                  Adopt DOE?
   Why:                                             “To call in the statistician
                                                     after the experiment is ...
       It‟s the scientific, structured, objective   asking him to perform a
        way to build better ops tests                postmortem examination: he
       DOE is faster, less expensive, and more      may be able to say what the
        informative than alternative methods         experiment died of.”
                                                       Address to Indian Statistical
       Uniquely answers deep and broad                             Congress, 1938.
        challenge: Confidence & Power across
        a broad battlespace
       Our less-experienced testers can
        reliably succeed
       Better chance of getting it right!
   Why not ...
    “If DOE is so good, why haven‟t I heard of
        it before?”
                                                              DOE Founder
    “Aren‟t these ideas new and unproven?”
                                                          Sir Ronald A. Fisher
                                                                                   59
What‟s Your Method of Test?

   DOE: The Science of Test




        Questions?
Backups – More Cases




                       61
                 Where we intend to go with
                  Design of Experiments
   Design of Experiments can aid acquisition in many areas
       M&S efforts (AoA, flyout predictions, etc.)
       Model verification (live-fire, DT, OT loop)
       Manufacturing process improvements
       Reliability Improvements
       Operational effectiveness monitoring
   Make DOE the default for shaping analysis and test program
     Like any engineering specialty, DOE takes time to educate and mature in
      use
     Over next 3-5 years, make DOE the way we do test - policy



          2008              2009              2010              2011
           •Initial Awareness              •Hire/Grad Ed Industrial/OR Types
                     •Begin Tech Trn                   •Mature DOE in all programs
                         •Mentor Practitioners
                       •Adjust PDs, OIs, trn plans
                     Case 1 Problem
   Binary data – tendency to score hit/miss or kill/no kill
   Bad juju – poor statistical power, challenge to normal theory
   Binary leads to bimodal distributions
                Normal theory fit is
                   inadequate
   Typically get surfaces
    removed from responses –
    poor fit, poor predictions
   Also get predicted
    responses outside 0-1:
    poor credibility with clients
    
        Solution is GLM – with binary
               response (error)
                                               
   We call this the logit
    link function
                             Logit ( )  ln      
   Logit model restricts
    predictions in 0/1
                                             1  
   Commonly used in
    medicine, biology,                1
    social sciences             
                                ˆ        β X
                                   1 e
             Logit Solution Model Works Well
Term                                      Estimate        L-R      Prob>ChiS Lower CL              Upper
                                                       ChiSquare        q                            CL
Visibility                                  1.72         43.40       <.0001     1.08                2.57
Aspect                                     -0.02         27.22       <.0001    -0.03               -0.01
Intercept                                   3.34          7.96        0.00      0.94                6.34
(Aspect-270)*(Aspect-270)                   0.00          1.59        0.21      0.00                0.00
Visibility*(Aspect-270)                     0.01          0.98        0.32      0.00                0.02
(TGT_Range-2125)*(Aspect-270)               0.00          0.61        0.43      0.00                0.00
Visibility*(TGT_Range-2125)                 0.00          0.60        0.44      0.00                0.00
Visibility*(TGT_Range-2125)*(Aspect-        0.00          0.13        0.72      0.00                0.00
270)
TGT_Range                                   0.00            0.04     0.84            0.00           0.00
Visibility*Visibility                      -0.04            0.01     0.93           -0.85           0.74
          With 10 replicates good model
          With only 1 replicate (90% discard) same model!
                                                     Term               Estimate L-R ChiSquare Prob>ChiSq           Lower CL   Upper CL
                             Visibility                                     2.44            18.13          <.0001     0.98       5.34
                             Aspect                                         -0.03           12.76           0.00     -0.07      -0.01
                             Intercept                                      8.23            5.53            0.02      1.05      21.28
                             (Aspect-270)*(Aspect-270)                      0.00            3.82            0.05      0.00       0.00
                             Visibility*(TGT_Range-2125)*(Aspect-270)       0.00            2.48            0.12      0.00       0.00
                             Visibility*(Aspect-270)                        0.02            2.44            0.12      0.00       0.05
                             TGT_Range                                      0.00            1.48            0.22      0.00       0.00
                             (TGT_Range-2125)*(Aspect-270)                  0.00            0.84            0.36      0.00       0.00
                             Visibility*(TGT_Range-2125)                    0.00            0.52            0.47      0.00       0.00
                             Visibility*Visibility                          0.36            0.24            0.62     -1.15       1.85
          Problem – Blast Arena
    Instrumentation Characterization




   Do the blast measuring devices read the same at various
    ranges/angles? Fractional Factorial
   Graphic of design geometry in upper right
                                                           Solution – unequal measurement
                                                                  variance - methods
       Normal theory confidence intervals
                                        95% CI BASED ON CLASSICAL STATS
                                      Vertical bars denote 0.95 confidence intervals
                      200

                      180

                      160


                                                                                                                        Rank Transform confidence intervals
Peak Pressure (PSI)




                      140                                                                                                                   95% CI BASED ON RESAMPLING STATS
                                                                                                                                           Vertical bars denote 0.95 confidence intervals
                      120
                                                                                                                        200
                      100
                                                                                                                        180
                      80
                                                                                                                        160
                      60
                                                                                                                        140
                      40

                                                                                                  Peak Pressure (PSI)
                                                                                                                        120
                      20
                                                                                                      100
                                                                                                Proximity
                       0
                            Device Type:       Type I          Device Type:            Type I   Near
                                                                                                       80
                                                                                                Proximity
                                    Type II                            Type II
                                                                                                Far
                                                                                                                         60
                                   Particle Stripper: No               Particle Stripper: Yes
                                                                                                                         40

                                                                                                                         20
                                 Closer to weapon,                                                                       0
                                                                                                                                                                                                                                                 95% CI BASED ON RANK STATS
                                                                                                                                                                                                                                            Proximity


                                  measurement
                                                                                                                              Device Type:
                                                                                                                                       Type II
                                                                                                                                                      Type I              Device Type:
                                                                                                                                                                                   Type II
                                                                                                                                                                                                   Type I
                                                                                                                                                                                                                              Bootstrap confidence intervals
                                                                                                                                                                                                                            200
                                                                                                                                                                                                                                            Near
                                                                                                                                                                                                                                            Vertical bars denote 0.95 confidence intervals
                                                                                                                                                                                                                                            Proximity
                                                                                                                                                                                                                                            Far
                                                                                                                                       Particle Stripper: No                        Particle Stripper: Yes
                                                                                                                                                                                                                            180
                                  variability is large                                                                                                                                                                      160

                                                                                                                                                                                                                            140




                                                                                                                                                                                                      Peak Pressure (PSI)
                                 Note confirmation                                                                                                                                                                         120


                                  with three methods                                                                                                                                                                        100

                                                                                                                                                                                                                            80

                                                                                                                                                                                                                            60

                                                                                                                                                                                                                            40

                                                                                                                                                                                                                            20

                                                                                                                                                                                                                             0                                                                        Proximity
                                                                                                                                                                                                                                  Device Type:          Type I        Device Type:           Type I   Near
                                                                                                                                                                                                                                          Type II                             Type II                 Proximity
                                                                                                                                                                                                                                                                                                      Far
                                                                                                                                                                                                                                         Particle Stripper: No               Particle Stripper: Yes
        Problem – Nonlinear problem
          with restricted resources




   Machine gun test – theory says error will increase with the
    square of range
   Picture is not typical, but I love Google
         Solution – efficient nonlinear
           (quadratic) design - CCD




   Invented by Box in 1950‟s
   2D and 3D looks – factorial augmented with center points and
    axial points – fit quadratics
         Power is pretty good with 4-
            factor, 30 run design
       Power at 5 % a level to detect signal/noise ratios of
    Term         StdErr** 0.5 Std. Dev. 1 Std. Dev. 2 Std. Dev.
Linear Effects   0.2357       16.9 %           51.0 %      97.7 %
 Interactions    0.2500       15.5 %           46.5 %      96.2 %
  Quadratic
   Effects       0.6213       11.7 %           32.6 %      85.3 %
                     **Basis Std. Dev. = 1.0
    Prediction Error Flat over Design
                 Volume
      Design-Expert® Software                                                 FDS Graph
      Min StdErr Mean: 0.311                        1.000


      Max StdErr Mean: 0.853
      Cuboidal
      radius = 1
      Points = 50000                                0.750
      t(0.05/2,15) = 2.13145
      FDS = 0.90




                                S td E rr M e a n
      StdErr Mean = 0.677

                                                    0.500




                                                    0.250




                                                    0.000




                                                            0.00   0.25              0.50            0.75   1.00




                                                                          Fraction of Design Space

   This chart (recent literature innovation) measures prediction
    error – mostly used as a comparative tool to trade off designs –
    Design Expert from StatEase.com
                         Summary
   Remarkable savings possible
    esp in M&S and HWIL
   Questions are scientifically
    answerable (and debatable)
   We have run into 0.00
    problems where we cannot
    apply principles
   Point is that DOE is widely
    applicable, yields excellent
    solutions, often leads to
    savings, but we know why N
   To our good science – add
    the Science of Test: DOE
                                   Logit fit to binary problem
             Case: Secure SATCOM for
                F-15E Strike Eagle
                                                                                  Test Objective:
                                                                                   To achieve secure A-G comms in
                                                                                     Iraq and Afghanistan, install ARC-
                                                                                     210 in fighters
                                                                                   Characterize P(comms) across
                                                                                     geometry, range, freq, radios, bands,
                                                                                     modes
                                                                                   Other ARC-210 fighter installs show
                                                                                     problems – caution needed here!


             DOE Approach:                                                                  Results:
 • Use Embedded Face-Centered CCD design                                             • For higher-power
      (created for X-31 project, last slide)                                              radio – all good
• Gives 5-levels of geometric variables across                                       • For lower power
                radios & modes                                                         radio, range problems
  • Speak 5 randomly-generated words and                                              • Despite urgent
              score number correct                                                         timeframe and
                                                                                         canceled missions,
  • Each Xmit/Rcv a test event – 4 missions
                                                                                       enough proof to field
                     planned
             Source: F-15E Secure SATCOM Test, Ms. Cynthia Zessin, Gregory Hutto, 2007 F-15 OFP CTF 53d Wing / 46 Test Wing, Eglin AFB, Florida
         Case: GWEF Large Aircraft IR
             Hit Point Prediction
                                                   Test Objective:
                                                    IR man-portable SAMs pose threat to
                                                      large aircraft in current AOR
                                                    Dept Homeland Security desired Hit
                                                      point prediction for a range of
                                                      threats needed to assess
                                                      vulnerabilities
                                                    Solution was HWIL study at GWEF
IR Missile C-5 Damage                                 (ongoing)


                 DOE Approach:                                      Results:
           •   Aspect – 0-180 degees, 7each         • Revealed unexpected hit point behavior
           •   Elevation – Lo,Mid,Hi, 3 each        • Process highly interactive (rare 4-way)
         •   Profiles – Takeoff, Landing, 2 each
           •    Altitudes – 800, 1200, 2 each
                                                   • Process quite nonlinear w/ 3rd order curves
      • Including threat – 588 cases                 • Reduced runs required 80% over past
    • With usual reps nearly 10,000 runs             • Possible reduction of another order of
                                                            magnitude to 500-800 runs
   • DOE controls replication to min needed
                       Case: Reduce F-16 Ventral Fin
                        Fatigue from Targeting Pod                                                                                Face-Centered CCD

              Test Objective:
                 blah



                                                                                                                             Embedded F-CCD


                                                                                         Expert Chose 162 test points
                      Ventral fin

                            DOE Approach:
       •         Many alternate designs for this 5-dimensional                                                 Results:
                    space (a, b , Mach , alt, Jets on/off)                               •   New design invented circa 2005 capable of efficient
                                                   Percent                                   flight envelope search
                                             Test     of                                 •   Suitable for loads, flutter, integration, acoustic,
              Choice    Design Name         Points Baseline           Model
                                                                                             vibration – full range of flight test
               Base     Subject-expert       324    100%        none - inspection
                1             CCD             58     18%      quadratic + interactions   •   Experimental designs can increase knowledge while
Experiments
 Designed




                2             FCD             58     18%      quadratic + interactions       dramatically decreasing required runs
                3       Fractional FCD        38     12%      quadratic + interactions
                4    1/3rd fraction 3-level   54     17%       quadratic interactions    •   A full DOE toolbox enables more flexible testing
                5       Box-Behnken           54     17%      quadratic + interactions
                6    S-L Embedded FCD         76     23%        Up to cubic model
        Case: Ejector Rack Forces for
         MAU-12 Jettison with SDB
                                                                                                Test Objective:
                                                                                                 AF Seek Eagle characterize forces to
                                                                                                   jettison rack with from 0-4 GBU-39
                                               120

                                               100
                                                                      ARD-446/ARD-446
                                                                      ARD-446/ARD-863
                                                                                                   remaining
                          Acceleration (g's)
                                                                      ARD-863/ARD-863
                                               80                                                Forces feed simulation for safe
                                               60


                                               40
                                                                                                   separation
                                               20                                                Desire robust test across multiple
                                                0
                                                10
                                                  2
                                                                10
                                                                  3


                                                      Log10 Store Weight(lbs)
                                                                                         4
                                                                                        10         fleet conditions
                                                                                                 Stores Certification depend on
                                                                                                   findings
                                                                                                                                                      4500


                                                                                             Results:                                                            y = 5.07e-008*x3 - 0.00038*x2 + 1.14*x + 1632.64
                DOE Approach:                                                                                                                                    y = 7.62e-008*x3 - 0.00067*x2 + 1.85*x + 1540.69
                                                                                             •   From Data Mined „99 force                            4000


•   Multiple factors: rack S/N, temperature,                                                     data with regression
    orifice, SDB load-out, cart lot, etc




                                                                                                                               Peak Fwd Force (lbs)
                                                                                                                                                      3500

                                                                                             •   Modeled effects of temp,
•   AFSEO used innovative bootstrap                                                              rack, cg etc                                         3000

    technique to choose 15 racks to characterize                                             •   Cg effect insignificant
    rack-variation                                                                           •   Store weight, orifice, cart                          2500




•   Final designs in work, but promise 80%                                                       lot, temperature all
                                                                                                                                                      2000
                                                                                                                                                                                                                              -10/-3
                                                                                                                                                                                                                              -10/-4


    reduction from last characterization in                                                      significant
    FY99                                                                                                                                              1500
                                                                                                                                                             0       500   1000   1500   2000   2500   3000   3500   4000   4500       5000
                                                                                                                                                                                         Store Weight (lbs)
                 Case: AIM-9X Blk II
              Simulation Pk Predictions
                                                    Test Objective:
                                                     Validate simulation-based
                                                       acquisition predictions with live
                                                       shots to characterize performance
                                                       against KPP
                                                     Nearly 20 variables to consider:
                                                       target, countermeasures,
                                                       background, velocity, season, range,
                                                       angles …
                                                     Effort: 520 x 10 reps x 1.5 hr/rep =
                                                       7800 CPU-hours = 0.89 CPU-years
                                                     Block II more complex still!


DOE Approach:                                                      Next Steps:
• Data mined actual “520 set” to ID critical    •     Ball in Raytheon court to learn/apply DOE
  parameters affecting Pk  many conditions                              …
  have no detectable effect …
• Shot grid  greatly reduce grid granularity
• Replicates  1-3 reps per shot maximum
• Result – DOE reduces simulation CPU
  workload by 50-90% while improving live-
  shot predictions
                                                      Notional 520 grid          Notional DOE grid
             Case: AIM-9X Blk II Simulation Pk
                    Predictions Update
                                                 Test Objective:
                                                  Validate simulation-based Pk
                                                  “520 Set”: 520 x 10 reps x 1.5
                                                    hr/rep = 7800 CPU-hours = 0.89
                                                    CPU-years
                                                  Block II contemplates “1380 set”
                                                  1 replicate-1 case: 1 hour =>
                                                    1380*1*10 =13,800 hours = 1.6
                                                    CPU-years


DOE Approach:                                                    Next Steps:
• We‟ve shown: many conditions have no           •   Raytheon responds to learn/apply DOE …
  effect, 1-3 reps max, shot grid 10x too fine
• Raytheon response – hire PhD OR-type to
  lead DOE effort
• Projection – not 13,800, but 1000 hours max
  saving 93% of resources
• 2 June week DOE meeting

                                                     Notional 520 grid        Notional DOE grid
                     Case: CFD for NASA CEV
                                                                                 Test Objective:
                                                                                  Select geometries to minimize total
                                                                                    drag in ascent to orbit for NASA‟s
                                                                                    new Crew Exploration Vehicle (CEV)
                                                                                  Experts identified 7 geometric
                                                                                    factors to explore including nose
                                                                                    shape
                                                                                  Down-selected parameters further
                                                                                    refined in following wind tunnel
                                                                                    experiments
DOE Approach:
                                                                            Results:
• Two designs – with 5 and 7 factors to vary
                                                                            • Original CFD study
• Covered elliptic and conic nose to                                           envisioned 1556
  understand factor contributions                                              runs
• Both designs were first order polynomials                                 • DOE optimized
  with ability to detect nonlinearities                                        parameters in 84
• Designs also included additional                                             runs – 95%!
  confirmation points to confirm the                                        • ID‟d key interaction
  empirical math model in the test envelope                                    driving drag

             Source: A Parametric Geometry CFD Study Utilizing DOE Ray D. Rhew, Peter A. Parker, NASA Langley Research Center, AIAA 2007 1616
                 Case: Notional DOE for Robust
               Engineering Instrumentation Design
                                                  Test Objective:
                                                     Design data link recording/telemetry
                                                      design elements with battery backup
                                                     Choose battery technology and antenna
                                                      robust to noise factors
                                                     Environmental factors include Link-16
                                                      Load (msg per minute) and temperature
                                                     Optimize cost, cubes, memory life, power
                                                      out



                 DOE Approach:                                       Battery*Temperature; LS Means

•
                                                               160
    Set up factorial design to vary multiple
    design and noise parameters simultaneously                 140



•
                                                               120
    DOE can sort out which design choices and
    environmental factors matter – how to build        Hours
                                                               100



    a robust engineering design                                 80



•   In this case (Montgomery base data) one                     60



    battery technology outshone others over                     40


    temperature – Link 16 messages do not                       20
                                                                                                     Battery
                                                                                                     LithiumIon
                                                                                                     Battery
    matter                                                       0
                                                                        -65          15         70
                                                                                                     NiCad
                                                                                                     Battery
                                                                                                     NiMH
                                                                                 Temperature
                      Case: Wind Tunnel X-31 AOA
                                                                                              Test Objective:
                                                                                               Model nonlinear aero forces in
 0.06

 0.04                                                                                             moderate and high AoA and yaw
0.02

   0
                                                                                                  conditions for X-31
                                                                20

                                                                                               Characterize effects of three control
-0.02
                                                           10
-0.04

                                                                                                  surfaces – Elevon, canard, rudder
   -20                                                 0
         0
                    20                         -10
                               40
                                    60
                                          80
                                         -20
             Angle of attack                         Yaw
                                                                                               Develop classes of experimental
                                                                                                  designs suitable for efficiently
                                                                                                  modeling 3rd order behavior as well
                                                                                                  as interactions
                                                                                           Results:
 DOE Approach:                                                                             • Usual wind tunnel trials encompass 1000+
 • Developed six alternate response surface                                                   points – this was 104
   designs                                                                                 • Revealed process nonlinear (X3), complex and
 • Compared and contrasted designs‟ matrix                                                    interactive
                                                                                                                            C vs. a
   properties including orthogonality,                                                     • Predictions accurate < 1%            1.2
                                                                                                                                                                    L
                                                                                                                                                      b = 0°, dc = 0°, d r = 0°, da = 0°)


   predictive variance, power and confidence                                                                                        1

                                                                                                                                  0.8

 • Ran designs, built math models, made                                                                                           0.6
                                                                                                                                                                                                  Baseline




                                                                                                                             CL
                                                                                                                                                                                                  RSM Low
   predictions and confirmed                                                                                                      0.4

                                                                                                                                  0.2
                                                                                                                                                         Coeff Lift (CL)
                                                                                                                                                                                                  RSM High

                                                                                                                                    0
                                                                                                                                                          Vs AoA (a)
                                                                                                                                  -0.2
                                                                                                                                         0   5   10          15         20      25           30   35         40
                          Source: A High Performance Aircraft Wind Tunnel Test using Response Surface Methodologies Drew Landman, NASA Langley Research Center,
                                                                                                                                                      a
                                                                    James Simpson Florida State University, AIAA 2005-7602
             Case: SFW Fuze Harvesting
                       LAT
                                                                 Test Problem/Objective:
                                                                  Harvest GFE fuzes from excess
                                                                    cluster/chem weapons
                                                                  Textron “reconditions” fuzes, installs
                                                                    in SFW
                                                                  Proposed sequential 3+3 LAT-
                                                                    pass/reject lot
                                                                  Is this an acceptable LAT? Recall:
                                                                  Confidence = P(accept good lot)

                                           800
                                                                  Power = P(reject bad lot)
DOE/SPC Approach:                          700                    Results:
                                           600   1-a
• Remember – binary vars weak              500                    • Shift focus from 1-0 to fuze function time
                                  Counts




                                           400

• Test destroy fuze                        300
                                           200
                                                                  • With 20kft delivery d >> 250 ms
• Establish LAT performance:               100
                                             0
                                                                  • Required n is now only 4 or 5 fuzes
     –
                                                 0   1   2   3
         Confidence = 75%                                                                 Statistical Power to Detect Escapement Delay


     –
                                                                                1.2
         Power = 50%                       600
                                                                                 1

•   Q: What n is required then?            500
                                                 1-b                            0.8
                                                                        Power




                                           400
                                  Counts




•
                                                                                0.6
    A: destroy 20+ of 25 fuzes             300
                                                                                0.4
                                           200
                                                                                0.2
                                           100
                                                                                 0
                                             0                                        0   2          4             6              8      10   12
                                                 0   1   2   3                                           Number of Fuses Tested
           Case: DOE for Troubleshooting
            & False Alarms (& software)
                                                      Test Problem/Objective:
                                                         782 TS has malf photos during
                                                          weapons tests
                                                         What causes it?
                                                         Possibilities include camera, laptop,
                                                          config file, and operator – 4 each
                                                         256 combinations – how to test?
                                                         Remember – binary vars weak
                                                         But – d is also huge: 0 or 1 not 0.8 to
                                                          0.9
           DOE/SPC Approach:                                              Results:
     •     We can design several ways:                •   TE Marshall solves with “Most Likely” screen
     –    44 design is 256 combos (not impossible)        • Soln: Laptop x Config file interaction
 –       “Likely” Screen with 24 design (16 combos)   •   Others: MWS, Altimeter false alarms, software
         – Use Optimal or Space-filling design




                                                             JMP Latin Hypercubes n=8, n=10
JMP Sphere-Packing n=8, n=10
              Case: DOE Characterizing Onboard
                 Squib Ignition on Test Track
                                                           Test Problem/Objective:
                                                              Track wants to test ability to initiate
                                                               several ignitors / squibs both on- and -
                                                               off board sleds
                                                              What is the firing delay – average &
                                                               variance?
                                                              Min of three squibs/ignitors
                                                              On and off board sled, voltages,
                                                               number, and delays are variables
                                                              Standard approach shown on next
                                                               slide

DOE/SPC Approach:                                         Results:
•  We can design several ways:                            •   Still TBD – late summer execution
     –   Choose to split design on and off board          •   Want to characterize also gauge reliability using
     –   Can choose nested or confounded design               harvested data mining
•   Objective search reveals two:                         •   Improved power across test space while examining
     –   Characterize firing delay for signal                 more combinations
     –   Characterize ignition delay from capacitor       •   Sniff out two objectives with process – two test
•   Build two designs to characterize with dummy/actual       matrices
    loads                                                 •   Power improved from measuring each firing event
     –   Dummy resistors for signal delay
     –   Actual (scarce) ordnance for ignition delay
                 Comparison of Designs
                                                                              Combos
Original Cfig Tests   Ordnance     NumAlloc On/Offbd Power (max) UsingNum Examined
         A           Pupfish-300V      19        Off   0.17-0.755   2 to 4       4
         B           BBU-35-75V        19        Off    0.17-0.97   2 to 6       4
         C          BBU-35 - 100V      24        On   0.17 to 0.97  2 to 6       4
         D           CI-10 -100V       9         On     0 to .755   1 to 4       2
         E           Pupfish-100V      19        On   0.17 to .755  2 to 4       4
                      Total Rqd        90
                      Number required (max - 2 reps)
                                                                              Combos
DOE Configs cover      Pupfish      BBU-35      CI-10     Power      Note    Examined
  AB - Offboard           24           12         --    0.90-0.99 New PF-75V    24
 CDE* - Onboard           16           16         8       .999+                 48
                      Total Rqd        76                                       72


   As commonly found – more combos (4x), improved power (4x), new
                        ideas and objectives
                Case: Arena 500 Lb Blast
               Characterization vs. Humans
                                                     Test Problem/Objective:
                                                      780 TS Live fire testers want to
                                                       characterize blast energy
                                                       waveform outside & inside
                                                       Toyota HiLux
                                                      Map Pk as a function of range,
                                                       aspect, weapon orientation,
                                                       gauge, vehicle orientation, glass
                                                      How repeatable are the
                                                       measurements bomb to bomb
             DOE/SPC Approach:
                                                                     gauge?
                                                       and gauge toResults:
                                                     •   Brilliant design from Capt Stults, Cameron, with
 • Blast test response – peak pressure, rise time,
                                                                assists from Higdon/Schroeder/Kitto
       fall time, slopes (for JMEM models)                                       Det 3: Tail / Horizontal
                                                         • Great collaborative, Det 4: Nose / Vert design
                                                                                    creative test
 • Directly blast test symmetry usually assumed                                                       300.00




  • Critical information on gauge repeatability                                                       250.00




   • Test is known as a “Split Plot” as each                                                          200.00




         detonation restricts randomization                       ft
                                                                                                      150.00




• Consequence: less information on bomb-bomb                                                          100.00




          variations (but of lesser interest)                                                          50.00



                                                                                                        0.00
                                                                       -300.00   -200.00   -100.00          0.00             100.00   200.00   300.00
                                                                                                               ft


                                                                                                     pencil         ground
                    Case: Computer Code
                           Testing
  Mission Planning          Cyber Defense           Test Problem/Objective:
                                CWDS                   OK-DOE works for CEP, TLE, Nav Err,
                                                        Penetrators, blast testing, SFW Fuze
                                          UAI           Function … But what about testing
   CFD            JAWS/JMEM
                                                        computer codes??
 Campaign Models                  IOW                  Mission Planning: load-outs, initial fuel,
                                                        route length, refueling, headwinds, runway
                              TBMCS                     length, density altitude, threats, waypoints,
                                                        ACO, package, targets, deliveries/LARS …
  NetWar                                               How do we ascertain the package works?
      F-15E Suite 5E OFP
                                                   Results:
          Whitebox       Code Verif       NA
                                                   • An evolving area – NPS, AFIT, Bell Labs
SW                         Physics                 • Several space-filling design algorithms
                                         Space
       Blackbox                                    • Aircraft OFP testing a proof-case: weapons,
DOE Approach:                                         comm, sensors work fine
                          Performance Classic
• We can design several ways:                      • More to do: Mission Planning, C4I, IW, etc.
   – Code verification – not too much contribution
                                                   • Proposal – try some designs and get some help
   – Physics or SW code – “Space Filling” Designs     from active units and universities (AFIT, Univ
   – SW Performance – errors, time, success/fail,     of Fl, Va Tech, Army RDECOM, ASU, Ga
      degree = classical DOE designs
                                                      Tech, Bell Labs …
Some Space-Filling 3D designs


        10 point            50 point
        10x10x4             10x10x4
        Sphere                Latin
        Packing            Hypercube
         Design              Design




        30 point           27 point
        10x10x4             3x3x3
        Uniform            Classical
         Design              DOE
                           Factorial
                            Design
                                         CV-22 IDWS
                                                                Issues/Concerns
                                               Accelerated testing
                                               Primary performance issues: vibration,
                                                performance and flying qualities
                                               Full battlespace tested, except
                                                    Climbs/dives excluded
                                                    400-round burst (gun heating effect)
                                                      excluded
                 For Official Use Only




            Program Description                              DOE approach
                                              •   Rescoped current phase of testing
• IDWS: Interim Defensive Weapon System
                                              •   Completed process flow diagrams
• Objective: Evaluate baseline weapons fire   •   Completed test matrix
accuracy for the CV-22.                       •   Developed run matrix
• Flight test Jan 09                                • Initial proposal = 108 runs
                                                    • Factorial with center points = 92 runs
                                                    • FCD with center points = 40 runs
                                                    • FCD with 2 center points = 36 runs
                                                    • FCD in 2 blocks = 30 runs

                                                                                               31 Dec 08
                             MC-130H BASS

                                                             Issues/Concerns

                                                    LMAS proposed test matrix
                                                Inconsistent test objective – overall
                                                               purpose




           Program Description                            DOE approach
• MC-130H Bleed Air System study             • Completed process description and
• Test Objective: Collect bleed air system     decomposition
  performance data on the MC-130H            • Led to fewer, better-defined test factors
                                             • “Augmented” LMAS test matrix of 114
  Combat Talon II aircraft
                                               required cases
• Flight test Jan 09                         • Split-split plot design of 209 runs
                                             • Increases power, confidence; decreases
                                               confounding
                                                                                      31 Dec 08
                                SABIR/ATACS
                                                                Issues/Concerns

                                                      Scope provided by LMAS Palmdale
                    Picture                             Acceptable confidence, power,
                                                        accuracy, precision TBD




            Program Description
• SABIR: Special Airborne Mission Installation &              DOE approach
  Response
• ATACS: Advanced Tactical Airborne C4ISR System   • Flight test spring 2009
• Test Objectives:                                 • Gov’t proposed matrix ~720 runs
    • Maintain aircraft pressurization             • DOE proposed approach = embedded FCD
    • Determine vibration                             factor of 5 reduction
    • Functional test at max loads
    • Determine aerodynamic loading
    • Performance and flying qualities
    • EMC/EMI

                                                                                    31 Dec 08
                                          AFSAT
                                                               Issues/Concerns
                                                Limited Funding
                                                Full orthogonal design limited by test
                                                 geometry
                                                Limited time of flight at test conditions
                                                Safety concerns- flying in front of unmanned
                                                 vehicle

                 For Official Use Only




             Program Description                            DOE approach
• AFSAT: Air Force Subscale Aerial Target      • Used minimal, factorial design.
•Objective: Collect IR signature data on the
AFSAT for future augmentation and threat       • Final design is a full factorial in some factors
representation                                   augmented with an auxiliary design.
•Flight test Jan-Feb 09
                                               • Minimized risk of missing key data points and
                                                 protected against background variation with
                                                 series of baseline cases
                                                                                           1 Jan 09
                                               LASI HPA
                                                                             Issues/Concerns
                                                            Bi-modal data distribution
                                                            Near-discontinuities in responses
                                                            High-order interactions and effects




        TE: ?                                 OA: ?                   Legend:  FYI     Heads-Up  Need Help


                Program Description                                       DOE approach
Large Aircraft Survivability Initiative (LASI) Hit-Point   • Harvested historic data sets for knowledge
Analysis (HPA)                                             • Identified intermediate responses with richer
• High Fidelity Models of Large Transport Aircraft           information
• Supports Susceptibility Reduction Studies                • Identified run-savings for CM testing
• Supports Vulnerability Reduction Studies




                                                                                                        31 Dec 08
        Anubis MAV Fleeting Targets QRF

                                     Micro-munitions Air Vehicle proof of
                                     concept & lethality assessment
                                     – Objective
                                        – Successful tgt engagement
                                        – Verify lethality




   Original MoT                     DoE used to
       Factor effects confounded      – Fixed original problems
       Imbalanced test conditions     – Used 8 instead of 10
       Susceptible to background        prototypes
        noise                          – Inserted pause after first test to
       No measure of wind effects       make midcourse correntions

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:17
posted:4/8/2011
language:English
pages:95
niusheng11 niusheng11
About Those docs come from internet,if you have the copyrights of one of them,tell me by mail niutianshang@163.com,and i will delete it on the first time. I just want more peo learn more knowledge. Thank you!