It's the Law_ - INCOSE by zhouwenjuan

VIEWS: 31 PAGES: 27

									                    Systems Engineering: It’s the Law!

                            An Overview of the
                Defense Acquisition Management System and
                  Weapon Systems Acquisition Reform Act
                                  (WSARA)

                                           Geoff Draper
                        Harris Government Communications Systems Division




Systems Engineering: It’s the Law! -1                                       April 2010
INCOSE Space Coast Chapter
Systems Engineering: It’s the Law!

Topics:                                      References:
•     Performance issues in Major           P. L. 111-23, Weapon Systems Acquisition Reform Act of 2009.
                                            May 22, 2009. http://thomas.loc.gov/cgi-bin/query/z?c111:S.454.enr:
      Defense Acquisition Programs          Defense Technical Memorandum, DTM-09-27.
      (MDAPs)                               “Implementation of the Weapon Systems Acquisition Reform Act of 2009”.
                                            Dec 2009. http://www.dtic.mil/whs/directives/corres/pdf/DTM-09-027.pdf
         – Summary of studies and reports   DoDI 5000.02, The Defense Acquisition Management System. Dec 2008.

•     Overview of the Defense               http://www.dtic.mil/whs/directives/corres/pdf/500002p.pdf
                                            DoD Strategic Management Plan. July 2009.
      Acquisition Management System         http://dcmo.defense.gov/documents/2009SMP.pdf

         – DoD policies and guidance        Defense Acquisition Guidebook (DAG), Chapter 4: Systems Engineering.
                                            https://acc.dau.mil/CommunityBrowser.aspx?id=332951&lang=en-US
         – DODI 5000.2                      Implementation of Weapon Systems Acquisition Reform Act (WSARA) of

•     Weapon Systems Acquisition            2009. Mona Lush, OUSD (AT&L). Oct 2009.
                                            http://www.acq.osd.mil/damir/2009%20Conference/Future%20of%20DoD%20
                                            Acquisition%20Plenary/WSARA%20Implementation%20Lush.pdf
      Reform Act of 2009 (WSARA)
                                            DODI 5000.02 and WSARA Impacts on Early System s Engineering.
•     Potential Implications to the         Sharon Vannucci, OSD (AT&L) / DDRE. NDIA SE Conference, Oct ’09.
                                            http://www.dtic.mil/ndia/2009systemengr/8925WednesdayTrack2Vannucci.pdf
      Defense Industry                      Weapon System Acquisition Reform Act (WSARA) 2009.
                                            Mr. Nicholas Torelli, OUSD (AT&L) / DDRE. NDIA SE Div mtg, July 2009.
                                            http://www.ndia.org/Divisions/Divisions/SystemsEngineering/Documents/Past
                                            %20Projects/August%2009%20Division%20Meeting/2009_08_06_SE-
                                            WSARA%20Brief%20to%20NDIA%20SED-final.pdf
                                            A Multi-Level Approach to Addressing SE Issues in Defense Programs.
                                            NDIA SE Division, June 2009.
                                            http://www.ndia.org/Divisions/Divisions/SystemsEngineering/Documents/Past
                                            %20Projects/

Systems Engineering: It’s the Law! -2                                                                      19-Jul-12
INCOSE Space Coast Chapter
        The Problem –
        Improving Program Performance through SE



                                                      Numerous studies and reports
                                                     document program performance
                                                      issues and the role of effective
                                                           systems engineering


GAO-09-362T - Actions Needed to Overcome Long-                    NRC/USAF Study - Pre-Milestone A and Early-Phase Delete this
standing Challenges with Weapon Systems Acquisition               Systems Engineering: A Retrospective Review and               box (from
and Service Contract Management                                   Benefits for Future Air Force Acquisition                     the slide
                                                                                                                                master)
  • “DOD’s major weapon systems programs continue to take         http://www.nap.edu/catalog/12065.html                         after
    longer to develop, cost more, and deliver fewer quantities      • “Attention to a few critical systems engineering processescreating
    and capabilities than originally planned.”                         and functions particularly during preparation for Milestones your
                                                                                                                                all of
  • “costs … of major defense acquisition programs increased           A and B is essential…”                                   slides.
    26 percent and development costs increased by 40                • “…critically dependent on having experienced systems Stay
                                                                                                                                inside this
    percent from first estimates”                                      engineers with adequate knowledge of the domain…” “safe
  • “programs … failed to deliver capabilities when promised—       • “Decisions made prior to Milestone A should be supported for
                                                                                                                                area”
    often forcing warfighters to spend additional funds on             by a rigorous systems analysis and systems engineering   correct
    maintaining legacy systems”                                        process involving teams of users, acquirers, and industrydisplay.
  • “current programs experienced, on average, a 21-month              representatives.”
    delay in delivering initial capabilities to the warfighter”




      Addressing SE Issues in the Defense Industry                                                                           3
      July 2009
                                                            (Version 1.0)
     Addressing SE Issues
     - Key Studies and Reports
Study / Report                             Summary Issues / Findings                                                            Details

Pre-Milestone A and Early-                 •Inexperienced leadership              •Incomplete/unstable reqts at MS B
Phase Systems Engineering                  •External interface complexity         •Reliance on immature technology
[report] (NRC/USAF study)                  •System complexity                     •Reliance on large amts of new SW
NDIA Top 5 SE Issues (2006)                •Inconsistent SE practices             •Requirements not well-defined and
[report, briefing]                         •Insufficient SE early in life cycle   managed
                                           •Lack SE resources (qty, quality)      •Inadequate SE tools/environments
NDIA Top SW Issues (2006)                  •SE decisions without SW involvement •Requirements impact on SW
[report, briefing]                         •Ineffective SW life cycle plans      •SW assurance (predictable, secure)
                                           •Lack SW resources (qty, quality)     •COTS/NDI impacts                              Delete this
                                           •SW verification: costly, ineffective                                                box (from
                                                                                                                                the slide
Systemic Root Cause Analysis •Unrealistic acq. strategies and plans               •Staffing shortfalls (quantity, skills,       master)
(SRCA) (2008)                •Milestone decision gates, criteria                  experience)                                   after
(publication 2009, pending)                                                                                                     creating
                                                                                                                                all of your
Systems Engineering of                     •Incomplete design-to requirements     •Component qualification test                 slides.
Tactical Air Launched                      •Requirements creep, not stable        •Weapon level integration tests               Stay
Systems: An Industry                       •Unplanned reqts verification effort   •Weapon level qualification tests             inside this
Examination (2008) [briefing]              •Unplanned performance/design tests    •Weapon flight testing                        “safe
                                                                                                                                area” for
(Air Armament Center,                      •Configuration variation issues        •Redesign for producibility/cost              correct
NDIA Gulf Coast Chapter)                                                          •Field reliability, sustainability            display.

House Armed Services Panel                 •Manage performance of acq. system     •Ineffective financial mgmt system
on Defense Acq. Reform                     •Lacking good reqts process (JCIDS)    •Protect the industrial base (all sizes)
(final report, March 2010)                 •Ad hoc acquisition of services        •Responsiveness of core acq system
                                           •Develop/incentivize acq. workforce

  Addressing SE Issues in the Defense Industry                                                                              4
  July 2009
                                                                 (Version 1.0)
                                        Systemic Issues of Big “A” Acquisition
Big “A” Acquisition*                                                                             •Funding instability
                                                                                                 •Insufficient resource trade space
                                                                                                 •Budget not properly phased/magnitude to
                                                  Resources                                                       support planned development
                                                   (PPBE)


                                                                                                           Small “a” Acquisition
                         Requirements                                 Defense
                                                                                                         •Immature technologies
                           (JCIDS)                                  Acquisition                          •Inadequate systems engineering
                                                                       System                            •Inadequate requirements flow-down/
                                                                         (DAS)                                           traceability/ decomposition
                                                                                                         •Insufficient schedule trade space
                                                                                                         •Inadequate implementation of Earned
                                                                                                                         Value Management System
                                                                                                         •Lack of time and assets for testing
•Lack of JROC-validated requirements document
                for basic program (ORD, CDD, CPD)
•Inadequate requirements for basic program and                  •Lack of inter- and intra-departmental
                any increments                                  stakeholder coordination and support
•Critical dependence on external programs with
                developmental issues
                                                                                                  Joint Capabilities Integration and Development System (CJCSI-3170)
                                                                                                      Program, Planning, Budgeting, and Execution (DoD-7000, FMRs)
        *Systemic Issues of Nunn-McCurdy “Class of 2007” Programs                                                              Defense Acquisition System (DoDI-5000)


                   Synchronize JCIDS, DAS, and PPBE to deliver capabilities to Warfighters.
                                                                                                                                                                 5
    Summary of Key SE Initiatives and
    Reports
Common program issues:                                                      Other sources:
-Poor program planning               -Unrealistic estimates                 •Program Support Reviews (PSRs)
-Unstable requirements               -Immature technology                   •Nunn-McCurdy breaches
-Not following SE processes          -Insufficient SE expertise             •Congressional oversight
-Ineffective reviews                 -Poor system reliability               •Defense Science Board
                                                                            •Other studies, reviews, reports


          2003                              2006                          2008              2009               2010

                                                                                                                            Delete this
                                                                                                                            box (from
     •NDIA Top 5                •Defense Acq Guide (DAG);          •NRC/USAF Study:       •PL 111-23        •QDR [2/10]     the slide
    SE Issues [1/03]                  Chap 4: SE [7/06]            Pre-MS A and Early   (WSARA) [5/09]    •House Armed      master)
                                                                                                                            after
  •DODI 5000.2 [5/03]              •OSD policy, guidance,               SE [1/08]        •DTM-09-027          Services      creating
                                     reviews: SEP; IMS;               •DODI 5000.2      Implementation       Committee      all of your
                                 training; risk mgmt; PSRs           updates [12/08]      of WSARA          report [3/10]   slides.
                                •NDIA Top SE Issues [7/06]           •Systemic Root         [12/09]        •OSD WSARA       Stay
                                •NDIA Top SW Issues [9/06]           Cause Analysis                           report to     inside this
                                                                                                                            “safe
    NDIA Reports                                                      (SRCA) [draft]                      Congress [3/10]   area” for
    DoD policy and actions                                                                                                  correct
                                                                                                                            display.
    Other (Congress, studies, reviews)


                There is no shortage of sources confirming we have issues –
             How do government and industry collaborate to act upon solutions?
  Addressing SE Issues in the Defense Industry                                                                         6
  July 2009
                                                                  (Version 1.0)
                Secretary of Defense Direction
Chief among institutional challenges facing the Department is acquisition.”




                                                                              7
Weapons Systems Acquisition Reform Act
                “The key to successful acquisition programs is
                getting things right from the start with sound
                systems engineering, cost estimating, and
                developmental testing early in the program cycle.
                The bill that we are introducing today will require the
                Department of Defense to take the steps needed to
                put major defense acquisition programs on a sound
                footing from the outset. If these changes are
                successfully implemented, they should help our
                acquisition programs avoid future cost overruns,
                schedule delays, and performance problems.”
                –Senator Carl Levin, Chairman, Senate Armed Services
                Committee

                “The Weapon System Acquisition Reform Act of
                2009 is an important step in efforts to reform the
                defense acquisition process. This legislation is
                needed to focus acquisition and procurement on
                emphasizing systems engineering; more effective
                upfront planning and management of technology
                risk; and growing the acquisition workforce to meet
                program objectives.”
                –Senator John McCain, Ranking Member, Senate
                Armed Services Committee

                                                                      8
       P. L. 111-23: Weapon Systems Acquisition
       Reform Act (WSARA) – May 2009

Key Elements of Legislation:
Organizational
 • Establishes Directorates for SE and DT&E as principal advisors
 • Joint tracking of component and MDAP progress against plans and
   measurable criteria with annual reporting to Congress
 • Independent cost estimation and cost analysis (Director, CAPE)
 • Provide adequate trained staff for SE and development planning
 • Conduct MDAP performance assessments and root cause analysis
                                                                                                                     Delete this
Role of SE across program lifecycle                                                                                  box (from
 • Developmental planning, lifecycle management, sustainability                                                      the slide
                                                                                                                     master)
 • Ensure reliability, availability, maintainability (RAM)                                                           after
                                                                                                                     creating
Mandates                                                                                                             all of your
 •    Measurable performance criteria in SE/DT&E plans                                                               slides.
                                                                                                                     Stay
 •    Competitive prototypes for MDAPs; prime make/buy analyses                                                      inside this
                                                                                                                     “safe
 •    System PDR before MS B, with formal MDA assessment                                                             area” for
 •    Assessment of technical maturity and integration risk of critical                                              correct
                                                                                                                     display.
      technologies during Technology Development (TD)
                                                                          http://thomas.loc.gov/cgi-bin/query/z?c111:S.454.enr
Technical and cost oversight
 • Independent estimates, Problem Assessment Root Cause Analysis
 • Technical analysis of cost/schedule breaches; presumed termination

     Addressing SE Issues in the Defense Industry                                                               9
     July 2009
                                                    (Version 1.0)
                                                                                                                     19-
Systems Engineering: It’s the Law! -10   19-Jul-12
Engineering Week 2010
                                  Key Acquisition Business Process Changes

  Increased Emphasis on Milestone A
    • Mandatory for MDAPs with Technology
      Development Programs
    • Likely for Most Programs                                                                     X
                                                                                When PDR is Conducted after Milestone B an MDA
                                                                                Post-PDR Assessment is Required



                             Preliminary Design Review (PDR) before Milestone
                             B to Enhance Understanding of Derived
                             Requirements and                                                     Post-Critical Design Review Assessment –
                             Improve Cost Estimation                                              A Mandatory Decision Point to Review Progress



                         MS A                               MS B                                                 MS C

                                                                                                                           FRP DR
   ICD

   MDD
            Materiel
            Solution
            Analysis
                                     Technology
                                     Development
                                                      PDR



                                  Competitive Prototyping
                                                            CDD
                                                                   XPDR
                                                                       P-PDRA       PCDRA
                                                                                              Engineering &
                                                                                              Manufacturing
                                                                                              Development

                                                                        Re-structured “EMD” Phase
                                                                                                                 CPD
                                                                                                                         Production &
                                                                                                                         Deployment
                                                                                                                                            Operations &
                                                                                                                                              Support


                                               Effective Contracting via Pre-Award Peer Reviews


Materiel Development Decision –                                                                      Enhanced Emphasis on:
Mandatory Process Entry Point
                                                                                                     • Technology Maturity
                                                                                                     • Systems Engineering
               Competitive Prototyping          Configuration Steering Boards
                                                                                                     • Integrated Testing and Test Planning
                                                Established to Stabilize Requirements                • Manufacturing and Producibility
                                                                                                     • Logistics and Sustainment Planning
                                                                                                                                                           11
Systems Engineering: It’s the Law! -12   19-Jul-12
INCOSE Space Coast Chapter
Systems Engineering: It’s the Law! -13   19-Jul-12
INCOSE Space Coast Chapter
                                       Technology and Manufacturing Readiness
                                                                                                                                      IOC                 FOC
                                   A                                    B                                          C
                   Materiel                TECHNOLOGY                              Engineering &                        PRODUCTION &                   OPERATIONS &
                   Solution                DEVELOPMENT                             Manufacturing                         DEPLOYMENT                      SUPPORT
                   Analysis                                                        Development

                  Materiel                                                                                                            FRP
                                                                                                   Post CDR
                  Development                                                                                                         Decision
                                                                                                   Assessment
                  Decision                                                                                                            Review



   TRLs 1-3            TRL 4             TRL 5             TRL 6                           TRL 7                         TRL 8          TRL 9             Technology
   Analytical/
  Experimental
                   Component
                      And/or
                                       Component
                                          And/or
                                                           System/
                                                         Subsystem
                                                                                           System
                                                                                          Prototype
                                                                                                                          Actual
                                                                                                                          System
                                                                                                                                         Actual
                                                                                                                                        System
                                                                                                                                                           Readiness
    Critical
   Function/
                   Breadboard
                    Validation
                                       Breadboard
                                        Validation
                                                          Model or
                                                          Prototype
                                                                                         Demonstrate
                                                                                              d
                                                                                                                        Completed
                                                                                                                         Qualified
                                                                                                                                       “Mission
                                                                                                                                       Proven”
                                                                                                                                                             Levels
  Characteristi        In a                In a         Demonstrate                         In an                        Through       Through          Defense Acquisition
       c            Laboratory           Relevant             d                          Operational                     Test and
    Proof of       Environment         Environment      In a Relevant                    Environment                   Demonstratio
                                                                                                                                      Successful
                                                                                                                                      Operations
                                                                                                                                                            Guidebook
    Concept                                             Environment                                                         n                               para. 10.5.2

                       MRL 4             MRL 5              MRL 6                 MRL 7                MRL 8             MRL 9         MRL 10
   MRLs 1-3
 Manufacturing Capability to produce   Capability to     Capability to Capability to Produce        Pilot Line        Low Rate        Full Rate        Manufacturing
  Feasibility
  Assessed.
                Technology In Lab
                  Environment.
                                        Produce
                                        Prototype
                                                       Produce System/ Systems, Subsystems
                                                          Subsystem     Or Components in a
                                                                                                    Capability
                                                                                                  Demonstrated.
                                                                                                                     Production      Production
                                                                                                                   Demonstrated. Demonstrated.           Readiness
   Concepts
   defined/
               Manufacturing Risks
                    Identified
                                       Components         Prototypes        Production
                                                                          Representative
                                                                                                  Ready for LRIP    Capability In Lean Production
                                                                                                                    Place for FRP Practices In Place       Levels
  developed                                                                Environment
                   Manufacturing       Cost Model       Detailed Cost       Cost Model Updated     Engineering       LRIP Cost         FRP Unit        Draft MRA Deskbook
                    Cost Drivers       Constructed        Analysis           To System Level       Cost Model        Goals Met        Cost Goals
                     Identified                           Complete          Unit Cost Reduction     Validated      Learning Curve        Met                 May 2008
                                                                             Efforts Underway                        Validated



   Section 2366b of Title 10, United States Code, requires certification that: the technology in the program has
                      been demonstrated in a relevant environment to enter Milestone B. [TRL 6]
Version 1.1, 9-14-09                                                                                                                                     DoD DSS, vg 14
     DoD Strategic Management Plan (7/09)
     Key acquisition-related excerpts

Goals                                             Measures                                      Key Initiatives
Improve acquisition processes and                 • # of months to IOC                          • Acquire systems through evolutionary
execution to support warfighter reqts                                                             acquisition
Increase use of fixed price contracts             • % of contracts at MS B that are FP          • Milestone decision reviews
                                                  • % of contracts at MS C that are FP
Increase % of MDAPs initiated with low            • # of MDAPs initiated in GFY                 • Competitive prototyping
technical risk                                    • % of MDAPs with technologies                • PDR before MS B
                                                    demonstrated in a relevant                  • Independent Technical Readiness
                                                    environment (i.e., TRL 6)                     Assessments (TRA)
Assess programs as directed in                    • % of Problem Assessment Root                • Root cause analysis                        Delete this
WSARA                                               Cause Analysis (PARCA) offices                                                           box (from
                                                  • % of programs assessed                                                                   the slide
                                                                                                                                             master)
Ensure supportability, maintainability            • % programs meeting DoD                      •   Establish sustainment metrics            after
and costs are considered in lifecycle               sustainment metrics                             reporting (availability, reliability, TOC)
                                                                                                                                             creating
                                                                                                •   Implement recommended next-gen all of your
                                                                                                                                             slides.
                                                                                                    sustainment strategies
                                                                                                                                             Stay
Focus R&D to address warfighting                  • % of completing demonstration               •   Track programs transitioning with CY     inside this
                                                                                                                                             “safe
requirements                                        programs transitioned per year              •   Review RDT&E funding for transition
                                                                                                                                             area” for
Sponsor technology leadership                     • # of 2-day dialogues with academia          •   Understand investment drivers and correct
                                                                                                                                             display.
strategy outreach to industry, academia             technology leaders                              strategies to sustain tech leadership
Right-shape and re-balance the                    • % certification positions filled (annual)   • Grow the acquisition workforce
acquisition workforce - goal to grow by           • Annual increase in end-strength             • Achieve 10,000 acquisition positions
20,000 positions by FY2015                        • # contractor work-yr equiv in-sourced         through in-sourcing

http://dcmo.defense.gov/documents/2009SMP.pdf
   Addressing SE Issues in the Defense Industry                                                                                          15
   July 2009
                                                                  (Version 1.0)
     What’s Next?

                                                Areas of emphasis:
                                                   • Defense Strategy
                                                      - Defense objectives, emerging threats
                                                   • Rebalancing the Force
                                                      - Counterinsurgency, counterterrorism, WMD, cyber
                                                      - U.S. force structure: sizing, shaping, evolution
                                                   • Defense workforce
                                                      - Supporting troops, deployment                       Delete this
                                                      - Recruiting, retention, development                  box (from
                                                                                                            the slide
                                                   • Strengthening relationships                            master)
                                                                                                            after
                                                      - U.S. defense posture, interagency, abroad           creating
                                                   • Reforming How We Do Business                           all of your
                                                                                                            slides.
                                                      - Rapid acquisition, security, how we buy, exports    Stay
                                                                                                            inside this
                                                      - Strengthening the industrial base                   “safe
                                                      - Strategic approach to climate and energy            area” for
                                                                                                            correct
                                                   • Defense Risk Management Framework                      display.

                                                      - Operational, force management, institutional,
http://www.defense.gov/QDR/                             future challenges, strategic, military, political

 A U.S. force “prepared to conduct a wide variety of missions under a range of different circumstances.”
 Addressing SE Issues in the Defense Industry                                                         16
 July 2009
                                                      (Version 1.0)
Summary

• DODI 5000.2 and WSARA are changing the game
         –     Early life cycle planning (SEP)
         –     Adherence to effective SE practices
         –     Independent cost estimates
         –     Competitive prototyping
         –     Managed technology risks (TRL)
         –     Increased emphasis on reliability and supportability (RAM)
         –     Congressional reporting and oversight
         –     More fixed priced contracts and evolutionary acquisition likely
         –     Revitalization of DoD SE and acquisition workforce

• Questions?
         – Geoff Draper (gdraper@harris.com, 321-727-5617)


Systems Engineering: It’s the Law! -17
INCOSE Space Coast Chapter
                                         Backup




Systems Engineering: It’s the Law! -18
INCOSE Space Coast Chapter                                19-
                                                  April 2010
                                                                    Delete this
                                                                    box (from
                                                                    the slide
                                                                    master)
                                                                    after
                                                                    creating
                                                                    all of your
                                                                    slides.
                                                                    Stay
                                                                    inside this
                                                                    “safe
                                                                    area” for
                                                                    correct
                                                                    display.




Addressing SE Issues in the Defense Industry                   19
July 2009
                                               (Version 1.0)
  Addressing SE Issues –
  Key NDIA Task Groups and Milestones
               •Government Studies/Reports              •Program Support Reviews (PSRs)         •Working Groups
                (GAO, NRC, DCMA, etc.)                  •Congressional oversight                •Conferences
               •Reviews (QDR, etc.)                      (Nunn-McCurdy, McCain-Levin, …)        •Forums




        2003                    2006                          2007                2008                      2009

                                                                                                                             Delete this
                                                                                                                             box (from
    •NDIA Top 5           •NDIA Top 5 SE Issues            •SE Effectiveness    •Systemic Root           •Top SE/SW Issues Update
                                                                                                                             the slide
     SE Issues            •NDIA Top SW Issues               Survey               Cause Analysis          (planned)           master)
                                                                                                                             after
                          •Defense Software                                      (SRCA) [draft]          •Industrial Committee
                                                                                                                             creating
                           Strategy Summit                                      •Industrial Committee     on SE (ICSE)       all of your
                                                                                 on Program Mgmt          (planned)          slides.
  Common program issues:                                                         (ICPM)                                      Stay
                                                                                                                             inside this
  -Poor program planning                 -Unrealistic estimates                                                              “safe
  -Unstable requirements                 -Immature technology                   •SE of Tactical Air
                                                                                                                             area” for
  -Not following SE processes            -Insufficient SE expertise              Launched Systems                            correct
  -Ineffective reviews                   -Poor system reliability                (Gulf Coast Chapter)                        display.
                                                                                 (Air Armament Center)


              There is no shortage of sources confirming we have issues –
           How do government and industry collaborate to act upon solutions?
Addressing SE Issues in the Defense Industry                                                                               20
July 2009
                                                                (Version 1.0)
    NRC Study –
    Pre-MS A and Early-Phase SE
 Findings                                            Recommendations
 Attention to a few critical systems engineering     Require that Milestones A and B be treated as critical
 processes and functions particularly during         milestones in every acquisition program and that … the
 preparation for Milestones A and B is essential     “Pre-Milestone A/B Checklist” … be used to judge
 to ensuring that Air Force acquisition programs     successful completion.
 deliver products on time and on budget.
 The creation of a robust systems engineering        Assess needs for officers and civilians in the systems
 process is critically dependent on having           engineering field and evaluate whether either internal
 experienced systems engineers with adequate         training programs … or external organizations are able to
 knowledge of the domain relevant to a               produce the required quality and quantity of systems            Delete this
 contemplated program.                               engineers and systems engineering skills. …                     box (from
                                                                                                                     the slide
 The government, FFRDCs, and industry all have       Decisions made prior to Milestone A should be supported         master)
 important roles to play throughout the              by a rigorous systems analysis and systems engineering          after
 acquisition life cycle.… Source selection for       process involving teams of users, acquirers, and industry       creating
                                                                                                                     all of your
 system development and demonstration should         representatives.                                                slides.
 not be made until after the work associated with                                                                    Stay
 Milestones A and B is complete.                                                                                     inside this
                                                                                                                     “safe
                                                                                                                     area” for
 The Air Force used to have a development            A development planning function should be established           correct
                                                                                                                     display.
 planning organization that applied pre-Milestone    in the military departments to coordinate the concept
 A systems engineering processes to a number of      development and refinement phase
 successful programs, but that organization was      of all acquisition programs to ensure that the capabilities
 allowed to lapse.                                   … as a whole are considered and that unifying strategies
                                                     such as … interoperability are addressed.

  Addressing SE Issues in the Defense Industry                                                                  21
Reference: Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air
  July 2009
Force Acquisition. http://www.nap.edu/catalog/12065.html(Version 1.0)
    NDIA Top 5 SE Issues
    NDIA SE Division (July 2006)
Provides an update and status from a previous task group report conducted in 2003.

Issues                                                        Recommendations
1. Key systems engineering practices known to be              Ensure institutionalization of effective SE practices into
effective are not consistently applied across all phases      program planning and execution
of the program life cycle.
2. Insufficient systems engineering is applied early in the Integrate engineering planning within the acquisition life
program life cycle, compromising the foundation for         cycle to ensure adequate time and effort for SE early in
initial requirements and architecture development.          the program life cycle
                                                                                                                       Delete this
3. Requirements are not always well-managed, including        Emphasize the application of SE practices and         box (from
the effective translation from capabilities statements into                                                         the
                                                              resources to the capability definition process to addressslide
                                                                                                                    master)
executable requirements to achieve successful                 warfighter needs and translation into executable
                                                                                                                    after
acquisition programs                                          programs.                                             creating
                                                                                                                       all of your
4. The quantity and quality of systems engineering            Grow SE expertise through training, career incentives,
                                                                                                                   slides.
expertise is insufficient to meet the demands of the          and broadening “systems thinking” into other         Stay
government and the defense industry.                          disciplines.                                         inside this
                                                                                                                       “safe
5. Collaborative environments, including SE tools, are        Strengthen and clarify policy and guidance regarding area” for
inadequate to effectively execute SE at the joint             use of collaborative environments, models, simulations,
                                                                                                                   correct
capability, system of systems (SoS), and system levels.       and other automated tools.                           display.




Reference: NDIA Top 5 Systems Engineering Issues Report. July 2006. NDIA Systems Engineering Division. [report, briefing]
http://www.ndia.org/Divisions/Divisions/SystemsEngineering/Pages/Past_Projects.aspx

  Addressing SE Issues in the Defense Industry                                                                    22
  July 2009
                                                       (Version 1.0)
      NDIA Top Software Issues
      NDIA SE Division (Sept 2006)
Issues                                                     Recommendations
1. The impact of requirements upon software is not         Enforce effective software requirements development and
consistently quantified and managed in development         management practices, including assessment of change
or sustainment.                                            impacts, for both the acquirer and the supplier
                                                           organizations.
2. Fundamental system engineering decisions are         Institutionalize the integration and participation of software
made without full participation of software engineering engineering in all system engineering activities.


3. Software life-cycle planning and management by          Establish a culture of quantitative planning and
acquirers and suppliers is ineffective.                    management, using proven processes with collaborative Delete this
                                                           decision-making across the software life cycle.       box (from
                                                                                                                            the slide
4. The quantity and quality of software engineering        Collaborate on innovative strategies to staff to appropriate  master)
expertise is insufficient to meet the demands of           levels, and to attract, develop, and retain qualified talent to
                                                                                                                         after
government and the defense industry.                       meet current and future software engineering needs in creating
                                                           government and industry.                                      all of your
5. Traditional software verification techniques are                                                                   slides.
                                                           Study current software verification practices in industry, and
costly and ineffective for dealing with the scale and      develop guidance and training to improve effectiveness in  Stay
complexity of modern systems.                              assuring product quality across the life cycle.            inside this
                                                                                                                   “safe
                                                                                                                   area” for
6. There is a failure to assure correct, predictable, safe,                                                        correct
                                                           Collaborate with industry to develop approaches, standards,
secure execution of complex software in distributed        and tools addressing system assurance issues throughout display.
environments.                                               the acquisition life cycle and supply chain.
7. Inadequate attention is given to total lifecycle issues Improve and expand guidelines for addressing total lifecycle
for COTS/NDI impacts on lifecycle cost and risk.           COTS/NDI issues.
Reference: NDIA Top Software Engineering Issues Report. Sep 2006. NDIA Systems Engineering Division. [report, briefing]
http://www.ndia.org/Divisions/Divisions/SystemsEngineering/Pages/Past_Projects.aspx
    Addressing SE Issues in the Defense Industry                                                                       23
    July 2009
                                                        (Version 1.0)
     Systemic Root Cause Analysis (SRCA)
     - Conducted 2007-2008, publication pending
                     •SRCA workshops and task group                   •95 preliminary recommendations
                     •Based on PSR findings (44 programs)             •48 systemic issues
                     •Tagged to core/systemic root causes             •3 Recommendation Areas


Recommendation                Problem Statement                         Industry Actions*
Area

Implement                     Acquisition strategies and plans are      •Require the government, prior to RFP release, to
Achievable                    incomplete, ineffective and               provide industry with govt. expectations and common
Acquisition Strategy          unrealistic, resulting in                 understanding for IMP/IMS/risk management/business
and Planning                  unachievable program expectations         rhythms                                           Delete this
                                                                                                                           box (from
                                                                                                                           the slide
                                                                                                                           master)
Enhance Gate                  Lack of timely process and                •Define criteria for trigger conditions
                                                                                                                           after
Review Process                adequately defined and enforceable                                                           creating
                              criteria to assess program maturity                                                          all of your
                              at milestones and linkage to                                                                 slides.
                              technical reviews                                                                            Stay
                                                                                                                           inside this
                                                                                                                           “safe
Enhance Staff                 Staffing shortfalls (numbers, skills,     •Develop and validate a representative staffing model for
                                                                                                                           area”
Capabilities                  and experience) lead to adverse           for DoD based on industry that can be applied to the
                                                                                                                           correct
                              acquisition consequences                  govt. given the current OSD acquisition guidance display.
                              specifically in the areas of      •Develop a workload analysis to estimate the numbers
                              requirements, planning, execution and expertise needed in the acquisition workforce
                              and expectations                  •Broaden expertise to enhance cross-functional and
                                                                domain knowledge and skills
* Reference SRCA briefing for additional details and recommended government actions.

   Addressing SE Issues in the Defense Industry                                                                           24
   July 2009
                                                              (Version 1.0)
     SE of Tactical Air Launched Weapons
     NDIA Gulf Coast Chapter (2008)

SE Deficiencies                       Root Causes
Program Structure and                 •Insufficient Maturity Of Design At Critical Decision Points
Control                               •Insufficient Testing And Analysis Planned To Achieve Maturity
                                      •Late Integration Of Production Critical Processes And Controls
                                      •Program Funding Profiles Not Structured For Improved Practices
Requirements And                      •Lack Of Service Use Profile Leaves Interpretive Requirements
Verification                          •Insufficient Mapping Of Requirements To Design
                                      •Ineffective Maturation And Verification Planning
Design Best Practices                 •Inadequate Design Analysis
                                          - Fault Tree Analysis On All Subsystems During Design                 Delete this
                                          - Single Point Failure Analysis During Design                         box (from
                                      •Inadequate Maturation Analysis And Testing                               the slide
                                                                                                                master)
                                          - COTS Integration                                                    after
                                          - Design Margin And Sensitivity Development                           creating
                                          - Critical Manufacture And Assembly Process ID / Control              all of your
                                      •Use Of “Production Representative” Configuration For Verification        slides.
                                                                                                                Stay
Risk Identification And               •Inadequate Relationship To “Knowledge” Of Design                         inside this
Management                                                                                                      “safe
                                                                                                                area” for
Supply Chain Practices                •Inconsistent Approaches To Design Characterization                       correct
                                                                                                                display.


Reference: “Systems Engineering of Tactical Air-Launched Weapons: An Industry Examination.”
Air Armament Symposium, 2008. [briefing]
http://www.ndiagulfcoast.com/events/archive/34th_Symposium/34_Day1/11_SysEngNDIASymposiumBriefOct208.pdf

  Addressing SE Issues in the Defense Industry                                                             25
  July 2009
                                                             (Version 1.0)
  A Survey of Systems Engineering
  Effectiveness (2007)
 Joint NDIA/SEI survey of 46 programs
 correlating the effectiveness of SE
 processes with program performance.
 SE processes most strongly correlated
 with better program performance:
  • Architecture
  • Trade Studies
  • Technical Solution
  • IPT Capability                                                                                                                                                 Delete this
                                                                                                                                                                   box (from
  • Requirements Development and                                                                                                                                   the slide
                                                                                                                                                                   master)
    Management                                                                                                                                                     after
                                                                                     Relationship of SE Processes to Program Performance                           creating
 Report:                                                                              Architecture                                                           40%
                                                                                                                                                                   all of your
                                                                                                                                                                   slides.
 http://www.sei.cmu.edu/publications/                                               Trade Studies
                                                                               Technical Solution
                                                                                                                                                        37%
                                                                                                                                                        36%        Stay
 documents/08.reports/08sr034.html                                                  IPT Capability                                                    34%          inside this
                                                                                                                                                                   “safe
                                                           SE Capability
                                                                             Reqts Devel & Mgmt                                                       33%
                                                                                        Validation                                              28%                area” for
                                                                                       Risk Mgmt                                                28%                correct
                                                                                      Verification                                            25%
                                                                                                                                                                   display.
         Projects with better Systems                                          Product Integration                                        21%
                                                                                     Config Mgmt                                  13%
       Engineering Capabilities deliver                                          Project Planning                                 13%
                                                                           Project Monitor/Control
          better Project Performance                                                                 -13%

                                                                                                 -20%       -10%   0%       10%         20%     30%         40%    50%
        (cost, schedule, functionality)                                                                            Gamma (strength of relationship)




Addressing SE Issues in the Defense Industry                                                                                                                 26
July 2009
                                               (Version 1.0)
                                          Defense Acquisition System Weighted Expenditures
            DoDI 5000.02 Perspective
                                                                                  Program
                            A                                                 B   Initiation                     C            IOC            FOC
   Materiel                                                                   Engineering and
                                  Technology                                                                         Production &         Operations &
   Solution                                                                   Manufacturing Development
   Analysis                       Development                                                                        Deployment           Support
           Materiel                                                                                                            FRP
           Development                                                                           Post-CDR                      Decision
           Decision                                                                              Assessment                    Review
                                                                                                                 LRIP/IOT&E

            Pre-Systems Acquisition                                                        Systems Acquisition                             Sustainment


            Warfighter and Sustainment Organization Perspective
                            (Program
                            Initiation)
              A         B                       C
  Materiel    Technology    Engineering and              Production &
  Solution    Development   Manufacturing                Deployment
  Analysis                  Development


Materiel
Development                                         LRIP/IOT&E
                                                                   FR5P
                                                                   Decision
                                                                   Review
                                                                                           Operations & Support
Decision                             Post-CDR
                                     Assessment

Pre-Systems                          Systems Acquisition
Acquisition
                                                                                                        Sustainment
                  20-35%                                                                                 65-80%
                                                                                                30+ YEARS
                                                     Nominal Life Cycle Cost Distribution                                                                27

								
To top