Space Systems O_S Costs Estimation Practices and

Document Sample
Space Systems O_S Costs Estimation Practices and Powered By Docstoc
					  Space and Missile O&S Cost
Estimation: Current Practices and
      Potential Directions

         Date: Feb 2012

        Bruce Thompson
         Lisa Colabella
           Brian Kim
          Art Sandoval
           Brett Smith
    O&S Cost Estimation- Agenda

•   Background
•   Research Approach
•   Preliminary Observations
•   Way Forward

                                         Concepts: O&S, O&M

• Definition of Operations & Support (O&S) Costs
   • Cost of manpower (organic & contractor), equipment, supplies,
     software, and services associated with operating, modifying,
     maintaining, supplying, training, and supporting a system

• O&M versus O&S
   • In the context of defense budget appropriations, Operations &
     Maintenance (O&M) is narrower in scope than O&S
       • Although O&S is often funded by O&M (3400) appropriation, it can be
         funded by other appropriations

   • In other contexts, O&M is often treated as interchangeable with O&S

                          From Defense Acquisition Guidebook (2011)
                 page 7/66 & Kolesar (2003)
                                  Space & Missile O&S Costs were Historically
                                                    Not a Significant Concern

  1999 report suggested O&S was a relatively small fraction of
  lifecycle costs for space systems
 System Type                                                    R&D                          Investment   O&S/ Disposal

 Space                                                          18%                          66%          16%
 Fixed Wing Aircraft                                            20%                          39%          41%
 Rotary Wing Aircraft                                           15%                          52%          33%
 Missiles                                                       27%                          33%          39%
 Electronics                                                    22%                          43%          35%
 Ships (note 1)                                                 1%                           31%          68%
 Surface Vehicles                                               9%                           37%          54%
 AIS                                                                                         30%          70%

 Notes: (1) Most ship design costs are included in production cost of lead ship of a class

Source: Status of DoD’s Capability to Estimate the Cost of Weapon
Systems: 1999 Update (IDA report cited in DAU Business Management
Reference Book (2010, page 7)
       Space & Missile O&S Costs Now Receiving More Attention

• Reasons for more attention:
   • Budget for operating and sustaining space programs is increasing as
     programs transition from development to sustainment (Johnson,
     SMC/SL, 2011)
   • Space O&S costs can be more substantial than early studies suggest
   • Tighter defense budget prompting greater efforts to reduce costs of
     all types
   • Heavy reliance on satellites for data gathering makes support of
     those satellites critical
• Examples of greater attention to space O&S:
   • 2009: SMC/FM and SMC/SL conducted “deep dive” into O&M
   • 2009: AFPEO/SP issued O&M Cost Reduction memo
   • 2011: Launch Ranges, AF Satellite Control Network, Milsatcom,
     SBIRS, and GPS have dedicated O&S cost analysts in Space
     Logistics Group at Peterson AFB
                Purpose of Study and Potential Contributions

• Purpose: To examine current approaches to space O&S cost
  estimation and identify challenges and opportunities for

• Intended Contributions
   • Provide information and guidance to cost analysts preparing to
     develop space O&S cost estimates
   • Encourage methodological improvements that increase the accuracy
     of O&S estimates
   • Support Better Buying Power initiative “Mandate Affordability as a
       • Promote better tracking of O&S costs (“quantified goals for sustainment”)
       • Provide information decision-makers need in order to consider
         sustainment along with acquisition costs
   • Support SAF/FM Strategic Plan goal 4.3.3 “reduce costs through
     standardization and continuous process improvement”
    O&S Cost Estimation- Agenda

•   Background
•   Research Approach
•   Preliminary Observations
•   Way Forward

• Literature review
   •   Cost handbooks
   •   Briefings on cost methodologies
   •   Policy and guidance documents
   •   Articles on commercial & military cost estimating practices
• Interviews with subject matter experts (SMEs) in the space
  cost estimating community
• Review of cost resources, databases, & estimating tools-- e.g.,
   • AFTOC
   • AFI 65-503 (Cost Factors)
   • CAFDEx
   • Historical life cycle cost estimates in ACEIT
   • NASA Cost Estimating Web Site

            Key Government Organizations in Space and Missile
                            O&S Cost Estimating Community

                                  DoD                                           Non-DoD

                 Air Force                            OSD                       NASA Cost
      AFSPC                  SAF/FM       OSD-CAPE              NRO             (including
                                           (formerly                               Cost
                                          PAE & CAIG)                            Analysis
      SMC                 SAF/FMC                                                Division)

FMC   Program      SLD/      AFCAA      *SLD/FM manages O&S for SMC programs and estimates
                                        O&S costs for management purposes.
      Offices*     FM*
                               PM       Program Offices (Directorates) sometimes include SLD/FM
                              AFTOC     O&S estimates in the Program Office Estimates, but often
                                        the program offices use their own O&S cost estimates for
                                        Life Cycle Costing.

    O&S Cost Estimation- Agenda

•   Background
•   Research Approach
•   Preliminary Observations
•   Way Forward

                         What are the Key O&S Costs for SMC Space Systems?

•        Ground station maintenance, primarily including
                    • Software maintenance
                    • Facilities maintenance
                    • Equipment (e.g., antenna, computer) maintenance

•        On-orbit support, primarily including
                    • Space vehicle (bus and payload) commanding
                    • Data evaluation and trending             Ground
                    • Anomaly resolution                       terminal

                                                                         Ground Ops
                                                                         & Processing

                                                                        MCC, POCC, SOCC


    Adapted from (Miller, Keesee, Jilla, 2003);
    Larson & Wertz, Space Mission Analysis & Design; and interview notes; diagram from Du, Yu. “A Satellite Ground Station Control
    System”, Informatics and Mathematical Modeling, Technical University of Denmark (2005:36)                                            11
                Summary of Preliminary Observations about
                   Space and Missile O&S Cost Estimation

• Organizations in space community use diverse work breakdown
  structures for O&S cost estimation
   •   Standard WBS (MIL-STD 881C) is used for Pre-Systems Acquisition and
       Systems Acquisition phases, but not for Sustainment (O&S) Phase

• Space community faces impediments to precise calculation of O&S
  WBS costs for new program

• No single, centralized database for space O&S information
   •   O&S cost estimates typically rely on multiple data sources

• When early O&S cost estimates are done, they tend to be rough
  order-of-magnitude estimates based on anticipated headcounts,
  cost-to-cost factors, and/or rules of thumb
   •   Specific cost-to-cost factors used differ among programs

• In DoD space community, O&S cost estimation tools are primarily
  used for software maintenance portions of estimates
            Diverse Space O&S Work Breakdown Structures
•   OSD O&S Cost Element Structure – used by a subset of programs at SMC
         1992 Version                         2007 Version
         1. Mission Personnel                 1. Unit-level Manpower
         2. Unit-level Consumption            2. Unit Operations
         3. Intermediate Maintenance
         4. Depot-level Maintenance           3. Maintenance
         5. Contractor Support
         6. Sustaining Support                4. Sustaining Support
                                              5. Continuing System Improvements
         7. Indirect Support                  6. Indirect Support

•   SMC Program-Unique O&S Work Breakdown Structures
•   AFCAA O&S Work Breakdown Structure (draft)
     •     Keeps high-level OSD categories; adds subcategories (within Sustaining
           Support and Continuing System Improvements) to tailor to space systems
•   NRO O&S Work Breakdown Structure (draft)
     •     Detailed WBS based on Mission Operations Directorate (MOD) elements; four
           tiers of categories; highest tier has O&M SEITPM, Operations, Maintenance,
           and Recap/Tech Refresh; lowest tier has up to 12 categories           13
                  Potential Benefits & Challenges of Moving Toward
                                         Standard Space O&S WBS

    • Potential Benefits
          • Facilitates comparison and reconciliation of independent estimates
          • Facilitates interpretation of O&S cost estimates by other
          • Supports data archiving, CER development, future cost estimates
                • Reduces time and money spent collecting and normalizing data

    • Challenges
          • Uniqueness of space programs
          • Availability and nature of O&S data corresponding to WBS
          • Existing O&S work breakdown structures have strengths and
            weaknesses; need criteria for selecting standard

Sources: interviews, GAO (2009)                                             14
                                 Rough Comparison of Alternative
                        Work Breakdown Structures for Space O&S
                                   OSD 2007                   SMC                      NRO                 AFCAA
                   O&S WBS           CES                   (SLD/FM)
 Desired WBS                                             Program-
 feature*                                               Unique WBS
 Easily matched to OSD            Same Structure        Not typically aligned     Not easily           Uses OSD
                                                        with OSD Cost             aligned to OSD       categories but adds
 structure needed for                                   Element Structure         structure            some sub-
 SAR, DAES, & AFTOC?                                                                                   categories; maps
                                                                                                       well to structure

 Tailored to Space                OSD O&M               Customized to each        Designed to be       Categories added
                                  categories are        program                   applicable to        to tailor the OSD
 Systems?                         “designed for                                   multiple space       structure to space
                                  [equipment in]                                  programs
                                  units like fighter
                                  wings” rather than
                                  for space systems

 Balances need for                Some elements         Programs determine        Highly detailed;     Adds some detail to
                                  may lack sufficient   level of detail           however, level of    OSD structure to
 conciseness & need for           detail/ clarity for   needed for estimate       detail on par with   clarify and facilitate
 comprehensiveness?               space systems                                   MIL-STD 881C.        application to
                                                                                                       space systems

 Readily available                2007 Operating        WBS dictionary not        NRO CAIG has         Much of 2007 OSD
                                  and Support Cost      readily available         O&M WBS              CAIG O&S Guide is
 documentation (manual)?          Estimating Guide                                manual               applicable

*May be appropriate to consider additional criteria
      Available                        In Progress                              Not Yet Available
           When Available, Space O&S Data are Decentralized--
                              Found in Multiple Data Sources

• Cost Analysis Requirements Document (CARD)
   • Manpower Estimate Requirement (MER) can be a source of O&S
     headcounts – but may not be current
   • Most program CARDs lack sufficient information for space O&S
       • An exception: AEHF has dedicated O&S CARD

• Pay rates often obtained from AFI 65-503
   • Table A19-2 – Military Pay Rates (annual composite for officers in BY;
     annual composite for enlisted in BY)
   • Table A 27-1 – Civilian Pay Rate (overall average for BY)
   • Table AF 56-1 – Installation Support Non-Pay Cost Factors (AF
     Composite, $ per authorized person)
• Some programs use organic software maintenance pay rate based
  on OO-ALC (Ogden Air Logistics Center) RGC (Repair Group
  Composite) bid rate
          When Available, Space O&S Data are Decentralized--
                             Found in Multiple Data Sources

• Centralized Access for Data Exchange (CAFDEx) used by
  SLD/FM cost analysts during sustainment, but not typically for
  early life cycle cost estimates
   • Supports enterprise logistics requirements determination process
       • Within CAFDEx, logistics requirements are developed, defined,
         validated, prioritized, & published (Air Force Materiel Command, 2010)
   • Provides information about methodologies and assumptions

• Other sources of historical data
   • Federal Procurement Data System (FPDS)
       • For historical predecessor O&S Data
   • Integrated Maintenance Data System (IMDS)
       • For program sustainment metrics (e.g., MTBF, depot-level maintenance
         actions for ground segment)

        AFTOC Serves as Centralized Data Source for Aircraft
                  but is Not Yet Widely Used for Spacecraft

• Air Force Total Ownership Cost (AFTOC) = VAMOSC
  information system containing historical O&S cost data
   • Up to 15 years of historical space and missile O&M costs,
     categorized into 1992 and 2007 OSD Cost Element Structures
       • Space programs included vary by year (from 1996 – 2011)
       • As of 2011, over 20 space programs in AFTOC, including EELV, GPS,
         MILSATCOM, SBIRS, and others

   • Not yet widely used for space O&S
       • As of mid-2011, total of 608 AFMC users vs. 26 AFSPC users
       • Data not sufficiently specific for some space & missile systems

   • Further refinement of data may improve utility to space cost
       • With historical RCCC lists from AFSPC installations, data could be
         further delineated
       • A caveat is that AFTOC captures Air Force expenditures, which may not
         cover the full program (if program is joint)
                   Space O&S Costs Often Estimated Based on
                                   Approximate Headcounts

   • Examples of equations incorporating headcounts based on CARD,
     program office judgment, or prior contract:
         • Contractor maintenance cost
           = (# FTEs) x (annual labor hours/FTE) x (CLS labor rate $/hr) x (CLS

         • Organic maintenance cost
             = [(# officers)(annual salary) + (# enlisted)(annual salary) +
               (# civilians)(annual salary)] x (O&S years)

         • Installation support cost
             = (annual Installation Support Non-Pay Cost per authorization*) x
               (# organic maintenance personnel) x (O&S years)
         *recall that Installation Support Non-Pay Cost Factor is in AFI 65-503 Table A56-1

CLS = Contractor Logistics Support

Sources: CAFDEx; Fox, Brancato, Alkire (2008:28); historical ACEIT estimates                  19
                          Space Programs Use a Variety of Cost-to-Cost
                                         Factors and Rules of Thumb

•   ACEIT instructor and cost estimator for an SMC program office:
     •   Annual software maintenance cost ≈ 10% of software devel cost

•   Borgoltz, Hagan, Lennox, & Zuchowski (2001); Wertz & Larson (1999) :
     •   Annual total O&S cost ≈
          (10%)(software + equipment + facilities development cost) +
          (# contract maintainers)(contract labor cost per staff-year) +
          (# government maintainers)(government labor cost per staff-year)

•   Aerospace (Hovden, 2011):
     •   Cost-to-cost factors for calculating logistics, hardware, sustaining engineering,
         modifications, and facilities O&S as a percentage of Development & Construction
     [Aerospace cost-to-cost factors are sensitive – used by SMC analysts but not releasable.]

•   NASA (from “O&M NASA Facilities” briefing by Glenn Butts, 31 May 2011, slides 10 & 11):
     •   “Facility maintenance costs run 2-4% of facility [current replacement value]”
     •   Caveat: “Many, perhaps most CRVs are incorrect”

                      In DoD Space Community, O&S Cost Estimation Tools
                       are Primarily Used for Software Maintenance Costing

• Key tools used for space O&S estimates
      • SEER-SEM
      • PRICE TRUE PLANNING (Price True S)
      • COCOMO II
• Prior studies (Madachy & Boehm 2008; Lum et al. 2001) comparing these
  models found that
      • Each can be used effectively if parameters are calibrated properly
      • Best to use at least 2 of the tools to estimate costs
• Other O&S estimating tools –e.g., Mission Operations Cost Model
  (MOCM)– are available but not prevalent in DoD space community
      • NASA (2008) identifies tools (of varying resolution) in their Cost Model
      •    Price & Coolahan (2011) survey of O&S estimating tools observed that
           current tools have limited focus but noted potential for “super tool”
  - Price & Coolahan (2011) of Raytheon and Johns Hopkins: “Modeling of Life Cycle Operations & Support Costs: To What Degree is Commonality
  Achievable?” NDIA presentation.
  -Madachy R, Boehm B, “Comparative Analysis of COCOMO II, SEER-SEM and True-S Software Cost Models”, USC-CSSE-2008-816, 2008
  -Lum K, Powell J, Hihn J, “Validation of Spacecraft Software Cost Estimation Models for Flight and Ground Systems”, JPL Report, 2001
              Software Maintenance is Significant Part of
                        Space O&S, but Not Full Picture

                        O&S Cost Percentages by Cost Element
Space Systems
                                                                UNIT-LEVEL MANPOWER (e.g., labor for systems, mission
                                                                planning, and unit-level maintenance)
                                                                UNIT OPERATIONS (e.g., consumable operating materials such
                                                                as fuel and electricity)

                                                                MAINTENANCE (e.g., labor for maintenance other than unit-
                                                                level; materials for all levels of maintenance)

      66% : Sustaining Support &
      Continuing System                              5%         SUSTAINING SUPPORT (e.g., support equipment replacement;
      Improvements (including SW                                sustaining engineering support) & CONTINUING SYSTEM
      maintenance and mods)                                     IMPROVEMENTS (e.g., software maintenance)

66%                                                             INDIRECT SUPPORT (e.g., medical and PCS support for military
                                                                personnel; installation support)

          Sources: Adapted (to use 2007 OSD categories instead of 1992 categories) from Kolesar (2003),
          Ownership Costing (OSD CAIG briefing);
    O&S Cost Estimation- Agenda

•   Background
•   Research Approach
•   Preliminary Observations
•   Way Forward

             Opportunities for Improving Space O&S Cost Estimation

•   Pursue greater standardization in space O&S Work Breakdown Structures.
    Consider weighting/prioritizing multiple criteria when evaluating or
    developing possible standard:
     •   Easily matched to OSD structure needed for SAR, DAES, & AFTOC
     •   Tailored to space systems
     •   Balances need for conciseness and need for comprehensiveness
     •   Readily available documentation (WBS dictionary)
     •   Other potential criteria: feasibility of measurement, distinctness (vs. overlap) of
         elements, outcome/result-orientation

•   Strive for strong, centralized source of historical O&S data, assumptions,
    and inputs to estimates
     •   AFTOC can potentially be further refined with historical RCCC codes

•   Ensure that CARD has sufficient detail (e.g., accurate headcounts, types,
    changes over time, SLOC & ELOC, MTBCF for HW, risk ) for more precise
    O&S cost estimates
     •   Critical to have involvement of O&S cost analysis team in design decision-
         making process during product development phase (Hendricks, 2009)
        Opportunities for Improving Space O&S Cost Estimation

•   Conduct validation studies (and review prior validation studies) to
    identify preferred set of cost-to-cost factors
•   Ensure estimating tools are sufficiently comprehensive to capture
    multiple O&S cost categories, not only software maintenance
•   When possible, use multiple methods (e.g., more than one software cost
    estimating tool) to promote O&S cost realism and validation
         • “The key to arriving at sound estimates is to use a variety of
           methods and tools and then investigate the reasons [that]
           estimates provided by one might differ significantly from those
           provided by another.” (Madachy & Boehm, 2008: 35)

•   Pursue greater collaboration among commercial and government
    organizations concerned with space O&S cost estimation—for example,
    via O&S Cost IPT or Joint Space Cost Council (JSCC)


           Space Community Faces Impediments to Precise
           Calculation of O&S WBS Costs for New Programs

•   From NASA (2008) Cost Estimating Handbook:

    • “Historical data for O&S CER development non-existent or sparse”
    • “Operations concept(s) [e.g., CARD info on O&S] not established or
    • “Cost estimates dependent on activity levels that are not yet known”
    • “O&S teams not yet formed; hard to identify O&S discipline experts”
    • “Maintenance data (e.g., failure rates and repair times) subject to
      great uncertainty”
    • “Independent validation of models usually not possible until late in


Shared By: