Learning Center
Plans & pricing Sign in
Sign Out
Your Federal Quarterly Tax Payments are due April 15th Get Help Now >>



  • pg 1
									Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

                Systems Engineering Sizing
              in the Age of Acquisition Reform

   Ricardo Valerdi                     Michael Ernstoff                  Paul H. Mohlman
   Center for Software Engineering     Consultant                        The Aerospace Corporation
   University of Southern California   11940 Victoria Avenue             2350 E. El Segundo Blvd.
   Los Angeles, CA 90089               Los Angeles, CA 90066             El Segundo, CA 90245
   rvalerdi@sunset.usc.edu             m.ernstoff@attbi.com              Paul.H.Mohlman@aero.org

   Don Reifer                          Evin Stump
   Reifer Consultants, Inc             Galorath, Inc
   P.O. Box 4046                       100 N. Sepulveda #1801
   Torrance, CA 90501                  El Segundo, CA 90245
   dreifer@earthlink.net               estump@galorath.com

As organizations develop more complex systems, increased emphasis is being placed
on Systems Engineering (SE) to ensure that cost and schedule are within
budget. Correspondingly, the failure to adequately plan and fund the systems
engineering effort appears to have contributed to a number of cost overruns and
schedule slips, especially in the development of complex ground and space
systems. Government and commercial organizations have recently placed increased
emphasis on accurately planning the SE function and on understanding the factors that
influence the resources needed to implement and perform SE.

In an attempt to better quantify the SE activity, the DoD acquisition community has been
exploring Systems Engineering Revitalization and migrating back to a reinstatement of
prescribed military standards. Several models and tools have become available to aid
in forecasting systems engineering resource needs, but few guidelines exist to help
engineers and program managers determine which approach is best suited for
estimating any particular effort.

To provide such guidelines, this paper provides an overview of eight existing cost
models - three of which focus on software and five on hardware. These models include
SE components and employ unique approaches to sizing the SE effort. An overview of
the genesis and assumptions of each model shed some light on their individual
applicability. To complete the paper, future research topics are discussed along with
recommendations on tasks to enhance the accuracy of SE sizing.

                     18 Annual Forum on COCOMO and Software Cost Modeling

                                            Page 1 of 17
 Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

 I. Introduction
 This paper represents the beginning of collaborative efforts between industry and
 academia to refine the process of estimating SE size and costs. The authors hope that
 this paper establishes the initial step in that direction and will continue to work towards a
 relevant model that can be used to better quantify SE in different application domains.

   The U.S. Air Force recently initiated the Systems Engineering Revitalization program
 which highlights the importance of Systems Engineering in military ground and space
 systems (Air Force 2003). While it has been shown that the appropriate level of SE
 effort leads to better control of project costs (Honour 2002), identifying the necessary
 level of SE effort is not yet a mature process. Some projects use the traditional 15% of
 the prime mission product or prime mission equipment to estimate SE, while other
 projects tend to use informal rules of thumb. These simplified and inaccurate methods
 can lead to excessively high bids by allocating too many hours on SE or even worse, by
 underestimating the amount of SE needed.

 SE is a sine qua non 1 of the acquisition and implementation process. This paper
 focuses on providing a starting point to enhance the estimation of SE resources. The
 objectives of this paper are to (1) describe current estimating models that address SE
 components, (2) compare the genesis, assumptions, and applicability of these models,
 and (3) suggest future research in this area that can help mature the process of
 estimating SE size, resources and cost.

II. Methodology
 The authors reviewed the available cost estimation models with respect to SE and
 selected what was believed to be the most influential models currently in use by the
 acquisition community. One finding during the review was that SE costs were extremely
 sensitive to the “sizing” rules that formed the basis of these models. These rules help
 estimators answer questions like “how big will this system be?” and “what is the size of
 the job?” The emphasis on sizing also allowed the authors to compare common
 aspects of the different cost models. A similar comparative analysis of cost models was
 previously completed (Kemerer 1987), which focused exclusively on models for
 software development. In contrast, the following considers both software and hardware
 cost models as they both involve SE.

III. Overview of Cost Models
 Cost models have been an essential part of DoD acquisition since the 1970s. Hardware
 models were the first to be developed and were followed by software models in the
 1980s. To put the analysis in context, we provide a timeline (Figure 1) (Ferens 1999)
 and background on several models that shows the progression of parametric cost

     essential element
                         18 Annual Forum on COCOMO and Software Cost Modeling

                                             Page 2 of 17
Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

                               Figure 1. Parametric Model Timeline
                             Source: Ferens 1999. Used with permission.

The eight cost models studied are shown in Table 1. The authors believe that these
models are representative of the cost estimation field and are the most commonly used
in the acquisition process. The scope of this paper considers models for which data
were available.

                 Table 1. Cost Models With Systems Engineering Components
                   Model Name                      Owner/Developer         Domain
       COCOMO II                              USC                        Software
       PRICE-H                                PRICE Systems, LLC         Hardware
       PRICE-S                                PRICE Systems, LLC         Software
       Raytheon SE Resource Forecasting Tool  Raytheon                   Hardware
       SEER-H                                 Galorath, Inc.             Hardware
       SEER-SEM                               Galorath, Inc.             Software
       SSCM                                   The Aerospace Corporation  Hardware
       USCM8                                  Los Angeles Air Force Base Hardware

       COCOMO II: The Constructive Cost Model (COCOMO) was developed by Dr.
       Barry Boehm when he was at TRW and made popular by his classic textbook,
       Software Engineering Economics, Prentice-Hall, 1981, which was updated in
       2000 and made more relevant to current methods used by software engineers in
       building their products. A new calibration for the model will be available in 2004,
       which will contain data from approximately 200 software projects.
                     18 Annual Forum on COCOMO and Software Cost Modeling

                                            Page 3 of 17
Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

       PRICE-H: The Parametric Review of Information for Costing and Evaluation
       (PRICE) Hardware model was the first widely available parametric cost
       estimation model, developed in 1973 by Martin Marietta Price Systems (initially
       RCA Price, then GE Price) to estimate development and production costs of
       hardware, based on parameters such as weight and manufacturing complexity.
       It has been continuously updated since its creation to make it more relevant to
       current best practices.

       PRICE-S: The Parametric Review of Information for Costing and Evaluation
       (PRICE) Software model was originally developed in 1977 by Martin Marietta
       Price Systems to estimate software development cost and schedule, using
       design parameters such as software size, application, and complexity. It has
       also been updated to reflect the current state-of-the-practice in software

       Raytheon's System Engineering Resource Forecasting Tool: Raytheon's System
       Engineering Resource Forecasting tool is the result of work begun by Michael
       Ernstoff at Hughes Aircraft Company's Electro-Optical and Data Systems Group,
       prior to its acquisition by Raytheon in December 1997. Management's objective
       was to obtain a tool that could consistently set quantitative boundaries on the
       scope of SE efforts for space-based sensors that were constrained by budgetary
       limitations. Subsequently, utilization of the tool has served as a sanity check on
       the magnitude of bottom-up estimates. Ivan Vincenzini at Raytheon El Segundo
       has pioneered efforts to apply the tool to space-based radar sensors in addition
       to the originally envisioned space-based infrared sensors.

       SEER-H: The System Evaluation and Estimation of Resources (SEER) Hardware
       model was developed by Galorath Incorporated and initially released in 1991.
       Development has continually progressed since then. SEER-H version 5.1 reflects
       what the hardware world is currently shipping.

       SEER-SEM: The SEER Software Engineering Model (SEER-SEM) was also
       developed by Galorath Incorporated. Originally based on the Jensen model of
       software effort and schedule estimation, it has evolved over the years to address
       modern practices for software engineering. SEER-SEM was initially released in
       1988. Development has been ongoing since then. SEER-SEM version 7.0 is
       expected for release in the fall of 2003.

       SSCM:     The Small Satellite Cost Model, developed by The Aerospace
       Corporation, is a parametric cost model consisting of a series of mathematical
       cost estimating relationships that relate spacecraft bus cost to physical, technical,
       and performance parameters that are known or believed to strongly influence
       spacecraft costs. The basis for these relationships is a database of actual costs
       and technical parameters for 35 modern small satellites.

                     18 Annual Forum on COCOMO and Software Cost Modeling

                                            Page 4 of 17
Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

       USCM8: The Unmanned Space Vehicle Cost Model version 8 (USCM8) was
       developed by Tecolote Research, Inc. under direction from the Space and
       Missile Systems Center (SMC) at the Los Angeles Air Force Base as a
       parametric cost estimating tool based on cost estimating relationships (CERs)
       derived from a historical database. USCM has formed the basis of numerous
       Independent Cost Estimates (ICE) for long-range planning and tradeoff studies,
       as well as detailed Component Cost Analyses (CCA) in support of the Defense
       Acquisition Board (DAB) process. USCM1, the first edition, was published in
       November 1969 by the Cost Analysis Division of the Space and Missile Systems
       Organization and was based on a historical database of 5 military, 5 NASA and 1
       commercial space vehicle programs. USCM8 was published in October 2001
       and is based on 23 military, 11 NASA and 9 commercial programs.

The next section compares the eight aforementioned models in five key areas relevant
to systems engineering:
    1. Model inputs for software or hardware size
    2. Definition of Systems Engineering
    3. Model inputs for Systems Engineering
    4. Life Cycle stages used in the model
    5. Domain of applicability

These areas provide information on the applicability of each model to Systems
Engineering sizing.

   1. Model inputs for software or hardware size

       COCOMO II: Software size can be estimated using SLOC (Source Lines of
       Code), FPs (Function Points), or APs (Application Points). Modified and reuse
       software size is calculated as equivalent SLOC of new code based on a non-
       linear reuse model. A separate non-linear reuse model is used to size software
       designed for reuse.

       PRICE-H: Hardware size can be estimated by using weight, manufacturing
       complexity, platform, quantities, schedule, engineering difficulty, the skill level of
       those that will perform the work, and the amount of new work generated.

       PRICE-S: Software size can be estimated using either SLOC FPs, or POPs
       (Predictive Object Points). Additional effort multipliers include the programming
       languages employed, the function performed by the software, the amount of new
       work generated, the skill level of those that will perform the work, the
       development process used, the constraints imposed by hardware, and how the
       customer plans to use the software.

       Raytheon's System Engineering Resource Forecasting Tool: The key-driving
       factor in this model is system complexity. Complexity may correlate with
                     18 Annual Forum on COCOMO and Software Cost Modeling

                                            Page 5 of 17
Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

       software lines of code and hardware weight or volume, but it is complexity, rather
       than the physical parameters, that drives systems engineering resource needs.
       The Raytheon System Engineering Resource Forecasting tool estimates system
       complexity by assigning a score to each subsystem interface and compiling a
       weighted sum of the scores. Learning curves are used to scale down the
       contribution from repeating interfaces and to account for design re-use. In many
       ways, the procedure is analogous to that of counting function points.

       SEER-H: The key sizing inputs used by SEER-H depend on the type of hardware,
       specifically electronics versus mechanical. Weight is the main driving parameter
       for mechanical hardware, printed circuit board count and size for electronics.
       Other influential parameters are materials used, certain design and
       manufacturing options, team experience, and team tools.

       SEER-SEM: The software effort can be sized using SEER-SEM based on
       function points, source lines of code, or user-defined metrics. Users can
       combine or select a single metric for any project element or for the entire project.
       COTS WBS elements also have specific size inputs defined either by Features,
       Object Sizing, or Quick Size, which describe the functionality being integrated.

       SSCM: SSCM 2002 calculates the size for each hardware subsystem using
       several attributes, including weight and other primary parameter(s) that are
       particular to a system, such as structural material or pointing knowledge. The
       cost size can be determined for either government or commercial missions.

       USCM8: Hardware size is calculated for each USCM8 Work Breakdown
       Structure item using weight in conjunction with primary parameter(s) that are
       particular to a system, such as number of communication channels. The size is
       segregated into nonrecurring costs and recurring costs for the first production
       article and for the production build quantity. The end of the nonrecurring costs
       phase is denoted by the completion of prototype qualification. The initialization of
       the recurring costs phase is signaled by the release of design drawings to flight
       hardware manufacturing.

   2. Definition of SE

       COCOMO II: Support to SE is defined in terms of the additional effort that
       software organizations expend in supporting SE activities which include, but are
       not limited to, the following: Integrated Product Teams developing system
       specifications and operational concept documents, Interface Control Working
       Groups defining hardware/software interfaces, equation definition teams
       developing     validated  algorithms,   implementation     teams     developing
       systems/subsystems, and test and evaluation teams verifying and validating that
       systems requirements have been satisfied.

                     18 Annual Forum on COCOMO and Software Cost Modeling

                                            Page 6 of 17
Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

       PRICE-H: SE is bundled into the Systems Integration and Project Management
       components of the model. The Systems Integration component includes the
       merging of hardware products into a single unified system, integrating hardware
       and software components into one system, and assembly integration and test.
       The Project Management component includes the efforts to assure the proper
       and most cost-effective performance of the contract between the customer and
       the company. Included in this is also technical management which involves:
       design review activities, reliability and maintainability studies, product assurance
       efforts, configuration management duties, and production engineering tasks such
       as assembly and test methods.

       PRICE-S: The SE/Project Management element includes the Systems
       Engineering effort to define the software system, and the Project Management
       Effort to manage the software development project. The Systems Engineering
       activity encompasses the effort to define system level requirements, the
       integrated planning and control of the technical program efforts of design
       engineering, specialty engineering, development of test procedures, and system
       oriented testing and problem resolution.

       Raytheon's Systems Engineering Resource Forecasting Tool: The scope of SE is
       defined in terms of a set of SE products that were compiled from the Systems
       Engineering product lists in EIA632, IEEE1220, the NASA Systems Engineering
       handbook and other sources. What results is an indentured list of products and a
       dictionary defining each product. The scope of each proposed SE effort is
       custom-defined by restricting the list of products to only include those that must
       be delivered as part of a proposed program. All resource projections are
       associated with required physical deliverables; there are no "level-of-effort" tasks.

       SEER-H: The functions included in the SE and Integration (SE&I) element
       encompass: (1) the SE effort to transform an operational need into a description
       of system requirements and/or a preferred system configuration; (2) the logistics
       engineering effort to define, optimize, and integrate logistics support
       considerations to ensure the development and production of a supportable and
       cost effective system; and (3) the planning, monitoring, measuring, evaluation,
       and directing of the overall technical program. Specific functions include those
       for control and direction of engineering activities, cost/performance tradeoffs,
       engineering change support and planning studies, technology utilization, and the
       engineering required for safety, reliability, and quality control and assurance.
       Also included is the effort for system optimization, configuration requirements
       analysis, and the submittal and maintenance of Interface Control Documents

       SEER-SEM: The System Requirements Design activity includes the creation of
       initial system requirements and related tasks. If the system has software and
       hardware components, specific functions are typically allocated to software.

                     18 Annual Forum on COCOMO and Software Cost Modeling

                                            Page 7 of 17
Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

       SSCM: SSCM combines systems engineering, program management and
       Integration Assembly & Test as a single cost. SE and Program Management
       include quality assurance, reliability, requirements activities, program
       management, data/report generation, and special studies not covered by or
       associated with specific satellite subsystems. Integration Assembly & Test
       (IA&T) includes research/requirements specification, design and scheduling of
       IA&T procedures, ground support equipment, systems test and evaluation, and
       test data analyses.

       USCM8: USCM8 combines SE, Program Management and Data as a single
       cost, which is given the acronym SEPMD. SEPMD, also called Program level
       costs, include those accounts for program management, reliability, planning,
       quality assurance, systems analyses, project control, and other costs that cannot
       be related to any other specific area of activity. SE includes all effort associated
       with the engineering organization, which allocates and controls the distribution of
       system-level requirements and specifications to lower level subsystems and
       equipment items. Also included are costs associated with controlling system-
       level documents such as specifications, weight management, system reliability,
       and overall quality assurance.       Program management includes all effort
       associated with defining, planning, directing, and controlling company functions,
       subcontractors, and suppliers in order to accomplish program objectives. Data
       includes costs for program-related graphic and written information, whether
       technical or non-technical. Most data requirement costs that fall into this
       category are controlled by a contract data requirements list (CDRL) attached to
       the system's contract.

   3. Model inputs for SE

       COCOMO II: Software support to SE is estimated as an additional cost using
       heuristics or rules of thumb developed by polling experts. COCOMO is
       essentially a software-only estimation model. This support effort is then allocated
       to life cycle phases using additional rules of thumb developed for that purpose.

       PRICE-H: SE is estimated as part of Systems Integration and Project

       PRICE-S: SE is included in the Systems Engineering/Project Management cost
       element. This cost element is estimated for each phase of the project life cycle.
       The core equation relates labor effort to software size (or volume) and
       productivity. These two factors are combined to calculate the labor hours for the
       SE effort during each phase of the project.

       Raytheon's System Engineering Resource Forecasting Tool: The Raytheon
       System Engineering Resource Forecasting tool requires numerous input
       parameters, so that any moderate error does not excessively skew the overall
                     18 Annual Forum on COCOMO and Software Cost Modeling

                                            Page 8 of 17
Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

       result. Furthermore, the input parameters have been carefully chosen so as to
       solicit consistent replies. For example, estimators are not asked to estimate the
       percent reuse; they are asked to provide the number of times the design team
       has previously used each specific subsystem. The parameters include: system
       complexity, requirements volatility, schedule flexibility, schedule aggressiveness,
       security classification, platform, and staffing.

       SEER-H: The system level cost capability (currently in development, not in the
       released product) includes parameters specific to systems engineering cost
       estimation. They include development complexity, development experience,
       production complexity, and production experience.

       SEER-SEM: SEER-SEM focuses on software projects; therefore its inputs are
       geared toward software project developments. All inputs would influence SE,
       and affect other outputs as well. However, there are no inputs that are
       exclusively dedicated to the estimation of the SE effort.

       SSCM: SSCM uses several drivers including design life, development time, type
       of orbit (planetary or earth orbit), and type of customer (government or
       commercial) as inputs to size the cost of SE, program management and
       Integration Assembly & Test (IA&T). SSCM2002 does not treat SE as a separate

       USCM8: USCM8 does not break out systems engineering as a separate cost.
       USCM8 combines systems engineering, program management and data as a
       single cost, represented by the acronym SEPMD. The input to calculate SEPMD
       is a factor multiplied by the sum of the space vehicle cost plus the cost of the
       integration, assembly and system test. The SEPMD is segregated into
       nonrecurring costs and recurring costs for the first article and for the production
       build quantity.

   4. Life cycle stages

       COCOMO II: COCOMO allows its users to allocate effort and duration estimates
       to either a waterfall or MBASE life cycle. MBASE is a life cycle generator that is
       fully compatible with modern incremental and spiral life cycle models like the
       Rational Unified Process (RUP). These phases include: (1) Inception, (2)
       Elaboration, (3) Construction, and (4) Transition.

       PRICE-H: The model identifies five stages of the hardware life cycle: (1) initial
       concept, (2) design, (3) production, (4) operation, and (5) disposal.

       PRICE-S: PRICE uses the nine DoD-STD-2167A development phases: (1)
       Concept, (2) System Requirements, (3) Software Requirements, (4) Preliminary

                     18 Annual Forum on COCOMO and Software Cost Modeling

                                            Page 9 of 17
Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

       Design, (5) Detailed Design, (6) Code/Unit test, (7) Integration & Test, (8)
       Hardware/Software Integration, and (9) Field Test.

       Raytheon's System Engineering Resource Forecasting Tool: The products of SE
       are usually specific to a life cycle phase. By limiting the scope of the products
       included in deliverables for the proposed program, one ensures that the resource
       projection reflects the appropriate portion of the life cycle, if not using a cradle-to-
       grave resource projection. These products are organized into an indentured list
       of product categories whose main headings are: (1) Management Plan, (2)
       System Design, (3) System Analysis, (4) Specifications & Interfaces, (5) Status
       Reports and Reviews, (6) Assembly, Integration & Test Requirement Plans &
       Procedures, (7) Test Equipment, Facilities & Delivery, (8) Assembly, Integration
       & Test, (9) Validation, and (10) Post-Delivery Support.

       SEER-H: The stages included in SEER-H are: (1) development, (2) production,
       (3) operations and (4) support.

       SEER-SEM: The following list encompasses various activities of development
       covered by SEER-SEM, and definitions of each. Activities may be defined
       differently across development organizations, although they usually can be
       mapped to SEER-SEM’s designations. The effort associated with each activity
       would still occur in some form or another. These activities include: (1) System
       Concept, (2) System Requirements Design, (3) Software Requirements Analysis,
       (4) Preliminary Design, (5) Detailed Design, (6) Code and Unit Test, (7)
       Component Integration and Testing, (8) Program Test, (9) Systems Integration
       through OT&E & installation, and (10) Operation Support.

       SSCM: The SSCM2002 life cycle covers the inception and development of the
       spacecraft bus, from Authority To Proceed through to launch and initial on-orbit
       checkout. SSCM2002 also includes Integration Assembly & Test, Program
       Management (PM), Systems Engineering (SE) and Launch and Orbital
       Operations Support (LOOS). LOOS consists of pre-launch planning, trajectory
       analysis, launch site support, launch-vehicle integration (spacecraft portion), and
       initial on-orbit operations before ownership is turned over to the operational user
       (typically 30 days).

       USCM8: The USCM8 life cycle covers the inception and development of the
       space vehicle, through launch and initial on-orbit operations. USCM8 includes
       hardware costs as well as the costs for analysis, design, production, integration,
       and testing efforts directly associated with each component/subsystem of the
       space vehicle. USCM8 also includes aerospace ground equipment (AGE) and
       launch and orbital operations support (LOOS). AGE consists of the equipment
       required to support the space vehicle during ground test and preparation for flight
       operations. LOOS includes any effort associated with pre-launch planning,
       launch and ascent, and initial on-orbit operations. This period ends when the

                     18 Annual Forum on COCOMO and Software Cost Modeling

                                            Page 10 of 17
Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

       newly deployed satellite is turned over to the operational user, typically after a
       period of two to three weeks.

The life cycle stages for the three software and five hardware models are shown in
Figure 2. Separate mapping has been done between the software models and
hardware models because of their fundamental difference in scope.
                            Life Cycle Stages Compared
                   Model                         Life Cycle Stages
                 COCOMO II             Inception           Elaboration Construction                       Transition

                                                System    S/W       Pre      Detailed Code/unit            H/W &     Field
                 PRICE-S             Concept    Reqs      Reqs      Design   Design test
                                                                                                           S/W Int   Test

                                     Sys    Reqs   S/W Req Pre    Detail Code &                                      Ops
                 SEER-SEM            Concpt Design Analys  Design Design Unit Test
                                                                                   I&T            Program OT&E
                                                                                                  Test               Support

                 PRICE-H              Initial concept      Design            Production       Operation         Disposal

                                      Mgmt System System         Specs Status &   Test    Test &
                 RSERFT               Plan Design Analys         & I/F Reviews    Plans   Delivery
                                                                                                   AI&T     V&V      Support

                 SEER-H               Development            Production             Operations             Support

                                                                                                            Orbital Ops
                 SSCM                  Inception           Development               Launch                 Support

                                                                                                            Orbital Ops
                 USCM8                 Inception           Development               Launch                 Support

                                 Figure 2.Forum on COCOMO and Software Cost Modeling                                           6
              10/22/03           18th Annual Life Cycle Stages Compared

   5. Domains of applicability

       COCOMO II: Commercial & government software development.

       PRICE-H: Electronic and mechanical hardware assemblies and systems for
       avionics and space systems.

       PRICE-S: Commercial & government software development.

       Raytheon's System Engineering Resource Forecasting Tool: Infrared and radar
       sensor development programs.

       SEER-H: Electronic and mechanical hardware development and production, and
       operations and support.

       SEER-SEM: Software                   intensive                projects,               including                    development   and

       SSCM: Government and commercial space vehicle buses for earth orbit or
       planetary missions. Payloads are not included. Focus tends to be on Class C
       and D vehicles (DoD HDBK 343).

                         18 Annual Forum on COCOMO and Software Cost Modeling

                                                        Page 11 of 17
 Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

        USCM8: Military, NASA and commercial space vehicle bus and communications
        payloads development. Focus tends to be on Class A and B vehicles (DoD
        HDBK 343).

IV. Discussion & Recommendations
    The increasing frequency and number of programs that have run significantly over-
    budget and behind schedule (GAO 2003) because SE problems were not
    adequately understood should, by itself, be reason enough for the acquisition
    community to press for improvement in forecasting SE resource needs. However,
    even if the history of SE problems is ignored, the future paints an even more
    demanding picture. The undeniable trend is toward increasingly complex systems
    dependent on the coordination of interdisciplinary developments where effective
    system engineering is no longer just another technology, but the key to getting the
    pieces to fit together. It is known that increasing front-end analysis reduces the
    probability of problems later on, but excessive front-end analysis may not pay the
    anticipated dividends. The key is to accurately estimate early in a program the
    appropriate level of SE to ensure mission success within cost and schedule budgets.

    Our findings have shown that most current estimation tools treat SE as a subset of a
    software or a hardware effort. In that complex systems are not dominated by either
    hardware or software, SE ought not to be viewed as a subset of hardware or
    software. Rather, because many functions can be implemented using either
    hardware or software, SE is becoming the discipline for selecting, specifying and
    coordinating the various hardware and software designs. Given that role, the right
    path is to forecast SE resource needs based on the tasks that systems engineering
    must perform and not as an arbitrary percentage of another effort. Hence, SE
    estimation tools must provide for aligning the definition of tasks that SE is expected
    to do on a given project with the program management's vision of economic and
    schedule cost, performance and risk.

    Tools that forecast SE resources largely ignore factors that reflect the scope of the
    SE effort, as insufficient historical data exists from which statistically significant
    algorithms can be derived. To derive cost-estimating relationships from historical
    data using regression analysis, one must have more data points than variables. It is
    difficult to obtain actual data on systems engineering costs and on factors that
    impact those costs. For example, a typical factor may be an aggressive schedule,
    which will increase the demand for SE resources. The result is a tool set that
    inadequately characterizes the proposed program and therefore inaccurately
    forecasts SE resource needs.

V. Conclusion & Directions for Future Research
    The estimation of SE size is a time-consuming and complex process. The
    acquisition community is pushing for more accurate quantification of the SE work
    that is performed on hardware and software systems. In order to realize more
                      18 Annual Forum on COCOMO and Software Cost Modeling

                                             Page 12 of 17
Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

   sophisticated methods for estimating SE effort, the models should be based on
   empirical data. Rather than using rules of thumb, models should be data-driven.
   Instead of having limited relevance to only standard life cycles, they should be more
   flexible. Further work is required to make the task of managing SE more quantitative.
   One of the first steps would be to more fully understand the factors that drive the
   cost and why there exists so much variation.

   The development of a tool comparison matrix would be of value to enable the
   selection of the most appropriate tool for a particular project. Such a comparison
   would establish uniform definitions and factors that would encompass the breadth of
   how the different tools assess SE. Such a comparison would facilitate identification
   of areas lacking sufficient detail and establish understanding to appropriately
   quantify SE. These findings would suggest common areas of enhancement and
   development for addition research.

   If industry and academia are to develop and validate a tool to more accurately
   forecast SE resources, they must first have good information on numerous projects
   involving SE. Getting this data is nearly impossible in the present environment where
   there is little standardization on the scope of the various Systems Engineering
   activities. For example, what is systems integration? Does SE include those
   redesign activities found necessary to make the components work together as
   intended in a system? If SE includes component redesign, might one project
   discipline that is running out of resources merely throw their responsibility over the
   fence into the SE bullpen by merely ignoring certain issues? The one discipline
   claims they completed their work within budget, but SE overruns its budget because
   it was stuck with resolving design issues that should have been completed
   previously. The first step, then, for developing a better tool for forecasting SE
   resource needs is to standardization on a Work Breakdown Structure and the
   definition of SE, so that one engineer’s integration includes the same items as

   Knowing what the cost is of a program is not the only data point needed. Additional
   information is also needed, such as identifying which factors have driven the size
   and effort elements of the program cost. Experts in the field can identify with
   reasonable certainty the parameters that drive SE costs, such as complexity,
   requirements volatility, and schedule aggressiveness. Standardized methods need
   to be established to identify these parameters and other variables based on what the
   Systems Engineer knows at the outset of a program. These methods must be
   designed so that same results can be generated consistently; without consistency
   any observed correlation would be meaningless. The cost drivers for the proposed
   program, solicited from the program manager, systems engineer, and other
   knowledgeable individuals, must be in agreement. The second step to developing a
   better SE forecasting tool is the development of a methodology to consistently and
   accurately characterize a program would enable the synthesis of a more accurate
   SE forecasting tool.

                     18 Annual Forum on COCOMO and Software Cost Modeling

                                            Page 13 of 17
Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

   Once one has established a solid description of what is to be estimated (i.e., the
   scope of the effort) and a set of methods to quantitatively characterize the
   parameters impacting systems engineering resource needs, a tool can be developed
   using cost estimating relationships derived from expert opinion. The output of this
   tool can then be tested against actual history to determine accuracy and precision.
   Over time, as the tool is exercised, data should be collected that would gradually
   validate more and more of the cost estimating relationships. The process takes
   time; thus, the third step is to adopt a suitable model for actual use. It is only by
   using the model that its weaknesses and strengths can be discovered.

       AGE             Aerospace Ground Equipment
       AP              Application Points
       CCA             Component Cost Analysis
       CDRL            Contract Data Requirements List
       COCOMO          Constructive Cost Model
       CER             Cost Estimating Relationship
       COTS            Commercial off the shelf
       DAB             Defense Acquisition Board
       DoD             Department of Defense
       EIA             Electronic Industries Alliance
       FP              Function Points
       GAO             General Accounting Office
       IA&T            Integration Assembly & Test
       ICD             Interface Control Document
       IEEE            Institute of Electrical & Electronics Engineers
       ICE             Independent Cost Estimate
       LOOS            Launch & Orbital Operations Support
       MBASE           Model Based Architecting & Systems Engineering
       PM              Program Management
       POP             Predictive Object Points
       PRICE           Parametric Review of Information for Costing and Evaluation
       RUP             Rational Unified Process
       SE              Systems Engineering
       SEER            System Evaluation and Estimation of Resources
       SE&I            Systems Engineering & Integration
       SEPM D          Systems Engineering Program Management and Data
       SLOC            Source Lines of Code
       SMC             Space and Missile Systems Center
       SSCM            Small Satellite Cost Model
       OT&E            Operational Test & Evaluation
       USCM8           Unmanned Satellite Cost Model version 8
       WBS             Work Breakdown Structure

                     18 Annual Forum on COCOMO and Software Cost Modeling

                                            Page 14 of 17
Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

Air Force Systems Engineering Revitalization website:
       http://ax.losangeles.af.mil/se_revitalization/main.htm, accessed on 09/15/03.

Boehm, B. W., Abts, C., Brown, A. W., Chulani, S., Clark, B. K., Horowitz, E., Madachy,
     R., Reifer, D., Steece, B., “Software Cost Estimation with COCOMO II”, Prentice
     Hall, 2000.

Ernstoff, M., “Estimation of System Complexity”, International Society of Parametric
      Analysts – Los Angeles Chapter, April 2001.

Ernstoff, M., Vincenzini, I., “Guide to Products of System Engineering”, International
      Council on Systems Engineering, Las Vegas, NV, August 1999.

Ferens, D., “Parametric Estimating – Past, Present, and Future”, 19th PRICE European
      Symposium, October 1999.

GAO-03-1073, Defense Acquisitions Improvements Needed in Space Systems
     Acquisition Management Policy, www.gao.gov/cgi-bin/getrpt?GAO-03-1073,
     September 2003.

Honour, E. C., “Toward an Understanding of the Value of Systems Engineering”, First
     Annual Conference on Systems Integration, Hoboken, NJ, 2002.

Kemerer, C., “An Empirical Validation of Software Cost Estimation Models”,
     Communications of the ACM, Vol. 30, No. 5, pp. 416-429, 1987.

Military Handbook for Design, Construction, and Testing Requirements for one of a kind
        Space Equipment, DoD HDBK 343, 01, February, 1986.

Military Standard for Defense System Software Development, DoD-STD-2167, 4 June
        1985, and DoD-STD-2167A, 27 October 1987.

PRICE-H, “Your Guide to PRICE-H: Estimating Cost and Schedule of Hardware
     Development and Production”, PRICE Systems, LLC, Mt. Laurel, NJ, 2002.

PRICE-S, “Your Guide to PRICE-S: Estimating Cost and Schedule of Software
     Development and Support”, PRICE Systems, LLC, Mt. Laurel, NJ, 1998.

SSCM Pro edition, The Aerospace Corporation, 2002.

USCM8 Knowledge Management System, Tecolote Research, Inc., 2002.

                     18 Annual Forum on COCOMO and Software Cost Modeling

                                            Page 15 of 17
Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

The authors would like to thank those who contributed to the manuscript as references
or reviewers specifically: Daniel Ferens (Air Force Research Labs), Yue Chen (USC-
CSE), Marilee Wheaton (The Aerospace Corporation), Dave Hickman (The Aerospace
Corporation), Carl Billingsley (The Aerospace Corporation), Karen McRitchie (Galorath),
Elaine Lim (The Aerospace Corporation), Larry Sidor (The Aerospace Corporation),
Daniel Nigg (The Aerospace Corporation) and Arlene Minkiewicz (PRICE Systems).

Ricardo Valerdi
Ricardo is a Research Assistant at the Center for Software Engineering and a PhD
student at the University of Southern California in the Industrial and Systems
Engineering department. His research is focused on the cost modeling of systems
engineering work. While completing a Masters degree in Systems Architecting &
Engineering at USC he collaborated in the creation of COSYSMO (Constructive
Systems Engineering Model).       He earned his bachelor’s degree in Electrical
Engineering from the University of San Diego. Ricardo is currently a Member of the
Technical Staff at the Aerospace Corporation in the Economic & Market Analysis
Center. Previously, Ricardo worked as a Systems Engineer at Motorola and at General
Instrument Corporation.

Michael Ernstoff
Michael Ernstoff, formerly with Raytheon Systems and Hughes Aircraft Company. Mr.
Ernstoff retired from Hughes Aircraft Company’ Electro-Optical and Data Systems
Group in 1996 after spending over 30 years engaged in the development of traveling
wave tubes, active-matrix liquid-crystal displays, and tactical and strategic thermal
imaging systems. Shortly after retiring, he returned to the same but renamed
organization, Raytheon Systems, to complete the formulation of an Excel based
parametric estimation tool for predicting system engineering resource needs on
complex sensor technology development programs. Since those tool development
efforts were substantially completed two years ago, the tool has reportedly been used
with surprisingly outstanding success as a sanity check on several major bottom's up
estimates. Mr. Ernstoff earned his Bachelor’s Degree in Electrical Engineering and his
Master of Science degrees from Cornell University. He is a Senior Member of the IEEE,
principle inventor on over a dozen patents, and has presented papers at national
conferences hosted by the IEEE (Institute of Electrical & Electronic Engineers), SID
(Society of Information Display), AIAA (American Institute of Aeronautics and
Astronautics) and INCOSE (International Council of System Engineers).

                     18 Annual Forum on COCOMO and Software Cost Modeling

                                            Page 16 of 17
Valerdi, Ernstoff, Mohlman, Reifer & Stump: Systems Engineering Sizing

Paul Mohlman
Mr. Mohlman is in the Cost and Requirements Department at The Aerospace
Corporation, which is the corporate focal point for space systems cost analysis and
estimation. His experience spans over 30 years in the aerospace and petroleum
industries and has previously worked at TRW, Rockwell, Texaco, Getty Oil, Mobil and
Halliburton. Mr. Mohlman received his Bachelor of Science in Mechanical Engineering
from the California State University, San Luis Obispo, California. He is a life member of
the United States Air Force Association, since 1991. Mr. Mohlman has issued and
presented reports on cost, schedule, systems engineering, launch vehicle integration,
strategic planning, launch vehicle and satellite systems readiness, component failure
assessment, space systems database, satellite orbital experience, refinery capital
project costs, oil-shale synthetic fuel facility costs, and management of oil company
refining and production subsidiaries.

Don Reifer
Donald Reifer is one of the leading figures in the field of software engineering and
management with over 30 years of experience in academia, industry and government.
He is currently President of RCI, a firm specializing in software security and anti-
tampering research.

Evin Stump
Mr. Evin Stump is a Senior Consultant for Galorath Incorporated. His experience as an
engineer spans 50 years in the aerospace industry, the last 25 of which have been
primarily in engineering cost analysis and modeling. His undergraduate engineering
studies were at Loyola University of Los Angeles. He earned the MS in Operations
Research from the University of Texas at Austin, and the Professional Designation in
Government Contract Management from UCLA/NCMA. He is a former president of the
Southern California chapter of the International Society of Parametric Analysts and has
written several award winning papers for that and other professional organizations.

                     18 Annual Forum on COCOMO and Software Cost Modeling

                                            Page 17 of 17

To top