Addie Contract by rwb15457

VIEWS: 0 PAGES: 43

More Info
									  Joint ADL Co-Lab BAA


COSCOMO Prototype Project


   Contract #: N61339-05-C-0115


      Final Project Review
       01 September 2006



                                  1
        COSCOMO Prototype Project
           Final Review Agenda

• Project Overview
• Management Review Material
• Technical Project Essentials
   – COCOMO Background
   – COSCOMO Model Derivation
   – How the COSCOMO Model Works
• The COSCOMO Prototype Tool
   – live Demo
• Model Calibration and Validation
   – Initial Model Calibration
   – Historical Project Data Collection
   – Model Validation Tests
• Wrap-up / Discussion



                                          2
Project Overview




                   3
                COSCOMO Prototype Project:
                       Concept
       It is challenging for both sponsors and developers to estimate the
       expected level of effort, duration, and cost of developing web-based
       SCORM conformant courseware.

• Project Goal: create an interactive project
  estimation tool “COSCOMO” for
  ISD/SCORM content

    – Domain focus: ADL Sharable Content
      Object Reference Model (SCORM)
      conformant content

    – ISD methodology: Analysis, Design,
      Development Implementation, Evaluation
      (ADDIE) model

    – Algorithmic foundation: COCOMO II
      model for software project estimation


                                                                              4
        COSCOMO Prototype Project:
              Description

• Create a COSCOMO algorithm to estimate the person-months
  required to create a SCORM conformant learning product

• Collect historical project data & expert knowledge

• Calibrate and Validate COSCOMO Model
   – Calibrate COSCOMO model to fit historical data and expert
      knowledge
   – Validate the algorithm for various sets of data within the
      bounded conditions

• Develop a prototype tool for the COSCOMO cost model

• Build a community of interest around the method & tool
    –   Training 2006 conference
    –   COSYSMO conference
    –   INCOSE 2006 symposium
    –   Implementation Fest 2006
    –   I/ITSEC 2006 conference
                                                                  5
             Applicability & Value to
                   Community

• Consistent, objective, and reliable estimation tool for
  SCORM content and projects
• First step in formalizing an estimation method in the
  ADL community
• Create a tool that other projects can apply, modify, and
  mature
   – COCOMO II has been evolving for 25 years.
   – COSCOMO prototype from this project will be the first
      step in the long evolution and improvement of a tool for
      this community




                                                                 6
Project Management




                     7
               Project Personnel


• SPARTA
   – Mike Garnsey, PM
   – Lacey Edwards
• GDIT (formerly Anteon)
   – Kelly Ward
   – Denise Stevens
   – Tatjana Pitts
   – Tim Richey




                                   8
                        Project Tasks

Task                            Status

1.1 Project Management          In Progress

1.2 Project Kick-off Meeting    Complete

1.3 Research COCOMO Variables   Complete

1.4 Research ISD Cost Items     Complete

1.5 COSCOMO Algorithms          Complete

1.6 In-Process Review           Complete

1.7 COSCOMO Software Model      Complete

1.8 Prototype User Interface    In Progress

1.9 Algorithm Documentation     In Progress

1.10 Final Project Review       In Progress

1.11 Validation Tests           In Progress

                                              9
                        Project Deliverables
   Deliverable                          Description                               Delivery Date
DI-MGMT-81117
(A001)
                 Technical and Management Work Plan.
                 1) Project Schedule
                                                                            12 October 2005
                                                                                                     
DI-MGMT-80227    Contractor‟s Progress, Status and Management Report.       Monthly Progress Reports
(A002)                                                                      07 November 2005     
                 1) Monthly progress reports (Due: 7th day of each month)
                                                                            07 December 2005     
                                                                            09 January 2006      
                                                                            07 February 2006     
                 2) In-Process Review presentation material                 07 March 2006        
                                                                            07 April 2006        
                                                                            07 May 2006          
                                                                            07 June 2006         
                                                                            07 July 2006         
                                                                            07 Aug 2006          
                                                                            07 September 2006

                                                                            IPR Material
                                                                            01 March 2006    
                                                                            31 August 2006   
DI-MGMT-81117
(A005)
                 Technical and Management Work Plan.
                 1) Detailed Design Document
                                                                            07 August 2006
                                                                                                     
DI-MISC-80711A
(A004)
                 Scientific and Technical Report.
                 1) Project Final Report
                                                                            29 August 2006
                                                                                                     
DI-MCCR-80700    Computer Software Product End Item.                        08 September 2006
(A003)           1) COCOMO-SCORM Application Software
                 2) COCOMO-SCORM User Documentation



                                                                                                         10
COCOMO Background




                    11
               COCOMO II Tool Example



  User‟s
Perspective

Quantitative




Qualitative




                                        12
                        COCOMO II Drivers
                        (29 Input Variables)
•   Size                                •   Personnel EM
      – Source Lines of Code (SLOC)          – Analyst Capability (ACAP)
      – Design Modification (DM)             – Programmer Capability (PCAP)
      – Code Modification (CM)               – Personnel Continuity (PCON)
      – Integration (IM)                     – Applications Experience (APEX)
      – Assessment (AA)                      – Platform Experience (PLEX)
      – Understanding (SU)                   – Language/Toolset Experience
      – Unfamiliarity (UNFAM)                   (LTEX)
      – Requirements Evolution (REVL)   •   Project EM
•   Product Effort Multipliers (EM)          – Use of Software Tools (TOOL)
      – Required Reliability (RELY)          – Multisite Development (SITE)
      – Database Size (DATA)                 – Required Development Schedule
      – Product Complexity (CPLX)               (SCED)
      – Required Reuse (RUSE)           •   Scale Drivers
      – Documentation (DOCU)                 – Development Flexibility (FLEX)
•   Platform EM                              – Process Maturity (PMAT)
      – Execution Time Constraints           – Precedentedness (PREC)
        (TIME)                               – Arch/Risk Resolution (RESL)
      – Main Storage Constraints             – Team Cohesion (TEAM)
        (STORE)
      – Platform Volatility (PVOL)
                                                                                13
                     COCOMO II Model Form

                                                * P EMi
                                                    17
                   PM =      A*(Size)E
                                                    i=1

                                                          5
                                  E = B + .01*S SFj
                                                      j=1
Where:
 PM = effort in Person Months
 A = calibration constant derived from historical project data
 Size = Adjust Source Lines of Code
 E = represents diseconomy of scale (composed of 5 scale factors)
 EM = effort multiplier for the ith cost driver. The geometric product results in an
 overall effort adjustment factor to the nominal effort.
 B = calibration constant derived from historical project data
 SF = scale factor for the jth cost driver. Provides organization specific adjustments
 to the size of the project.

                                            3

                                            P EMi = EM1 x EM2 x EM3
                                            i=1
                                                                                    14
               Reliability = PRED(30)


• Reliability of COCOMO family of models is often
  measured by the percentage of test cases that it will
  estimate within 30% of the actual project costs
   – e.g. If a project requires 300 person-months to
     complete, then its PRED(30) range would be (210 to
     390)
   – If the model estimates 70% of its test cases within this
     range then the model‟s PRED(30) = 70%
• COCOMO Family Model Levels
   – COCOMO II (2000): PRED(30) = 69%
   – COSYSMO: PRED(30) = 56%




                                                                15
COSCOMO Model Derivation




                           16
      COCOMO II Research Approach


• Studied the definitive source on COCOMO II: Software
  Cost Estimation with COCOMO II by Barry Boehm et.al.
   – Model form and variables
   – Methodology used to create the COCOMO II model

• Researched how related models (COSYSMO,
  COPSEMO, etc.) modified COCOMO II for their
  purposes
   – COSYSMO workshop
   – Relationship with Dr. Ricardo Valerdi

• Surveyed software implementations of COCOMO II



                                                         17
                        COCOMO Model Family
 Software Cost Models                      DBA COCOMO
                                                                        Other Independent
                                               2004                     Estimation Models
 COCOMO 81
    1981            COCOMO II
                       2000               COINCOMO
                                                                   COCOTS
                                                                    2000
                                                                                      COSYSMO
                                                                                         2002
                                                                                                 
                                              2004
                                                                  COSoSIMO          Costing Secure
                                                                    2004             System 2004




 COQUALMO
   1998     
                     iDAVE
                      2003
                                      COPLIMO
                                        2003              COPSEMO
                                                             1998                 Security
                                                                                Extension 2004


                             COSCOMO
                               2006
                                                  COPROMO
                                                    1998
                                                                       CORADMO
                                                                         1999          
Software Extensions

       Legend:
       Model has been calibrated with historical project data and expert (Delphi) data
       Model is derived from COCOMO II
       Model has been calibrated with expert (Delphi) data
            Dates indicate the time that the first paper was published for the model
                   COCOMO II Modeling Methodology



Analyze existing
literature

Step 1    Perform Behavioral
          analyses

          Step 2    Define relative
                    significance, data,
                    ratings
                    Step 3     Perform expert-
                               judgment Delphi
                               assessment, formulate
                               a priori model
                               Step 4     Gather project data
                                         Step 5     Determine Bayesian
                                                    A-Posteriori model
                                                    Step 6
                                                                Gather more data;
                                                                refine model

                                                                Step 7
                                                                                    19
                COSCOMO Mods to the
              COCOMO II Input Variable Set
•   Size                                •   Personnel EM
      – Source Lines of Code (SLOC)          – Analyst Capability (ACAP)
      – Design Modification (DM)             – Programmer Capability (PCAP)
      – Code Modification (CM)               – Personnel Continuity (PCON)
      – Integration (IM)                     – Applications Experience (APEX)
      – Assessment (AA)                      – Platform Experience (PLEX)
      – Understanding (SU)                   – Language/Toolset Experience
      – Unfamiliarity (UNFAM)                   (LTEX)
      – Requirements Evolution (REVL)   •   Project EM
•   Product Effort Multipliers (EM)          – Use of Software Tools (TOOL)
      – Required Reliability (RELY)          – Multisite Development (SITE)
      – Database Size (DATA)                 – Required Development Schedule
      – Product Complexity (CPLX)               (SCED)
      – Required Reuse (RUSE)           •   Scale Drivers
      – Documentation (DOCU)                 – Development Flexibility (FLEX)
•   Platform EM                              – Process Maturity (PMAT)
      – Execution Time Constraints           – Precedentedness (PREC)
        (TIME)                               – Arch/Risk Resolution (RESL)
      – Main Storage Constraints             – Team Cohesion (TEAM)
        (STORE)
      – Platform Volatility (PVOL)
                                                                                20
              More COSCOMO Mods to the
              COCOMO II Input Variable Set
•   Size of Product                       •   Personnel EM
      – Hours of Courseware                    – Senior Capability (SCAP)
      – Level of Instruction                   – Developer Capability (DCAP)
      – New, Reused, & Modified                – Personnel Continuity (PCON)
        Courseware                             – Applications Experience (APEX)
      – Requirements Evolution (Content        – Platform Experience (PLEX)
        Discarded)                             – Development Tools Experience
•   Product Effort Multipliers (EM)               (DTEX)
      – Required Reliability (RELY)       •   Project EM
      – Product Complexity (CPLX)              – Lifecycle Tools (LIFE)
      – Required Reuse (RUSE)                  – Multisite Development (SITE)
      – Documentation (DOCU)                   – Required Development Schedule
•   Platform EM                                   (SCED)
      – Bandwidth (BAND)                  •   Scale Factors
      – Platform Volatility (PVOL)             – Precedentedness (PREC)
                                               – Development Flexibility (FLEX)
                                               – Arch/Risk Resolution (RESL)
                                               – Team Cohesion (TEAM)
                                               – Process Maturity (PMAT)


                                                                                  21
How the COSCOMO Model Works




                              22
                     Project Cost Estimation
                                                  Dollar Costs
    Project
 Requirements




Quantitative Input
                        Qualitative Adjustments
  Reuse                                             Person
     Level of                                      Months to
   Instruction                                      Develop
        Course
         Hours




                                                    Time to
                                                    Develop
                                                                 23
                                        COSCOMO Algorithm



       PM = A*(Size)E * P EMi                                                  COCOMO II
                                                                              equation form
                              E = B + .01*S SFj


                                                                                 E    15
                                                                    PM  A * (Size) *  EM i
                                                                                     i 1
                     COSCOMO                                        where
                     algorithm                                                        5
                                                                      E  B  0.01*  SF
                                                                                    j 1 j
Note: in COSCOMO, Size = adjusted # of courseware hours (vs SLOC)                              24
                                         Effort Multipliers: Effect on
                                           Project Cost Estimates
                                  15 Effort Multipliers for Nominal Project
                                  Cumulative Effect of Effort Multipliers



                           RELY   CPLX RUSE DOCU BAND         PVOL SCAP      DCAP PCON APEX         PLEX   DTEX   LIFE   SITE   SCED
                                                                                                                                       Extra High
Low
                    1.80

                                                                                                                                       Nominal
nal
                    1.60


High
                                                                                                                                       Very Low
High                1.40

lative Multiplier
                    1.20


                           1.00   1.00   1.00   1.00   1.00   1.00    1.00   1.00   1.00     1.00   1.00   1.00   1.00   1.00   1.00
      Course        1.00
                                                                                                                                         Project
      Hours                                                                                                                               Cost
                    0.80




                    0.60




      Multiplier Project Cost                            =                   A        * EMi                * Course Hours
      Form:      (Person Months)                                     (Constant)            (Modifiers)              (Student Hours)


                                                                                                                                                    25
                  Effort Multipliers: Effect on
                 Project Cost Estimates (con‟t)




One use of the COSCOMO tool output graph is to aid in identifying which variables are
driving the estimate. As shown is this notional graph, high senior team capability
(SCAP) and high development team capability (DCAP) are contributing significantly to
reducing the project cost. (increasing the experience and capability of SCAP and DCAP
by one level each decrease the estimate by 37%)
                                                                                        26
                    Scale Factors: Effect on Project
                    Cost Estimates on Project Cost
when the input values for the
five scale factors are at their
default „Nominal‟ levels, the
scale factors have no impact on
the estimated cost value


higher than nominal scale factor
value levels reduce the estimated
cost; lower than nominal scale
factor value levels increase the
estimated cost. This graph
depicts a notional “real world”
project, with:
  „Very High‟ development flexibility
  „Low‟ Process Maturity
  „Nominal‟ precedentedness
  „Very Low‟ architecture/ risk resolution
  „Very Low‟ team cohesion
                                                       27
               Scale Factors: Maximum +/-
                Effects on Estimated Cost
All SF‟s = „Very Low‟     16.0%
                          14.0%

    PREC = 1              12.0%


    FLEX = 1
                          10.0%
                          8.0%
                                                                        Impact:
    RESL = 1
                          6.0%
                          4.0%
                                                                     14.8% Higher
    TEAM = 1              2.0%                                       Cost Estimate
                          0.0%

    PMAT = 1              -2.0%
                          -4.0%
                                  PREC   FLEX   RESL   TEAM   PMAT




                          4.0%

All SF‟s = „Extra High‟   2.0%


    PREC = 6              0.0%
                                                                        Impact:
    FLEX = 6              -2.0%
                                                                      7.1% Lower
    RESL = 6              -4.0%
                                                                     Cost Estimate
                          -6.0%
    TEAM = 6
                          -8.0%
    PMAT = 6                      PREC   FLEX   RESL   TEAM   PMAT




                                                                                 28
COSCOMO Prototype




                    29
              Prototype Tool - Overview

• The COSCOMO tool GUI prototype was developed in Microsoft
  Excel. Why?
   – Spreadsheets are great for Cost Model implementations
   – Easy to deploy (just about everyone has Microsoft Excel!)
   – This is an R&D project (not a shrink-wrap s/w devel contract)

• Specifically, the prototype tool was developed as a custom
  Microsoft Excel „Add-in‟…
   – Look and feel more like a custom application vs spreadsheet
   – Easy to constrain and simplify user input via „user forms‟
   – Decent GUI rapid prototyping environment & widget set
   – Protection from all sorts of undesirable user actions
       » resizing / hiding / deleting columns
       » deleting cells that contain important formulas/parameters
       » Etc
   – Enables password protection of the “source code”



                                                                     30
COSCOMO Prototype Demo




                         31
Initial Model Calibration




                            32
           Delphi Calibration Survey


• Using Delphi Method to gather independent opinions
  about the level of effort associated with each setting on
  each variable.
   – Collect individual responses
   – Calculate group average
   – Present averages to each person and allow them to
     change their responses if they would like
   – Convene group work session to settle on acceptable
     values
• Round 1 of Delphi Conducted Via Private Survey
• Round 2 & 3 Conducted in Seminar Form




                                                              33
                Delphi Variable Survey Excerpt
  The center value is "Nominal" or "Standard." Estimate the relative amount of effort
  required for each of the other levels. For example, if you believe that a project
  ranked as "Very High" would take 2 times the effort of a nominal project, then enter
  2.0 in the column for "Very High."


                       Very Low         Low                Nominal            High             Very High   Extremely High




                                                                        PV OL data


                                  3.5

                                   3
  actual ratings                  2.5
                                                                                                            Series1
from the 5 survey                  2                                                                        Series2
                                                                                                            Series3

   experts after                  1.5                                                                       Series4
                                                                                                            Series5
                                   1
 Delphi round #2                  0.5

                                   0
                                        PVOL_VL   PVOL_L       PVOL_N    PVOL_H      PVOL_VH   PVOL_EH
                                                                                                                            34
Historical Project Data Collection




                                     35
         Projects Collected


• Historical Data from 9 projects collected:
   – Defense Nuclear Weapons School
   – Air Force Accounting Liaison Course
   – Explosive Hazard Awareness Course
   – Joint Forces Staff College
   – Navy CNET
   – Field Examiner TPSS
   – LGY Construction Complaints
   – ACTS VAW FY02
   – ACTS VAW FY03




                                               36
Model Calibration and Validation




                                   37
Historical Data Calibration Tool




                                   38
       Model Validation Test Approach

• Perform Model Validation Tests on 5 Historical Projects

   – Select 5 of the 9 collected project data sets
      » main selection criteria: project size diversity

   – For each validation test, re-calibrate the model w/o the
     project data set under test




                                                                39
COSCOMO: What‟s Next?




                        40
• This project is the first step in formalizing a cost estimation method
  in the ADL community
     – COSCOMO tool prototype is the first step in the long evolution and
       improvement of a tool for the ADL community
         » keep in mind: COCOMO II has been evolving for 25 years

•   Historical project data collection is essential, but it is also very difficult to
    get access and cooperation from the people with this information
     – Have currently collected data on 9 projects
     – 40+ projects needed to calibrate the model appropriately

•   GUI prototype of the COSCOMO tool is ready for early adoption by the ADL
    community
     – Not a polished, “shrink-wrapped” product, but more refined and user friendly
        than a raw spreadsheet




                                                                                        41
                  Transition Discussion

•   Provide Long-Term Support for Collecting Historical Project Data
     – COSCOMO.admin@jointadlcolab.org ???


•   Encourage the ADL Community to Provide Historical Project Data
     – Keep records of who downloads/uses the prototype
     – Periodically contact these people and request their data


•   Plan to Periodically Revise the COSCOMO Model Calibration Data File
     – It‟s not that time-consuming
     – It‟s a critical element in the long-term success of COSCOMO


•   Technical Support?
     – Lots of things to consider!


                                                                          42
              COSCOMO Prototype - Files
•COSCOMO Prototype uses three main types of files:
    – one or more .xla (Excel Add-In) files
    – one “COSCOMO_parameters.xls” (in same folder as .xla file(s)!)
    – a set of .html help files

•Each .xla file contains the following:
    – Excel Worksheet for a single Courseware project
    – the COSCOMO “logic” as a combination of both:
          » worksheet formulas
          » A password-protected project that contains custom VBA code
             and Userforms

•Intended usage: multiple .xla files supported by a single
“COSCOMO_parameters.xls” file. For example:
     – Field Examiner TPSS.xla
     – DNWS.xla
     – Navy CNET Hi Freq.xla
     – etc

Note: The intent of keeping the “COSCOMO_parameters.xls” stand-alone is to
facilitate model re-calibration as well as reuse among multiple .xla files. This
is the file that could be updated by the COSCOMO support organization and
made available to the ADL community.                                               43

								
To top