Integrated Metrics for CMMI and SW-CMM by zrl90908


									                 Best Practices

                         Integrated Metrics for CMMI and SW-CMM
                                                                                                                                                      Gary Natwick
                                                                                                                                                    Harris Corporation
                                                           As organizations move toward the Capability Maturity Model ® (CMM®) Integration SM
                                                           requiring the integration of technical and management processes across functional disci-
                                                           plines, the tool suites used to plan, manage, and monitor these integrated processes must also
       Wednesday, 30 April 2003                            evolve to support them. One example of this is an integrated engineering metrics set to rein-
         Track 8: 3:50 - 4:30                              force process deployment, provide effective management oversight, and ensure alignment with
           Room 251 D - F                                  organizational business goals. Harris Corporation used a Goal-Question-Metric approach
                                                           to develop an integrated metrics set for quantitative management of performance, progress,
                                                           cost, schedule, and resources across systems, software, and hardware engineering disciplines.
                                                           Institutionalization of this metrics approach resulted in achieving CMM for Software
                                                           Level 4.

A    t the Government Communications
     Systems Division (GCSD) of Harris
Corporation, Melbourne, Fla., integrated
                                                              •    Evaluate – to determine status with
                                                                   respect to plans. Measures are indica-
                                                                   tors of when projects and processes
                                                                                                                         ment actions are working as intended,
                                                                                                                         and what the side effects may be.
                                                                                                                         Good measures also help communi-
metrics is a key element of successful                             are drifting off-track so they can be                 cate goals and convey reasons for
quantitative management of every pro-                              brought back under control. Eval-                     improving. This helps engage and
gram and engineering discipline. Harris                            uations also assess achievement of                    focus the support of those working
Corporation achieved the Software                                  quality goals and the impacts of tech-                within processes to make them suc-
Engineering Institute’s (SEISM) Capability                                                                               cessful.
Maturity Model® (CMM®) for Software
(SW-CMM®) [1] Level 4 and is advancing                                                                               Goal-Driven Metrics
to CMM IntegrationSM (CMMI®) [2] Level                             “One metric does not                              Using the Goal-Question-Metric (GQM)
4 using integrated metrics across engi-                                                                              [3] approach, integrated engineering met-
                                                                     tell the whole story.                           rics was derived from strategic business
neering disciplines. A SEI authorized lead
appraiser performed the Level 4 SW-                               You need integrated, and                           goals and best practices of our organiza-
CMM appraisal of Harris GCSD in June                                                                                 tion, the industry, and government. The
2002.                                                              many times, orthogonal                            main objective for integrated engineering
    Integrated engineering metrics focus                                                                             metrics is to objectively measure the pro-
on quality, productivity, and predictability                       views of metrics to get                           gram health and status in relation to the
providing support data for estimating                                                                                following organization’s goals:
future jobs, tracking ongoing jobs, and                              a complete picture;                             • Project Management. Planning, esti-
identifying and evaluating process                                                                                       mating, monitoring, and controlling a
improvements.                                                          trending is key.”                                 project’s costs, schedules, and quality.
                                                                                                                     • Process Improvement. Providing
                                                                                                                         baseline data and measuring trends,
Why Measure?                                                       nology and process improvements on                    tracking root causes of problems and
Harris is recognized in the industry for                           products and processes.                               defects, and identifying and imple-
developing and delivering quality prod-                       •    Predict – to gain an understanding of                 menting changes for process improve-
ucts; however, to advance itself in a com-                         relationships among processes and                     ment.
petitive industry the company has to con-                          products so the values observed could             • Organizational Vision. Effectively
tinually improve its overall program per-                          be used to predict others. This is done               applying unified end-to-end engineer-
formance. The reason many companies in                             to establish achievable goals for cost,               ing processes and methods encom-
the industry are advancing their capabili-                         schedule, and quality so appropriate                  passing proven and emerging stan-
ties by measuring engineering processes,                           resources can be applied. Predictive                  dards/approaches for the purpose of
products, and resources is to accomplish                           measures are also the basis for trend-                delivering high-quality, cost-competi-
the following:                                                     ing so estimates for cost, time, and                  tive system solutions to our customers.
• Characterize – to gain understanding of                          quality can be updated based on cur-
    processes, products, resources, and                            rent evidence.                                    Approach
    environments, and to establish base-                      •    Improve – to identify roadblocks, root            An action team (composed of systems
    lines for comparisons with future                              causes, inefficiencies, and other oppor-          engineers, software engineers, program
    assessments.                                                   tunities for improving product quality            managers, and assessment experts)
     SEI, CMM Integration, and SCAMPI Lead Assessor are            and process performance. Measures of              focused on defining the Harris GCSD’s
     service marks of Carnegie Mellon University.                  current performance give us baselines             goals and ensuring the metrics needed to
®    Capability Maturity Model, CMM, and CMMI are regis-
     tered in the U.S. Patent and Trademark Office.                to compare whether or not improve-                measure the achievement of those goals

4 CROSSTALK The Journal of    Defense Software Engineering                                                                                                     May 2003
                                                                                                    Integrated Metrics for CMMI and SW-CMM

were being captured at all stakeholder         the industry literature, and key practices    of the currently used division control
levels of the organization. Structured         from the SW-CMM [1] and CMMI [2],             panels containing up to nine metrics each
interviews were conducted with individu-       metrics were identified to measure the        (3 x 3). The control panels represented
als representing the following four levels     success in achieving the goals and sub-       the integrated project engineering met-
(from the highest to lowest) of stake-         goals. GQM [3] concepts were used to          rics distributed across systems engineer-
holders:                                       validate the results of the metrics derived   ing, software engineering, hardware engi-
• Division management.                         from the other sources and to identify        neering, and project engineering
• Business area leadership.                    any metrics that might have been over-        resources. The derived metrics differed
• Project management and technical             looked. It should also be noted that sev-     from the pre-existing division metrics in
     leadership teams.                         eral existing division metrics were           two major areas:
• Functional owners of division                dropped, as they were not directly attrib-    • More emphasis on product quality via
     processes.                                utable to the defined division business           defect measurement and tracking.
     The protocols for the interviews          goals. An example of a division goal          • Additional measurement of the per-
(individual and group) at each level of        mapped to metrics using GQM follows:              sonnel resources’ training, develop-
the organization were based on the             • Goal: Project Management, i.e., plan,           ment, and tool support.
results of the interviews from the previ-          estimate, monitor, and control project        The metrics set supported the SW-
ous level of the organization. Division            quality.                                  CMM [1] and CMMI [2] Level 4 objec-
management was asked to rate the               • Sub-Goal: Improve customer satis-           tives of defined measurement standards.
importance of the goals in the division            faction by reducing defects.              Each metric has a specified value that
Strategic Guide Plan (and the goals sup-       • Question: Where are defects intro-          represents an enterprise performance
porting those in the plan). The business           duced and removed?                        goal. As data are collected, the goals are
area leaders then were interviewed and         • Metric: Defects detected in peer            converted to control limits. The top six
asked to identify the subclass, questions,         reviews and testing.                      metrics in the example of Integrated
and metrics that they used, or would like          A red team consisting of six project      Engineering Cost and Schedule Control
to use, to achieve the goals identified by     teams reviewed the resulting metrics.         Panel, shown in Figure 1 (see page 6),
division management. The project lead-         Each project team was composed of the         address the GQM as follows:
ership interviewees were asked to identi-      project’s program manager, chief system       • Goal: Project Management, i.e., plan,
fy the questions and metrics that they         engineer, chief software engineer, chief          estimate, monitor, and control project
used, or would like to use, to achieve the                                                       cost and schedule.
division goals and business area sub-                                                        • Sub-Goal: Perform within planned
goals. The process owners were then                                                              cost and schedule.
asked to identify the questions and met-                   “Having an                        • Question: How effective is the
rics that they would use to measure the          organizational standard                         process execution versus the plan?
process goals identified in the prior inter-                                                 • Metric: Cost performance index
views as well as to achieve the improve-            tool is a must for                           (CPI), schedule performance index
ment goals that the process owners iden-                                                         (SPI), and to-complete performance
tified.                                          consistency; it should be                       index (TCPI).
     The interviews were structured to                                                           Additional information provided in
correspond to the GQM [3] methodolo-                user friendly with                       the footer of these metrics is cost vari-
gy, where issues, problems, and objec-                                                       ance (CV), schedule variance (SV), and
tives led to the identification of meas-              easy access.”                          variance at completion (VAC).
ures. The interviewees were also asked to
prioritize both the reasons for desiring                                                     Integrated Metrics Process
the measurement information and the                                                          Integrated engineering metrics is used to
importance of the specific measures they       hardware engineer, and quality assurance      gauge a project’s progress and to alert
recommended.                                   engineers. A structured evaluation tech-      program management of any potential
     The Functional Analysis System            nique was used against each metric using      risks to its quality, cost, and schedule.
Technique (FAST) [4] was used to graph-        the following criteria:                       Each metric provides insight into sys-
ically depict the linkage of each higher-      • Utility to the customer.                    tems/software/hardware engineering
level goal to lower-level goals. FAST pro-     • Utility to the project leadership.          development products and processes and
vided a mechanism for obtaining impor-         • Utility to division management.             process improvement and/or organiza-
tance ratings, by interviews, on more          • Difficulty to collect.                      tional improvement through one of the
than 100 goals without losing the goals’                                                     following four major indicator categories:
context. The team analyzed the impor-          Results                                       • Progress. The achievement or com-
tance rating and selected the highest-level    The metrics definition effort identified          pletion of goals or commitments.
goals that spawned a set of lower-level        metrics covering all aspects of project       • Resources. The availability or capabil-
goals with a 90 percent or greater cover-      management and engineering perform-               ity of organizational assets.
age. This generated a set of top-level         ance across systems engineering, soft-        • Quality. The problems and/or defects
goals that were briefed to division man-       ware engineering, and hardware engineer-          with a product or process.
agement and used as the foundation for         ing. The metrics were grouped into sets       • Stability. The degree of change, com-
organizational metrics.                        that represented a theme or view of per-          pleteness, or effectiveness.
     The analysis identified division goals    formance familiar to each of the four             Everyone who uses engineering
and sub-goals: Based on the metrics cur-       levels of the organization.                   processes and/or develops engineering
rently used in the division, metrics from          The metric groupings took the form        products utilizes engineering metrics.

May 2003                                                                                                          5
Best Practices

Program team members are responsible                           ing the engineering metrics process.            Metrics are indicators that give warn-
for collecting and analyzing individual                        Division control limits are statistically   ings of problems associated with issues.
metrics. Project team leaders are respon-                   based upon historical data. Projects use       An issue may be tracked with several met-
sible for collecting, analyzing, and report-                the division control limits or statistically   rics that may be based on different meas-
ing metrics to the program team and divi-                   determine their own.                           ures. Insight into an issue typically
sion management. Division management                                                                       requires statistical analysis of metrics over
ensures the collecting and reporting of                     Collecting                                     time and is trend-based or limit-based as
metrics, and the engineering process                        Collecting measurement data is the sec-        follows:
group conducts metrics analysis and                         ond and continuing step in the integrated      • Trend-based metrics are used when
trending. The integrated engineering                        engineering metrics process. The collec-           expected or planned values change
metrics process has four steps: planning,                   tion occurs at periodic intervals defined in       regularly over time. The analysis of a
collecting, analyzing, and reporting.                       the project plans and is monitored for             trend-based metric involves determin-
                                                            completeness, integrity, and accuracy. The         ing whether the performance implied
Planning                                                    primary source for planning data is in the         in the trend is achievable.
Planning is the first step in the integrated                project plans. The primary source for          • Limit-based metrics are used when the
engineering metrics process. The collecti-                  actual data is in the accounting systems           expected or planned values remain rel-
ing, analyzing, and reporting of metrics                    used to manage the project (e.g., financial        atively constant over time. The analy-
are integrated into the project plans iden-                 management, configuration management,              sis of a limit-based metric requires
tifying the following:                                      change management, and risk manage-                determining whether the performance
• Metrics used to support quantitative                      ment) and is input into the division stan-         crosses its established bounds. Limits
    management.                                             dard metric tool each period.                      can represent norms, expected values,
• Planned and/or expected perform-                                                                             or constraints.
    ance in the metrics, including any                      Analyzing                                          Detecting a difference, limit or trend,
    required goals and/or control limits.                   Analyzing metrics and making objective         between planned and actual recognizes
• Variance implication and corrective                       quantitative management decisions is the       problems. If the difference exceeds the
    action for metrics falling outside their                true benefit step in the integrated engi-      threshold of acceptable risk, then the sit-
    control limits.                                         neering metrics process.                       uation is investigated and corrected.
• Source and collection mechanism of                            Metrics are most often communicated
    the measurement data.                                   graphically conveying a clear and easily       Reporting
• Responsible persons for collecting                        understood message. It is better to have       Reporting integrated engineering metrics
    measurement data, analyzing of met-                     many graphs than it is to have many mes-       is the final step in making quantitative
    rics, reporting the results, and manag-                 sages on one graph.                            management decisions and communicat-
Figure 1: Integrated Engineering Cost and Schedule Control Panel

6 CROSSTALK The Journal of   Defense Software Engineering                                                                                        May 2003
                                                                                                      Integrated Metrics for CMMI and SW-CMM

ing to project team members, manage-             References
ment, and customers. Reporting and               1. Paulk, Mark C., Charles V. Weber,                  COMING EVENTS
reviewing metrics are integrated into the           Suzanne M. Garcia, Mary Beth Chrissis,
management process and occurs as soon as            and Marilyn W. Bush. Key Practices of
possible after analysis has been completed          the Capability Maturity Model ®. Ver. 1.1.                  June 2-6
to assure that there is time for corrective         CMU/SEI-93-TR-25. Pittsburg, PA:              Applications of Software Measurement
action. Any metric falling outside the con-         Software Engineering Institute, Feb. 1993.                San Jose, CA
trol limits is reviewed for variance, and cor-   2. Carnegie Mellon University. CMMI SM for
rective actions are recorded and tracked to                                                      
                                                    Systems Engineering/Software Engi-
closure. Meeting minutes are kept that              neering, Ver. 1.1, Staged Represen-
record the variance explanations.                   tation. CMU/SEI-2002-TR-002. Pitts-
                                                                                                              June 7-14
                                                    burg, PA: Carnegie Mellon University,            Federated Computing Research
Integrated Metrics Tool                             Dec. 2001.                                                 Conference
Integrated engineering metrics are collect-                                                                 San Diego, CA
                                                 3. Park, Robert E., Wolfhart B. Goethert,
ed, analyzed, and reported via the division-                                                    
                                                    and William A. Florac, Goal-Driven
standard metric tool (Web client/database
server) for consistency in application              Software Measurement – A Guide-
across the division. A required set of inte-        book. CMU/SEI-96-HB-002. Pitts-                            June 9-13
grated engineering metrics is used by all           burg, PA: Software Engineering Insti-         International Conference on Practical
projects to advance the engineering process         tute, Aug. 1996.                                   Software Testing Techniques
maturity of the division.                        4. Kaufman, J. Jerry. Value Engineering for                Washington, DC
    Projects utilize additional metrics such        the Practitioner. Raleigh, NC: North     
as customer-required metrics, to comple-            Carolina State University, 1990.
ment the division-standard metric tool. A                                                                     June 25-28
detailed definition of each engineering                  About the Author
                                                                                                    Agile Development Conference
metric is built into the metric tool, includ-                     Gary Natwick is the
ing description, audience, purpose,                                                                       Salt Lake City, UT
method, measures, metrics, control limits,                        metrics leader for the
formulas, range of values, graphic informa-                       engineering     process
tion, and references. Control panels are the                      group responsible for                    August 25-29
most common method for communicating                              Harris      Corporation          QAI’s Annual eXtreme Conference
an integrated view of engineering metric                          achieving the Software                   Las Vegas, NV
frames. A subset of the standard division         Engineering       Institute’s   (SEISM)       
metrics is presented at all program reviews.      Capability Maturity Model® for
                                                  Software (SW-CMM®) Level 4 and                          September 14-19
Lessons Learned
Lessons learned from implementing a met-          advancing to CMM IntegrationSM Level             International Function Point Users
rics program and tool within an integrated        4. Previously, he led the software engi-             Group Annual Conference
discipline work force are as follows:             neering process group responsible for                      Scottsdale, AZ
• One metric does not tell the whole              Harris Corporation achieving SW-                           www.ifpug,org
    story. You need integrated, and many          CMM Level 3. Natwick has 30 years of
    times, orthogonal views of metrics to         software and systems engineering                       September 22-25
    get a complete picture; trending is key.      experience (management, develop-                      AUTOTESTCON 2003
• Project planning is key, and data collec-       ment, and process improvement) with
    tion is the hardest.                                                                                   Anaheim, CA
                                                  the U.S. Air Force (USAF) and Harris        
• Having an organizational standard tool
    is a must for consistency; it should be       Corporation. He received the USAF
    user friendly with easy access.               Commendation Medal for Meritorious                       October 21-24
• Cultural change is hard, so train every-        Service, and the Engineering Process                  18 International Forum

    one about the organizational metrics          and Golden Quill awards for advancing               on COCOMO and Software
    program and tool to increase accept-          Harris Corporation. Natwick has a                         Cost Modeling
    ance and buy-in.                              Bachelor of Science in Electrical                        Los Angeles, CA
                                                  Engineering from the University of            
Conclusion                                        Miami, Fla. He is a SEI Authorized
Integrated engineering metrics are required
to provide effective management oversight         Lead Appraiser (SCAMPISM and CBA                        April 19-22, 2004
and to ensure alignment with organization-        IPI methods), and “Introduction to              Software Technology Conference 2004
al business goals. As organizations move          CMMI®” course instructor.
toward the CMMI [2] requiring the integra-
                                                      Harris Corporation
tion of technical and management process-
es across functional disciplines, the tool            P.O. Box 37
                                                      Melbourne, FL 32902-0037                            Salt Lake City, UT
suites used to plan, manage, and monitor
these integrated processes must also evolve           Phone: (321) 729-3970                     
to support them.◆                                     E-mail:

May 2003                                                                                                            7

To top