Capability Maturity Model (CMM) for Software Development
Geir Amsjø
What do a customer want?
Customers wants: Predictable performance
• • • • On time deliveries Costs according to budget All functionality Reliable promises
Visibility of progress and quality
SW-CMM
2
Crisis-Driven Organizations
Processes:
• ad hoc - improvised and reinvented each time • abandoned under stress • reactionary
Schedules and budgets:
• unpredictable • usually exceeded
Success factors:
• heroes • overtime • fire fighting
Blaming others: bad subcontractors, hardware problems...
SW-CMM
3
Executive Problems in Software
Low visibility into progress and quality Unpredictable performance Constant surprises Excessive costs
Difficult to plan for company strategies
SW-CMM
4
Project Manager Problems
Hard to give accurate estimates
• under-scoped requirements • schedule and resource shortfalls
Unmanaged work
• no corrective action • unruly subcontractors
Volatile baselines
• changing requirements • uncontrolled versions
SW-CMM
5
What Frustrates Developers
Being given half the time they need to do the work Continual overtime Guessing what requirements means Contending with uncontrolled changes in requirements Finding they are not working with the current version Redoing builds forever Delivering products they are not proud of Fixing old defects without design documentation Constantly patching up old software that has no architecture
SW-CMM
6
Sources of Crisis
Software development specifics: Product complexity Conformity – everything is possible Continuous changes Invisible product Explosive code growth Defect removal costs Culture
Partly from Brooks
SW-CMM
7
Cost (i.e Effort) to Remove Defects
Cost/Effort 1-2 orders of magnitude Cost to repair
Time/phase
SW-CMM
8
Improvement Areas
Technology
Product
People
Process
SW-CMM
9
The Role of the Process
Management
Environment Process
Disciplined Methods
Staff
Technical assets
Reference: Bill Curtis, TeraQuest Metrics
SW-CMM 10
Definition of Software Process
”…Software Process can be defined as a set of activities, methods, practices, and transformations that people use to develop and maintain software and the associated products (e.g., project plans, design documents, code, test cases, and user manuals).”
Definition from SEI technical report SEI-93-TR-24, ”Capability Maturity Model for Software, Version 1.1”
SW-CMM
11
Potentials in Process-Based Development
Documents our common experience
• alive and updated • learn from our mistakes
Promotes teamwork
• defines roles, interfaces and dependencies
Promotes individual training
• needs of the company and the individual
Defines activities and output
• nothing neglected nor superfluous
SW-CMM
12
Misconceptions About Process
I do not need process if I have
• • • really good people advanced technology an experienced manager
Process Process Process Process
interferes with creativity = bureaucracy hinders agility in fast-moving markets is only useful on large projects
SW-CMM
13
What is an Effective Process?
Defined Documented Used Supported by management (enforced) Trained Measured/Known in quantitative terms Supported by tools and technology Tailored Improved/maintained
SW-CMM
14
The Process is a Part of the Competition
Process is a competitive weapon in almost every industry except software development Our software process decides quality, cost and cycle-time Software Development
• Historically not considered to be engineering “The quality of a software system is governed by the quality of the process used to develop and evolve it.”
Watts Humphrey (Father of the CMM)
SW-CMM
15
Internal Delivery Precision
Deviation between Max/Min of 12 Month Rolling Average
% 105 100 95 90 85 80 75 70 65 60 92/01 92/05 92/09 93/01 93/05 93/09 94/01 94/05 94/09 95/01 95/05 95/09 96/01 96/05 96/09 97/01 97/05 97/09 0 4 3 2 1 CMM Level 5
TG2 Rolling Average TG2 Max Roll Ave
TG2 Min Roll Ave CMM Level
SW-CMM
16
Fault Density, First 6 Months
Fault Density, First 6 Months of Operations (Plotted against PR-A Date)
Fault/KPlex 5 1.50 1.00 0.50 0.00
91-01 91-06 91-11 92-04 92-09 93-02 93-07 93-12 94-05 94-10 95-03 95-08 96-01 96-06 96-11 97-04 97-09
4 3 2 1 0
CMM Level
ETM/R Goal
Failure Density
12 mth Rolling avarage
CMM Level
SW-CMM
17
The Start of CMM
Millions of dollars wasted for DoD, Ada etc didn’t solve all problems. Management problem, not technical! Watts Humphrey, then at IBM, had an idea! SEI and MITRE Corporation began the development late 1986 Brief description - September 1987 Managing the Software Process by W. Humphrey- 1989 CMM Version 1.0 - 1991 Version 1.1 - 1993 Integrated CMM (CMMI) released in 2001
SW-CMM
18
Software-CMM by SEI:
“The Capability Maturity Model for Software (CMM) is a framework that describes the elements of an effective software process. The CMM describes an evolutionary path from an ad hoc, chaotic process, to a mature disciplined process”
SW-CMM
19
Capability Maturity Model Objectives
CMM is TQM with a software view, is based on state-of-the-art SW Engineering practices and helps organizations: Characterize the maturity of their process Establish goals for process improvement Set priorities for immediate actions Envision culture of software excellence
SW-CMM
20
Improvement Challenges
The process cannot be continuously improved if:
• • • • • Sound engineering practices are sacrificed to schedule There is no feedback on process performance Each person does something different Wide variation occurs in performing identical tasks Commitment to improve is not organization-wide
The CMM
• Overcomes these hurdles one by one
SW-CMM
21
CMM Design Rationale
Level 5 - Continuously improve process performance Level 4 - Control process variation Level 3 - Install a common process Level 2 - Stabilize the environment
SW-CMM
22
Maturity Framework
Continuous process improvement
Level 5: Optimizing Level 4: Managed
Change management
Process control
Process definition
Level 3: Defined
Engineering management
Quantitative management
Process discipline
Level 2: Repeatable Level 1: Initial
Project management
SW-CMM 23
CMM Architecture
Level Optimizing (5) Managed (4) Defined (3) Focus
Continous process improvement Product and process quality Defined engineering process
Key Process Areas
Defect prevention Technology change management Process change management Quantitative process management Software quality management Organisational process focus Organisational process definition Integrated software management Software product engineering Intergroup coordination Training program Peer reviews Requirements management Software project planning Software project tracking Software subcontract management Software quality assurance Software configuration management
Repeatable (2)
Project management and commitment process
Initial (1)
Heroes
SW-CMM
24
Key Process Area (KPA)
Key process area: Important component of behavior at a maturity level Collects a number of related activities The the lower Key Process Areas will evolve as we reach higher maturity levels. There are 18 Key Process Areas in the CMM
SW-CMM
25
Key Process Area Goals
Key process area goals: Enhance the capability of the processes related to the KPA Requirements for the maturity level Are used to guide: Process assessments Alternative practices to fulfill a KPA The most important description of a KPA!
SW-CMM
26
Key Practices
Practices
Maturity Levels
Key Process Area
Goals
Practices
Goals Key Process Area Goals
Practices
Practices
Practices
SW-CMM
27
Key Practices
Implements the goals of the KPA: Describe what to accomplish Does not describe how to accomplish it Neither exclusive, nor exhaustive There are 316 key practices in the CMM The Key practices are organized in Common Features
SW-CMM
28
Common Features
Goals Key Practices:
Commitment
Ability
Activities
Measurement
Verification
Institutionalization
Implementation
SW-CMM
29
Common Features
Common features: Organize the key practices Supports institutionalization of process performance It is important that the organization: Commits to perform - Commitment Supply the capabilities for performing - Ability Performs in a repeatable way - Activities Measure how well it performs - Measurement Follow-up and ensures that it performs - Verification
SW-CMM
30
Implementation vs. Institutionalization
Note that: Institutionalization leads to implementation Implementation does not have to lead to institutionalization
Do not forget the Institutionalization Key practices
SW-CMM
31
Commitment to Perform
Actions to ensure that the organization is committed to the activities in the KPA: Responsibilities are assigned Policies for the organization supports the activities Senior management sponsors the activities
SW-CMM
32
Ability to Perform
Preconditions for performing the activities in the KPA: Providing resources Providing training Establishing the necessary organizations (i.e. new roles and responsibilities)
SW-CMM
33
Activities
Tasks that implement the goals of a KPA: Planning of activities Performing activities Tracking activities Taking corrective actions Informing about activities and results
SW-CMM
34
Measurement and Analysis
Ensure that the activities are measured and analyzed: Measure effectiveness and degree of implementation Analyze to take corrective actions
SW-CMM
35
Verifying Implementation
Ensure that the activities are performed according to: Commitments Policies Processes Documented procedures
SW-CMM
36
The Structure of CMM - Summary
Commitment
Ability
Maturity Levels
Key Process Areas
Goals
Activities
Measurement & Analysis
Verification
SW-CMM
37
The Repeatable Process
Basic project management installed Projects are planned and tracked Changes to plans and baselines are managed Good practices are observed Commitments are made and approved Successful projects can be repeated in a stable managed environment Process capability exists for meeting schedules
SW-CMM
38
Responsibilities at Level 2
Organization
State expectations for project behavior Motivate - set incentives Allow the process to be followed Responsible for change Establish disciplined management Control baselines
Projects
Participate in making commitments Keep to agreed upon standards and requirements
Individual
SW-CMM
39
Moving to Level 2
Focus on projects - one by one Train Project Managers - skill by skill Implement measurements - do not complicate Institute Quality Assurance - project support Institute baseline control - all deliverables
Agreement
Manage Commitments
SW-CMM
40
The Commitment Process
A pact freely assumed, visible and expected to be kept by all parties.
• based on realistic plans • includes all affected parties
Participation increases commitment
A commitment process is the core of level 2
making - reviewing - approving commitments
SW-CMM
41
The Repeatable Level
5. Optimizing
4. Managed 3. Defined Requirements management Software project planning Software project tracking and oversight Software subcontract management Software quality assurance Software configuration management
2. Repeatable
1. Initial
SW-CMM
42
Requirement Management
customer
software project
SW-CMM
43
Why Requirements Management?
Establish - between customer and SW project a common understanding agreement on requirements maintain understanding and agreement
The agreement covers both technical and non-technical requirements is the basis for estimating, planning, performing and tracking
xxxxx xxx xxxxx
SW-CMM
44
Requirements Management Goals
1. System requirements allocated to software are controlled to establish a baseline for software engineering and management use. 2. Software plans, products, and activities are kept consistent with the system requirements allocated to software.
SW-CMM
45
Typical Problems
Requirements are volatile
• • • • customer wants changes and additions customers don’t fully understand their needs/requirements required functionality originally under-scoped technology and competition evolve
Customers don’t want to document requirements Software developers suffer from creeping elegance
• implement more functionality than required • solution more elegant than required
Commitments made before scope is understood
SW-CMM
46
What is Requirements Management
Software development requirements design implementation
gather
analyze/ specify verify
manage change
xxxx xxx xxxx.
understand - describe - agree
SW-CMM 47
Maturity framework
Continuously improve process performance
Level 5: Optimizing Level 4: Managed
Control process variation
Install a common process
Level 3: Defined Level 2: Repeatable
Stabilize the environment
Level 1: Initial
SW-CMM 48
Level 3 - the Defined Process
Standard software processes are
• defined • documented • applied throughout the organization
Shared understanding of how the process works and each person’s role is established Software engineering process group created to focus on process improvement Process capability to meet schedule, cost, and functionality targets exists within established product lines
SW-CMM
49
Moving to Level 3
At Level 2 the focus is on projects At level 3 the emphasis shifts to the organization:
• best practices are gathered from across the organization • standard processes are developed for the organization by integrating these best practices • processes are tailored for use on projects
Organization process capability
• Meet schedule, cost and functionality targets within established product lines
Process definition
Level 3: Defined
Engineering management
50
Level 2: Repeatable
SW-CMM
Problems to avoid on Level 3
Bureaucratic - the same process everywhere Overhead - tailoring is extra work Few best practices - nothing collected Too complicated - processes governing not supporting Over-documentation - tries to document everything in processes
SW-CMM
51
What is Defined?
Organization's Software process database
Library of software process related documentation
Organization's Approved software life cycles
Project Project 1 1 Project 1 Size Size Size $$$ $$$ $$$ Defects Defects Defects Results Results Results Lessons Lessons Lessons
Guidelines and criteria for tailoring the organization’s standard software process
Specification of organization's standard software process
Software process Architecture
•Developed from best practices •Tailored for each project •Consistent work products •Comparable measurements •Transfer of learning •Coordination of improvement
SW-CMM
Specifications of Software process elements
Paulk (1995) 52
Level 4 - the Managed Process
Statistical process control principles are used to address special causes of process variation Detailed process and product data are available - quality targets are set Process is capable of performing within narrowly defined quantitative limits - targets are predictable Detailed measures of the software process and product quality are collected Both the software process and products are quantitatively understood and controlled using detailed measures
SW-CMM
53
Moving to Level 4
At Level 3 systematic measures have been defined and collected At Level 4 decisions are made based on data analyses Projects are quantitatively managed and process and quality variation are reduced
Process control
Level 4: Managed
Level 3: Defined
SW-CMM
Quantitative management
54
What is Managed?
Control process variation
• statistical process control • remove special causes of process variation
Quantitative software management
• establish quantitative goals • manage process performance
Special cause of process variation (e.g. defect prone module)
Control Chart
Upper performance boundary Average process performance
Process performance
Lower performance boundary
SW-CMM
55
Level 5 - the Optimizing Process
Chronic causes of poor performance are identified and eliminated. Software process is continually improved New technologies are prototyped, piloted, and if successful, introduced into the process Process capability is continuously raised Continuous process improvement is enabled by quantitative feedback from the process and from testing innovative ideas and technologies
SW-CMM
56
Moving to Level 5
The organization's focus shifts from special to common causes • At Level 4 process and quality variation are reduced • At Level 5 the average process and quality levels are continuously improved Continuous improvement becomes a way of life • At the lower maturity levels worker participation on continuous improvement may be on the order of 20-30% • World-class companies have 70-80% participation in improvement programmes at any given time The organization institutionalizes change management
Continuous process improvement
Level 5: Optimizing Level 4: Managed
Change management
57
SW-CMM
What is Optimizing
Continuous process improvement
• Eliminate chronic causes of poor performance • Continuous evaluation of processes and technologies • Individuals empowered to improve their processes
Chronic cause of inefficiency eliminated (e.g. a new req. process)
Special cause
Initial capability
Upper performance boundary Improved capability Lower performance boundary
SW-CMM
58
Management Visibility by Level
Level 5
In
Out
Level 4 Level 3 Level 2 Level 1
In In In In
SW-CMM
Out Out Out Out
59
Process Capability by Level
Probability Probability Probability
5 4 3 2 1
target
target
target
Probability
target
Probability
target
SW-CMM
60
Changing the Organization
Organization
lit y St ab i
Organization establishes process framework Management establishes process discipline Project reduces process variation
t us Tr
Project
Individual
Ad hoc, Inconsistent undisciplined process
Individual improves personal process
SW-CMM
61
Cultural Changes by CMM Level
Level 5 Level 4 Level 3 Level 2 Level 1
Culture of empowerment and innovation Culture of performance excellence Culture of engineering Culture of commitment Imported culture
SW-CMM
62
Experiences - Raytheon’s Equipment Division
Raytheon’s Equipment Division, USA;
• 400 software engineers • real-time embedded radar and air traffic control
Started with process improvements 1988 Level 2 1990 Level 3 1992 Level 4 1995
Dion (1993), Haley (1995)
SW-CMM
63
Raytheon’s Cost Analysis Method
Crosby’s cost of quality model: Performance - cost of building it right the first time Nonconformance - cost of rework Appraisal - cost of testing Prevention - cost of preventing nonconformance Average% project time by cost type:
Perform 1988 1990 1992 1994 37 55 66 76
Nonconf 41 18 11 6
Appraisal 15 15 23 18
SW-CMM
Prevent 7 12
64
Cost of Quality
SW-CMM
65
Return on Investment
Invests $1.000.000 annually Reports an ROI of 5,7 times per year 7.7 times in 1990 Received a $9.600.000 bonus for early delivery on one system (not included in ROI numbers!)
SW-CMM
66
Better quality
SW-CMM
67
Higher productivity
SW-CMM
68
Predictability
SW-CMM
69
SEI data and experiences
CBA IPIs and SPAs conducted since 1987 through June 2002 and reported to the SEI by July 2002: 2325 assessments
• • 1840 CBA IPI 485 SPA
1756 organisations 512 different companies 451 reassessed organisations 9632 projects Kilde: http://www.sei.cmu.edu/sema/pdf/2002aug.pdf
SW-CMM
70
SEI Industry Analysis results
Benefit Area Productivity growth Pre-test defect detection Time to market Field error reports Cost of improvements Return on investment
Companies 4 3 2 5 5 5
Annual result 37% increase 18% increase 19% decrease 45% reduction $481.000 5.7 * investment
Herbsleb et al 1994
SW-CMM
71
Characteristics of successful organizations
Senior management actively monitors SPI progress Clearly stated, well understood SPI goals Staff time/resources dedicated to process improvement Clear, compensated assignment of responsibility SEPG staffed by highly respected people Technical staff is involved in improvement
(Bill Peterson (SEI), E-SEPG, 1996)
SW-CMM
72
Characteristics of less successful organizations
Turf guarding Organizational politics Cynicism from previously unsuccessful improvement experiences Beliefs that SPI gets in the way of real work Need more guidance on how to improve, not just what
(Bill Peterson (SEI), E-SEPG, 1996)
SW-CMM
73
More info
A collection of articles and papers mainy by Mark Paulk: http://www.sei.cmu.edu/activities/cmm/cmm.articles.html
SW-CMM
74