SW-CMM

Capability Maturity Model (CMM) for Software Development Geir Amsjø What do a customer want? Customers wants: Predictable performance • • • • On time deliveries Costs according to budget All functionality Reliable promises Visibility of progress and quality SW-CMM 2 Crisis-Driven Organizations Processes: • ad hoc - improvised and reinvented each time • abandoned under stress • reactionary Schedules and budgets: • unpredictable • usually exceeded Success factors: • heroes • overtime • fire fighting Blaming others: bad subcontractors, hardware problems... SW-CMM 3 Executive Problems in Software Low visibility into progress and quality Unpredictable performance Constant surprises Excessive costs Difficult to plan for company strategies SW-CMM 4 Project Manager Problems Hard to give accurate estimates • under-scoped requirements • schedule and resource shortfalls Unmanaged work • no corrective action • unruly subcontractors Volatile baselines • changing requirements • uncontrolled versions SW-CMM 5 What Frustrates Developers Being given half the time they need to do the work Continual overtime Guessing what requirements means Contending with uncontrolled changes in requirements Finding they are not working with the current version Redoing builds forever Delivering products they are not proud of Fixing old defects without design documentation Constantly patching up old software that has no architecture SW-CMM 6 Sources of Crisis Software development specifics: Product complexity Conformity – everything is possible Continuous changes Invisible product Explosive code growth Defect removal costs Culture Partly from Brooks SW-CMM 7 Cost (i.e Effort) to Remove Defects Cost/Effort 1-2 orders of magnitude Cost to repair Time/phase SW-CMM 8 Improvement Areas Technology Product People Process SW-CMM 9 The Role of the Process Management Environment Process Disciplined Methods Staff Technical assets Reference: Bill Curtis, TeraQuest Metrics SW-CMM 10 Definition of Software Process ”…Software Process can be defined as a set of activities, methods, practices, and transformations that people use to develop and maintain software and the associated products (e.g., project plans, design documents, code, test cases, and user manuals).” Definition from SEI technical report SEI-93-TR-24, ”Capability Maturity Model for Software, Version 1.1” SW-CMM 11 Potentials in Process-Based Development Documents our common experience • alive and updated • learn from our mistakes Promotes teamwork • defines roles, interfaces and dependencies Promotes individual training • needs of the company and the individual Defines activities and output • nothing neglected nor superfluous SW-CMM 12 Misconceptions About Process I do not need process if I have • • • really good people advanced technology an experienced manager Process Process Process Process interferes with creativity = bureaucracy hinders agility in fast-moving markets is only useful on large projects SW-CMM 13 What is an Effective Process? Defined Documented Used Supported by management (enforced) Trained Measured/Known in quantitative terms Supported by tools and technology Tailored Improved/maintained SW-CMM 14 The Process is a Part of the Competition Process is a competitive weapon in almost every industry except software development Our software process decides quality, cost and cycle-time Software Development • Historically not considered to be engineering “The quality of a software system is governed by the quality of the process used to develop and evolve it.” Watts Humphrey (Father of the CMM) SW-CMM 15 Internal Delivery Precision Deviation between Max/Min of 12 Month Rolling Average % 105 100 95 90 85 80 75 70 65 60 92/01 92/05 92/09 93/01 93/05 93/09 94/01 94/05 94/09 95/01 95/05 95/09 96/01 96/05 96/09 97/01 97/05 97/09 0 4 3 2 1 CMM Level 5 TG2 Rolling Average TG2 Max Roll Ave TG2 Min Roll Ave CMM Level SW-CMM 16 Fault Density, First 6 Months Fault Density, First 6 Months of Operations (Plotted against PR-A Date) Fault/KPlex 5 1.50 1.00 0.50 0.00 91-01 91-06 91-11 92-04 92-09 93-02 93-07 93-12 94-05 94-10 95-03 95-08 96-01 96-06 96-11 97-04 97-09 4 3 2 1 0 CMM Level ETM/R Goal Failure Density 12 mth Rolling avarage CMM Level SW-CMM 17 The Start of CMM Millions of dollars wasted for DoD, Ada etc didn’t solve all problems. Management problem, not technical! Watts Humphrey, then at IBM, had an idea! SEI and MITRE Corporation began the development late 1986 Brief description - September 1987 Managing the Software Process by W. Humphrey- 1989 CMM Version 1.0 - 1991 Version 1.1 - 1993 Integrated CMM (CMMI) released in 2001 SW-CMM 18 Software-CMM by SEI: “The Capability Maturity Model for Software (CMM) is a framework that describes the elements of an effective software process. The CMM describes an evolutionary path from an ad hoc, chaotic process, to a mature disciplined process” SW-CMM 19 Capability Maturity Model Objectives CMM is TQM with a software view, is based on state-of-the-art SW Engineering practices and helps organizations: Characterize the maturity of their process Establish goals for process improvement Set priorities for immediate actions Envision culture of software excellence SW-CMM 20 Improvement Challenges The process cannot be continuously improved if: • • • • • Sound engineering practices are sacrificed to schedule There is no feedback on process performance Each person does something different Wide variation occurs in performing identical tasks Commitment to improve is not organization-wide The CMM • Overcomes these hurdles one by one SW-CMM 21 CMM Design Rationale Level 5 - Continuously improve process performance Level 4 - Control process variation Level 3 - Install a common process Level 2 - Stabilize the environment SW-CMM 22 Maturity Framework Continuous process improvement Level 5: Optimizing Level 4: Managed Change management Process control Process definition Level 3: Defined Engineering management Quantitative management Process discipline Level 2: Repeatable Level 1: Initial Project management SW-CMM 23 CMM Architecture Level Optimizing (5) Managed (4) Defined (3) Focus Continous process improvement Product and process quality Defined engineering process Key Process Areas Defect prevention Technology change management Process change management Quantitative process management Software quality management Organisational process focus Organisational process definition Integrated software management Software product engineering Intergroup coordination Training program Peer reviews Requirements management Software project planning Software project tracking Software subcontract management Software quality assurance Software configuration management Repeatable (2) Project management and commitment process Initial (1) Heroes SW-CMM 24 Key Process Area (KPA) Key process area: Important component of behavior at a maturity level Collects a number of related activities The the lower Key Process Areas will evolve as we reach higher maturity levels. There are 18 Key Process Areas in the CMM SW-CMM 25 Key Process Area Goals Key process area goals: Enhance the capability of the processes related to the KPA Requirements for the maturity level Are used to guide: Process assessments Alternative practices to fulfill a KPA The most important description of a KPA! SW-CMM 26 Key Practices Practices Maturity Levels Key Process Area Goals Practices Goals Key Process Area Goals Practices Practices Practices SW-CMM 27 Key Practices Implements the goals of the KPA: Describe what to accomplish Does not describe how to accomplish it Neither exclusive, nor exhaustive There are 316 key practices in the CMM The Key practices are organized in Common Features SW-CMM 28 Common Features Goals Key Practices: Commitment Ability Activities Measurement Verification Institutionalization Implementation SW-CMM 29 Common Features Common features: Organize the key practices Supports institutionalization of process performance It is important that the organization: Commits to perform - Commitment Supply the capabilities for performing - Ability Performs in a repeatable way - Activities Measure how well it performs - Measurement Follow-up and ensures that it performs - Verification SW-CMM 30 Implementation vs. Institutionalization Note that: Institutionalization leads to implementation Implementation does not have to lead to institutionalization Do not forget the Institutionalization Key practices SW-CMM 31 Commitment to Perform Actions to ensure that the organization is committed to the activities in the KPA: Responsibilities are assigned Policies for the organization supports the activities Senior management sponsors the activities SW-CMM 32 Ability to Perform Preconditions for performing the activities in the KPA: Providing resources Providing training Establishing the necessary organizations (i.e. new roles and responsibilities) SW-CMM 33 Activities Tasks that implement the goals of a KPA: Planning of activities Performing activities Tracking activities Taking corrective actions Informing about activities and results SW-CMM 34 Measurement and Analysis Ensure that the activities are measured and analyzed: Measure effectiveness and degree of implementation Analyze to take corrective actions SW-CMM 35 Verifying Implementation Ensure that the activities are performed according to: Commitments Policies Processes Documented procedures SW-CMM 36 The Structure of CMM - Summary Commitment Ability Maturity Levels Key Process Areas Goals Activities Measurement & Analysis Verification SW-CMM 37 The Repeatable Process Basic project management installed Projects are planned and tracked Changes to plans and baselines are managed Good practices are observed Commitments are made and approved Successful projects can be repeated in a stable managed environment Process capability exists for meeting schedules SW-CMM 38 Responsibilities at Level 2 Organization State expectations for project behavior Motivate - set incentives Allow the process to be followed Responsible for change Establish disciplined management Control baselines Projects Participate in making commitments Keep to agreed upon standards and requirements Individual SW-CMM 39 Moving to Level 2 Focus on projects - one by one Train Project Managers - skill by skill Implement measurements - do not complicate Institute Quality Assurance - project support Institute baseline control - all deliverables Agreement Manage Commitments SW-CMM 40 The Commitment Process A pact freely assumed, visible and expected to be kept by all parties. • based on realistic plans • includes all affected parties Participation increases commitment A commitment process is the core of level 2 making - reviewing - approving commitments SW-CMM 41 The Repeatable Level 5. Optimizing 4. Managed 3. Defined Requirements management Software project planning Software project tracking and oversight Software subcontract management Software quality assurance Software configuration management 2. Repeatable 1. Initial SW-CMM 42 Requirement Management customer software project SW-CMM 43 Why Requirements Management? Establish - between customer and SW project a common understanding agreement on requirements maintain understanding and agreement The agreement covers both technical and non-technical requirements is the basis for estimating, planning, performing and tracking xxxxx xxx xxxxx SW-CMM 44 Requirements Management Goals 1. System requirements allocated to software are controlled to establish a baseline for software engineering and management use. 2. Software plans, products, and activities are kept consistent with the system requirements allocated to software. SW-CMM 45 Typical Problems Requirements are volatile • • • • customer wants changes and additions customers don’t fully understand their needs/requirements required functionality originally under-scoped technology and competition evolve Customers don’t want to document requirements Software developers suffer from creeping elegance • implement more functionality than required • solution more elegant than required Commitments made before scope is understood SW-CMM 46 What is Requirements Management Software development requirements design implementation gather analyze/ specify verify manage change xxxx xxx xxxx. understand - describe - agree SW-CMM 47 Maturity framework Continuously improve process performance Level 5: Optimizing Level 4: Managed Control process variation Install a common process Level 3: Defined Level 2: Repeatable Stabilize the environment Level 1: Initial SW-CMM 48 Level 3 - the Defined Process Standard software processes are • defined • documented • applied throughout the organization Shared understanding of how the process works and each person’s role is established Software engineering process group created to focus on process improvement Process capability to meet schedule, cost, and functionality targets exists within established product lines SW-CMM 49 Moving to Level 3 At Level 2 the focus is on projects At level 3 the emphasis shifts to the organization: • best practices are gathered from across the organization • standard processes are developed for the organization by integrating these best practices • processes are tailored for use on projects Organization process capability • Meet schedule, cost and functionality targets within established product lines Process definition Level 3: Defined Engineering management 50 Level 2: Repeatable SW-CMM Problems to avoid on Level 3 Bureaucratic - the same process everywhere Overhead - tailoring is extra work Few best practices - nothing collected Too complicated - processes governing not supporting Over-documentation - tries to document everything in processes SW-CMM 51 What is Defined? Organization's Software process database Library of software process related documentation Organization's Approved software life cycles Project Project 1 1 Project 1 Size Size Size $$$ $$$ $$$ Defects Defects Defects Results Results Results Lessons Lessons Lessons Guidelines and criteria for tailoring the organization’s standard software process Specification of organization's standard software process Software process Architecture •Developed from best practices •Tailored for each project •Consistent work products •Comparable measurements •Transfer of learning •Coordination of improvement SW-CMM Specifications of Software process elements Paulk (1995) 52 Level 4 - the Managed Process Statistical process control principles are used to address special causes of process variation Detailed process and product data are available - quality targets are set Process is capable of performing within narrowly defined quantitative limits - targets are predictable Detailed measures of the software process and product quality are collected Both the software process and products are quantitatively understood and controlled using detailed measures SW-CMM 53 Moving to Level 4 At Level 3 systematic measures have been defined and collected At Level 4 decisions are made based on data analyses Projects are quantitatively managed and process and quality variation are reduced Process control Level 4: Managed Level 3: Defined SW-CMM Quantitative management 54 What is Managed? Control process variation • statistical process control • remove special causes of process variation Quantitative software management • establish quantitative goals • manage process performance Special cause of process variation (e.g. defect prone module) Control Chart Upper performance boundary Average process performance Process performance Lower performance boundary SW-CMM 55 Level 5 - the Optimizing Process Chronic causes of poor performance are identified and eliminated. Software process is continually improved New technologies are prototyped, piloted, and if successful, introduced into the process Process capability is continuously raised Continuous process improvement is enabled by quantitative feedback from the process and from testing innovative ideas and technologies SW-CMM 56 Moving to Level 5 The organization's focus shifts from special to common causes • At Level 4 process and quality variation are reduced • At Level 5 the average process and quality levels are continuously improved Continuous improvement becomes a way of life • At the lower maturity levels worker participation on continuous improvement may be on the order of 20-30% • World-class companies have 70-80% participation in improvement programmes at any given time The organization institutionalizes change management Continuous process improvement Level 5: Optimizing Level 4: Managed Change management 57 SW-CMM What is Optimizing Continuous process improvement • Eliminate chronic causes of poor performance • Continuous evaluation of processes and technologies • Individuals empowered to improve their processes Chronic cause of inefficiency eliminated (e.g. a new req. process) Special cause Initial capability Upper performance boundary Improved capability Lower performance boundary SW-CMM 58 Management Visibility by Level Level 5 In Out Level 4 Level 3 Level 2 Level 1 In In In In SW-CMM Out Out Out Out 59 Process Capability by Level Probability Probability Probability 5 4 3 2 1 target target target Probability target Probability target SW-CMM 60 Changing the Organization Organization lit y St ab i Organization establishes process framework Management establishes process discipline Project reduces process variation t us Tr Project Individual Ad hoc, Inconsistent undisciplined process Individual improves personal process SW-CMM 61 Cultural Changes by CMM Level Level 5 Level 4 Level 3 Level 2 Level 1 Culture of empowerment and innovation Culture of performance excellence Culture of engineering Culture of commitment Imported culture SW-CMM 62 Experiences - Raytheon’s Equipment Division Raytheon’s Equipment Division, USA; • 400 software engineers • real-time embedded radar and air traffic control Started with process improvements 1988 Level 2 1990 Level 3 1992 Level 4 1995 Dion (1993), Haley (1995) SW-CMM 63 Raytheon’s Cost Analysis Method Crosby’s cost of quality model: Performance - cost of building it right the first time Nonconformance - cost of rework Appraisal - cost of testing Prevention - cost of preventing nonconformance Average% project time by cost type: Perform 1988 1990 1992 1994 37 55 66 76 Nonconf 41 18 11 6 Appraisal 15 15 23 18 SW-CMM Prevent 7 12 64 Cost of Quality SW-CMM 65 Return on Investment Invests $1.000.000 annually Reports an ROI of 5,7 times per year 7.7 times in 1990 Received a $9.600.000 bonus for early delivery on one system (not included in ROI numbers!) SW-CMM 66 Better quality SW-CMM 67 Higher productivity SW-CMM 68 Predictability SW-CMM 69 SEI data and experiences CBA IPIs and SPAs conducted since 1987 through June 2002 and reported to the SEI by July 2002: 2325 assessments • • 1840 CBA IPI 485 SPA 1756 organisations 512 different companies 451 reassessed organisations 9632 projects Kilde: http://www.sei.cmu.edu/sema/pdf/2002aug.pdf SW-CMM 70 SEI Industry Analysis results Benefit Area Productivity growth Pre-test defect detection Time to market Field error reports Cost of improvements Return on investment Companies 4 3 2 5 5 5 Annual result 37% increase 18% increase 19% decrease 45% reduction $481.000 5.7 * investment Herbsleb et al 1994 SW-CMM 71 Characteristics of successful organizations Senior management actively monitors SPI progress Clearly stated, well understood SPI goals Staff time/resources dedicated to process improvement Clear, compensated assignment of responsibility SEPG staffed by highly respected people Technical staff is involved in improvement (Bill Peterson (SEI), E-SEPG, 1996) SW-CMM 72 Characteristics of less successful organizations Turf guarding Organizational politics Cynicism from previously unsuccessful improvement experiences Beliefs that SPI gets in the way of real work Need more guidance on how to improve, not just what (Bill Peterson (SEI), E-SEPG, 1996) SW-CMM 73 More info A collection of articles and papers mainy by Mark Paulk: http://www.sei.cmu.edu/activities/cmm/cmm.articles.html SW-CMM 74