中華民國品質學會第 42 屆年會暨第 12 屆全國品質管理研討會
Quality-Enhanced Roadmap for ISO 9001 Registered Organizations
Wen-Kui Chang1 Shih-Fang Huang2 Jeng-Feng Yang3 1 Dept. of Computer Science & Information Engineering, Tunghai University, Taichung 2 Motive Power Industry Co., Ltd. Changhua 3 GIS Research Center, Feng Chia University, Taichung Email: 1wkc@thu.edu.tw, 2carson@pgoscooter.com.tw, 3gary@gis.tw
Abstract Normally starting a software quality program from scratch is time consuming and a task often doomed to failure before it is begun. Inadequate preparation and failure to specify the tailored process sequences in the organization are only a few of the pitfalls waiting for the overanxious practitioner. Standards are the keystone of a software quality system to provide the basis against which activities can be measured and evaluated. In nowadays, there exist many international standards and development guides such as DO-178B, ISO/IEC 12207, ISO/IEC 15504, CMMI, etc., to provide common methods and practices while developing a complex project. In the CMMI continuous representation, a target profile establishes a target of capability level profile for process improvement. In particular, an organization, before deploying an improvement program, has to specify her target staging in terms of a sequence of target profiles to describe the path of process improvement. In this paper, we investigate the issue of process improvement in a software-intensive organization and establish a quality system for emphasizing software verification and validation (V&V) in the aspects of capability levels and its integrity levels. Considering the practical resources at a middle-scale software-intensive organization in Taiwan area, we propose a feasible, efficient and economical roadmap for software process improvement, no matter the company is ISO 9001:2000 registered or not yet, which then provides a shortcut to enhance V&V tasks for the ISO-9001 registered software organizations to achieve the second integrity level.
Keywords: Process Improvement, Verification and Validation, Capability Maturity Model Integration (CMMI), Target Profile, Integrity Level
1. Introduction Recently more organizations play much emphasis on verification and validation (V&V) tasks in order to deliver quality product. To reach the objectives of the software validation process, both static and dynamic techniques of system checking and analysis are usually employed. However, static techniques can only check the correspondence between software and its specification, i.e., the so-called verification process. They cannot demonstrate sufficiently that the software is operationally valid.
Standards are the keystone of a software quality system to provide the basis against activities, which can be measured and evaluated. In nowadays, there exist many international standards and development guides that may provide common methods and practices so that the same task can be
1
accomplished the same way each time it is done. For instance, DO-178B provides guidance for determining the software aspects of airborne systems and equipment complied with airworthiness requirements; and ISO/IEC 15504 establishes a quantitative standard in the area of process assessment, etc. In this research, we investigate the full set of Capability Maturity Model Integration (CMMI) models (Chrissies, Konrad & Shrum, 2003) released by the Software Engineering Institute (SEI) in 2002. In the CMMI continuous representation, a target profile establishes a baseline of capability level profile to represent an objective for process improvement. In particular, an organization before deploying an improvement program has to specify her target staging to denote a sequence of target profiles that describes the path of process improvement to be followed.
The CMMI has been proven that it is efficient for achieving product and process improvement and widely accepted and implemented in the IT area (Carter et. al., 2002). However, it usually requires lots of project budget and effort for an organization to implement the CMMI framework for reaching a higher maturity level. It may become a bottleneck on promoting CMMI to middle-scale software organizations, which popularly exist in Taiwan areas. Accordingly, we propose a software quality-enhanced framework for the middle-scale, ISO 9001:2000 registered software organization, in performing the required V&V tasks from the perspective of the Process Areas (PAs) of the CMMI model, by the fact that most of current information technology companies in Taiwan area have been registered ISO 9000. In our belief, once an organization that has achieved ISO 9001 registration, but she hopes to improve processes continuously, CMMI can be a strong candidate because it provides a more detailed roadmap for process improvement.
Furthermore, in referring with IEEE Standard 1012-2004-Standards for Software Verification and Validation Plans (IEEE, 2004), we summarize the minimum V&V tasks and use a software-integrity-level scheme based upon software intended use and quantify application of the system to criticality. By integrating the software integrity level involves capability level in the continuous representation CMMI model; these minimum V&V tasks establish a stepwise roadmap for capability level from the first to the fourth level. The benefit of our proposed roadmap of improving software process within a middle-scale organization provides an effective, efficiency and economical approach, no matter the companies is ISO 9001:2000 registered or not yet. In the following, we first give a bird-view on the related international standards in the subsequent section. Section 3 investigates relative mappings of CMMI PAs to ISO 9001:2000. In Section 4, we summarize the minimum V&V tasks and use a software-integrity-level scheme upon which the proposed roadmap bases. Finally, target profiles for V&V efforts with the second integrity level in introduced in Section 5. 2. Major Standards Essentially, the full set of CMMI models released by SEI in January 2002 aims to provide guidance for improving an ogn ao’poess n ait t m ng t dvl m nt, acquisition, rai t ns rcs ad b i o aaeh ee p e zi e ly e o and maintenance of his software products (Ahern, Clouse & Turner, 2001). Furthermore, CMMI model may be useful for appraising its organizational maturity or process area capability, establishing priorities for improvement, and implementing these improvements. In CMMI models, process areas describe key aspects of such processes as requirements management, configuration management, verification, validation, and many others. Specifically, a process area (PA) provides a list of the required practices to achieve its intended goals, but it does not describe how an effective process is executed, e.g., entrance and exit criteria, roles of participants and
2
resources. Currently there are two types of CMMI model representations: staged and continuous. The staged representation uses predefined sets of process areas to define an improvement path for an organization. A maturity level is a well-defined evolutionary plateau toward achieving the improved organizational processes. Oppositely, the continuous representation allows an organization to select a specific process area and improve relative to it. This representation uses capability levels to characterize relative improvement to an individual process area. On the other hand, the ISO 9000 family of international quality management standards and guidelines has become an international reference basis for establishing quality management systems. In particular, ISO 9001:2000 specifies requirements for a quality management system where an organization needs to demonstrate its ability to consistently produce product that meets customer and applicable regulatory requirements, while aims to enhance customer satisfaction through the processes for continual improvement of the system. Since their popularity of the ISO 9000 family and CMMI models, relationships between the two models have been studied in this paper. Paulk studies the relationship between ISO 9001 paragraphs and CMM Key Process Areas (Paulk, 1995). Due to the CMMI published by SEI in 2002, its content of models is quite different with that of CMM. In this paper, some mapping results of the ISO 9001:2000 clauses to their corresponding CMMI PAs will be discussed in the subsequent section. 3. Coverage of CMMI PAs with ISO 9001 Since late 2000, significant interest has been seen in certification and registration for many organizations under ISO 9001:2000 and in transition from the CMM to the CMMI. In contrast to ISO 9001:2000 that can be applied to any organization regardless of its field in which it operates. The CMMI specifically focuses on organizations that develop products and systems containing software. While the CMMI provides a roadmap for achieving process capability or maturity levels (Mutafelija, 2001), ISO requires all of its requirements to be fulfilled before certification can be issued. Furthermore, both ISO and the CMMI are based on principles of systems engineering and a process approach. In the following, we strive to compare ISO requirements to CMMI PAs and specific practices to depict their corresponding mappings. To be specific in this paper, we limit our discussion to the ML2 and ML3 PAs of CMMI SE/SW disciplines. 3.1 Mapping CMMI process areas to ISO 9001:2000 clauses As stated in the CMMI technical report (Carter et. al., 2002), the Requirements Management (REQM) PA essentially maintains the project requirements. It describes activities for obtaining and controlling requirement changes to ensure other relevant plans and data currently kept. Furthermore, it provides traceability of requirements from customer to product, till the product component. After analyzing the corresponding interpretations, the goals of the REQM PA may be equally performed by the clauses 4.1 and 4.2 of ISO 9001, as shown in the first row on Table 1. Similarly, according to (Ahern, Clouse & Turner, 2001; Yoo, 2006), the Project Planning (PP) PA involves the various tasks such as developing the project plan, interacting with stakeholders appropriately, getting commitment to the plan and maintaining the plan. By the interpretation rationale of reaching the same purposes, we conclude that the tasks of the PP PA correspond to the clauses 4.1, 5.1, 5.4 and 7.1 of ISO 9001, as listed on the second row on the same table.
On the same way, we summarize the result for all PAs of the CMMI model in Table 1 after careful study, the cross mappings from CMMI process areas at different capability levels to the corresponding clauses under Sections 5– in ISO 9001:2000. Naturally, the illustrated mapping between ISO 8
3
9001:2000 and CMMI seems a subjective association. Actually, we have concluded this result based on our several experimental case studies, which have tried to verify its correctness and evaluate its contribution. Due to the space limitation, the detailed information will not repeat here. Generally, the ISO 9001:2000 allows an organization more flexible in the way chose to document its quality management system. ISO 9001 does not contain any explicit requirements for the software development process, because it was originally designed for application in a broad number of topics, including development of products, systems or services. In a sense, this "flexibility" makes ISO 9001 quite difficult to implement. CMMI add value and detail to ISO 9001:2000 clause descriptions (Mutafelija, 2001).
3.2 Primary & auxiliary process areas According to the CMMI framework, the verification (VER) PA ensures that selected work products meet the specified requirements. VER is generally an incremental process, starting with product-component verification and usually concluding with verification of fully assembled products. The validation (VAL) PA incrementally validates products against t cs m r nes h ut e s ed and may be e o ’ performed in the operational environment or a simulated operational environment. With studying the generic goals of the VER and VAL PAs in a software development process, some related process areas including RD, REQM, and TS as listed in (Chrissis, Konrad & Shrum, 2003) are required to establish the baseline infrastructure. The RD PA is needed for the generation and development of customer, product, and product-component requirements in order to validate requirements, while the REQM PA aids for managing requirements. Moreover, the TS PA may provide assistance to transform requirements into product specifications for the corrective action when validation issues are identified to affect the product or product-component design. Thus, in this paper, these five PAs will be regarded as the primary process areas to perform the required V&V tasks as briefed in Table 3. On the other hand, while in software project development, both the PP and MA PAs are usually the key to successful implementation of a variety of process areas. Furthermore, from the perspective of IEEE Std 1012-2004, some other PAs such as PPQA, CM, PI, PMC are essentially needed in implementation the V&V tasks. In the following, PAs such as PP, MA, PPQA, CM, PI and PMC are named the auxiliary process areas to suggest a requisite framework for an organization who has not yet obtained ISO 9001 registration but desires to improve her software improvement in the interest of V & V areas.
4
Table 1. Comparison of CMMI ML 2 -3 PAs and ISO Sections 5– 8. CMMI process areas
Requirement Management
Acronyms REQM
Capability level mapping 0 1 2 3 4 5
ISO 9001:2000 clauses
4.1 General requirements 4.2 Documentation requirements 7.3.2 Design and development inputs 7.3.A.1Design and development process management 4.1 General requirements 5.1 Management commitment 5.4 Planning 7.1 Planning of product realization 7.3.1.1Establishing design and development plan 4.1 General requirements 5.1 Management commitment 7.3.4Design and development review 7.6 Control of monitoring and measuring devices 8.2 Measurement and monitoring 4.1 General requirements 7.4 Purchasing 7.4.1Purchasing process 7.4.3Verification of purchased product 7.5 Production and service operations 8.2 Measurement and monitoring 8.2.3.1Monitoring and measurement of processes 8.2.3.2Monitoring and measurement of product 8.4 Analysis of data 8.4.A Measurement management 4.1 General requirements 5.1 Management commitment 5.2 Customer focus 5.3 Quality policy 8.2.2Internal audit 4.2 Documentation requirements 7.3 Design and/or development 7.3.7Control of design and development changes 7.5 Produuction and service operations 7.5.3Identification and traceability 5.2 Customer focus 7.2 Customer-related processes 7.2.1Determination of requirements related to the product 7.2.2Review of requirements to the product 7.3 Design and/or development 7.3 Design and/or development 7.3.A.2Technical solution 7.3 Design and/or development 7.3.A.3Production integration 7.5.5Preservation and delivery of product 7.1 Planning of product realization 7.3 Design and/or development 7.3.5Design and development verification 7.5 Production and service operations 7.1 Planning of product realization 7.3 Design and/or development 7.3.6Design and development validation 5.5 Administration 8.2.2Internal audit 8.4 Analysis of data-? 8.5.2.1Deploying improvement 4.2 Documentation requirements 4.2.1General 4.2.2.1Organization’set of standard process s 4.2.2.2 Organization’set of standard process tailoring criteria s and guidelines 4.2.5Process assets mangement 4.2.6Measurement management-MA? 5.3 Quality policy 5.4 Planning 5.5 Administration
Project Planning
PP
Project Monitoring and Control Supplier Agreement Management
PMC
SAM
Measurement and Analysis
MA
Process and Product Quality Assurance
PPQA
Configuration Management
CM
Requirement Development Technical Solution Product Integration
RD TS PI VER
Verification
Validation
VAL
Organizational Process Focus
OPF
Organizational Process Definition
OPD
5
Organization Training
OT
Integrated Project Management
IPM
Integrated Supplier Management
ISM
6.2 Human resources 6.2.2Competence, awareness and training 4.2.3Control of documents 4.2.5Process assets mangement 5.4 Planning 5.5 Administration 7.1 Planning of realization processes 7.3.1.1Establishing design and development plan 7.3.1.2Team composition and operation 7.3.4 Design and development review 6.1 Provision of resources 7.4 Purchasing 7.4.1Purchasing process 7.4.2Purchasing information 7.4.3Verification of purchased product 5.1 Management commitment 7.3.1.3Risk management 7.3.4Design and development review
Risk Management
RSKM
4. V&V efforts under the second integrity level In practice, software systems exhibit different levels of criticality based upon their intended purposes and cost impact due to their system failures. To consider the trade-offs between the criticality levels and the paid effort, software-development organizations may strategically choose a lower integrity level to save the development effort (IEEE, 2004), if its cost impact, once it is happened, is acceptable or negligible. More detailed, software integrity levels denote a range of software criticality values that are necessary to maintain risks within an acceptable limit. These software quality metrics include safety, security, software complexity, performance, reliability, correctness or other characteristics. Generally, critical and high-integrity software typically requires a larger set and more rigorous application of V&V tasks. To identify the minimum V&V tasks that apply to the different integrity-level software systems, the software developers may refer to the IEEE STD 1012-2004 for the complete list. The IEEE STD 1012-2004 is a technical discipline of systems engineering. The purpose of IEEE STD 1012-2004 is to help the development organization build quality into the software during the software life cycle. In this paper, we are limited ourselves to the non-critical commercial applications that exist the most popular in the medium-scale organizations in Taiwan. Thus, for the system in non-critical uses, Table 2 delineates the minimum V&V tasks assigned to integrity level 2, in correspondence with the ISO 9001:2000 clauses and CMMI PAs and capability level (CL) as well. In that table, we accept the general framework on dividing the whole software life cycle (SLC) into 5 periods: concept phase, requirement phase, design phase, implementation phase and test phase. For the concept phase, the minimum V&V tasks include two tasks, i.e., the concept-documentation evaluation and criticality analysis, which derive from the IEEE STD 1012-2004. Each task is further corresponding to the associate clauses of ISO 9001. The final column of the table shows the corresponding PA and its capability level that will be attained after the concerned task is performed. For instance, the concept-documentation evaluation task in (IEEE, 2004) will perform the same effect as both the REQA PA at the capability-level 3 and the PP PA at the capability-level 2 as well. In terms of CMMI terminology, a capability-l e 2poesi ca c r e a a“ aae poes e l rcs s hr t i d s m ngd rcs v a ez ,”while a capability-level 3 as a “ defined process.” A critical distinction between a managed process and a defined process is the scope of application of the process descriptions, standards, and procedures. For a managed process, the process descriptions, standards, and procedures are applicable to a particular project, group, or organizational function. As a result, the managed processes for two projects within the same organization may be very different. Whereas, at the defined capability level, the organization is interested in deploying standard processes that govern all related projects.
6
More detailed, the essential process elements for each minimum V&V tasks are investigated and summarized in Table 2 in order to be ready for implementation.
Table 2. Minimum V&V tasks at the second integrity level. SLC Phase Minimum V&V Tasks
Concept Documentation evaluation Criticality Analysis
ISO 9001:2000 Clauses
4.1 General requirements 4.2 Documentation requirements 7.1-Planning of product realization 7.2.1-Determination of requirements related to the product 7.2.2-Review of requirements related to the product 7.3.A.1Design and development process management 5.2 Customer focus 5.3 Quality policy 5.4.1-Quality objective 5.4.2-Quality management system planning 7.2.1- Determination of requirements related to the product 7.5.3-Identification and Traceability 7.2 Customer-related processes 7.2.1Determination of requirements related to the product 7.3.2-Design and development inputs 7.3.3-Design and development outputs
associated CMMI PAs & CL
REQM (CL4) PP (CL3) REQM (CL4)
Concept
Acceptance V&V test plan generation and verification Requirement
PPQA (CL4) PP (CL3)
Criticality Analysis
RD (CL4)
Design
Implementation
Component V&V test 7.3.1-Design and development planning plan generation and 7.3.5-Design and development verification verification 7.2.2Review of requirements to the product Criticality Analysis 7.3.4-Design and development review Component V&V test 7.3.1-Design and development planning execution and 7.3.6-Design and Development Validation verification Criticality Analysis 7.3.7-Control of design and development changes Acceptance V&V test 7.5.1-Control of production and service provision execution and 8.2 Measurement and monitoring verification 5.4.1-Quality objectives Acceptance V&V test 5.4.2-Quality management system planning procedure generation 7.3.6-Design and Development Validation and verification 7.5.2-Validation of process for production and service provision
VER (CL4) PP (CL3) RD (CL4) VER (CL4) VAL (CL3) CM (CL4) PI (CL3) CM (CL4) PMC (CL3) VER (CL4)
Test
VER (CL4) VAL (CL3) PP (CL3)
5. Proposed roadmap to process improvement Naturally, there are many ways to enhance software process improvement within a software organization by the approach of implementing CMMI model. In the following, we propose a practical and systematic sequence for those middle-scale software organizations, from the perspective of minimum V &V effort at the second integrity level to save the software development cost. At this stage, we set the goal of software improvement from none to the third capability-level with the purpose of making the proposed roadmap is easily implemented for an organization that starts to employ the philosophy of CMMI model. In general, the amount of V&V effort required for software project depends on its performance requirement and naturally, it does not related directly to the size of a software organization. It is noted that software-integrity levels relate to the project criticality instead of software organization. In this paper, we have observed that most local middle-scale software organizations in Taiwan area are developing non-critical business projects. That is why we are concerned with the software projects of the integrity-level 2, although some higher integrity-level software of specific domain are actually kept in a small size to reduce V&V cost and carried by a relatively small team.
7
Furthermore, in considering with some software-related organizations in Taiwan, which have been ISO 9001:2000 registered, the suggested roadmap may have two different options basing on their current situations as illustrated in Fig. 1 and the proposed target profile in Table 3. As shown in Fig. 1, a software organization that has been ISO-9001 registered may directly perform the primary process areas (i.e., RD, REQM, TS, VER and VAL) to benefit her achievement from ISO efforts. Taking REQM as an example, she needs to enhance the concept-documentation evaluation task by improving her original 7.1 clause (Planning of product realization) to the capability level 4, as illustrated on the first row of Table 2. Alternatively, if a software organization has not ever practices of ISO-9001 but desires to implement her process improvement through the continuous CMMI model, she has to start with those auxiliary process areas as suggested in Fig. 1. Accordingly, the proposed roadmap provides an obvious shortcut to enhance process improvement for the ISO-9001 registered software organizations.
CL 0 ISO 9001:2000 Registered? YES Equivalent ML3
Level 5 Level 4
CL 0 ISO 9001:2000 Registered? NO
Level 3 Level 2 Level 1 Level 0
RD REQM TS VAL VER
Capability Level
Primary process areas
Level 5 Level 4 Level 3 Level 2 Level 1 Level 0
PP MA PPQA CM PI
PMC
Auxiliary process areas
Fig. 1 The proposed roadmap for minimum V&V tasks at the second integrity level
Table 3. Target profiles for V&V efforts with the second integrity level.
Category PA
RD REQM
CL1
CL2
CL3
CL4
CL5
Primary
TS VER VAL PP MA PPQA
Auxiliary
CM PI PMC
Dug into a software life cycle, the proposed framework identifies the most important tasks for performing the minimum V&V tasks in order to ensure the developed software in accordance with functional specifications and customer’ expected performance. Emphasis on the V&V tasks lies from s
8
the fact that identification and correction of errors early in the development cycle are less costly than that in later phases, and thus the quality of software are significantly improved. As a result, a middle scale organization will benefit greatly from its software process improvement by the proposed budget-acceptable, feasible and effective approach. 6. Conclusion To learn lessons from the common practices on the previous projects in order to enhance process implication, the proposed roadmap will be much helpful to the ISO 9001 registered organizations. Furthermore, the roadmap will assist the implemented organizations in performing gap analysis and maintaining their quality manual without any difficulty while adopting the CMMI model. With the desire to making the suggested roadmap easily implemented for an organization starting to employ philosophy of the CMMI model, we set the target goal of improvement path from none to the third capability level in the continuous representation CMMI model. In summary, the proposed framework establishes the bottom line to be performed for software process improvement in a software organization. Within a software project life cycle, the effort on verification and validation is highly emphasized to ensure that both quality control and quality assurance are implemented as scheduled plans. Based on several observations on practical application demonstration, significant improvements have been found over some interested metrics such as productivity, defect injection rate and defect removal rate. The benefit of our proposed roadmap provides an effective, efficiency and economical approach no matter the middle-scale company is ISO 9001:2000 registered or not yet.
References
[1] [2] Ahern, D.M., Aaron Clouse and Richard Turner (2001). CMMI Distilled. Addison-Wesley. Carter, L., C. Graettinger, M. Patrick, G. Wemyss, S. Zasadni (2002). The Road to CMMI: Results of the First Technology Transition Workshop. CMU/SEI-2002-TR-007, Pittsburgh, PA: Software Engineering Institute, February. Chrissis, M.B., M. Konrad, S. Shrum. (2003). CMMI: Guidelines for Process Integration and Product Improvement, Addison Wesley. IEEE. (2004). STD 1012-2004, IEEE Standards for Software Verification and Validation. ISO. (1995). ISO/IEC 12207: 1995, Information Technology—Software Life Cycle Processes. ISO. (2001). Software and Systems Engineering—Guidelines for the Application of ISO 9001:2000 to Software, TC/SC SC7 WG18N61, ISO/IEC CD 9000-3, and Version for FCD Ballot, Geneva, Switzerland: ISO. ISO. (2003). ISO/IEC 15504-2:2003. Information Technology-Process Assessment-Part 2: A Reference Model for Process and Process Capability, Geneva, ISO. Mutafelija, B. (2001). Software Process Improvement: Synergy between ISO 9001:2000 and CMMI, in: SEPG Conference, New Orleans, LA. Paulk, M.C. (1995). How ISO 9001 Compares with the CMM. IEEE Software, January, 74– 83.
[3] [4] [5] [6]
[7] [8] [9]
[10] Yoo, C., J.Yoon, B. Lee, C. Lee, J. Lee, S. Hyun, and C. Wu (2004). An Integrated Model of ISO 9001:2000 and CMMI for ISO Registered Organizations. 11th Asia-Pacific Software Engineering C ne neA S C0) 5-157. of ec ( P E ’4 10 r [11] Yoo, C., J.Yoon, B. Lee, C. Lee, J. Lee, S. Hyun, and C. Wu (2006). A unified model for the implementation of both ISO 9001:2000 and CMMI by ISO-certified organizations. Journal of Systems and Software, Web published, June.
9