The Frameworks Quagmire, A Brief Look
Sarah A. Sheard
Software Productivity Consortium
2214 Rock Hill Rd, Herndon VA 22070
sheard@software.org (703) 742-7106
Abstract. Organizations wishing to remain developers are discovering that their ability to
competitive often want to comply with all win and perform on contracts is subject to
possible contractor evaluation criteria, process investigations of their processes as well as the
models, and quality standards, but the field is quality, cost, or effectiveness of their prod-
evolving rapidly. This paper describes the ucts. The frameworks against which their
categories of compliance frameworks and processes are evaluated are multiplying, as
characteristics of seven important frame- shown in Figure 1.
works: the CMM, SE-CMM, IPD-CMM, ISO In the late 1980s the CMMsm for Software
9000, SDCE, MIL-STD-498, and Trillium. [CMM] was created to help software develop-
The paper also discusses trends and recom- ers mature their software development pro-
mendations for how to deal with the Frame- cesses, to better meet contractual require-
works Quagmire. ments. Recently, the engineering community
presented two capability models ([SE-CMM]
INTRODUCTION
and [SECAM]) for companies to use in im-
More and more, software and system proving their systems engineering processes.
MIL-STD
SDCCR MIL-Q 1679
MIL-STD-1803
CMM SDCE 9858
SCE DOD-STD
People CMM 2167A
IEEE Stds. 730,828
SA-CMM
829, 830,1012,1016 DOD-STD
Trusted CMM*
1028,1058,1063 7935A
SSE-CMM ISO Baldrige MIL-STD
SPICE 498
MIL-STD Trillium
499B* EIA IS 640/
EQA, IEEE 1498
ISO/CD
IEEE AF IPD ISO/IEC
9004-8*
1220 SE-CMM Guide 12207 EIA/IEEE/
J STD 016
EIA IS ISO 9000
DOD
632 SECAM Series
IPPD ISO
(INCOSE) ISO
10011
EIA/ 15288* US Draft
IPD-CMM* TickIT
ANSI SECM 12207-1996*
632* Merged Model*
* = Not yet released
Figure 1. The Frameworks Quagmire
page 1 of 8
Meanwhile, the international community Organization for Standardization (ISO) Soft-
developed several different groups of process ware Process Improvement Capability
improvement and quality standards ([ISO dEtermination (SPICE), MIL-STD-499B
9000] and ISO’s [SPICE]), and the military (draft), and the Institute of Electrical and
developed ways to evaluate bidders during Electronics Engineers standard for systems
source selection, such as the [SDCE]. engineering [IEEE 1220]. The SE-CMM was
Even now, new standards are appearing subsequently used in creating the Integrated
regularly. Asterisks in Figure 1 indicate Product Development CMM [IPD-CMM], the
frameworks that have not yet been publicly Security Systems Engineering CMM (SSE-
released. Omitted from this chart were ef- CMM) [Hefner 96], and a merged systems
forts, such as the Testing Maturity Model, engineering capability model (SECM) that is
documented in [Burnstein 96], that were not currently being developed with facilitation
driven by large, standards-setting or profes- from the Electronics Industries Association
sional groups. The set of available frame- (EIA).
works is large and confusing, at best.
TYPES OF COMPLIANCE
Clearly, organizations need help determin-
FRAMEWORKS
ing which standards and other frameworks are
most beneficial. As a consortium of member The first step toward making sense of the
companies, the Software Productivity Consor- Quagmire is to categorize the frameworks by
tium has studied the compliance frameworks purpose. One or more of the six categories in
that are relevant to companies building soft- Table 1 apply to most of the frameworks.
ware-intensive systems. This paper highlights
some of the main points of a Consortium 1. Standards and Guidelines
course on Compliance Frameworks, including 2. Process Improvement (PI) Models and
framework types, characteristics, trends, and Internal Appraisal Methods
recommendations. 3. Contractor Selection Vehicles
A MULTITUDE OF FRAMEWORKS 4. Quality Awards
5. Software Engineering Life-Cycle Models
From Figure 1 it is evident that many, many
6. Systems Engineering Models
frameworks exist which developers may need
to consider. The field is truly a quagmire, in Table 1. Compliance Framework
which process improvement efforts can bog Categories
down, if an organization is not careful. 1. Standards and Guidelines
The arrows in the Figure 1 show the usage Standards and guidelines establish what
of one framework in developing another. For must be done in a contractual situation. Most
example, the Systems Engineering Capability can be tailored as desired by both parties, and
Maturity Model (SE-CMM) of EPIC1 devel- are used as recommendations of good practic-
oped from the Capability Maturity Model es in general. Guidelines may interpret asso-
(CMM)2,3 for Software, the International ciated standards or collect practices not in-
tended to be specified in a contract.
1 Standards include:
Enterprise Process Improvement Collaboration, a
collaboration of Industry, Government, and Academic
United States (US) Military Standards,
institutions. such as MIL-STD-498 (Software
2
“CMM”, and other names and acronyms including
3
“Capability Maturity Model” and “CMM,” are service The acronym “CMM,” when used alone, refers to the
marks of Carnegie-Mellon University. CMM for Software.
page 2 of 8
Development and Documentation), and 4. Quality Awards
guidelines, such as the Air Force guide- In 1987, the US Government established
lines for Integrated Product Development an award program, the Malcolm Baldrige Na-
Commercial Standards, such as [EIA IS tional Quality Award [Baldrige] to improve
632], an interim standard on the Systems American businesses’ focus on quality and
Engineering Process customer satisfaction. Subsequently, Europe
International Standards, such as the [ISO established an award with very similar criteria
9000] series for Quality Systems (which and selection methodology, and recently ISO
includes guidelines as well as standards) has begun a draft standard with identical cate-
gories to the Baldrige’s seven categories.
2. Process Improvement Models and
Quality awards include:
Internal Appraisal Methods
Malcolm Baldrige National Quality
In general, these frameworks define the
Award
characteristics of good processes, and avoid
European Quality Award
prescribing how the processes must be enact-
ed. The purpose of process improvement ISO/CD 9004-8, draft standard on Quality
models is to establish a roadmap by which a Management Principles
route can be drawn from “where we are to- 5. Software Engineering Life-Cycle Models
day” to “where we want to be.” In order to MIL-STD-498 developed from DOD-STD
determine “where we are today,” an organiza- 2167A (for software development), DOD
tion performs an appraisal, sometimes with 7935A (for documentation), and MIL-Q-9858
the aid of an outsider with specific expertise (for quality). Approved after the official date
in the model. These models include: of “no more military standards” in 1993, this
CMM relatives, including the CMM for standard was intended to be an interim stand-
Software, the SE-CMM, IPD-CMM, Peo- ard until commercial standards replaced it in
ple CMM [P-CMM], and others about two years. [ISO/IEC 12207] is an inter-
SECAM national standard in the same area, and MIL-
The [Trillium] model, for telecommunica- STD-498 is being adapted (in several steps) to
tions add ISO/IEC 12207-type requirements on
acquirers, maintainers, and operators, parties
3. Contractor Selection Vehicles
not mentioned in MIL-STD-498. Life-cycle
These frameworks specify the examina-
models include:
tion of an organization’s processes by an out-
MIL-STD-498
sider, either a second-party (the potential ac-
EIA/IEEE J STD 016 [J STD 016]
quirer) or a third party, usually one hired by
the potential acquirer. The purpose is a de- ISO/IEC 12207
tailed comparison of competitors’ strengths US Draft 12207-1996
and weaknesses, in order to minimize pro- 6. Systems Engineering Models
curement risk to the acquiring agency. Two The SE-CMM was developed in 1994 by
methods in use are: half- or full-time authors working on the
Software Capability Evaluation (SCE) model for a year. Funding was provided by
(associated with the CMM) the authors’ companies, as a provision of their
Software Development Capability Eval- participation in EPIC, then called the Industri-
uation (SDCE), from the US Air Force al Collaboration. The model was completed
in a year, and revised, along with an accom-
page 3 of 8
panying appraisal method, in the next eight- EIA IS 632 and EIA/ANSI 632
een months. ISO 15288
At the same time, an INCOSE working
CHARACTERISTICS OF SEVEN
group developed the Systems Engineering
FRAMEWORKS
Capability Assessment Model (SECAM) from
several systems engineering assessment mod- Table 2 compares characteristics of the seven
els used internally by companies in the important frameworks which are circled on
aerospace and defense industry. Figure 1. Brief notes about the frameworks
One primary difference is that the follow.
SE-CMM was first developed as a model, and CMM for Software
the SECAM as an assessment method, intend- The CMM “nucleus” includes the CMM
ed to work with systems engineering stand- for Software, the P-CMM, the Software
ards (from the EIA and IEEE) serving as the Acquisition CMM [SA-CMM], and the Trust-
model. Another difference is that the SE- ed CMM (T-CMM).
CMM confines its scope to process character- The CMM, P-CMM, and T-CMM address
istics, and the SECAM includes non-process the software development organization within
characteristics, such as work quality and sys- an enterprise. The SA-CMM applies to an
tems engineering team experience. Finally, acquisition agency. In contrast, the SE-CMM
the SE-CMM consideres all practices as “base addresses the organization building systems,
practices,” whose performance in an informal which will be larger than the software devel-
manner would earn the organization a rating opment organization if the systems include
of 1 in the process area, but the SECAM al- hardware and software. The IPD-CMM ad-
lowed some practices to be required only of dresses the product development enterprise,
higher-capability organizations, so that Level including such groups as marketing, manufac-
1 organizations need not perform them at all. turing, and business management, as well as
In the SECM merger effort being the development organizations.
facilitated by the EIA, these differences have
been resolved and the models are being SE-CMM
merged. Initial public release is scheduled for Frameworks centered around the SE-
mid-1997. CMM nucleus all involve systems engineer-
[ISO 15288] is an effort to create an ing. The systems engineering standards’ defi-
international system life-cycle standard, initi- nition for systems engineering includes most
ated by the group that created the ISO soft- of the twelve systems engineering roles de-
ware life-cycle standard, ISO/IEC 12207, and scribed in [Sheard 96]. IEEE 1220 takes pri-
augmented by people with systems engineer- marily a Technical Management view, while
ing expertise. An INCOSE past president, EIA IS 632 leans more toward requirements
who was also an author of MIL-STD-499B, development and system design. The latter
EIA Interim Standard 632 [EIA IS 632], and two are apparently being consolidated now,
IEEE 1220, represents INCOSE in the 15288 into the anticipated EIA/ANSI 632.
effort. Scheduled release is for 2001. IPD-CMM
Systems engineering models include: The IPD-CMM is being created by EPIC,
MIL-STD-499B (Systems Engineering) which also wrote the SE-CMM. The authors
SE-CMM used the Air Force IPD guide and the DOD
SECAM Guide to Integrated Product and Process
IEEE 1220 Development [DOD IPPD] as input, and
page 4 of 8
Framework Scope Purpose Length, Major Notes
pages Focus
CMM for Software PI 500 SW Process Staged architecture provides “Triptik” (a
Software developing AAA map with exact roads, stops, and
organization times highlighted) for improvement.
SE-CMM Organization PI 250 SE Continuous architecture provides map of
developing Process terrain.
systems
IPD-CMM Enterprise PI 220 Process Staged-continuous architecture provides
map plus “Triptik.”
ISO 9000 Product Trade 16 Quality Registration certifies a minimum quality
producing process system compliance.
organization
SDCE Bidding Contractor 600 Process, Evaluates risks to acquirer for each bid,
organization selection Capacity, and reduces risks with winning contractor.
Technology
Software Software Contract 60-200 Manage- Standards are evolving to include role of
Life Cycle developing compliance ment acquirer and others, as well as supplier.
Standards organization process
Trillium Enterprise PI 130 Process Combines requirements from CMM, ISO,
Baldrige, and software quality standards.
Table 2. Characteristics of Seven Frameworks
visited organizations practicing IPD well to CMM and SE-CMM processes will fall under
verify principles and practices. the ISO 9000 umbrella.
A goal of the IPD-CMM is to establish a TickIT, a program described by a roughly
framework into which other CMMs can fit. 75-page document, was developed by the
Its architecture is a hybrid of the staged and British Standards Institute. TickIT provides
continuous models. Implementation of the detailed ISO 9000 guidance for software de-
integration depends, in part, on the existence velopment. TickIT registration is generally
of minimal “plug-and-play” models in specif- optional, except in the United Kingdom.
ic domains. These models would be com- The ISO 10011 series of standards speci-
bined with the basic product development, fies requirements and training for ISO 9000
integration, or supporting processes already auditors.
covered by the IPD-CMM, to form a model
SDCE
that is appropriate for the specific organiza- The SDCE evolved from the Air Force’s
tion using it. Software Development Critical Capacity Re-
ISO 9000 view (SDCCR) and from the SCE, which is
Despite the fact that ISO 9000 registration an assessment method associated with the
is for the purpose of trade, the quest for regis- CMM for Software. To CMM-type questions,
tration often has the same effect on the organ- the SDCE adds several areas of focus, includ-
ization as the quest for a CMM level (general- ing systems engineering and technology (such
ly Level 2 or Level 3): the organization’s pro- as artificial intelligence). The SDCE’s tech-
cesses are documented, and discipline is tight- nology areas, in particular, may not apply on
ened to ensure documented processes are fol- all procurements, and the SDCE is intended to
lowed. Thus, although in theory, ISO address- be tailored by deleting up to half of the ques-
es only “quality-impacting” processes, this tions for any one procurement.
definition is broad enough that virtually all the
page 5 of 8
Software Life-Cycle Standards comment-logging and solution-approval
The standards specify life cycles, includ-
ing reviews that approve moving from one processes, and are often updated on a prede-
phase to the next. These standards provide termined schedule. This is good in that the
processes, in contrast to CMMs, which pro- models need to be improved, and in that im-
vide requirements that good processes will provements can include better integration with
meet. other models. However, any changes must be
Initial modification of MIL-STD-498 to read, understood, and responded to by anyone
become more commercial proceeded under who has used the old model. Because there
the numbers EIA IS 640 and IEEE 1498. The- are over 2000 organizations with investment
se numbers were retired before the standard in the Software CMM, and over 10,000 with
was released, and the standard was given a J investment in the ISO 9000 series, the inertia
series number [J STD 016], meaning a Joint against change can be considerable.
standard between the two organizations. The
Proliferation
current number for the expected commercial More models are continually being
version is US Draft 12207-1996. developed. The T-CMM and SSE-CMM are
Trillium examples of recent addtitions.
The Trillium model, created by Bell Proliferation is both good news and bad
Canada, combines requirements from the ISO news. It is good in that new models capture
9000 series, the CMM for Software, and the wisdom and best practices, because develop-
Malcolm Baldrige criteria, with software ers have been seeking “best practices” for
quality standards from the IEEE. The model years.
cannot necessarily be adopted as-is because in But implementing best practices in a real
some cases, goals of the frameworks are used organization is at least as hard as collecting
rather than their detailed requirements, and lessons from the groups who have learned
because the model includes process infor- them the hard way. Someone depositing a
mation that is unique to the telecommunica- new 300-page model on a process engineer’s
tions field. However, the model serves as desk is not likely to walk away feeling appro-
proof that the requirements of several of the priately appreciated. Time must be devoted to
popular frameworks can be combined, and it reading new models and to understanding
provides a template for additional efforts in changes to existing models as they emerge. If
this area. a new model is not seamlessly and obviously
integratable with current frameworks, half a
FRAMEWORKS TRENDS
staff-year can easily go into determining what,
Four trends are evident to those following the if anything, a new model suggests should be
field of compliance frameworks. done differently.
Evolution
Integration and Coordination
Proliferation
As mentioned above, the IPD-CMM
Integration and Coordination provides a framework for future integration of
Consolidation CMMs. In addition, the SEI is sponsoring the
Evolution CMM Integration effort, which may impose
Current models are being improved and (on CMM revisions) requirements that will
adapted to better meet stakeholder needs. Pro- make model integration easier for users. The
fessional standards and the CMMs all have SEI is also coordinating with other model
page 6 of 8
TickIT should organizations integrate compliance
SDCE CMM
and ISO
ISO/IEC with a new model and compliance with other
SPICE
12207 frameworks?
IPD-CMM and US
Trillium Version RECOMMENDATIONS FOR
DEVELOPERS
ISO 9000
People CMM Series Cost competitiveness and time-to-market
SA-CMM SECM
Trusted CMM dominate the factors that will keep companies
SSE-CMM Baldrige-type alive, not to mention profitable, in the 1990s
ISO 15288 and quality
EIA/ANSI 632 models and beyond. But the act of defining and im-
plementing process changes costs significant
Figure 2. A Better Frameworks Quagmire
money how can this be efficient?
makers, notably ISO 9000 (at least keeping up The key lies in adopting only a few, high-
with them and publishing comparisons be- leverage frameworks. In addition to the CMM
tween the models) and ISO SPICE. Further, as for Software (which most have already adopt-
various frameworks evolve, authors are read- ed), the Consortium’s member companies are
ing other frameworks and incorporating the generally considering the SE-CMM, ISO
best features. It is reasonable to expect fewer 9000, and the SDCE (when bidding on Air
differences in the next cycle of all the models. Force contracts). Many members already
have contracts requiring compliance with
Consolidation MIL-STD-498 or its predecessors, so they are
Retirement of multiple models as they are looking to the future, when MIL-STD-498
consolidated into a single new model is a very will be merged with ISO/IEC 12207. Mem-
positive trend. This appears to be happening bers are also looking to the IPD-CMM to pro-
with software life-cycle standards and, to a vide a needed integration framework.
smaller extent, systems engineering standards It is recommended that developers delay
and models. implementation of most of the other frame-
Figure 2 shows an idealized picture of works. Some will disappear from lack of
what the Quagmire should look like if similar support. Others, which may prove long-lived,
frameworks are consolidated. Each of the can be adopted after better integration meth-
lines would indicate a defined interface, for a ods have been made available.
defined purpose. Frameworks listed together
would be consolidated. Even more consolida- CONCLUSIONS
tion may occur with the CMMs and the Companies should focus on identifying a
SECM. small set of high-value frameworks to adopt.
RECOMMENDATIONS FOR Those already working with the CMM for
CREATORS OF FRAMEWORKS Software may add the SE-CMM and the IPD-
CMM to help broaden their process improve-
Clearly, those writing standards, process ment effort, and may delay adopting other
models, and contractor selection vehicles need CMMs until methods for integrating CMMs
to understand the predicament of developers. are better defined.
While there certainly is a need for well- Other frameworks worth investigating are
crafted collections of best practices, creators the ISO 9000 series of standards, the SDCE
of frameworks must take note of the frame- (if bidding on Air Force contracts), Trillium
works that already exist, and must tailor their (particularly for telecommunications compa-
additions to fit into some of them. How nies), and the IPD-CMM.
page 7 of 8
Organizations with frameworks questions [IPD-CMM] Software Engineering Institute and EPIC,
or problems in this area should consider An Integrated Product Development Capability
Maturity Model, Carnegie Mellon University, Ver-
joining with other industrial, academic, and sion 0.9, 28 October 1996.
government institutions, in order to leverage [J STD 016] Electronics Industry Association, EIA/
their efforts in navigating the Frameworks IEEE/J STD 016, Standard for Information Tech-
Quagmire. nology - Software Life Cycle Processes - Software
Development - Acquirer-Supplier Agreement, 6
REFERENCES February 1996.
[P-CMM] Curtis, Bill, William E. Hefley, and Sally
Square brackets in the text denote references, as shown
Miller. People Capability Maturity Model. Soft-
below. Because most frameworks are written by insti-
ware Engineering Institute, CMU/SEI-95-MM-02,
tutions rather than individual authors, the framework
September 1995.
documents are cited by acronym. Other articles are
[SA-CMM] Software Acquisition Capability Maturity
cited by first author and year. WWW references are
Model. See www.sei.cmu.edu/technology/risk
included where practical.
/Risk_SW_Acq/SA-CMM.html.
[SDCE] Software Development Capability Evaluation,
[Baldrige] National Institute of Standards and Technol- see www.wpafb.af.mil/indguide/sdce.htm.
ogy, Malcolm Baldrige National Quality Award, [SECAM] INCOSE Capability Assessment Working
see www.nist.gov/director/quality_program/. Group, Systems Engineering Capability Assess-
[Burnstein 96] Burnstein, Ilene, Taratip Suwannasart, ment Model, Version 1.50, June 1996.
and C. R. Carlson, “Developing a Testing Maturity [SE-CMM] EPIC, A Systems Engineering Capability
Model,” Parts I and II., Crosstalk, August and Sep- Maturity Model, Version 1.1. Available from
tember, 1996. http://www.software.org/secmminfo.html.
[CMM] M. C. Paulk, B. Curtis, M. B. Chrissis, and C. [Sheard 96] “Twelve Systems Engineering Roles,”
V. Weber, Capability Maturity Model for Soft- Proceedings of INCOSE, 1996.
ware, Version 1.1, Software Engineering Institute, [SPICE] International Organization for Standardiza-
CMU/SEI-93-TR-24, February 1993. tion, SPICE: Software Process Improvement
DOD IPPD] DoD Guide to Integrated Product and Capability dEtermination; see www-sqi.cit.gu.edu
Process Development, February 5, 1996, .au /spice/.
OUSD(A&T)/DTSE&E; ATTN: Mr. Mark D. [Trillium] Bell Canada, Northern Telecom, and Bell-
Schaeffer, 3110 Defense Pentagon, Washington Northern Research. The Trillium Model. See
DC 20301. http://ricis.cl.uh.edu/trillium/trillium.html.
[EIA IS 632] Electronics Industry Association, EIA IS
632, Draft For Technical Committee Review, AUTHOR BIOGRAPHY
Systems Engineering, 20 Sept. 1994.
[Hefner 96] Hefner, Rick, David Hsiao, and Warren Sarah A. Sheard has seventeen years’
Monroe, “Experience with the Security Systems
experience in systems engineering. Ms.
Engineering CMM,” Proceedings of INCOSE,
1996. Sheard worked as a satellite engineer at
[IEEE 1220] IEEE Computer Society, IEEE Trial-Use Hughes Aircraft Space and Communications
Standard for Application and Management of the Group, and in software systems at the Federal
Systems Engineering Process, IEEE Std Systems group of IBM and Loral. Currently
1220-1994, New York, 28 February 1995.
she coordinates systems engineering efforts at
[ISO 9000] International Organization for Standardiza-
tion, ISO 9000 International Standards for Quality the Software Productivity Consortium in
Management. Switzerland, 1991. See www.iso.ch Herndon, Virginia, where she also develops
or www.exit109.com/~leebee/. technical products and consults and teaches in
[ISO 15288] Lake, Dr. Jerome G., “ISO Standard the areas of systems engineering, process im-
15288, System Life Cycle Processes,” INSIGHT,
provement, and integrated product teams. Ms.
Spring 1997.
[ISO/IEC 12207] Information technology -- Software Sheard received an MS in chemistry from the
life cycle processes, International Organization for California Institute of Technology in 1979.
Standardization and International Electrotechnical
Commission, ISO/IEC 12207:1995 (E), 1 Aug 95.
page 8 of 8