ACM SIGSOFT Software Engineering Notes vol 26 no 1 January 2001 Page 61
COTS Workshop: Continuing Collaborations for Successful COTS Development
John Dean, National Research of Council Canada
Patricia Oberndorf, Software Engineering Institute
Mark Vigder, National Research Council Canada
Chris Abts, Universityof Southern California
Hakan Erdogmus, National Research Council of Canada
Nell Maiden, City University, UK
Michael Looney, DERA, UK
George Heineman, Worcester Polytechnic Institute
Michael Guntersdorfer, Universityof California, Irvine
Introduction summary of their discussions which are included, without major edit-
In early June of 2000 a COTS Workshop entitled "Continuing ing, below. A more complete description of the workshop, as well as
Collaborations for Successful COTS Development" was held all the participants' position papers, can be found at:
in Limerick, Ireland in conjunction with ICSE 2000. The pur- http://seg.iit.nrc, ca/projects/cots/icse2000wkshp/index.htnd
pose of the workshop was to collect experience reports re-
garding the use of commercial off-the-shelf (COTS) software Workshop Results
to build systems, identify best-practices for the use of COTS
software, and to establish a research agenda for those re- The Workshop brought together a diverse group of researchers and
practitioners experienced in many areas of software engineering. This
searchers interested in COTS-based software systems. This
one and a half day workshop was an extension of the work allowed the Workshop to identify a number of best practices when
begun during the workshop entitled "Ensuring Successful building COTS-based systems, as well as directing the participants to
COTS Development" held in conjunction with ICSE '99. Re- problems requiring further research.
sults from that workshop demonstrated that there were a num- A complete summary of the breakout sessions is provided below, but
ber of common research areas, including acquisition, planning some of the key findings that emerged during the plenary included:
and management, architecture and implementation, and
evaluation and testing, for which researchers saw the possibil- * The Economic and Financial Issues breakout agreed that improved
ity of collaboration. These areas included specific topics such models need to be developed for cost-estimation associated with
as estimating the effort required to implement COTS-based the entire lffecycle of COTS-based systems. This includes esti-
systems, classification of architectural styles, and certification mating cost, schedule, and effort for each stage of development
of COTS products for reliability and safety. The group will and maintenance. Significant data gathering is still required for
reconvene at ICSE'01 (w_~_ww.c_s_r..u.w_'c;.ca/icse200Dto discuss calibration of these models.
further the results achieved. As well, managers need models and tools to support making deci-
The ICSE 2000 Workshop had about 26 participants and was sions based on economic models associated with the technical as-
formatted as a combination of plenary sessions and small pects of the system. This allows managers to decide, for example,
breakout groups that worked on specific issues related to whether to buy or build a component, or whether to upgrade a
COTS-based systems. The breakout groups investigated the component now or wait for a future release.
impact of COTS software usage in the following areas: • The Requirements Definition breakout focussed on the relation-
ship between requirements and COTS selection and usage. Given
• Economic and financial issues. that much of the functionality of a system is being bought, not de-
• Requirements definition_ veloped, product selection becomes critically intertwined with re-
. Software engineering process. quirements definition_ The group used two case studies to ground
• Integration, maintenance and system management. their discussions and used them as a basis for setting a research
• Business models. agenda for the next three to five years. The most important issue
was determined to be appropriate Product Description Languages
Each breakout group tried to identify the current state of the for COTS products by which integrators can determine operational
art in COTS software usage as well as open questions that
characteristics of a product within different contexts. The product
could provide the basis for fiLrther research in the coming
descriptions would ideally be developed by independent third par-
years. Each group was responsible for producing a written
ACM SIGSOFT Software Engineering Notes vol 26 no 1 January 2001 Page 62
ties and cover properties of a product well beyond itsop- Application of Financial Valuation Models
erational characteristics. The COTS approach must provide clear fiuancial benefits for it to be a
• The Software Engineering Process breakout identified ncial valuation tech-
viable alternative to traditional development Fin.q_
how the development process changes when a system is niques such as Net Present Value and Real Options Analysis aid in this
COTS-based. The number of development activities re- kind of assessment by linking development decisions to their economic
quired and the constraints that exist between them di- impact. Within this framework, the problems being addressed include:
rected the discussion. The activities of system architecture
• Comparison of a COTS-based development strategy with a custom
and infrastructure design, product selection, and require-
development strategy for a given system.
ments definition all have much more complex relation-
• Comparison of two different COTS-based development strategies
ships among them for COTS-based systems than custom-
built systems. There are also new project management for a given system.
functions relating to software licencing and warranty • Assessment of the value of flexibility inherent in a COTS-based
management and COTS product evaluation and upgrade system.
decisions.Many of these issues were discussed as well in • Assessment of migration strategies, both from a COTS-based
the Economic and Financial Issues and the Requirements system to a custom system and from a custom system to a COTS-
Definition Breakout. based system.
• The Integration, Maintenance and System Management • Timing and value of COTS product upgrade and system release
breakout agreed that the integration costs for COTS-based decisions.
systems are often higher than for custom-built systems. • Management of COTS product license costs to mitigate risk.
The group felt that there must be ways that products can CO 7"5 vs. Custom Decisions
be described to allow system builders to determine the The viability of COTS-based development with respect to custom de-
integration issues and costs before building the systems. velopment can be assessed using a simple financial model based on the
Similar to the Requirements Definition group, they felt concept of Net Present Value (NPV). In corporate fin_ance, NPV is the
that there was a need for independent evaluators to pro- gold standard for making capital budgeting decisions. The NPV of a
vide profiles of COTS products. Maintenance and man- project is given by the sum of its discounted future cash flows. Dis-
agement become significant issues as the system is being counting takes into account time value of money, i.e., the notion that
maintained at the COTS product level rather than at the money that is yet to be expended or received is worth less today than it
source code level, and many of the maintenance and man- is in the future. The NPV rule states that only projects with positive
agement decisions are being driven by the COTS vendor NPV are worth undertaking, and a project with a higher NPV is prefer-
rather than the system builder. able to a project with a lower NPV.
• The Business Models breakout identified all the
stakeholders involved in developing a COTS-based sys- An NPV-based model of software development can compare two de-
tem and the various risks faced by each of these velopment strategies using a metric that captures the economic incen-
stakeholders. They then proposed a set of risk mitigation tive to choose one strategy over the other. The five parameters that
strategies that can be practiced by each of these a~ect this metric are:
stakeholders. Although stir requiring a significant amount • the ratio of the development cost of the two strategies;
of research to complete, they have provided a direction • the ratio of the time-to-market of the two strategies;
that allows those involved to manage the risk.
• the ratio of the operation cost of the two strategies;
• the ratio of the asset value (a measure of post-development bene-
fits or market value) of the two strategies; and
Each Breakout Coordinator prepared a summary of the results • the market risk of the end product, as captured by the discount
of their individual sessions. These summaries are presented in rate.
With the estimates of these pmmneters, it is possible to compare dif-
Breakout 1: Economic and Financial Issues ferent development strategies. These include not only different types of
pure COTS-based and pure custom development sWategies, but also
Chris Abts, (University of Southern California, USA)
hybrid strategies, and with the use of Real Options Analysis, strategies
Hakan Erdogmus, (National Research Council Canada)
that start with one type of system and gradually migrate to another type
Patricia Obemdoff (Software Engineering Institute, CMU,
of system. In the latter case, the incremental migration of system com-
ponents is viewed as a series of options that are exercised provided that
The purpose of the breakout session was to explore possible
the long-tea-a~ benefit of each migration phase exceeds the immediate
areas of collaborationthat pertainto the economic and finan-
cost of that phase.
cial aspects of COTS-based software development. The dis-
cussions were centered on two topics: Two rules of thumb that emerge from an NPV-based analysis are:
1) Application of financial vahmt__ionmodels to evaluate • When the COTS approach has both siotmificanfly higher asset
COTS-based development decisions. value and operation cost, increasing product risk increases the in-
2) The COCOTS cost estimation model. centive for the COTS approach only if that approach also has suf-
ficient time-to-market advantage over a custom system.
ACM SIGSOFT Software Engineering Notes vol 26 no 1 January 2001 Page 63
If time-to-market is important and the custom system has for the developer, but a liability for the vendor when license fees ex-
significantly higher market value, COTS-to-custom mi- hibit a great deal of fluctuation. Therefore the vendor is justified to ask
gration strategy is usually most valuable. for an up-front premium from the developer in return. The fair value of
this premium, again, can be determined by Real Options Analysis.
Value of Replacement Flexibility
One advantage of the COTS approach is flexibility. Flexibility COCOTS (COCOMO for COTS) Update
arises from the abitity to replace the COTS components used Cost estimation is a vital input for any economic analysis, including
in the systems at a fraction of the cost normally required to financial valuation. The COCOTS cost model is being developed at
develop those components from scratch. However, to achieve USC Center for Software Engineering to address the needs of COTS-
such flexibility, the system must be designed accordingly. based systems. It is one of a suite of specialized models being devel-
This implies an up-front investment in a robust COTS archi- oped as extensions of the COCOMO (Constructive Cost Model) II cost
tecture. Such an investment buys the developer the option to estimation model. The COCOTS model has features that make it suit-
replace the COTS products at a reasonable cost in the future. able for effort and schedule estimation in cases where parts of a system
This early investment can be thought of as a risk mitigation are implemented by COTS products.
strategy to hedge against replacement cost uncertainty. The The two defining characteristics of COTS software, and thus which
cost of the extra investment in system architecture is not jusli- drive the whole COTS usage process, are: 1) the COTS product source
fled if it surpasses the value of the replacement flexibility, or
code is (typically) not available to the application developer, and 2) the
the option value of future replacement opportunities. There-
future evolution of the COTS product is not under the control of the
fore, it is important to identify the circumstances under which
application developer. (The market decides when and how a given
the benefits of flexibility exceed the cost of achieving it. COTS product evolves.) These characteristics are the underlying con-
A state-of-the-art financial valuation technique, Real Options stmints that shaped the current form of the COCOTS model.
Analysis, can address this issue by mapping the replacement
(Note: COCOTS as described here deals only with initial COTS inte-
scenario to an option pricing problem. By investing in a robust
gration efforts. Subsequent to the COTS workshop at ICSE 2000, work
COTS architecture, the developer is acquiring a future option has been done to expand the model to provide estimates of long-term
to upgrade the technologies underlying the COTS products
COTS-based system operation and maintenance effort through system
used. When sufficient information is available, the value of
retirement. That work is on-going, however, and remains outside the
this option can be estimated and compared to the amount of
scope of the discussion here.)
architectural investment needed.
COCOTS is actually a collection of four related submodels, each ad-
Upgrade Decisions dressing individually what has been identified by USC-CSE as the four
In a COTS-based system, upgrading a COTS product should primary sources of COTS softwar¢ integration costs (from a system
be a discretionary activity. Some upgrades will be minor and construction activity point of view). These sources include the effort
incur a relatively small cost, while others will be major and needed to perform (1) candidate COTS component assessment, (2)
require substantial rework. When the cost of a product up- COTS component tailoring, (3) the development and testing of any
grade is superior to its long-term benefits, it does not make integration or "glue" code needed to plug a COTS component into a
good economic sense to proceed with the upgrade. This larger system (or to make it function with another COTS component),
thinking gives upgrade decisions an option-like flavor. How- and (4) increased system level progranmung due to volatility in incor-
ever, product obsolescence must also be accounted for when porated COTS components.
technology evolves at a rapid pace. Delaying an upgrade in-
Assessment is the process by which COTS components are selected for
definitely may have serious long-term economic consequences
use in the larger system being developed. Tailoring refers to those ac-
that are not immediately apparent.
tivities that would have to be performed to prepare a particular COTS
In addition, in the presence of multiple COTS products, syn- program for use, regardless of the system into which it is being incor-
chrouization of a system's upgrade cycles with the release cy- poratecL or even if operating as a stand-alone item. These are things
cles of the COTS products used may be problematic. such as initializing parameter values, specifying input/output screens
or report formats, setting up security protocols, etc. Glue code devel-
We feel that integer programming combined with Real Op-
opment and testing refers to the new code external to the COTS com-
tions Analysis together can tackle the timing and economic
ponem itself that must be written in order to plug the component into
feasibility of upgrade decisions.
the larger system. This code by nature is unique to the particular con-
text in which the COTS component is being used and must not be con-
Management of COTS Product Licenses fused with tailoring activity as defined above. Volatility in this context
Any multi-user system that is based on COTS products will refers to the frequency with which new versions or updates of the
incur license costs. When license fees are volatile, the total COTS software being used in a larger system are released by the ven-
cost may be substantial, and their proper management is thus dors over the course of the system's development and subsequent de-
instrumental for mitigating the resulting risk. Special option- ployment.
like contracts may be negotiated with the COTS vendors, for
example to give the developer the future right to acquire a yet Schedule Estimator
unknown number of licenses of a certain COTS product with a As well as effort, COCOTS will attempt to estimate the associated
cap on the single-license fee. A contract such as this is an asset calendar schedule that wig be required to construct a system using
ACM SIGSOFT Software Engineering Notes vol 26 no 1 January 2001 Page 64
COTS components. The schedule estimator should define the This breakout session aimed at bridging the gap between these com-
relationship between development time and effort for a COTS- plementary approaches by exposing two promising threads of work to
based system. It is also through the schedule estimator that the each other. We hope that it will lead to fruitful collaborations among
connection between COCOTS and COCOMO II is most di- the parties involved.
rectly evident. COCOMO II can be used to determine effort
associated with those parts of a system that are to be created Breakout 2: Requirements Definition & COTS: Setting a
from scratch or by reusing components for which the source Research Agenda for COTS-Based Development 3-5 Years
code is available. COCOTS is to be used to estimate the effort Down the Line
associated with those parts of a system that are to be imple- Nell Maiden (City University, UK);
mented using COTS components. The required schedule Ljerka Beus-Dukic (University of Northumbria, UK);
needed to complete the entire system (including new, reused, Colin Paul Glasier (City University Dublin, Ireland);
and COTS elements) must be a function of the effort required Gillian Mallalieu (University of Sunderland, UK);
for all those elements combined. Thus, the use of the CO- Cornelius Ncube (City University London, UK);
COMO II schedule estimator equation is proposed for CO- Thuy Nguyen (EDF, France);
COTS, but the input effort parameter (defined in person- Joan Pastor (Universitat Polictecnica de Catalunya, Spain);
months) includes both the traditional COCOMO and the addi- William M~ Thomas (The MITRE Corporation, USA)
tional COCOTS derived effort (as well as a possible recalibra-
tion of some of the constants in the schedule model). Structure Of The Breakout Session
The work carried out in this breakout session was divided into 3 basic
Effort and Schedule Distribution Percentages sessions, each addressing a basic problem or issue concerned with de-
Compared to COCOMO II, the initial COCOTS calibration fining requirements and evaluating soflavare packages for their fit with
data set suggests a slightly different distribution of total devel- these requirements. Firstly, we elicited 2 real-world COTS selection
opment effort and schedule across development activities. For and configuration case studies, to ground the academic discussions and
our current set of 20 industrial data points, and within our de- provide a focus for research later in the session. The first of the 2 case
fined CoTS-related activities, preliminary analysis indicates studies looked at COTS package selection at EDF, the French electric-
that approximately 8% of the total COTS related effort is ity supplier, and the second reported problems in adaptation of a COTS
dedicated to assessment, 28% to tailoring, 45% to develop- software package to include missing funetion_~lity. Secondly, in order
ment of glue code, and 19% to activities that pertain to COTS to enable us to explore a research agenda for requirements engineering
volatility. See the position paper and the accompanying slides and COTS package selection we brainstormed a vision of the COTS
for further details. software marketplace 3-5 years down the line. This vision was to act as
a baseline for the research agenda. Thirdly we developed a comprehen-
COCO TS Future Directions sive research agenda for COTS software package selection 3-5 years
As stated previously, currently the COCOTS model covers into the future, with particnlar emphasis on determining what consti-
only the development phase. The initial model is actively be- tutes a COTS software package to enable new decision-making capa-
ing extended to address operation and maintenance activities bilities. The following describes each of these 3 sessions in more
as well. detail.
The next round of data collection for calibration purposes is Two Cots Product Selection Case Studies
expected to begin in fall 2000. A revised version of the model The breakout session investigated two case studies reported by its par-
is expected soon afterwards. The revised model will account ticipants.
for operational and maintenance costs in addition to the initial
development costs. Selecting COTS Products for Electn'city Power Plants
As new data becomes available, the initial development model This first case study was reported by Thuy Nguyen from EDF in
as well as the opemtious and maintenance model parameters France. Selecting COTS software packages for electricity power plants
will be recalibrated. is a large, long, and complicated process. Thuy gave us some stark
facts. In a typical process of selecting embedded COTS software pack-
Conclusions ages for power plant conlrol, there are 3000 pages of user require-
Constructing a sound business case for or against COTS prod- ments, up to 85% of the requirements are non-technical requirements,
uct usage requires economic models that can deal with the there are 200-300 COTS candidates to choose from, and the selection
peculiarities of COTS-based system development. Specialized process can take several years to complete. Furthermore the selection
cost estimation models and state-of-the-art financial valuation of the fight COTS software package is business critical: EDF lose $1M
techniques are expected to help in this endeavor. per day ff the plant is not working. If this problem were not complex
enough, there is a need to consider multiple power plant systems: ide-
Cost estimation and financial valuation are highly comple- ally EDF would like to reuse software packages and package evalua-
mentary in that cost data is a vital input of financial valuation. tion information for design of new and future power plants, however
Whereas cost estimates are traditionally of more direct interest the time-scale of each selection process currently makes this infeasible.
to project managers, financial valuation is of more direct in-
terest to executives and investors who are concerned not only The current COTS software package selection process in EDF is a 4-
with costs, but also with the overall long-term value. stage process:
ACM SIGSOFT Software Engineering Notes vol 26 no 1 January 2001 Page 65
i. Analyse eqm'pment facilities: EDF investigate complex • failure to recognise incremental development and future software
non-functional requirements (e.g., performance require- versions.
ments) with simple formal models, which also enables Again, non-technical issues were identified as more important than
them to demonstrate these requirements to other technical issues in the success of this COTS customisation.
stakeholders. At the same time EDF place contracts on all
software pacl~ge vendors to provide them ALL required Brainstorming 3-5 Year Market Vision To Baseline Our
information for subsequent decision-making phases, Research Agenda
ii. Write the user requirements document: This process can The next stage of the session established a baseline for long-term re-
often take 1-2 years. The requirements are often written search in requirements engineering and package selection in a COTS-
by domain experts rather than software experts, with im- driven development process. However, the session faced a problem:
plications for the style and content of the requirements. setting a research agenda based on current problems and processes
Most user requirements are written as untestable natural might invalidate the agenda if these problems and processes change
language statements. EDF must also take care to keep the sig~aificantly over the next 5 years. Hence to establish the baseline the
requirements' generic', to avoid bias towards one package session undertook a brainstorming session to discover future trends in
vendor. For critical requirements, EDF critique these re- COTS software packages and their use of software development. Ideas
quirements with formal models similar to the use of for- were collected on a whiteboard, then structured into themes. The re-
mal models applied in stage 1. The result of this stage is suiting market vision was written as the following scenario:
an invitation-to-tender (ITF) document,
iii. Collect vendor responses and generate system specifica- "Both customers and suppliers of COTS software products will use
tions: Vendor responses to the 1Tr provide the basis for third-party product evaluation agencies to deliver added-value to their
writing the system specification, a process that can take a selection, customisation and integration processes. These agencies will
further one year or more. This specification is a specifica- use in-depth knowledge of customer requirements, vendor products,
tion of the wider socio-technical system, and will include, customers and suppliers, and how to undertake decision-making proc-
for example, human accident handling procedures as well esses to broker deals and advise their sponsors. These product evalua-
as software procedures. Each system requirement is de- tions will be 'lighter' than current product logs, profiles and
fined as either negotiable or non-negotiable. Responses certifications, in that the agencies will recegnise the diverse range of
from vendors are often in two parts: (i) the 'standard' different situations in which COTS software is used and tailor their
vendor specification for their package, (ii) responses to advice accordingly. These agencies will emerge bottom-up to seize
critical customer requirements; market share when the opportunities arise, rather than through top-
iv. Select the soRware package: The package selection proc- down government or other intervention. R will be just as likely that the
ess is a highly mechanistic one, which is essential to agencies will recommend to purchase COTS software as to use Appli-
avoid being sued for unfavourable selection decisions. As cation Service Providers (ASPs), renting products and ~ n g open
such, due to this mechanistic nature, EDF see package source software. Due to the demand and scale of the problem, agencies
selection as something distinct fIom requirements engi- will maximise the use of semi-automated computer-based techniques
neering. that rely on standard requirements and product description languages,
Thuy reported that EDF find the current process too long, ex- that is an agreed lingua-ftanca for deciding about COTS products."
pensive and not 'reusable' to meet new market opportunities Other identified trends in this market vision not incorporated into the
such as new power plants to be developed in the developing scenario included:
world. He identified the following requirement that EDF
might have to overcome these problems: • using COTS products as 'free' prototypes to drive the require-
ments elicitation process;
Third-party assessors to do the difficult and time-consuming • tendering for software services as you tender for other commodi-
package selection work, guarantee requirements compliance, ties such as financial services;
and offer confidence in this guarantee; furthermore the pack- • greater componentisation of COTS products, leading to compo-
age vendors should pay for this work. nents-on-demand;
• the rise of architectures, middle-ware and integration frameworks;
Tailoring an Enterprise Resource Planning (ERP) • the rise of multi-disciplinary evaluation and decision-making
Package (The Back-Flushing Problem) teams to handle the increasing importance of non-technical fac-
This case study came from Gill Mallalieu of the University of tors;
Sunderland. She reported back on a UK organisation that had
• the emergence of more COTS product case studies and disaster
implemented a complex COTS (ERP) product without a cer-
stories, leading to guidelines and success stories for software man-
tain 'backflush' function. This omission was surprising given
agers to use;
the good processes that were followed, inch~ding: effective
requirements elicitation, product evaluation, and vendor in- • understanding that COTS products will evolve constantly;
volvement in the process. However, several problems • agencies use novel techniques such as intelligent agents;
emerged, including: • who will be responsible in the decision-making process when
third-party agencies will be around?
• lack of top-level management commitment;
• lack of ownership of decision-making; These trends provided the basis for the positing of the following re.
ACM SIGSOFT Software Engineering Notes vol 26 no I January 2001 Page 66
Software Product Description Languages third party who will make this information available to COTS-based
There was strong agreement in the breakout session that the systems integrators.
single most important issue related to the market vision was
The critical question is: what are the discriminating characteristics of a
the emergence of standard product description languages. The
software package? Our attempt to determine these characteristics is
need to understand the capabilities of a COTS software pack-
shown in the graphic below. Participants played the roles of software
age under specific operational environments has led to the
product suppliers, customers and brokers, and identified characteristics
concept of COTS package profiling. This involves developing
that were important to their roles in package selection. These charac-
over time a description of the known capabilities and other
teristics were then grouped according to broader categories.
characteristics of the package under consideration. We envis-
age that this profiling service will normally be performed by a
f VENDOR:~ / SOFTWARE~
location issues~ ~ n 6 d o n s , actions and se~ces
basic vendor details; ('~software packagearchitecture;
devt processquality (CMM); =jtability and version frequencie=
plans, credentials & viability; Y,,fitto inter-oparabilitystandards~ batch(es) features
,,~ future producttrends /
effective architectures; "~ wider system architecture;
wative/novel configurations ~t:to non-functional requirements
& integrations; \1 targetenvironment cult
inter-operables e f f w a m ~
amhitectuml styles :/STOMER REQUIREMEN
expressed reqts for system
project references; DEVT:
impl. strategies; relevant user expertise;
r reqt- product compliance; )
, work-arounds,fixes, use
of best-of-breedetc /
Discriminating Characteristics of a Software Package
The most important finding is that most important characteris- • multiple techniques needed, but need to know how to use which
tics are not specific to the COTS software. This has profound technique when sensitivity analysis is too complicated.
implications for software product description languages - We look forward to advancing this work on standard product descrip,
these languages need to describe more than the software prod- tion languages.
ucts; for example, they need to describe the wider system ar-
B r e a k o u t 3: C O T S . B a s e d Software E n g i n e e r i n g P r o c e s s
chitecture, the system environment, previous product
implementations and the customer development process re- Michael Looney (DERA, UK)
lated to the software product. Furthermore, the session ranked Shane Lunga (DERA, UK)
the characteristics according to the level of difficulty to obtain Mark Mosco (University of California, Santa Cruz, USA)
these characteristics reliably, where l=simple and 3--difficult. John Dean (National Research Council Canada)
The results of this ranking are also shown on the graphic
above - most information was perceived to be difficult to The group started off by discussing why there was a need to identify
elicit and use. development process and identifying the scope of what the gron1
might address within that subject.
Finally, the session discussed issues relating to software prod-
uet description languages. Important issues included: It was accepted that the process for COTS component based systen
development was different from that which had been applied for thq
• recognise the strong inter-dependencies between software development of bespoke systems and that there was a need to formali~
product characteristics that provide a basis for multi- that process. It was also agreed that the development process uncle
criteria decision making (MCDM). consideration would only be applicable at the 'system level' and wouh
• q~_mntitativeand qualitative decision-making; not apply to the actual development of the COTS components them
• make qualitative (e.g., requirements) and quantitative selves. It was felt that this was a task carried out by component ven
(e.g., through measurable fit criteria) dors, and at that level there are already well-established soflwar,
development processes in place.
ACM SIGSOFT Software Engineering Notes vol 26 no 1 January 2001 Page 67
It was agreed that, although component selection was a sig- required due to obsolescence. If the system was relatively small and
niiieant part in the development process, it was also the sub- was not expected to evolve or be supported over any extended period,
ject of one of the other groups and so would not be addressed it was felt the framework would be of less importance than the selec-
in any detail. tion of the components. This subject was taken further during the later,
more detailed discussions on the identification of possible solutions to
From the experiences of the group members it was clear that
meet the system requirements and con_straints.
there was a range of systems which needed to be considered,
and it was accepted that what might be appropriate at one end The discussion on the basic stages for the process started with re-
of the spectrum might be overkill at the other. This resulted in quirements and the need to identify the appropriate stakeholders for the
the conclusion that it was necessary to identify a basic process system and to get their 'buy in' to what was being undertaken as the
that could be specified as a set of activities or stages that first stage. This was then followed by the identification of possi-
might be elaborated to a greater level of detail when dealing ble/feasible candidate solutions using the requirements and any con-
with the more complex systems. swaints either from the first stage, as part of the requirement, or as a
result of looking at the various options. An example of this might be
At this point there was also some discussion on the relation-
that the selection of one possible component as a part of the solution
ship between the initial development and the sup-
might imply the use of a second compatible product and prohibit the
port/maintenance cycle. The consensus was that this should be
use of other alternatives. The main objective of this stage would be the
looked at once some form of initial process had been identi-
consideration of the technical options and how best the system re-
fied. There was also an extended discussion on the role of
qnirements might be achieved from that point of view. This would then
system framework or architecture to provide for the independ-
be followed by some form of prototyping to achieve actual measures of
ence and cooperation of the components and on where this
acceptability agnin~ the overall requirements. If there were a 'perfect
came into the development process. The feeling was that it
might be of considerable importance in a larger more complex match' with a selected set of components, which meet all the criteria,
then it would be a simple step to implement the system by integrating
system where it might be necessary to extend the system use
the parts and carrying out the necessary system acceptance activity.
over time and to replace components within that framework as
Constraints q I
Produce set of feasible
Project Management, System Designs
Negotiate Trade off
~ - ~ a t Technical System level
Implementation/ Presentation of Alternative /
Acceptance/ 4 - Options for Decision Maldng
Fielding Process at Business level
Basic DevelopmentProcess Representation
If there were two or three possible technical solutions or con- The discussion then identified the common threads, which would be
sideratiorts of a non-technical nature, then the next stage needed throughout all of the activities, including those associated with
would be some form of business level assessment of the op- the project management and control of such a development approach.
tious. This would look at factors outside the main technical While it was accepted that the appropriate levels of configuration man-
areas such as the 'Wade off' of cost against functionality or the agement and version control would be needed, there were some addi-
need to be 'in line' with the company strategy. If no accept- tional factors that had never been present in the development of
able solution was achieved for whatever reason at this point, bespoke systems, those of licenses and warranties. It was agreed that
then the initial requirements will need to be reconsidered and the concerns in the areas configuration and version control should be
the process repeated. If there is an acceptable solution, then addressed at the system level with the collection of information on the
the f i l l stage for the development would be the implementa- component relationships done within a system context rather than at
tion and acceptance followed by the fielding of the system. the component level. This subject was also covered during the sup-
port/maintenance discussions. There was some discussion on the rela-
ACM SIGSOFT Software Engineering Notes vol 26 no 1 January 2001 Page 68
tionship between the vendors and the developers during the development. It is under these circumstances that the need for elabora
whole process, and it was agreed that it was a significant fac- tion arises. If the system is a simple one, with no requirement for lon~
tor in how well some of the component assessment could be term support, then the selection of the components could well be con.
carried out. But, as the selection process discussion was with sidered the main task of this stage, with little or no effort being spent h
another group, this aspect was not taken further. identifying the architecture/framework and the quickest most simplistic
Following this identification of the basic form of the process approach being adopted for the integration of the components. How,
there was some discussion on the ways in which it might differ ever, if the system is large and complex then it may be necessary t(
for the range of possible systems and the levels of elaboration look at a wider set of issues before attempting to select the compo
that might be necessary. In the task of identifying the set of nents. It might be appropriate to consider a system architeo
possible technical solutions there are several possible activities ture/framework as well as the interface sta.d~rds to be preferred. Als~
that need to be carried out, the most obvious being the selec- it might be appropriate to give some consideration of the approach tc
tion of the appropriate components. Whether that is the first be adopted for the glue code software as shown in the figure below
activity is not at all clear and could, to a large extent, be de- which might comprise legacy parts as well as COTS products ant
some new custom items.
pendent on the type, size and complexity of the system under
VuUy New Development
~ Newly developed component
mq Glue code C O T S component
Illustration of the Use of the Glue Code with the Different Components
It might also be appropriate to consider how system complex- right decisions during the initial development. It was also fel~
ity will vary depending on how well the various parts of the that during this stage other socio/economic factors associate¢
system fit together and the requirements imposed by the pos- with the technical options would be considered. These mighl
sible need to support the system over some extended period. relate to the selection of the human factors related issues or t(
It was agreed that the aspects covered by the choice of archi- previous system compatibility, but whatever the source the3
tecture/framework, the interface standards, and the glue code would act as constraints on the possible solution space.
were the keys to the long term support of any system and that
through life costs would be very dependent on making the
Selection of A rch./Framework
Identification of Glue Code Standard
Identification of Interface Standards 4 - - - -
I Im p a c t Analysis,
S/W, H/W, Socio/economic C o m b i n a t o r y issues
I C o m p o n e n t Set Selection ..................... ]
Elaborated Version of the Produce Set of Feasible System Designs Activity
The final activity considered under this stage of the process Again the level of effort needed to address this might be de.
was the consideration of impact and combinatory issues. pendent .on the type of system under development but for the
ACM SIGSOFT Software Engineering Notes vol 26 no 1 January 2001 Page 69
larger more complex ones this would be an essential part of replacement component considered for integration nmst be as-
the support process if not the initial development. Any new or sessed for its impact on the other existing
components, the ripple effect. Is the new component larger, The group then went back to the question of whether the process for
slower, faster? These are all characteristics that might impact development and that for support for a system were the same. This
the final system capability and must be checked in the system went over a lot of the ground again in terms of the various activities
context to demonstrate the 'usability' of the new component that were needed as part of the support process. The general conclusion
and its relationship with existing components. Many products was that while there were basically the same stages, in the support pro-
satisfactorily pass stand-alone tests but fail the performance cess the constraints were different. The most obvious one is that there
requirements when integrated as part of a larger system. was a system in existence with the various versions of the component,
Worse still, unknown side effects caused by interdependencies which had been originally selected in place but which was being 'up-
could bring the system to a stop. graded' for some reason or other. Therefore there were components
and a framework that were functioning and for which any new changes
Two specific requirements were identified that might have a
made should not degrade current capability or performance. This was
significant impact on the amount of elaboration required in
considered to be of significance because, for any production line sys-
certain areas of the process if they were stated as being neces-
tem, there was a problem of down time caused by the necessity for
sary. These were safety and security, but due to the limited
testing or acceptance. This would entail some loss
time it was not felt appropriate to explore these further during
of production and loss of revenue that would need to be mini- and that if the framework had been selected satisfactorily then
mised. This raised the question of whether it was necessary to integrating the new component would not be too onerous a task.
have a 'back up' system on which to carry out the testing re- However, it was felt that with the possible divergence of a prod-
quired during the upgrading of the system. While this was uct being used from the needs of the system over time, there
considered important, it was not taken further during the dis- might be a more complex set of considerations to undertake as
cussions. part of the upgrade; more akin to the original development proc-
ess but with many more constraints giving a much reduced so-
The next point made was that as there was already a system,
the second stage was much reduced in that it was basically the
identification of the component or components to be changed
Illustration of the Reduced Solution Space as the Selection Criteria and Constraints Become More Precise
The conclusion reached on the support process was that while to consider these as changes to the level of elaboration or detail at each
the basic stages for support were very similar to those for de- stage. But it was felt that the area still warranted further study and in-
velopment there were differences and that it might be possible vestigation.
ACM SIGSOFT Software Engineering Notes vol 26 no 1 January 2001 Page 70
The final area of discussion arose from the input of some of essential to keep details of interfaces and component dependencies so
the latest SEI thoughts on how the process would function and that during the maintenance some form of software agent could be
the way in which it might be used 'for real'by any large proj- used to identify the impact of selecting any new component. There
ect. If the system was large and complex then it was essential would be other similar occasions when it might be possible to provide
to identify interface standards and retain all of the relation- automated checking or high lighting of conflicts thereby reducing the
ships between the components in order to provide the infor- chances of in~oducing errors. Another factor considered significant
mation required to allow through life support to take place. was to identify and capture cost information at as early a stage as pos-
This leads to the need for some sort of project database to hold sible, perhaps even 3~inst the requirement but certainly against the
all of this information. Consideration was given to the sorts of component such as the various license fees etc.
information that could be held, why it would be needed and
As a final conclusion the group thought that progress had been made
how it might be used. Due to the limited time available it was
and that while no final definitive process model had been generated
accepted that this was not something that could be addressed
there was now a much better understanding of what was required and
to any level during this workshop but that there were several
how some of it fitted together. It was hoped that the activity now
aspects of it that might be worth following up at some other
started would continue and that people would follow up on what had
opportunity, While it was obvious that such things as the re-
been achieved in this very important area.
quirements and constraints should be captured it would also be
; Framework/Interface t
Specify / Glue Code
. . . . . . . . . "J~ Req~drements/ :
Constraints ~t - ,
- - ' -%/' l , , Componmt Selection Task
Group Inputs / .
: ! .._.-.--"
Produce set of feasilte
Project Management, ~ I [ ~ . ~ System Desigm 4•m ¢ m
Configm'ation C o n t r o i ~ O j e c t I -~---- i
I Analyse System
Maintenance I Negotiate Trade off
l~OOeSs cycle . . . . l~ at Technical System level
I I I Prototype solution
! I I I
I Presentation of Alternative I
- Acceptancd Options for Decision Making 4t , -
Process at Business levd Process F l o w - ' -~
Information Flow ~-
A Possible Full Process Model for the Development and Support of COTS Component System
Breakout 4: Integration, Maintenance and System a) security
Management • confidentiality
Leigh Davis (University of Tulsa, USA) • identification/authentication
Jerry Gao (San Jose State University, USA) • authorization
George Heineman (WPI, USA) b) perform~n~
Alok Mehta (AFS, USA) c) platform(e.g.,which operatingsystem, which chip)
Robert Seacord (Software Engineering Institute, CMU, d) technologies (e.g., Java, CORBA, DCOM)
USA) If COTS software was specified using these non-functional properties
Mark Vigder (National Research Council Cavsda) (provided by the vendor, then a third party could independently verify
We started with a focus on non-functional properties of COTS
software The trouble is that non-functional properties are often dependent on
fimctionalproperties. Industry lacks a performance model for COTS
ACM SIGSOFT Software Engineering Notes vol 26 no i January 2001 Page 71
software, and such a model would be helpful to predict per- c) Rapid prototyping integration points, therefore, are suggested
formance of C O T S software. as a means to (1) verify the ability to integrate the COTS
software at all; and (2) attempt to measure what the overall
When ratingthe properties for COTS software,an application
costs will be.
builder can make tmdeoffs. How is this to be reflected in the
4. How can we mitigate the risk of maintenance costs?
properties for a product?
a) You must never forget that upgrading is driven by the COTS
1. How can we detect in advance a conflict with using a par-
ticular COTS software system? It should be possible to iden-
b) Could the industry select a consistent monitor-
tify candidates that can be disqualified. If there were a
ing/instmmentation standard to make it possible to track the
standard mechanism for specifications (and this would be a
use of COTS software within a system and determine where
big cultural change)...
defects are occurring?
a) How detailed would these specifications be? (shallow • The only way this would work is if the "value added" of
vs. deep) the standard mechanism were accepted by huge numbers
b) CoUld we develop algorithms to search potential of vendors.
COTS software being considered for use and detect 5. How do we do system management of COTS systems?
a) Configuration Management is addressed at the source code
• The collection of specifications does not need to
be centralized in a database.
b) Version checking at the COTS soRware level (i.e., this ver-
• It canbe incrementally updated.
sion of COTS system A works with any version number 2.5
• The specifications can be in heterogeneous lan- and higher for C O T S product B)
guages if we have sophisticated searching algo- c) There is a constantthreatthatan upgrade of C O T S software
rithms. will alteritsinterface.
2. How can we determine IN ADVANCE integration costs? to
(I) There isa lack of utilities manage such C O T S systems.
H o w can multiple in~allafionsbe maintained? H o w can we
a) What are the problems with integration? mange/control the system without directuser intervention
• Mismatch in interfaces (detected by compiler) (i.e.,automatic scripts)?H o w do we deal with licenseexpira-
• Inserting COTS software that can break other tion and re-installations?
parts of the system (i.e., the COTS software is e) There will always be a tmdeoff between upgrading to include
not at fault) the new version of COTS software or keeping and dealing
• Architectural mismatch (i.e., Ocker- with an older version that is known to work. Try to develop
bloom/Ccarlan) business model for selecting technology - only upgrade
• Tremendous glue code needed (no reusable COTS soRware if benefits exceed costs.
wrappers) 6. Does Open Source COTS Software change anything?
• Costs that depend on the system into which
COTS software is being injected. a) There are options available to the integrators, and it's easier to
b) This problem defies simplification. find defects.
b) We must avoid the temptation to modify the code and
• No standard means of composing COTS soft-
thereby create a long-term internal maimenance cost.
7. Are there success stories7
c) Does the use of C O T S lower integrationcosts?
• W e agreed thatusing C O T S will increaseinte- a) Flight Simulation software for F18 aircraft
grationcosts, and this goes against the idea of b) Domain-Specific Software Architectures
using COTS to reduce overall lifecycle costs. 8. Recommendations
d) Configurable C O T S software will increase integra-
a) Databases and compilers have benchmarks. Can one be de-
veloped to help COTS?
e) Since COTS vendors focus on feature differentiation,
b) Work with Underwriters Laboratories (UL, Inc.) to find how
often COTS soRware is chosen because of a particu-
to develop data sheets for COTS software
lar nifty feature. Instead, an organization should se-
lect a COTS software feature based on "the risk to the
Breakout 5:Business Models
system if the feature is not available."
3. How can we lower the integration costs? Aria Alvarez (European Software Institute,Spain)
Michael Guntersdoffer (Universityof California,Irvine,USA)
a) Organizations should build expertise in particular Rouald Kohl (AverStar,Inc.,USA)
black-box integration methods. This more than any- Maurizio Morisio (Universityof Maryland, USA)
thing will help lower the costs of integration by man- JeffreyVoas (ReliableSoftware Technologies, USA)
dating the use of tried methods.
b) Since the integration of COTS software is often a
failure point, the evaluation process for selecting The purpose of the breakout session on Business Models was to:
COTS software should include a prototype integra- • identify today's problems of COTS software components and how
tion effort into a toy system. they relate to their weak business model;
ACM SIGSOFT Software Engineering Notes vol 26 no 1 JanuarY 2001 Page 72
• discuss how those problems are currently mitigated; their system even though the component is technically sound.
• identify shortcomings in those mitigation strategies in 12) In case of the occurrence of problems with a component, often
order to identify the areas with need for improvement; another problem occurs: Who is to blame? For example, consider
• determine where improvement could be offered by the a system that is composed of components from more than one
sofi~are engineering community; and vendor. In the case of interaction problems between two compo-
• identify the corresponding research areas and topics. nents bought from different vendors, it is likely that both vendors
The breakout was carried out in two sessions. The first session will deny responsibility and point fingers at each other.
was used to identify both essential and accidental problems of The different types of parties affected by these problems were identi-
COTS components and their business model, as defined by fied as
Brooks. Groups of people (parties) that are affected by these v. The procurement personnel of the purchaser (buyer side).
problems (including the linkage of individual problems to in- vi. The user of the system, which integrates the component
dividual parties), and how mitigation is carried out today in (buyer side).
each case were identified. The second session was used to vii. The system integrators (buyer side).
identify the problems where mitigation is insufficient and viii. The publisher (seller side).
where software engineers could come to the rescue. Finally, ix. The (system) maintainers (buyer side).
corresponding research topics were identified. x. The COTS analyst (independent, third party).
The problems (or risks) were assigned as relevant to each party as fol-
Session I lows:
The following typical problems/risks of COTS components
and their application were identified: c) 2,5,8,10,12.
1) A component reveals too much functionality, i.e., the e) 1-12.
component exceeds the specification and offers additional f) 2 , 4 , 6 , 8 , 9 , 1 1 , 1 2 .
functionality in addition to the one specified or needed by g) 1 -6, 8 - 10, 12.
the purchaser. h) 4 & 5 .
2) A component reveals too little functionality, i.e., the com- The following mitigation strategies of each party were identified as
ponent cannot provide all the functionality specified or being current "state-of-the-art" for each case:
needed by the purchaser.
3) A component's documentation is incomplete, i.e., func- A:
tionality, features, or any other behavior of the component
is not described in its documentation. 2. Lawsuit (breach of contract), return to vendor, use alternative
4) A component's documentation is incorrect, i.e., the docu- component.
mentation is indeed flawed and error prone, containing 5. Understand vendor options, lawsuit (bankruptcy law).
incorrect statements about the component's behavior.
5) The component's vendor is not viable, i.e., the seller of the 8. Understand support options, lawsuit (breach of contract, if ap-
component is driven out of business and cannot further plicable).
maintain the product or support the customer. The buyer 10. Purchase alternative product, purchase additional functional-
is therefore left on its own. ity, hire personnel to write additional functionality.
6) The component's functions are unreliable, i.e., the buyer
of the component cannot depend on the component to 12. Threat of lawsuit (to all parties), contact third party to decide
carry out its tasks correctly. (arbitration), use internal resources to decide.
7) An operational assumption mismatch occurs, i.e., the B:
creator and the user (both parties are software developers
here) have different opinions on how the component 1. Write user procedures (guidelines for users, what not to use and
functions. This is the famous architectural mismatch how).
coined by Garlan. 2. Use another product, request more (missing) functionality.
8) A component vendor decides to limit or stop supporting a
component, i.e., even though the vendor is still in busi- 3. Document yourself, request better (complete) documentation.
ness and capable of supporting the customers, he decides
4. Document yourself, request better (correct) documentation.
to dump a product, e.g., due to lack of profitability.
9) .A component turns out to be volatile, i.e., the component 5. N/A (nothing the user can do about that).
has frequent downtimes where it doesn't function properly
6. Modify usage procedures, report failure.
(e.g., crashes) and cannot be used at all.
10) The system needs diverges from the component's capa- 7. Report failure, change hardware/operating system profile, if ap-
bilities, i.e., over time the vendor's ongoing development plicable.
of the component results in versions with different func-
tionality from the needs of the purchaser, whose system is 8. Complain to vendor.
also evolving. 9. Complain to vendor.
11) The composer won't integrate, i.e., the purchaser's system
integrators are unable to integrate the component into 10. Complain to system integrator.
ACM SIGSOFT Software Engineering Notes vol 26 no 1 January 2001 Page 73
12. Provide more data to determine culprit. spin-off product.
9. Listen to customers, modify release schedules.
1. Wrappers, firewalls, procedures for users, request 11. Use standards, redesign, test for additional configurations, of-
product modificatiorL fer on-site integration support.
2. Request product modification, extend product via 12. Surveillance instrumentation by vendor, hire third party, com-
wrappexs to add functionalityor via otherproducts. plex licenses that protect publishers, lawsuit.
3. Same asB3.
4. Same as B4. 4. Hire a third party expert, ask around to validate documentation
claims, in-house test lab.
5. Find competing alternatives, plan to build in-house,
contract to put source code into escrow. 5. Do independent analysis of company or product.
6. Find competing alternatives, plan to build in-house, re- Session Ih
quest fixes. The following software engineering research topics were derived from
the resultsof session I:
7. (C only) same as C6 + B7.
I) Detecting hidden functionalityand hidden interfaces(see also:
8. Same as B8 + subcontract support to 3rd party.
dormant code, malicious code). Targets risksC3, C4, C11, C12,
9. Same as B9 + refuse to upgrade. E3, E4, E12, and F4.
2) How to determine thata COTS component isunreliable(creation
10. Same as C7 + stand down on the requirements. Targets risksC6, C9,
of oracles,creationof testinfrastructures).
11. (C only) Return, find alternatives, request modifica- C11, C12, E6, E9, and E12.
tionfrom vendor, wrappers. 3) Waystocrcatewrappcrs. Targctsrisks CI, C2, Cll, El, andE2.
4) Definitionof volatilitymetrics.Targets risksC9, C12, E9, and
12. Hire third party expert, collect data via instnnnenta- El2.
tion. 5) Instrumentationof binariesto collectinformation about existing
D: behavior. Targets risksC3, C4, C6, C7, C9, CI0, C11, C12, E3,
E4, E6, E9, El0, El2, and F4.
2. Add functionality, deny, better requirements. 6) Definitionof metrics for similarityof competing products. Targets
4. Fix, deny. risksAt0, B2, C5, C7, C9, C10, Cll, E5, E9, and El0.
7) Definition/developmentof metrics/gauges for intcroperability.
6. Fix, deny, improve V&V, better requirements. Targets risksC5, C7, C9, CI0, Cli, C12, D7, DI I,E5, E9, El0,
8. Improve vendor support, subcontract (outsource) sup- and E12.
port, make explicit to customers that support is limited,