COTS Workshop Continuing Collaborations for Successful COTS

Document Sample
COTS Workshop Continuing Collaborations for Successful COTS Powered By Docstoc
					ACM SIGSOFT                                     Software Engineering Notes vol 26 no 1                             January 2001 Page 61

    COTS Workshop: Continuing Collaborations for Successful COTS Development
                                    John Dean, National Research of Council Canada
                                    Patricia Oberndorf, Software Engineering Institute
                                     Mark Vigder, National Research Council Canada
                                        Chris Abts, Universityof Southern California
                                Hakan Erdogmus, National Research Council of Canada
                                                 Nell Maiden, City University, UK
                                                    Michael Looney, DERA, UK
                                    George Heineman, Worcester Polytechnic Institute
                                  Michael Guntersdorfer, Universityof California, Irvine

Introduction                                                         summary of their discussions which are included, without major edit-
In early June of 2000 a COTS Workshop entitled "Continuing           ing, below. A more complete description of the workshop, as well as
Collaborations for Successful COTS Development" was held             all the participants' position papers, can be found at:
in Limerick, Ireland in conjunction with ICSE 2000. The pur-              http://seg.iit.nrc, ca/projects/cots/icse2000wkshp/index.htnd
pose of the workshop was to collect experience reports re-
garding the use of commercial off-the-shelf (COTS) software          Workshop Results
to build systems, identify best-practices for the use of COTS
software, and to establish a research agenda for those re-           The Workshop brought together a diverse group of researchers and
                                                                     practitioners experienced in many areas of software engineering. This
searchers interested in COTS-based software systems. This
one and a half day workshop was an extension of the work             allowed the Workshop to identify a number of best practices when
begun during the workshop entitled "Ensuring Successful              building COTS-based systems, as well as directing the participants to
COTS Development" held in conjunction with ICSE '99. Re-             problems requiring further research.
sults from that workshop demonstrated that there were a num-         A complete summary of the breakout sessions is provided below, but
ber of common research areas, including acquisition, planning        some of the key findings that emerged during the plenary included:
and management, architecture and implementation, and
evaluation and testing, for which researchers saw the possibil-      *   The Economic and Financial Issues breakout agreed that improved
ity of collaboration. These areas included specific topics such          models need to be developed for cost-estimation associated with
as estimating the effort required to implement COTS-based                the entire lffecycle of COTS-based systems. This includes esti-
systems, classification of architectural styles, and certification       mating cost, schedule, and effort for each stage of development
of COTS products for reliability and safety. The group will              and maintenance. Significant data gathering is still required for
reconvene at ICSE'01 (w_~_ww.c_s_r..u.w_'c;.ca/icse200Dto discuss        calibration of these models.
further the results achieved.                                            As well, managers need models and tools to support making deci-
The ICSE 2000 Workshop had about 26 participants and was                 sions based on economic models associated with the technical as-
formatted as a combination of plenary sessions and small                 pects of the system. This allows managers to decide, for example,
breakout groups that worked on specific issues related to                whether to buy or build a component, or whether to upgrade a
COTS-based systems. The breakout groups investigated the                 component now or wait for a future release.
impact of COTS software usage in the following areas:     •              The Requirements Definition breakout focussed on the relation-
                                                                         ship between requirements and COTS selection and usage. Given
•    Economic and financial issues.                                      that much of the functionality of a system is being bought, not de-
•    Requirements definition_                                            veloped, product selection becomes critically intertwined with re-
.    Software engineering process.                                       quirements definition_ The group used two case studies to ground
•    Integration, maintenance and system management.                     their discussions and used them as a basis for setting a research
•    Business models.                                                    agenda for the next three to five years. The most important issue
                                                                         was determined to be appropriate Product Description Languages
Each breakout group tried to identify the current state of the           for COTS products by which integrators can determine operational
art in COTS software usage as well as open questions that
                                                                         characteristics of a product within different contexts. The product
could provide the basis for fiLrther research in the coming
                                                                         descriptions would ideally be developed by independent third par-
years. Each group was responsible for producing a written
    ACM SIGSOFT                                  Software Engineering Notes vol 26 no 1                            January 2001 Page 62

      ties and cover properties of a product well beyond itsop-      Application of Financial Valuation Models
      erational characteristics.                                     The COTS approach must provide clear fiuancial benefits for it to be a
•     The Software Engineering Process breakout identified                                                              ncial valuation tech-
                                                                     viable alternative to traditional development Fin.q_
      how the development process changes when a system is           niques such as Net Present Value and Real Options Analysis aid in this
      COTS-based. The number of development activities re-           kind of assessment by linking development decisions to their economic
      quired and the constraints that exist between them di-         impact. Within this framework, the problems being addressed include:
      rected the discussion. The activities of system architecture
                                                                     •   Comparison of a COTS-based development strategy with a custom
      and infrastructure design, product selection, and require-
                                                                         development strategy for a given system.
      ments definition all have much more complex relation-
                                                                     •   Comparison of two different COTS-based development strategies
      ships among them for COTS-based systems than custom-
      built systems. There are also new project management               for a given system.
      functions relating to software licencing and warranty          •   Assessment of the value of flexibility inherent in a COTS-based
      management and COTS product evaluation and upgrade                 system.
      decisions.Many of these issues were discussed as well in       •   Assessment of migration strategies, both from a COTS-based
      the Economic and Financial Issues and the Requirements             system to a custom system and from a custom system to a COTS-
      Definition Breakout.                                               based system.
•     The Integration, Maintenance and System Management             •   Timing and value of COTS product upgrade and system release
      breakout agreed that the integration costs for COTS-based          decisions.
      systems are often higher than for custom-built systems.        •   Management of COTS product license costs to mitigate risk.
      The group felt that there must be ways that products can       CO 7"5 vs. Custom Decisions
      be described to allow system builders to determine the         The viability of COTS-based development with respect to custom de-
      integration issues and costs before building the systems.      velopment can be assessed using a simple financial model based on the
      Similar to the Requirements Definition group, they felt        concept of Net Present Value (NPV). In corporate fin_ance, NPV is the
      that there was a need for independent evaluators to pro-       gold standard for making capital budgeting decisions. The NPV of a
      vide profiles of COTS products. Maintenance and man-           project is given by the sum of its discounted future cash flows. Dis-
      agement become significant issues as the system is being       counting takes into account time value of money, i.e., the notion that
      maintained at the COTS product level rather than at the        money that is yet to be expended or received is worth less today than it
      source code level, and many of the maintenance and man-        is in the future. The NPV rule states that only projects with positive
      agement decisions are being driven by the COTS vendor          NPV are worth undertaking, and a project with a higher NPV is prefer-
      rather than the system builder.                                able to a project with a lower NPV.
•     The Business Models breakout identified all the
      stakeholders involved in developing a COTS-based sys-          An NPV-based model of software development can compare two de-
      tem and the various risks faced by each of these               velopment strategies using a metric that captures the economic incen-
      stakeholders. They then proposed a set of risk mitigation      tive to choose one strategy over the other. The five parameters that
      strategies that can be practiced by each of these              a~ect this metric are:
      stakeholders. Although stir requiring a significant amount     •   the ratio of the development cost of the two strategies;
      of research to complete, they have provided a direction        •   the ratio of the time-to-market of the two strategies;
      that allows those involved to manage the risk.
                                                                     •   the ratio of the operation cost of the two strategies;
                                                                     •   the ratio of the asset value (a measure of post-development bene-
Breakout Summaries
                                                                         fits or market value) of the two strategies; and
Each Breakout Coordinator prepared a summary of the results •            the market risk of the end product, as captured by the discount
of their individual sessions. These summaries are presented in           rate.
this section.
                                                               With the estimates of these pmmneters, it is possible to compare dif-
Breakout 1: Economic and Financial Issues                      ferent development strategies. These include not only different types of
                                                               pure COTS-based and pure custom development sWategies, but also
    Chris Abts, (University of Southern California, USA)
                                                               hybrid strategies, and with the use of Real Options Analysis, strategies
    Hakan Erdogmus, (National Research Council Canada)
                                                               that start with one type of system and gradually migrate to another type
    Patricia Obemdoff (Software Engineering Institute, CMU,
                                                               of system. In the latter case, the incremental migration of system com-
                                                               ponents is viewed as a series of options that are exercised provided that
The purpose of the breakout session was to explore possible
                                                               the long-tea-a~ benefit of each migration phase exceeds the immediate
areas of collaborationthat pertainto the economic and finan-
                                                               cost of that phase.
cial aspects of COTS-based software development. The dis-
cussions were centered on two topics:                          Two rules of thumb that emerge from an NPV-based analysis are:
1) Application of financial vahmt__ionmodels to evaluate             •   When the COTS approach has both siotmificanfly higher asset
   COTS-based development decisions.                                     value and operation cost, increasing product risk increases the in-
2) The COCOTS cost estimation model.                                     centive for the COTS approach only if that approach also has suf-
                                                                         ficient time-to-market advantage over a custom system.
ACM SIGSOFT                                     Software Engineering Notes vol 26 no 1                              January 2001 Page 63

    If time-to-market is important and the custom system has        for the developer, but a liability for the vendor when license fees ex-
    significantly higher market value, COTS-to-custom mi-           hibit a great deal of fluctuation. Therefore the vendor is justified to ask
    gration strategy is usually most valuable.                      for an up-front premium from the developer in return. The fair value of
                                                                    this premium, again, can be determined by Real Options Analysis.
Value of Replacement Flexibility
One advantage of the COTS approach is flexibility. Flexibility      COCOTS (COCOMO for COTS) Update
arises from the abitity to replace the COTS components used         Cost estimation is a vital input for any economic analysis, including
in the systems at a fraction of the cost normally required to       financial valuation. The COCOTS cost model is being developed at
develop those components from scratch. However, to achieve          USC Center for Software Engineering to address the needs of COTS-
such flexibility, the system must be designed accordingly.          based systems. It is one of a suite of specialized models being devel-
This implies an up-front investment in a robust COTS archi-         oped as extensions of the COCOMO (Constructive Cost Model) II cost
tecture. Such an investment buys the developer the option to        estimation model. The COCOTS model has features that make it suit-
replace the COTS products at a reasonable cost in the future.       able for effort and schedule estimation in cases where parts of a system
This early investment can be thought of as a risk mitigation        are implemented by COTS products.
strategy to hedge against replacement cost uncertainty. The        The two defining characteristics of COTS software, and thus which
cost of the extra investment in system architecture is not jusli-  drive the whole COTS usage process, are: 1) the COTS product source
fled if it surpasses the value of the replacement flexibility, or
                                                                   code is (typically) not available to the application developer, and 2) the
the option value of future replacement opportunities. There-
                                                                   future evolution of the COTS product is not under the control of the
fore, it is important to identify the circumstances under which
                                                                   application developer. (The market decides when and how a given
the benefits of flexibility exceed the cost of achieving it.       COTS product evolves.) These characteristics are the underlying con-
A state-of-the-art financial valuation technique, Real Options stmints that shaped the current form of the COCOTS model.
Analysis, can address this issue by mapping the replacement
                                                                   (Note: COCOTS as described here deals only with initial COTS inte-
scenario to an option pricing problem. By investing in a robust
                                                                   gration efforts. Subsequent to the COTS workshop at ICSE 2000, work
COTS architecture, the developer is acquiring a future option has been done to expand the model to provide estimates of long-term
to upgrade the technologies underlying the COTS products
                                                                   COTS-based system operation and maintenance effort through system
used. When sufficient information is available, the value of
                                                                   retirement. That work is on-going, however, and remains outside the
this option can be estimated and compared to the amount of
                                                                   scope of the discussion here.)
architectural investment needed.
                                                                   COCOTS is actually a collection of four related submodels, each ad-
Upgrade Decisions                                                  dressing individually what has been identified by USC-CSE as the four
In a COTS-based system, upgrading a COTS product should primary sources of COTS softwar¢ integration costs (from a system
be a discretionary activity. Some upgrades will be minor and construction activity point of view). These sources include the effort
incur a relatively small cost, while others will be major and needed to perform (1) candidate COTS component assessment, (2)
require substantial rework. When the cost of a product up- COTS component tailoring, (3) the development and testing of any
grade is superior to its long-term benefits, it does not make integration or "glue" code needed to plug a COTS component into a
good economic sense to proceed with the upgrade. This larger system (or to make it function with another COTS component),
thinking gives upgrade decisions an option-like flavor. How- and (4) increased system level progranmung due to volatility in incor-
ever, product obsolescence must also be accounted for when porated COTS components.
technology evolves at a rapid pace. Delaying an upgrade in-
                                                                   Assessment is the process by which COTS components are selected for
definitely may have serious long-term economic consequences
                                                                   use in the larger system being developed. Tailoring refers to those ac-
that are not immediately apparent.
                                                                   tivities that would have to be performed to prepare a particular COTS
In addition, in the presence of multiple COTS products, syn- program for use, regardless of the system into which it is being incor-
chrouization of a system's upgrade cycles with the release cy- poratecL or even if operating as a stand-alone item. These are things
cles of the COTS products used may be problematic.                 such as initializing parameter values, specifying input/output screens
                                                                   or report formats, setting up security protocols, etc. Glue code devel-
We feel that integer programming combined with Real Op-
                                                                   opment and testing refers to the new code external to the COTS com-
tions Analysis together can tackle the timing and economic
                                                                   ponem itself that must be written in order to plug the component into
feasibility of upgrade decisions.
                                                                   the larger system. This code by nature is unique to the particular con-
                                                                   text in which the COTS component is being used and must not be con-
 Management of COTS Product Licenses                               fused with tailoring activity as defined above. Volatility in this context
Any multi-user system that is based on COTS products will refers to the frequency with which new versions or updates of the
incur license costs. When license fees are volatile, the total COTS software being used in a larger system are released by the ven-
cost may be substantial, and their proper management is thus dors over the course of the system's development and subsequent de-
instrumental for mitigating the resulting risk. Special option- ployment.
like contracts may be negotiated with the COTS vendors, for
example to give the developer the future right to acquire a yet Schedule Estimator
unknown number of licenses of a certain COTS product with a As well as effort, COCOTS will attempt to estimate the associated
cap on the single-license fee. A contract such as this is an asset calendar schedule that wig be required to construct a system using
 ACM SIGSOFT                                      Software Engineering Notes vol 26 no 1                             January 2001 Page 64

COTS components. The schedule estimator should define the             This breakout session aimed at bridging the gap between these com-
relationship between development time and effort for a COTS-          plementary approaches by exposing two promising threads of work to
based system. It is also through the schedule estimator that the      each other. We hope that it will lead to fruitful collaborations among
connection between COCOTS and COCOMO II is most di-                   the parties involved.
rectly evident. COCOMO II can be used to determine effort
associated with those parts of a system that are to be created        Breakout 2: Requirements Definition & COTS: Setting a
from scratch or by reusing components for which the source            Research Agenda for COTS-Based Development 3-5 Years
code is available. COCOTS is to be used to estimate the effort        Down the Line
associated with those parts of a system that are to be imple-            Nell Maiden (City University, UK);
mented using COTS components. The required schedule                      Ljerka Beus-Dukic (University of Northumbria, UK);
needed to complete the entire system (including new, reused,             Colin Paul Glasier (City University Dublin, Ireland);
and COTS elements) must be a function of the effort required             Gillian Mallalieu (University of Sunderland, UK);
for all those elements combined. Thus, the use of the CO-                Cornelius Ncube (City University London, UK);
COMO II schedule estimator equation is proposed for CO-                  Thuy Nguyen (EDF, France);
COTS, but the input effort parameter (defined in person-                 Joan Pastor (Universitat Polictecnica de Catalunya, Spain);
months) includes both the traditional COCOMO and the addi-               William M~ Thomas (The MITRE Corporation, USA)
tional COCOTS derived effort (as well as a possible recalibra-
tion of some of the constants in the schedule model).                 Structure Of The Breakout Session
                                                                      The work carried out in this breakout session was divided into 3 basic
Effort and Schedule Distribution Percentages                          sessions, each addressing a basic problem or issue concerned with de-
Compared to COCOMO II, the initial COCOTS calibration                 fining requirements and evaluating soflavare packages for their fit with
data set suggests a slightly different distribution of total devel-   these requirements. Firstly, we elicited 2 real-world COTS selection
opment effort and schedule across development activities. For         and configuration case studies, to ground the academic discussions and
our current set of 20 industrial data points, and within our de-      provide a focus for research later in the session. The first of the 2 case
fined CoTS-related activities, preliminary analysis indicates         studies looked at COTS package selection at EDF, the French electric-
that approximately 8% of the total COTS related effort is             ity supplier, and the second reported problems in adaptation of a COTS
dedicated to assessment, 28% to tailoring, 45% to develop-            software package to include missing funetion_~lity. Secondly, in order
ment of glue code, and 19% to activities that pertain to COTS         to enable us to explore a research agenda for requirements engineering
volatility. See the position paper and the accompanying slides        and COTS package selection we brainstormed a vision of the COTS
for further details.                                                  software marketplace 3-5 years down the line. This vision was to act as
                                                                      a baseline for the research agenda. Thirdly we developed a comprehen-
COCO TS Future Directions                                             sive research agenda for COTS software package selection 3-5 years
As stated previously, currently the COCOTS model covers               into the future, with particnlar emphasis on determining what consti-
only the development phase. The initial model is actively be-         tutes a COTS software package to enable new decision-making capa-
ing extended to address operation and maintenance activities          bilities. The following describes each of these 3 sessions in more
as well.                                                              detail.
The next round of data collection for calibration purposes is Two Cots Product Selection Case Studies
expected to begin in fall 2000. A revised version of the model The breakout session investigated two case studies reported by its par-
is expected soon afterwards. The revised model will account ticipants.
for operational and maintenance costs in addition to the initial
development costs.                                                  Selecting COTS Products for Electn'city Power Plants
As new data becomes available, the initial development model       This first case study was reported by Thuy Nguyen from EDF in
as well as the opemtious and maintenance model parameters France. Selecting COTS software packages for electricity power plants
will be recalibrated.                                              is a large, long, and complicated process. Thuy gave us some stark
                                                                   facts. In a typical process of selecting embedded COTS software pack-
Conclusions                                                        ages for power plant conlrol, there are 3000 pages of user require-
Constructing a sound business case for or against COTS prod- ments, up to 85% of the requirements are non-technical requirements,
uct usage requires economic models that can deal with the there are 200-300 COTS candidates to choose from, and the selection
peculiarities of COTS-based system development. Specialized process can take several years to complete. Furthermore the selection
cost estimation models and state-of-the-art financial valuation of the fight COTS software package is business critical: EDF lose $1M
techniques are expected to help in this endeavor.                  per day ff the plant is not working. If this problem were not complex
                                                                   enough, there is a need to consider multiple power plant systems: ide-
Cost estimation and financial valuation are highly comple- ally EDF would like to reuse software packages and package evalua-
mentary in that cost data is a vital input of financial valuation. tion information for design of new and future power plants, however
Whereas cost estimates are traditionally of more direct interest the time-scale of each selection process currently makes this infeasible.
to project managers, financial valuation is of more direct in-
terest to executives and investors who are concerned not only The current COTS software package selection process in EDF is a 4-
with costs, but also with the overall long-term value.             stage process:
 ACM SIGSOFT                                    Software Engineering Notes vol 26 no 1                             January 2001 Page 65

i.   Analyse eqm'pment facilities: EDF investigate complex          •   failure to recognise incremental development and future software
     non-functional requirements (e.g., performance require-            versions.
     ments) with simple formal models, which also enables           Again, non-technical issues were identified as more important than
     them to demonstrate these requirements to other                technical issues in the success of this COTS customisation.
     stakeholders. At the same time EDF place contracts on all
     software pacl~ge vendors to provide them ALL required          Brainstorming 3-5 Year Market Vision To Baseline Our
     information for subsequent decision-making phases,             Research Agenda
ii. Write the user requirements document: This process can          The next stage of the session established a baseline for long-term re-
     often take 1-2 years. The requirements are often written       search in requirements engineering and package selection in a COTS-
     by domain experts rather than software experts, with im-       driven development process. However, the session faced a problem:
     plications for the style and content of the requirements.      setting a research agenda based on current problems and processes
     Most user requirements are written as untestable natural       might invalidate the agenda if these problems and processes change
     language statements. EDF must also take care to keep the       sig~aificantly over the next 5 years. Hence to establish the baseline the
     requirements' generic', to avoid bias towards one package      session undertook a brainstorming session to discover future trends in
     vendor. For critical requirements, EDF critique these re-      COTS software packages and their use of software development. Ideas
     quirements with formal models similar to the use of for-       were collected on a whiteboard, then structured into themes. The re-
     mal models applied in stage 1. The result of this stage is     suiting market vision was written as the following scenario:
     an invitation-to-tender (ITF) document,
iii. Collect vendor responses and generate system specifica-        "Both customers and suppliers of COTS software products will use
     tions: Vendor responses to the 1Tr provide the basis for       third-party product evaluation agencies to deliver added-value to their
     writing the system specification, a process that can take a    selection, customisation and integration processes. These agencies will
     further one year or more. This specification is a specifica-   use in-depth knowledge of customer requirements, vendor products,
     tion of the wider socio-technical system, and will include,    customers and suppliers, and how to undertake decision-making proc-
     for example, human accident handling procedures as well        esses to broker deals and advise their sponsors. These product evalua-
     as software procedures. Each system requirement is de-         tions will be 'lighter' than current product logs, profiles and
     fined as either negotiable or non-negotiable. Responses        certifications, in that the agencies will recegnise the diverse range of
     from vendors are often in two parts: (i) the 'standard'        different situations in which COTS software is used and tailor their
     vendor specification for their package, (ii) responses to      advice accordingly. These agencies will emerge bottom-up to seize
     critical customer requirements;                                market share when the opportunities arise, rather than through top-
iv. Select the soRware package: The package selection proc-         down government or other intervention. R will be just as likely that the
     ess is a highly mechanistic one, which is essential to         agencies will recommend to purchase COTS software as to use Appli-
     avoid being sued for unfavourable selection decisions. As      cation Service Providers (ASPs), renting products and ~ n g open
     such, due to this mechanistic nature, EDF see package          source software. Due to the demand and scale of the problem, agencies
     selection as something distinct fIom requirements engi-        will maximise the use of semi-automated computer-based techniques
     neering.                                                       that rely on standard requirements and product description languages,
Thuy reported that EDF find the current process too long, ex-       that is an agreed lingua-ftanca for deciding about COTS products."
pensive and not 'reusable' to meet new market opportunities         Other identified trends in this market vision not incorporated into the
such as new power plants to be developed in the developing          scenario included:
world. He identified the following requirement that EDF
might have to overcome these problems:                              •   using COTS products as 'free' prototypes to drive the require-
                                                                        ments elicitation process;
Third-party assessors to do the difficult and time-consuming •          tendering for software services as you tender for other commodi-
package selection work, guarantee requirements compliance,              ties such as financial services;
and offer confidence in this guarantee; furthermore the pack- •         greater componentisation of COTS products, leading to compo-
age vendors should pay for this work.                                   nents-on-demand;
                                                                    •   the rise of architectures, middle-ware and integration frameworks;
Tailoring an Enterprise Resource Planning (ERP)                     •   the rise of multi-disciplinary evaluation and decision-making
Package (The Back-Flushing Problem)                                     teams to handle the increasing importance of non-technical fac-
This case study came from Gill Mallalieu of the University of           tors;
Sunderland. She reported back on a UK organisation that had
                                                                    •   the emergence of more COTS product case studies and disaster
implemented a complex COTS (ERP) product without a cer-
                                                                        stories, leading to guidelines and success stories for software man-
tain 'backflush' function. This omission was surprising given
                                                                        agers to use;
the good processes that were followed, inch~ding: effective
requirements elicitation, product evaluation, and vendor in-        •   understanding that COTS products will evolve constantly;
volvement in the process. However, several problems                 •   agencies use novel techniques such as intelligent agents;
emerged, including:                                                 •   who will be responsible in the decision-making process when
                                                                        third-party agencies will be around?
•    lack of top-level management commitment;
•    lack of ownership of decision-making;                          These trends provided the basis for the positing of the following re.
                                                                    search agenda.
ACM SIGSOFT                                         Software Engineering Notes vol 26 no I                                          January 2001 Page 66

Software Product Description Languages                                     third party who will make this information available to COTS-based
There was strong agreement in the breakout session that the                systems integrators.
single most important issue related to the market vision was
                                                                           The critical question is: what are the discriminating characteristics of a
the emergence of standard product description languages. The
                                                                           software package? Our attempt to determine these characteristics is
need to understand the capabilities of a COTS software pack-
                                                                           shown in the graphic below. Participants played the roles of software
age under specific operational environments has led to the
                                                                           product suppliers, customers and brokers, and identified characteristics
concept of COTS package profiling. This involves developing
                                                                           that were important to their roles in package selection. These charac-
over time a description of the known capabilities and other
                                                                           teristics were then grouped according to broader categories.
characteristics of the package under consideration. We envis-
age that this profiling service will normally be performed by a

                    f          VENDOR:~                 /         SOFTWARE~
                            location issues~      ~ n 6 d o n s , actions and se~ces
                         basic vendor details;       ('~software packagearchitecture;
                     devt processquality (CMM); =jtability and version frequencie=
                     plans, credentials & viability; Y,,fitto inter-oparabilitystandards~          batch(es) features
                    ,,~ future producttrends /

                                           effective architectures; "~ wider system architecture;
                                          wative/novel configurations ~t:to non-functional requirements
                                               & integrations;        \1 targetenvironment cult
                                          inter-operables e f f w a m ~
                                             amhitectuml styles                                    :/STOMER REQUIREMEN
                                                                                                   expressed reqts for system

                                                                        project references;                             DEVT:
                                                                         consultant CVs;
                                                                         impl. strategies;            relevant user expertise;
                                  r reqt- product compliance;     )
                             interchangeable products
                                                                                        , work-arounds,fixes, use
                                                                                        of best-of-breedetc     /

                                              Discriminating Characteristics of a Software Package
The most important finding is that most important characteris-             •    multiple techniques needed, but need to know how to use which
tics are not specific to the COTS software. This has profound                   technique when sensitivity analysis is too complicated.
implications for software product description languages -                  We look forward to advancing this work on standard product descrip,
these languages need to describe more than the software prod-              tion languages.
ucts; for example, they need to describe the wider system ar-
                                                                           B r e a k o u t 3: C O T S . B a s e d Software E n g i n e e r i n g P r o c e s s
chitecture, the system environment, previous product
implementations and the customer development process re-                        Michael Looney (DERA, UK)
lated to the software product. Furthermore, the session ranked                  Shane Lunga (DERA, UK)
the characteristics according to the level of difficulty to obtain              Mark Mosco (University of California, Santa Cruz, USA)
these characteristics reliably, where l=simple and 3--difficult.                John Dean (National Research Council Canada)
The results of this ranking are also shown on the graphic
above - most information was perceived to be difficult to                   The group started off by discussing why there was a need to identify
elicit and use.                                                             development process and identifying the scope of what the gron1
                                                                            might address within that subject.
Finally, the session discussed issues relating to software prod-
uet description languages. Important issues included:            It was accepted that the process for COTS component based systen
                                                                 development was different from that which had been applied for thq
• recognise the strong inter-dependencies between software development of bespoke systems and that there was a need to formali~
     product characteristics that provide a basis for multi-     that process. It was also agreed that the development process uncle
     criteria decision making (MCDM).                            consideration would only be applicable at the 'system level' and wouh
• q~_mntitativeand qualitative decision-making;                  not apply to the actual development of the COTS components them
• make qualitative (e.g., requirements) and quantitative         selves. It was felt that this was a task carried out by component ven
     (e.g., through measurable fit criteria)                     dors, and at that level there are already well-established soflwar,
                                                                 development processes in place.
ACM SIGSOFT                                     Software Engineering Notes vol 26 no 1                            January 2001 Page 67

It was agreed that, although component selection was a sig-      required due to obsolescence. If the system was relatively small and
niiieant part in the development process, it was also the sub-   was not expected to evolve or be supported over any extended period,
ject of one of the other groups and so would not be addressed    it was felt the framework would be of less importance than the selec-
in any detail.                                                   tion of the components. This subject was taken further during the later,
                                                                 more detailed discussions on the identification of possible solutions to
From the experiences of the group members it was clear that
                                                                 meet the system requirements and con_straints.
there was a range of systems which needed to be considered,
and it was accepted that what might be appropriate at one end The discussion on the basic stages for the process started with re-
of the spectrum might be overkill at the other. This resulted in quirements and the need to identify the appropriate stakeholders for the
the conclusion that it was necessary to identify a basic process system and to get their 'buy in' to what was being undertaken as the
that could be specified as a set of activities or stages that first stage. This was then followed by the identification of possi-
might be elaborated to a greater level of detail when dealing ble/feasible candidate solutions using the requirements and any con-
with the more complex systems.                                   swaints either from the first stage, as part of the requirement, or as a
                                                                 result of looking at the various options. An example of this might be
At this point there was also some discussion on the relation-
                                                                 that the selection of one possible component as a part of the solution
ship between the initial development and the sup-
                                                                 might imply the use of a second compatible product and prohibit the
port/maintenance cycle. The consensus was that this should be
                                                                 use of other alternatives. The main objective of this stage would be the
looked at once some form of initial process had been identi-
                                                                 consideration of the technical options and how best the system re-
fied. There was also an extended discussion on the role of
                                                                 qnirements might be achieved from that point of view. This would then
system framework or architecture to provide for the independ-
                                                                 be followed by some form of prototyping to achieve actual measures of
ence and cooperation of the components and on where this
                                                                 acceptability agnin~ the overall requirements. If there were a 'perfect
came into the development process. The feeling was that it
might be of considerable importance in a larger more complex match' with a selected set of components, which meet all the criteria,
                                                                 then it would be a simple step to implement the system by integrating
system where it might be necessary to extend the system use
                                                                 the parts and carrying out the necessary system acceptance activity.
over time and to replace components within that framework as
                                                   Requirements/                                   1
                                                   Constraints   q                          I
                                                                                        Produce set of feasible
                  Project Management,                                                    System Designs
                  Configuration Control
                                                                                         Analyse System
                                                                                         Negotiate Trade off

                                                                             ~    - ~ a t Technical System level
                                                                                         Prototype selution

                  Implementation/     Presentation of Alternative /
                   Acceptance/    4 - Options for Decision Maldng
                     Fielding         Process at Business level
                                              Basic DevelopmentProcess Representation
If there were two or three possible technical solutions or con-     The discussion then identified the common threads, which would be
sideratiorts of a non-technical nature, then the next stage         needed throughout all of the activities, including those associated with
would be some form of business level assessment of the op-          the project management and control of such a development approach.
tious. This would look at factors outside the main technical        While it was accepted that the appropriate levels of configuration man-
areas such as the 'Wade off' of cost against functionality or the   agement and version control would be needed, there were some addi-
need to be 'in line' with the company strategy. If no accept-       tional factors that had never been present in the development of
able solution was achieved for whatever reason at this point,       bespoke systems, those of licenses and warranties. It was agreed that
then the initial requirements will need to be reconsidered and      the concerns in the areas configuration and version control should be
the process repeated. If there is an acceptable solution, then      addressed at the system level with the collection of information on the
the f i l l stage for the development would be the implementa-      component relationships done within a system context rather than at
tion and acceptance followed by the fielding of the system.         the component level. This subject was also covered during the sup-
                                                                    port/maintenance discussions. There was some discussion on the rela-
 ACM SIGSOFT                                     Software Engineering Notes vol 26 no 1                                     January 2001 Page 68

tionship between the vendors and the developers during the           development. It is under these circumstances that the need for elabora
whole process, and it was agreed that it was a significant fac-      tion arises. If the system is a simple one, with no requirement for lon~
tor in how well some of the component assessment could be            term support, then the selection of the components could well be con.
carried out. But, as the selection process discussion was with       sidered the main task of this stage, with little or no effort being spent h
another group, this aspect was not taken further.                    identifying the architecture/framework and the quickest most simplistic
Following this identification of the basic form of the process       approach being adopted for the integration of the components. How,
there was some discussion on the ways in which it might differ       ever, if the system is large and complex then it may be necessary t(
for the range of possible systems and the levels of elaboration      look at a wider set of issues before attempting to select the compo
that might be necessary. In the task of identifying the set of       nents. It might be appropriate to consider a system architeo
possible technical solutions there are several possible activities   ture/framework as well as the interface sta.d~rds to be preferred. Als~
that need to be carried out, the most obvious being the selec-       it might be appropriate to give some consideration of the approach tc
tion of the appropriate components. Whether that is the first        be adopted for the glue code software as shown in the figure below
activity is not at all clear and could, to a large extent, be de-    which might comprise legacy parts as well as COTS products ant
                                                                     some new custom items.
pendent on the type, size and complexity of the system under

                              VuUy                                     New Development
                                                                           ~               Newly developed component
              Legacy com

               Architecture/framework                                           Added/Modified

                              mq    Glue code                                         C O T S component

                               Illustration of the Use of the Glue Code with the Different Components

It might also be appropriate to consider how system complex-                  right decisions during the initial development. It was also fel~
ity will vary depending on how well the various parts of the                  that during this stage other socio/economic factors associate¢
system fit together and the requirements imposed by the pos-                  with the technical options would be considered. These mighl
sible need to support the system over some extended period.                   relate to the selection of the human factors related issues or t(
It was agreed that the aspects covered by the choice of archi-                previous system compatibility, but whatever the source the3
tecture/framework, the interface standards, and the glue code                 would act as constraints on the possible solution space.
were the keys to the long term support of any system and that
through life costs would be very dependent on making the
                                                     Selection of A rch./Framework
                                                     Identification of Glue Code Standard
                                                     Identification of Interface Standards 4 - - - -

                                  i ...........
                   Derived Constraints
                                                                          I                         Im p a c t Analysis,
                   S/W, H/W, Socio/economic                                                         C o m b i n a t o r y issues
                   Human Factors
                                  I                       C o m p o n e n t Set Selection         ..................... ]

                             Elaborated Version of the Produce Set of Feasible System Designs Activity

The final activity considered under this stage of the process                 Again the level of effort needed to address this might be de.
was the consideration of impact and combinatory issues.                       pendent .on the type of system under development but for the
ACM SIGSOFT                                    Software Engineering Notes vol 26 no 1                            January 2001 Page 69

larger more complex ones this would be an essential part of               replacement component considered for integration nmst be as-
the support process if not the initial development. Any new or            sessed    for    its    impact    on     the    other    existing
components, the ripple effect. Is the new component larger,        The group then went back to the question of whether the process for
slower, faster? These are all characteristics that might impact    development and that for support for a system were the same. This
the final system capability and must be checked in the system      went over a lot of the ground again in terms of the various activities
context to demonstrate the 'usability' of the new component        that were needed as part of the support process. The general conclusion
and its relationship with existing components. Many products       was that while there were basically the same stages, in the support pro-
satisfactorily pass stand-alone tests but fail the performance     cess the constraints were different. The most obvious one is that there
requirements when integrated as part of a larger system.           was a system in existence with the various versions of the component,
Worse still, unknown side effects caused by interdependencies      which had been originally selected in place but which was being 'up-
could bring the system to a stop.                                  graded' for some reason or other. Therefore there were components
                                                                   and a framework that were functioning and for which any new changes
Two specific requirements were identified that might have a
                                                                   made should not degrade current capability or performance. This was
significant impact on the amount of elaboration required in
                                                                   considered to be of significance because, for any production line sys-
certain areas of the process if they were stated as being neces-
                                                                   tem, there was a problem of down time caused by the necessity for
sary. These were safety and security, but due to the limited
                                                                   testing or acceptance.          This    would     entail    some    loss
time it was not felt appropriate to explore these further during
this workshop.
of production and loss of revenue that would need to be mini-            and that if the framework had been selected satisfactorily then
mised. This raised the question of whether it was necessary to           integrating the new component would not be too onerous a task.
have a 'back up' system on which to carry out the testing re-            However, it was felt that with the possible divergence of a prod-
quired during the upgrading of the system. While this was                uct being used from the needs of the system over time, there
considered important, it was not taken further during the dis-           might be a more complex set of considerations to undertake as
cussions.                                                                part of the upgrade; more akin to the original development proc-
                                                                         ess but with many more constraints giving a much reduced so-
The next point made was that as there was already a system,
                                                                         lution space.
the second stage was much reduced in that it was basically the
identification of the component or components to be changed


            Illustration of the Reduced Solution Space as the Selection Criteria and Constraints Become More Precise

The conclusion reached on the support process was that while to consider these as changes to the level of elaboration or detail at each
the basic stages for support were very similar to those for de- stage. But it was felt that the area still warranted further study and in-
velopment there were differences and that it might be possible vestigation.
ACM SIGSOFT                                        Software Engineering Notes vol 26 no 1                                                         January 2001 Page 70

The final area of discussion arose from the input of some of             essential to keep details of interfaces and component dependencies so
the latest SEI thoughts on how the process would function and            that during the maintenance some form of software agent could be
the way in which it might be used 'for real'by any large proj-           used to identify the impact of selecting any new component. There
ect. If the system was large and complex then it was essential           would be other similar occasions when it might be possible to provide
to identify interface standards and retain all of the relation-          automated checking or high lighting of conflicts thereby reducing the
ships between the components in order to provide the infor-              chances of in~oducing errors. Another factor considered significant
mation required to allow through life support to take place.             was to identify and capture cost information at as early a stage as pos-
This leads to the need for some sort of project database to hold         sible, perhaps even 3~inst the requirement but certainly against the
all of this information. Consideration was given to the sorts of         component such as the various license fees etc.
information that could be held, why it would be needed and
                                                                         As a final conclusion the group thought that progress had been made
how it might be used. Due to the limited time available it was
                                                                         and that while no final definitive process model had been generated
accepted that this was not something that could be addressed
                                                                         there was now a much better understanding of what was required and
to any level during this workshop but that there were several
                                                                         how some of it fitted together. It was hoped that the activity now
aspects of it that might be worth following up at some other
                                                                         started would continue and that people would follow up on what had
opportunity, While it was obvious that such things as the re-
                                                                         been achieved in this very important area.
quirements and constraints should be captured it would also be

                                                                                                     ;       Framework/Interface t
                                                        Specify                                  /           Glue Code
                    . . . . . . . . .               "J~ Req~drements/                        :
                                                          Constraints          ~t - ,
                                                                               - - ' -%/' l , ,              Componmt Selection Task
                                                                                                             Socio/economlc issues
                                                                                                             ImpacffComHnatory issues
                          Group Inputs                                              / .
                                                                                :        !                                .._.-.--"

                                                                                    Produce set of feasilte
                         Project Management, ~ I [ ~               . ~               System Desigm          4•m                       ¢   m

                         Configm'ation C o n t r o i ~ O j e c t          I          -~----                                                   i
                                                                                                               V                i
                                                         I                                                  Analyse System

                  Maintenance                            I                                                  Negotiate Trade off
                  l~OOeSs cycle                          .     .    .      .                             l~ at Technical System level
                   I                                     I                           I                      Prototype solution
                   !                                     I                           I                          I
                   I                                  Presentation of Alternative         I
                   I     Implementation/
                       -   Acceptancd                  Options for Decision Making 4t , -
                                                       Process at Business levd         Process F l o w  - ' -~
                                                                                        Information Flow       ~-

                   A Possible Full Process Model for the Development and Support of COTS Component System

Breakout 4: Integration, Maintenance and System                                 a) security
Management                                                                         • confidentiality
   Leigh Davis (University of Tulsa, USA)                                                •           identification/authentication
   Jerry Gao (San Jose State University, USA)                                            •           authorization
   George Heineman (WPI, USA)                                                   b) perform~n~
   Alok Mehta (AFS, USA)                                                        c) platform(e.g.,which operatingsystem, which chip)
   Robert Seacord (Software Engineering Institute, CMU,                         d) technologies (e.g., Java, CORBA, DCOM)
   USA)                                                                  If COTS software was specified using these non-functional properties
   Mark Vigder (National Research Council Cavsda)                        (provided by the vendor, then a third party could independently verify
                                                                         these properties.
We started with a focus on non-functional properties of COTS
software                                                     The trouble is that non-functional properties are often dependent on
                                                             fimctionalproperties. Industry lacks a performance model for COTS
ACM        SIGSOFT                             Software Engineering Notes vol 26 no i                           January 2001 Page 71

software, and such a model would be helpful to predict per-         c)  Rapid prototyping integration points, therefore, are suggested
formance of C O T S software.                                           as a means to (1) verify the ability to integrate the COTS
                                                                        software at all; and (2) attempt to measure what the overall
When ratingthe properties for COTS software,an application
                                                                        costs will be.
builder can make tmdeoffs. How is this to be reflected in the
                                                                4. How can we mitigate the risk of maintenance costs?
properties for a product?
                                                                    a) You must never forget that upgrading is driven by the COTS
1. How can we detect in advance a conflict with using a par-
ticular COTS software system? It should be possible to iden-
                                                                    b) Could the industry select a consistent monitor-
tify candidates that can be disqualified. If there were a
                                                                        ing/instmmentation standard to make it possible to track the
standard mechanism for specifications (and this would be a
                                                                        use of COTS software within a system and determine where
big cultural change)...
                                                                        defects are occurring?
     a) How detailed would these specifications be? (shallow            • The only way this would work is if the "value added" of
         vs. deep)                                                           the standard mechanism were accepted by huge numbers
     b) CoUld we develop algorithms to search potential                      of vendors.
          COTS software being considered for use and detect     5. How do we do system management of COTS systems?
         known problems?
                                                                    a) Configuration Management is addressed at the source code
          •   The collection of specifications does not need to
              be centralized in a database.
                                                                    b) Version checking at the COTS soRware level (i.e., this ver-
          • It canbe incrementally updated.
                                                                        sion of COTS system A works with any version number 2.5
          • The specifications can be in heterogeneous lan-             and higher for C O T S product B)
              guages if we have sophisticated searching algo-       c) There is a constantthreatthatan upgrade of C O T S software
              rithms.                                                   will alteritsinterface.
2. How can we determine IN ADVANCE integration costs?                                              to
                                                                    (I) There isa lack of utilities manage such C O T S systems.
                                                                        H o w can multiple in~allafionsbe maintained? H o w can we
      a) What are the problems with integration?                        mange/control the system without directuser intervention
         • Mismatch in interfaces (detected by compiler)                (i.e.,automatic scripts)?H o w do we deal with licenseexpira-
         • Inserting COTS software that can break other                 tion and re-installations?
             parts of the system (i.e., the COTS software is        e) There will always be a tmdeoff between upgrading to include
             not at fault)                                              the new version of COTS software or keeping and dealing
         • Architectural mismatch (i.e., Ocker-                         with an older version that is known to work. Try to develop
             bloom/Ccarlan)                                             business model for selecting technology - only upgrade
         • Tremendous glue code needed (no reusable                     COTS soRware if benefits exceed costs.
             wrappers)                                          6. Does Open Source COTS Software change anything?
         • Costs that depend on the system into which
             COTS software is being injected.                       a)   There are options available to the integrators, and it's easier to
      b) This problem defies simplification.                             find defects.
                                                                    b) We must avoid the temptation to modify the code and
         • No standard means of composing COTS soft-
                                                                         thereby create a long-term internal maimenance cost.
                                                                7. Are there success stories7
    c) Does the use of C O T S lower integrationcosts?
        • W e agreed thatusing C O T S will increaseinte-           a) Flight Simulation software for F18 aircraft
             grationcosts, and this goes against the idea of        b) Domain-Specific Software Architectures
             using COTS to reduce overall lifecycle costs.     8. Recommendations
    d) Configurable C O T S software will increase integra-
                                                                    a) Databases and compilers have benchmarks. Can one be de-
        tion costs.
                                                                        veloped to help COTS?
    e) Since COTS vendors focus on feature differentiation,
                                                                    b) Work with Underwriters Laboratories (UL, Inc.) to find how
        often COTS soRware is chosen because of a particu-
                                                                        to develop data sheets for COTS software
        lar nifty feature. Instead, an organization should se-
        lect a COTS software feature based on "the risk to the
                                                               Breakout 5:Business Models
        system if the feature is not available."
3. How can we lower the integration costs?                         Aria Alvarez (European Software Institute,Spain)
                                                                   Michael Guntersdoffer (Universityof California,Irvine,USA)
    a) Organizations should build expertise in particular          Rouald Kohl (AverStar,Inc.,USA)
        black-box integration methods. This more than any-         Maurizio Morisio (Universityof Maryland, USA)
        thing will help lower the costs of integration by man-     JeffreyVoas (ReliableSoftware Technologies, USA)
        dating the use of tried methods.
    b) Since the integration of COTS software is often a
        failure point, the evaluation process for selecting    The purpose of the breakout session on Business Models was to:
        COTS software should include a prototype integra-      • identify today's problems of COTS software components and how
        tion effort into a toy system.                              they relate to their weak business model;
 ACM SIGSOFT                                  Software Engineering Notes vol 26 no 1                            JanuarY 2001 Page 72

•    discuss how those problems are currently mitigated;               their system even though the component is technically sound.
•    identify shortcomings in those mitigation strategies in      12) In case of the occurrence of problems with a component, often
     order to identify the areas with need for improvement;            another problem occurs: Who is to blame? For example, consider
• determine where improvement could be offered by the                  a system that is composed of components from more than one
     sofi~are engineering community; and                               vendor. In the case of interaction problems between two compo-
• identify the corresponding research areas and topics.                nents bought from different vendors, it is likely that both vendors
The breakout was carried out in two sessions. The first session        will deny responsibility and point fingers at each other.
was used to identify both essential and accidental problems of    The different types of parties affected by these problems were identi-
COTS components and their business model, as defined by           fied as
Brooks. Groups of people (parties) that are affected by these     v.      The procurement personnel of the purchaser (buyer side).
problems (including the linkage of individual problems to in-     vi.     The user of the system, which integrates the component
dividual parties), and how mitigation is carried out today in             (buyer side).
each case were identified. The second session was used to         vii.    The system integrators (buyer side).
identify the problems where mitigation is insufficient and        viii.   The publisher (seller side).
where software engineers could come to the rescue. Finally,       ix.     The (system) maintainers (buyer side).
corresponding research topics were identified.                    x.      The COTS analyst (independent, third party).
                                                                  The problems (or risks) were assigned as relevant to each party as fol-
Session I                                                         lows:
The following typical problems/risks of COTS components
and their application were identified:                                c) 2,5,8,10,12.
                                                                      d) 1-10,12.
1) A component reveals too much functionality, i.e., the              e) 1-12.
    component exceeds the specification and offers additional         f) 2 , 4 , 6 , 8 , 9 , 1 1 , 1 2 .
    functionality in addition to the one specified or needed by       g) 1 -6, 8 - 10, 12.
    the purchaser.                                                    h) 4 & 5 .
2) A component reveals too little functionality, i.e., the com-   The following mitigation strategies of each party were identified as
    ponent cannot provide all the functionality specified or      being current "state-of-the-art" for each case:
    needed by the purchaser.
3) A component's documentation is incomplete, i.e., func-        A:
    tionality, features, or any other behavior of the component
    is not described in its documentation.                          2. Lawsuit (breach of contract), return to vendor, use alternative
4) A component's documentation is incorrect, i.e., the docu-        component.
    mentation is indeed flawed and error prone, containing          5. Understand vendor options, lawsuit (bankruptcy law).
    incorrect statements about the component's behavior.
5) The component's vendor is not viable, i.e., the seller of the    8. Understand support options, lawsuit (breach of contract, if ap-
    component is driven out of business and cannot further          plicable).
    maintain the product or support the customer. The buyer         10. Purchase alternative product, purchase additional functional-
    is therefore left on its own.                                   ity, hire personnel to write additional functionality.
6) The component's functions are unreliable, i.e., the buyer
    of the component cannot depend on the component to              12. Threat of lawsuit (to all parties), contact third party to decide
    carry out its tasks correctly.                                  (arbitration), use internal resources to decide.
7) An operational assumption mismatch occurs, i.e., the          B:
    creator and the user (both parties are software developers
    here) have different opinions on how the component               1. Write user procedures (guidelines for users, what not to use and
    functions. This is the famous architectural mismatch            how).
    coined by Garlan.                                               2. Use another product, request more (missing) functionality.
8) A component vendor decides to limit or stop supporting a
    component, i.e., even though the vendor is still in busi-       3. Document yourself, request better (complete) documentation.
    ness and capable of supporting the customers, he decides
                                                                    4. Document yourself, request better (correct) documentation.
    to dump a product, e.g., due to lack of profitability.
9) .A component turns out to be volatile, i.e., the component       5. N/A (nothing the user can do about that).
    has frequent downtimes where it doesn't function properly
                                                                    6. Modify usage procedures, report failure.
    (e.g., crashes) and cannot be used at all.
10) The system needs diverges from the component's capa-            7. Report failure, change hardware/operating system profile, if ap-
    bilities, i.e., over time the vendor's ongoing development      plicable.
    of the component results in versions with different func-
    tionality from the needs of the purchaser, whose system is      8. Complain to vendor.
     also evolving.                                                  9. Complain to vendor.
11) The composer won't integrate, i.e., the purchaser's system
     integrators are unable to integrate the component into          10. Complain to system integrator.
ACM SIGSOFT                                      Software Engineering Notes vol 26 no 1                            January 2001 Page 73

    12. Provide more data to determine culprit.                         spin-off product.
                                                                        9. Listen to customers, modify release schedules.
     1. Wrappers, firewalls, procedures for users, request              11. Use standards, redesign, test for additional configurations, of-
     product modificatiorL                                              fer on-site integration support.
     2. Request product modification, extend product via                12. Surveillance instrumentation by vendor, hire third party, com-
     wrappexs to add functionalityor via otherproducts.                 plex licenses that protect publishers, lawsuit.
     3. Same asB3.
     4. Same as B4.                                                     4. Hire a third party expert, ask around to validate documentation
                                                                        claims, in-house test lab.
     5. Find competing alternatives, plan to build in-house,
     contract to put source code into escrow.                           5. Do independent analysis of company or product.
     6. Find competing alternatives, plan to build in-house, re-   Session Ih
     quest fixes.                                                  The following software engineering research topics were derived from
                                                                   the resultsof session I:
     7. (C only) same as C6 + B7.
                                                                   I) Detecting hidden functionalityand hidden interfaces(see also:
     8. Same as B8 + subcontract support to 3rd party.
                                                                      dormant code, malicious code). Targets risksC3, C4, C11, C12,
     9. Same as B9 + refuse to upgrade.                               E3, E4, E12, and F4.
                                                                   2) How to determine thata COTS component isunreliable(creation
     10. Same as C7 + stand down on the requirements.                                                           Targets risksC6, C9,
                                                                      of oracles,creationof testinfrastructures).
     11. (C only) Return, find alternatives, request modifica-        C11, C12, E6, E9, and E12.
     tionfrom vendor, wrappers.                                    3) Waystocrcatewrappcrs. Targctsrisks CI, C2, Cll, El, andE2.
                                                                   4) Definitionof volatilitymetrics.Targets risksC9, C12, E9, and
     12. Hire third party expert, collect data via instnnnenta-       El2.
     tion.                                                         5) Instrumentationof binariesto collectinformation about existing
D:                                                                    behavior. Targets risksC3, C4, C6, C7, C9, CI0, C11, C12, E3,
                                                                      E4, E6, E9, El0, El2, and F4.
     2. Add functionality, deny, better requirements.              6) Definitionof metrics for similarityof competing products. Targets
     4. Fix, deny.                                                    risksAt0, B2, C5, C7, C9, C10, Cll, E5, E9, and El0.
                                                                   7) Definition/developmentof metrics/gauges for intcroperability.
     6. Fix, deny, improve V&V, better requirements.                  Targets risksC5, C7, C9, CI0, Cli, C12, D7, DI I,E5, E9, El0,
     8. Improve vendor support, subcontract (outsource) sup-          and E12.
     port, make explicit to customers that support is limited,

Shared By: