COCOTS Risk Analyzer and
Process Usage
Ye Yang, Barry Boehm
Center for Software Engineering
University of Southern California
Annual Research Review
Mar. 14th, 2006
3/14/2006 USC-CSE 1
Outline
• Motivation
• COCOTS Model
• COCOTS Risk Analyzer
• Evaluation
• Process Usage: Risk-Based Prioritization
• Conclusions
3/14/2006 USC-CSE 2
Motivation
• Enable COTS integration risk analysis with
COCOTS cost estimation inputs
• Identify relative risk levels of COTS-based
development (CBD)
• Provide recommendations to improve risk
management practices
3/14/2006 USC-CSE 3
COCOTS Model
- Calibrated to 20 industry projects
3/14/2006 USC-CSE 4
COCOTS Glue Code Sub-model
Cost Name Definition
Factors
Size Driver Glue Code Size The total amount of COTS glue code developed for the
system.
Scale AAREN Application Architectural Engineering
Factor
ACIEP COTS Integrator Experience with Product
Effort ACIPC COTS Integrator Personnel Capability
Multiplier AXCIP Integrator Experience with COTS Integration Processes
APCON Integrator Personnel Continuity
ACPMT COTS Product Maturity
ACSEW COTS Supplier Product Extension Willingness
APCPX COTS Product Interface Complexity
ACPPS COTS Supplier Product Support
ACPTD COTS Supplier Provided Training and Documentation
ACREL Constraints on Application System/Subsystem Reliability
AACPX Application Interface Complexity
ACPER Constraints on COTS Technical Performance
ASPRT Application System Portability
3/14/2006 USC-CSE 5
COCOTS Risk Analyzer
In p u t ( C o s t U ser O u tp u t
F a c to r R a tin g s ) ( R is k S u m m a r y )
5
6 . P r o v id e R is k
M it ig a t io n
1 . Id e n t if y r is k s M it ig a t io n
S tra te g y
o f r a t in g A d v ic e s
c o m b in a t io n s R is k R u le s
n o w le g g
KKn o w le d d e e
Base
R is k R u le s
B ase 4
5. A ssess
O v e r a ll R is k
R is k L e v e l
Schem e
2 . E v a lu a t e R is k
P r o b a b ilit y
3 . A n a ly z e R is k
S e v e r it y
3/14/2006 USC-CSE 6
Knowledge Base
• Contents
– Risk Rules (RR)
– Risk level scheme
– Common risk mitigation strategy
• Constructing approach
– Expert Delphi Survey
– Empirical study results
– Literature review
3/14/2006 USC-CSE 7
Risk Rule
• A CBD risk situation
– a combination of two cost attributes at their
extreme ratings
• Risk Rule (RR)
– An identified risk situation is formulated as a risk
rule. E.g. one example RR:
IF ((COTS Product Complexity > Nominal)
AND (Integrator’s Experience on COTS Product =50% 40% 20%
SIZE (Percentage of responses over total)
AAREN
ACIEP Total # of Delphi responses: 5
ACIPC
AXCIP # of % of # of risk
APCON responses responses situations
ACPMT >=3 >50% 24
ACSEW
2 40% 26
APCPX
ACPPS 1 20% 28
ACPTD
ACREL
AACPX
ACPER
ASPRT 24 Risk Rules
formulated in the
ACSEW
APCON
AAREN
ACPMT
AACPX
ACPER
ACREL
ACPTD
APCPX
ACPPS
ASPRT
knowledge base
ACIPC
AXCIP
ACIEP
SIZE
3/14/2006 USC-CSE 9
Risk Potential Rating for Cost Factors
Mapping between cost factor’s rating to
its risk potential rating:
Cost Factors Cost Factor Rating Risk Probability Rating
AAREN, ACIEP, Very Low Worst Case
ACIPC, AXCIP, Low Risk Prone
APCON, ACPMT, Nominal Moderate
ACSEW, ACPPS, High OK
ACPTD Very High OK
Very Low OK
Low OK
APCPX, ACREL, Nominal Moderate
AACPX, ACPER, High Risk Prone
ASPRT Very High Worst Case
3/14/2006 USC-CSE 10
Risk Level Scheme
Assignment of risk probability levels:
Attribute 1
Worst Case Risk Prone Moderate OK
Worst Case Severe Significant General
Attribute 2 Risk Prone Significant General
Moderate General
OK
Quantitative weighting scheme:
Risk level Quantifier
Severe 0.4
Significant 0.2
General 0.1
3/14/2006 USC-CSE 11
Productivity Range
• Reflects the cost consequence ACIPC 2.58
of risk occurring APCON 2.51
• Combines both expert judgment
ACPMT 2.10
and industry data calibration
AAREN 2.09
APCPX 1.80
ACIEP 1.79
Cost Factor
AACPX 1.69
ACPPS 1.48
ACREL 1.48
ACPTD 1.43
AXCIP 1.42
ACPER 1.22
ACSEW 1.22
ASPRT 1.14
0.00 0.50 1.00 1.50 2.00 2.50 3.00
Productivity Range
3/14/2006 USC-CSE 12
Project Risk Quantification
• Project Overall Risk:
– Riskprobij corresponds to the nonlinear relative probability of the risk occurring
– The product of PRi and PR j represents the cost consequence of the risk occurring
• Risk interpretation:
– Normalized scale: 0 ~ 100
– 100 represents the situation where each cost factor is
rated at its most expensive extremity
– 0 ~ 5: low risk; 5 ~ 15: medium risk; 15 ~ 50: high risk; 50
~ 100: very high risk
3/14/2006 USC-CSE 13
Risk Mitigation Recommendations
• Knowledge base built on previous empirical
study results, e.g.:
Risk Rule Risk Situation Mitigation Advice
APCPX_ACIPC Complex integration with inexperienced Consider more compatible
(High, Very Low) personnel COTS; re-staffing; training;
consultant mentoring
ACREL_ACPMT High-reliability application dependent on Consider more mature
(High, Low) immature COTS COTS; reliability-enhancing
COTS wrappers; risk-based
testing
ACPER_AAREN Unvalidated architecture with COTS Benchmark current and
(High, Very Low) performance shortfalls alternative COTS choices;
reassess performance
requirements vs.
achievables
3/14/2006 USC-CSE 14
Evaluation Results
45 50
40 45
y = 0.6749x - 2.3975 40
35 y = 45.75x + 0.6143
R2 = 0.8948
35 R2 = 0.6283
30
Analyzed Risks
Analyzed Risk
30
25
25
20
20
15
15
10
10
5 5
0 0
0 10 20 30 40 50 60 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Reported Risks Reported Prob.(Risk)
Data: 9 USC e-services projects Data: 7 COCOTS calibration projects
USC e-services Industry
Web-based campus-wide Generally large scale
e-services applications comminication, control
Domain such as library services systems
# COTS 1 ~ 6 1 ~ 53
Duration 24 weeks 1 ~ 56 months
Effort 6 person by 24 weeks 1 ~ 1411 person-month
Size 0.2 ~ 10 KSLOC 0.1 ~ 390 KSLOC
3/14/2006 USC-CSE 15
Process Usage – An Example
• COTS A and B are our strongest COTS
choices
– But there is some chance that they have
incompatible HCI’s
– Probability of loss P(L)
• COTS C is almost as good as B, and it is
compatible with A
3/14/2006 USC-CSE 16
Risk-Driven CBD Process Framework
S ta rt
C
P 1 : I d e n t i f y O b j e c t i v e, P 7 : C u s to m D e v e lo p m e n t D e p lo y
C o n s tr a in ts a n d
P r io r itie s (O C & P s )
Yes No P ro c e s s
A re a
P 2 : D o R e le v a n t C O T S P 6 : C a n a d ju s t
No D e c is io n
P r o d u c t s E x i s t? O C & P s?
/R e v i e w
N o a c c e p ta b le o r r is k y
Y e s o r U n s u re A ssess-
C O T S -B a s e d S o lu tio n A
A m ent
P3: A ssess C OTS P 5 : M u ltip le C O T S
P a r tia l C O T S s o lu tio n b e s t
C a n d id a te s c o v e r a ll O C & P s ? T T a ilo r in g
N o , C u s to m c o d e
S i n g l e F u l l- C O T S s o l u t i o n R e q u ir e d to s a tis fy G lu e -
G
s a tis fie s a ll O C & P s a ll O C & P s Code
Yes
P 8 : C o o r d in a te
C C u s to m
c u s to m c o d e a n d g lu e
P 4 : T a i l o r i n g R e q u i r e d? code
c o d e d e v e lo p m e n t
G
C P 9 : D e v e lo p C u s to m P 1 0 : D e v e lo p
N s
Y eo Code G lu e C o d e
No
T
P 1 2 : P r o d u c tiz e,
P 1 1 : T a ilo r C O T S D e p lo y
T e s t a n d T r a n s itio n
3/14/2006 USC-CSE 17
Different Risk Strategy Resulting in
Different Process
( a ) R i s k A v o i d a n c e: C hoose In te g r a te D e v e lo p
D e liv e r
C O T S C a d e q u a te COTS C COTS A, C A p p lic a tio n
( b ) R i s k T r a n s f e r: D e v e lo p
C hoose
COTS C not A p p lic a tio n, OK D e liv e r
COTS B
a d e q u a te In te g r a te A & B
D e v e lo p U s e r is k r e s e r v e
P r o b le m
A p p lic a tio n to fix p r o b le m
D e v e lo p r e s t
D e liv e r
o f a p p lic a tio n
(c ) R is k R e d u c tio n:
D e v e lo p p a r ts o f
C u s to m $ , IP
C hoose a p p lic a tio n , u s e
COTS B w ra p p e rs to
(d ) R is k A c c e p ta n c e:
in te g r a te A and B Package
D e v e lo p e r $ , IP
w ra p p e rs fo r
fu tu re u s e
3/14/2006 USC-CSE 18
Conclusions
• CBD brings a host of unique risk items
• Many risk techniques/tools require intensive user
inputs
• COCOTS Risk Analyzer provides a handy way to
automate the CBD risk analysis by leveraging on
existing knowledge and expertise in both cost
estimation and risk mgmt.
• Case study shows how it supports process decisions
following the risk based prioritization strategy
3/14/2006 USC-CSE 19
Backup Slides
3/14/2006 USC-CSE 20
Risk Potential Rating
• Captures the underlying relation between
cost attributes and the impact of their specific
ratings on project risk
– 4 Levels
• OK, Moderate, Risk Prone, and Worst Case
• Two types of treatments
– Transforming continuous Size representation into
discrete risk potential ratings
– Mapping cost driver ratings into risk potential
ratings
3/14/2006 USC-CSE 21
Risk Potential Rating for Size
Delphi Responses for Size Rating (Size in KSLOC):
Rating OK Moderate Risk Prone Worse Case
Response 1 1 2 10 50
Response 2 2 5 10 25
Response 3 1 3 10 10
Response 4 1 2 10 50
Response 5 1 2 10 50
Median 1 2 10 50
Stdev 0.447214 1.30384 0 18.5741756
3/14/2006 USC-CSE 22
Risk Based Prioritization Strategy
Risk Spiral CBD process Description
Strategy Quadrants Decision
Step Framework Step
S1 Q1 P1, P2 Identify OC&Ps, COTS/other alternatives
S2 Q2a P3 Evaluate COTS vs. OC&Ps (incl.
COCOTS)
S3 Q2a P3 Identify risks, incl. COCOTS risk analysis
S4 Q2b P3 Assess risks, resolution alternatives; If
risks manageable, go to S7
S5 Q2b, Q1 P6 Negotiate OC&P adjustments; If none
acceptable, drop COTS options (P7)
S6 Q2a P3 If OC&P adjustments successful, go to
S7; If not, go to S5
S7 Q3 P4 or P5 Execute acceptable solution
3/14/2006 USC-CSE 23