IV&V Facility
Real-World Software Reliability Assessment
(WVU UI#7: Sensitivity of Software Reliability to Operational Profile Errors: Architecture-Based Approach)
PI: Katerina Goseva – Popstojanova
Students: Sunil Kamavaram & Olaolu Adekunle
Lane Department of Computer Science and Electrical Engineering West Virginia University, Morgantown, WV katerina@csee.wvu.edu
What we are doing?
IV&V Facility
Anyone can see a fire
What we need are smoke detectors
But what about the sensitivity and accuracy of the alarms ?
2
Problem statement & Our goal
IV&V Facility
Traditional view: Point estimate of software reliability computed from the model using point estimates of input parameters Problem: Estimation of a trustworthy operational profile is difficult
IV&V information on operational profiles - limited, may be inaccurate Single operational profile could not be sufficient to describe the use by different users Software systems evolve - operational profile may change
Our goal: Reliability “sensitometer” that enables us to answer the question “How parameters uncertainty propagates into overall application reliability?”
Develop an architecture-based methodology for uncertainty analysis of software reliability & apply it on case studies
3
What we can do?
IV&V Facility
Forecast: Reliability
Entropy as a measure of uncertainty
10,000 Trials
.027
Frequency Chart
9,963 Displayed
270
.020
202.5
.014
135
Reliability frequency chart & distribution fitting
.007
67.5
.000 0.5000 0.6250 0.7500 0.8750 1.0000
0
Certainty is 99.63% from 0.5000 to 1.0000
Trend Chart
C
Benefits to IV&V Software reliability assessment throughout the life cycle (keeping track of the software evolution) Allocation of testing efforts Software certification
4
C
o
Execution rates & uncertainty of components
1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0
Un ce r ta in ty E xe c uti on rat e
F ta te S t1 en t2 t3
Certainty Bands - (Percentiles)
1.0000 95%
0.8750 75%
0.7500
50%
25% 0.6250
on
10% 0.5000
R e lia b ility
C
m
o
S
Certainty bands (percentiles)
on
en
en
on
m
p
m
o
p
ta
te
p
E
IV&V Facility
Architecture - based methodology for uncertainty analysis
Uninformed Approach
(maximum entropy)
Intended Approach
(historical data, UML)
Informed Approach
(component traces)
1 2 1-p12 3 1 E
R1
p12
Growth models
1-p23
R2
p23
Non-failed executions
R3
Fault injection
Uncertainty analysis
5
Methods for uncertainty analysis
IV&V Facility
Uncertainty analysis
Sensitivity studies Entropy Probability distributions Analytical Monte Carlo simulation
Method of moments Confidence intervals
Perturbation analysis
6
Choice of the method
IV&V Facility
Choose the method using the following criteria
Data requirements & ability to collect data Reliability measures Accuracy of the solution Scalability with respect to the number of components
Our goal: fill the table
Method Data requirements Reliability measures Accuracy of the solution Scalability
7
IV&V Facility
Construction of the software architecture model
1
p12
Structural phase – establishment of static software architecture Software specifications
Architectural design Parser-based or lexically based tools (SIAT tool - Titan Systems Corporation)
2
1-p12 p23
Statistical phase – estimation of the relative frequencies of component interactions, that is, transition probabilities
Uniform distribution – maximum entropy approach
3
1
1-p23
E
Historical data Software specification (e.g. UML use case & sequence diagrams) Component traces from profiles or test coverage tools (Testing tool for JSC AERCam project - Dr.Yann-Hang Lee, ASU)
8
European Space Agency case study
IV&V Facility
Almost 10.000 lines of C code The program has been extensively used after the last fault removal without failures; this gold version is used as an oracle
Component traces obtained during testing were used for constructing software architecture & estimating transition probabilities
Informed Approach
(component traces)
1 2 1-p12 3 1 E
R1
p12
R2
p23 1-p23
R3
Two faulty versions were obtained reinserting the real faults discovered during the integration testing and operational usage
Fault Injection
(real faults) 9
Parameter estimation
IV&V Facility
Two versions
Version A: faulty components 1&2, fault-free component 3 Version B: faulty components 2, fault-free components 1&3 nij Transition probabilities pij ni Version p12 p23 where nij is the number of times
control was transferred from component i to component j, and
A B
0.5933 0.7364
0.7704 0.6866
ni j nij
Component reliability Ri 1 lim
where f i is the number of failures and ni is the number of executions of component i in N randomly generated test cases
fi
R1 0.8428 1 R2 0.8346 0.8346 R3 1 1
10
ni ni
Version A B
IV&V Facility
Construction of the architecture – based software reliability model
1
p12 R1 1-R1 1-R2
2
(1-p12)R1 p23 R2
F
1-R3
3
R3
(1-p23)R2
E
1
C
11
IV&V Facility
Traditional View: Point estimates of software reliability
R 1 lim F N
N
Actual reliability of the software
where F is the number of system failures in N randomly generated test cases
Estimated reliability from the model Results
Version Actual reliability
R (1 p12 ) R1 p12 1 p23 R1R2 p12 p23 R1R2 R3
Estimated reliability 0.7601 0.8782
Error
A B
0.7393 0.8782
2.81% 0%
12
Methods for uncertainty analysis
IV&V Facility
Uncertainty analysis
Sensitivity studies Entropy Probability distributions Analytical Monte Carlo simulation
Method of moments Confidence intervals
Perturbation analysis
13
IV&V Facility
Sensitivity of software reliability to variations in operational profile
Version A reliability Version B reliability
Rmax = 0.8414 Rmin = 0.7048
Rmax = 0.9983 Rmin = 0.8363
14
Methods for uncertainty analysis
IV&V Facility
Uncertainty analysis
Sensitivity studies Entropy Probability distributions Analytical Monte Carlo simulation
Method of moments Confidence intervals
Perturbation analysis
15
Uncertainty study based on entropy
IV&V Facility
Entropy quantifies the uncertainty present in a stochastic source
H i pij log pij
where i represents the usage distribution and pij the transition probabilities Higher entropy implies an exponentially greater number of statistically typical paths Maximum entropy – all transitions that are exit arcs from each state are equiprobable
i
j
16
Uncertainty of the operational profile
IV&V Facility
Hmax = 0.5514 HHmax = 0.5514 min = 0.0404 Hmin = 0.0404
Operational profile A (H=0.4707) is more uncertain than operational profile B
(H=0.4604) Software systems that have uniform operational profile are more uncertain and thus would require more testing
17
Uncertainty of software reliability
IV&V Facility
Operational profile
Version A uncertainty
Version B uncertainty
Version A reliability
Version B reliability
Considering software failure behavior increases the uncertainty for both versions compared to the uncertainty due to operational profile Version B, which is more reliable, is less uncertain than version A
18
IV&V Facility
Uncertainty of components for the operational profile
i ij ij
Uncertainty of component i is estimated using the conditional entropy H p log p
j
Version A
1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0
1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0
Version B
t3
on
en
on
en
U n c e r ti n ty E x e c u ti o n ra te
en
t3
U n c e r ta in ty E x e c u ti o n ra te
t1
t1 en
en
t2
t2
en
on
on
E
p
p
m
on
on
m
te
p
p
m
ta
m
o
C
C
p
p
m
S
C
C
o
Uncertainty of component i will be higher if it transfers the control to more components and the transition probabilities are equiprobable
19
C
C
o
m
o
S
o
ta
o
te
E
IV&V Facility
Uncertainty of components for the software reliability model
Version A
1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0
Version B
c
on
on
en
on
on
en
U n c e r ta in ty E x e c u ti o n ra te
en
en
E
E
U n c e r ta in ty E x e c u ti o n ra te
t1
t1 en
en
t2
t2
t3
t3
F
te
on
m
on
m
te
p
p
te
ta
m
m
o
o
p
p
ta
ta
S
C
C
m
m
o
o
S
C
C
o
Uncertainty of component 1 version B remains the same because R1 1 For all other components uncertainty increases due to R i 1 Components that have higher expected execution rate, higher component uncertainty, and moderate reliability should be allocated more testing effort
20
C
C
o
S
ta
S
te
p
p
F
Methods for uncertainty analysis
IV&V Facility
Uncertainty analysis
Sensitivity studies Entropy Probability distributions Analytical Monte Carlo simulation
Method of moments Confidence intervals
Perturbation analysis
21
IV&V Facility
Uncertainty study based on the method of moments
Method of moments involves the following steps
1.Obtain the expression for the system reliability using the architecture-based software reliability model 2.Expand the expression for system reliability using Taylor series 3.Determine the moments of the components reliabilities 4.Estimate the mean and the variance of the system reliability using the parameter moments and Taylor series coefficients
22
First order Taylor series
IV&V Facility
First order Taylor series expansion
R a0
ai ( Ri i )
i 1
R ( , , ) 1 2 n
2 2
n
where i ERi is the mean component reliability, and
R a o f ( , , ); a 1 2 n i R i
Mean reliability is ER a0 Variance of the reliability is
n 2 R i 1
ai i
where i2 VarRi is the variance of the component reliability
23
Second order Taylor series
IV&V Facility
Second order Taylor series expansion
R a0
ai Ri i aii Ri i
i 1
n
1 n
2
2 i 1
aij Ri i R j j
i 1 j 1
R ( 1 , 2 ,, n )
n i 1
where a0 f ( 1 , 2 , , n ), ai
R R i
2R 2R , aii and aij R 2 R R R ( 1 , 2 ,, n ) i R ( 1 , 2 ,, n ) i j
Mean reliability is
E R a0
1 2
aii i
i 1
n
2
Variance of the reliability is
2
a a R
2 i 2 i 2 ij 2 i 2 j i 1 i 1 j 1
n
n
i 1
1 4
a E Ri i ai aii E Ri i
2 ii 4 3 i 1 i 1
n
n
a 4
1
n 2 ii i 1
2 2 i
24
IV&V Facility
Method of moments for the case study
First order Taylor series Mean reliability 0.7601 0.0825 0.0068 0.8782 0.0589 0.0035 Second order Taylor series 0.7601 0.0825 0.0068 0.8782 0.0589 0.0035
Version A
Standard deviation Variance Mean reliability
Version B
Standard deviation Variance
1 0.95 0.9
Second order approximation does not improve accuracy significantly Version B is more reliable with less variance of the reliability
Version A Version B
Reliability
0.85 0.8 0.75 0.7 0.65 0.6
25
Methods for uncertainty analysis
IV&V Facility
Uncertainty analysis
Sensitivity studies Entropy Probability distributions Analytical Monte Carlo simulation
Method of moments Confidence intervals
Perturbation analysis
26
IV&V Facility
Uncertainty study based on Monte Carlo simulation
Monte Carlo simulation involves the following steps
1. Obtain the expression for the system reliability using the architecture-based software reliability model 2. Assign probability distributions to the transition probabilities and components reliabilities 3. Sample the distributions 4. Compute the reliability of the system using the sampled values 5. Repeat steps 3&4 until the desired number of values of system reliability has been generated 6. Calculate the moments, frequency chart and percentiles for the system reliability, do the distribution fitting
27
IV&V Facility
Variation of the operational profile: Frequency chart and distribution fitting
Forecast: Reliability Frequency Chart 9,958 Displayed
154
10,000 Trials
.015
Overlay Chart Distribution Fitting
.021
.012
115.5
.016
.008
77
Weibull Distribution Loc. = 0.7021 Scale = 0.0648 Shape = 3.00
.011
.004
38.5
.005 Reliability
.000
.000 0.7060 0.7332 0.7603 0.7874 0.8146 0
0.7053
0.7326
0.7600
0.7873
0.8146
Mean Standard deviation (Spread of the distribution) Variance (Spread of the distribution) Skewness (Lean of the distribution) Kurtosis (Peakedness of the distribution)
0.7600 0.0210 0.0004 0.2072 2.6047 28
IV&V Facility
Variation of the operational profile: Percentiles
Trend Chart Certainty Bands - (Percentiles)
0.8500 95%
0.8125 75%
0.7750
50%
95%
75%
25% 0.7375 10% 0.7000
95% certainty band shows the range of values in which reliability has 95% chance of falling
29
Convergence of the mean
IV&V Facility
0.7650
Mean reliability =0.7600
Mean Reliability
0.7625 0.7600 0.7575 0.7550 1 1017 2033 3049 4065 5081 6097 7113 8129 9145 Number of Iterations
The estimation of the mean reliability converges after around 3000 iterations
30
IV&V Facility
Variation of the operational profile: Sensitivity measured by contribution to variance
Sensitiv ity Chart Target Forecast: Reliability
P1E P12 P3E P23 60.6% 39.4% 0.0% 0.0%
100%
50%
0%
50%
100%
Measured by Contribution to Variance
Reliability is more sensitive to p1E; the variance is positive Reliability is also sensitive to p12; the variance is negative
31
IV&V Facility
Variation of the operational profile and component reliabilities: Frequency charts
Version A
Forecast: Reliability
Version B
Forecast: Reliability 9,953 Displayed
261
10,000 Trials
.026
Frequency Chart
10,000 Trials
.037
Frequency Chart
9,997 Displayed
368
.020
195.7
.028
276
.013
130.5
.018
184
.007
65.25
.009
92
.000 0.5000 0.6250 0.7500 0.8750 1.0000
0
.000 0.5000 0.6250 0.7500 0.8750 1.0000
0
Version A Mean Standard deviation (Spread of the distribution) Variance (Spread of the distribution) Coefficient of variation (Relative measure of spread) Skewness (Lean of the distribution) Kurtosis (Peakedness of the distribution) 0.7589 0.0860 0.0074 0.1493 -0.5190 3.1367
Version B 0.8780 0.0660 0.0044 0.0752 -0.9646 4.2254 32
IV&V Facility
Variation of the operational profile and component reliabilities: Distribution fitting & percentiles
Version A
Ov erlay Chart Distribution Fitting
Version B
Ov erlay Chart Distribution Fitting
.037
.026
.020
Beta Distribution Alpha = 17.5014 Beta = 5.3662 Scale = 0.9916
.028
Beta Distribution Alpha = 20.1525 Beta = 2.7208 Scale = 0.9965
.013
.018
.007 Reliability
.009 Reliability
.000 0.5000 0.6250 0.7500 0.8750 1.0000
.000 0.5000 0.6250 0.7500 0.8750 1.0000
Trend Chart Certainty Bands - (Percentiles)
1.0000 95% 1.0000
Trend Chart Certainty Bands - (Percentiles)
95%
0.8750 75%
0.8750 75%
0.7500
50%
0.7500
50%
25% 0.6250 0.6250 10% 0.5000 0.5000
25%
10%
33
Making a choice
IV&V Facility
Method Sensitivity
Data requirements
Point estimates
Reliability measures
Sensitivity of the point estimate NA Moments
Accuracy of the solution
Exact analytical solution
Scalability
Large systems
Entropy Method of moments Monte Carlo simulation
Point estimates Moments of the parameters
Exact analytical solution Approximate solution: accuracy may be increased by higher order Taylor series Approximate solution: accuracy may be increased by increasing the sample size Sampling errors may be involved in case of long tail distributions
Large systems Small to medium systems
Distribution functions of the parameters Generation of random numbers
Distribution
Moments
Large systems
34
Accomplishments
IV&V Facility
Architecture-based methodology for uncertainty analysis of software reliability was developed Four different methods already developed These methods were illustrated on the European Space Agency software
35
Future work
IV&V Facility
Develop other methods for uncertainty analysis Complete “Make a choice” table Apply & validate all methods using NASA case studies
SIAT tool - Titan Systems Corporation Testing tool for JSC AERCam project - Dr.Yann-Hang Lee, ASU
36