Quality Determination for Web-based Applications

Document Sample
Quality Determination for Web-based Applications Powered By Docstoc
					                                 Outlines
• Overview of MCDA
  – General definition
  – MCDM process
  – MCDA methods
• Evaluation of WBA
  – Quality Attribute Relationships
  – Aggregation by Choquet Integral
  – Implementation
  – Case study and results
                    What is MCDA?

• Aims to give the decision-maker some
  tools in order to enable him to advance
  in solving a decision problem where
  several – often contradictory-points of
  view must be taken into account.
                      What is MCDM?

• Highly structured, disciplined and formal
  approach to decision making
• evaluating the alternatives in the given set A
  against the set C of criteria
• Aggregating the individual evaluations to
  produce global evaluation
• Could be used for selection the best possible
  alternatives or for ranking the alternatives
                                 MCDM Process
Set of Alternatives               Set of Criteria



                                   C1, C2,………Cn
                            A1   x11……..………x1n
                            A2   x21……..………x2n
                            .
                            .
                            Am xn1……..………xmn


  Weights wi /
 Importance of
   Criteria


                             Aggregation Measure




                      Overall worth of an alternative Ai
                Evaluation of MCDA
                           methods
• Criteria – interdependence,
  completeness, non-linear preferences
• Weights – transparency of process, type
  of weights, meaning
• Solution finding procedure – ranking,
  option
• Project constraints – cost, time
                      Evaluation of MCDA
                                 methods
• Structure of problem solving process –
  stakeholder participant, tool for learning
  transparency, actors communication
• Data Situation
   – Type of data - qualitative or quantitative
   – Risk/uncertainties – probabilities, thresholds, fuzzy
     numbers, sensitive analysis
   – Data processing amount
   – Non-substitutability
                    Evaluation of WBA
• Quality of web site is hard to evaluate
  – Consists of multiple criteria to be measured
• Simple weighted average cannot be used to
  summaries the various quality measurements
  into a single score.
• Inability to account for dependency among the
  quality criterion.
• Tend to construct independent criteria, or
  criteria that are supposed to be so
  – Causing some bias effect in evaluation
                      WBA Evaluation
                        Approaches
Single criteria
• usability aspects(Collins, 1996; Stefani
  & Xenos, 2001; Hassan & Li, 2005),
• content and structure (Bauer & Scharl,
  2000).
• accessibility (Vigo et al., 2007)
                    WBA Evaluation
                      Approaches
Multi-criteria
• WEBQEM (Olsina et al., 1999)
• EWAM (Schubert & Selz, 1998)
• WebQual (Barnes and Vidgen, 2002)
• WAI (Miranda et al., 2006)
• FQT4Web (Davoli et al., 2005)
              ISO/IEC 9126 Evaluation
                             Process
  Stated or implied needs
       ISO 9126 & other technical info
                                                                               Requirement
                                                               Managerial      definition
              Quality requirement
                                                               requirement
  Quality     specification
Requirement
 Definition
                                  Metric      Rating level     Assessment         Preparation
             Software            Selection     definition        criteria
           Development                                          definition
                           Products
                                             Measured
                               Measurement   value
                                                                               Evaluation
                                                             Rated
                                                Rating       value           Result
                                                                             (acceptable or
                                                                             unacceptable)
                                                               Assessment
                                                                           Quality Model
                                                                                             e-commerce
                                                                                               e-learning
                                                                                             e-education
                                                                                            e-government
                                                                                                    etc.
                                                         APPLICATION
                                                            DOMAIN




                                                                                                                                                                  C H A R A C T E RI S T IC S
                                                                                                                              Q U A L IT Y
   Functionality       Reliability                Usability                Efficiency    Maintainability        Portability




                                          understandability




                                                                                                                              S U B C H A R A C T E R IS T IC S
suitability        maturity               learnability                 time behaviour   Analysability       Adaptability
                   fault tolerance        operability                  resource         changeability       installability
accuracy                                                                                                    coexistence
                   recoverability         attractiveness               utilisation      stability
interoperability                          expliciteness                efficiency                           replaceability
                   availability                                                         testability
security           degradability          customisability              compliance       manageability       portability
traceability       reliability            clarity                                       reusability         compliance
functionality      compliance             helpfulness                                   maintainability
compliance                                user-friendliness                             compliance
                                          usability compliance




                                     Indicators, scales and preferred values
            Quality Attributes for WBA

• Define software product qualities as a
  hierarchy of factors, criteria and metrics.
• Quality factor represents behavioral
  characteristics of the system
• Quality criterion is an attribute of a quality
  factor that is related to software production
  and design
• Quality metrics is a measure that captures
  some aspect of a quality criterion.
                                                           Factor A is split up into
                                Overall
                              Quality Score                three criteria a1, a2, and a3.
                                                           Criteria a1 with the weight 4
                                                           is considered four times as
                                                           important as criteria a2 and
                                                           twice as important as
                Factor A         Factor B       Factor C
                                                           criteria a3.


                                                           Similarly, we can set
                                                           different weight for each
Criteria a1,   Criteria a2,      Criteria a3,              factor    to  indicate its
 weight 4       weight 1          weight 2                 importance.
                                 Definition of Quality
                                            Attributes
Name              Description

Functionality     The capability of the Web site to provide functions and properties which meet
                  stated and implied needs when the site is used under specified conditions
Usability         The capability of the Web site to be understood, learned and liked by the user,
                  when used under specified conditions

Reliability       The capability of the Web site to maintain a specified level of performance
                  when used under specified conditions.
Efficiency        The capability of the site to provide appropriate performance, relative to the
                  amount of resource used, under stated conditions
Maintainability   The capability of the site to be modified. Modifications may include
                  corrections, improvements or adaptation of the site to changes in environments,
                  and in requirements and functional specifications
Portability       The capability of the site to be transferred from one environment to another
                                Quality Attributes
                                  Relationships
Three types of relationships
• Positive, i.e. a good value of one attribute result in a
  good value of the other (synergistic goals).
   – Relationships definitions: If characteristics A is enhanced,
     then characteristics B is likely to be enhanced (+)
• Negative, i.e. a good value of one attribute result in a
  bad value of the other (conflicting goals).
   – Relationships definitions: If characteristics A is enhanced,
     then characteristics B is likely to be degraded (-)
• Independent, i.e. the attributes do not affect each
  other.
   – Relationships definitions: If characteristics A is enhanced,
     then characteristics B is unlikely to be affected (0)
 Interrelationships between
quality factors (Perry, 1987)
Relationship Chart (Gillies,
                     1997)
                            Techniques to explore the
                                        relationships
Ref      Attributes                           Purpose                                Techniques used


         Correctness, Reliability
         Integrity, Usability                To study the relations of different     Survey -questionnaire
[8, 9]
         Efficiency, Maintainability         quality goals attribute in developing
         Testability, Flexibility            software
         Portability. Reusability
         Interoperability
         Performance                          To address the importance of
[10]
         Adaptability                         design decision made during            Case Study - Interview
         Maintainability                      software development

         Usability
[11]
         Time to market                       To increase the understanding of       Research Literature and
         Reliability, Usability               software quality attributes and        Survey –structured interview
         Correctness, Portability             their relations

[12]
         Quality attributes in 3 different    To merge different view and            Discussion (meeting and
         perspectives: management,            discuss the relationships between      offline discussion)
         developer and user perspective       the quality attributes
     Quality Attributes
Relationships for WBA
               What is Aggregation?

• method of combining several numerical
  values into a single one, so that the
  result of aggregation takes into account
  in a given manner all the individual
  values
                        Aggregation issues
•   use simple weighted average approach
•   methods are not transparent
•   assume independency
•   the choice of summarization method somehow
    should depend on the certain parameters
    – E.g. the kind of importance parameters (weights) and
      the type of dependency and interaction
• the definition of the quality factors and their
  relationships must be clearly specified
• Quasi-arithmetic means (arithmetic, geometric,
  harmonic, etc.)
   – Not stable under linear transformation and
     consider criteria as non interacting
• Median
   – Typical ordinal operator – defined the middle
     value of the ordered list
• Weighted minimum and maximum
   – Possible to increase one of the weights without
     having any change in the result
• Ordered weighted averaging operators
   – Can express vague quantifiers
                                                       23
                      Properties of an
                  aggregation operator
mathematical properties
  – Properties of extreme values
  – Idempotence
  – Continuity
  – Monotonicity
  – Commutativity
  – Decomposability
  – Stability under the same positive
    linear transformation
                       Properties of an
                   aggregation operator
behavioural properties
  – express the decisional behavior,
    interaction between criteria,
    interpretability of the parameters
    and weights on the arguments
             Aggregation by fuzzy
                          integral
• Different methods have been
  developed according to
 – type of information to be aggregated
   and
 – the properties have to be satisfied.



                                          26
Definition 1: A fuzzy measure on the set X of criteria is a
     set function  : Ƥ (X) [0,1], satisfying the following
     axioms
i.    ()=0,  (X)=1.
ii. A  B  X implies (A)  (B)

(A) represent the weight of importance of the set of
      criteria A.
Additive : if (AB) = (A) + (B); A  B=
Superadditive: if (AB)  (A) + (B); A  B=
Subadditive if (AB)  (A) + (B); A  B=
If a fuzzy measure is additive, then it suffices to define n   27
      coefficients (weights) ({ I}), … ({ n})
 Definition 2: Let  be a fuzzy measure on X.
 The choquet integral of a function
 ƒ : (X) [0,1] with respect to  is defined by
                        n
C (f(x1),…. f(xn)):=  (f(x(i)) - f(x(i-1))) (A(i) )
                       i=1

ƒ ((0)) = 0

 •Fuzzy integral model does not need to assume
 independency
 •Fuzzy integral of ƒ with respect to  gives the overall
 evaluation of an alternative                               28
                      Importance and
                 interaction of criteria
• Problem of evaluation of student with
  respect to three subjects: mathematics (M),
  Physics (P) and literature (L).
• By weighted sum (3 , 3, 2) result:




                                                29
Solved by fuzzy measure  and the choquet integral

1.   Scientific subjects are more important than
     literature;
          ({M}) =  ({P}) =0.45;  ({L}) = 0.3
2.   M and P are redundant,
          ({M, P}) = 0.5 < 0.45 + 0.45
3.   Students equally good at scientific subjects and
     literature,
          ({L, M}) = 0.9 > 0.45 + 0.3
          ({L, P}) = 0.9 > 0.45 + 0.3
4.    ()=0,  ({M, P, L})=1

                                                        30
Result by applying fuzzy measure:
* The initial ratio of weight (3, 3, 2) is kept
  unchanged




                                                  31
•  Number of coefficients grows exponentially with the
   number of criteria to be aggregated.
• 3 approaches (to reduce the number of
   coefficients)
1. Identification based on semantics
   – Importance of criteria
   – Interaction between criteria
   – Symmetric criteria
   – Veto effects
2. Identification based learning data
   – Minimization of squared error
   – Constraint satisfaction
                                                         32
3. Combining semantics and learning
                  Proposed solution

• Apply 2-additive Choquet integral
• provide the information about the
  relationships among criteria
  (redundancy or support among criteria)
  and the preference among alternatives
• Derive fuzzy measures by constraint
  satisfaction
                Explore relationships
• Techniques to explore how the different
  attributes are related to each other:
   – Experience Based Approach
   – Mathematical Modeling
   – Statistical Technique (Correlation Analysis)
• measures the strength of a linear
  relationship among different quality factors
• The main result of a correlation is called the
  correlation coefficient (r)
Correlation Result
                        Implementation of
                         Choquet Integral

1.   Definition of the initial preferences.
2.   Convert into Choquet integral form
3.   Identify threshold values.
4.   If solution exists, calculate the
     Choquet integral, Shapley value and
     Interaction indices
                             Define preference
                                    thresholds
 A partial weak order A overA (ranking of the webs),
 A partial weak order N over (ranking of the importance
                                 N
   of the quality factor),
 Quantitative intuitions about the relative importance of
   some quality factor
 A partial weak order P over the set of pairs of quality
   factor (ranking of interactions),
 Intuitions about the type and the magnitude of the
   interaction between some quality factor,
 The behavior of some quality factor as veto or favor,
 Etc.
                               Convert into Choquet
                                        integral form
                   Preferences        Choquet Integral
Ranking of         x  A x'           C  (u ( x ))  C  (u ( x ' ))   C
alternatives       x ~ A x'             C  C  (u ( x ))  C  (u ( x ' ))   C
Ranking of         i N j               (i )    ( j )   Sh
criteria           i ~N j               Sh   (i )   ( j )   Sh
(weights)
Ranking of pairs   ij  P kl          I  (ij )  I  (kl)   I
of criteria        ij ~ P kl            I  I  (ij )  I  ( kl)   I
(interactions)
Sign of some       Range of           a  I  (ij )  b , a, b   1,1
interactions       interactions
                   Complementary or   I  (ij )  m or
                   Redundant           I  (ij )  m ; m  [0,1]
                         Define preference
                                thresholds
• Three preference thresholds C, Sh & I have
  to be determined before the aggregation take
  part.
• Range of : 0 to 1
• no rule to fix the , we need to compare the
  solutions obtain with different value of .
• Once the solution exist, Choquet integral will
  be calculated
                                                Calculate the Choquet
                                                               integral
For 2-additive fuzzy measure, we have for any KX:


                               n
  ( K )   ai xi                                          a         ij   xi x j
                             i 1                         {i , j } X


with   xi  1   if iK,      xi  0                           
                                      otherwise. We deduce that                  i    ai   for

all i, and      ij  ai  a j  aij  i   j  aij .
                          Calculate the Shapley
                                          value
          n 1
vi      
         k 0
                      k          (
                          k  x \ i ,| K |  k
                                                 iK    K )
                                       n = total number of criteria,
with
                                       k = number of elements in a sub-set
     (n  k  1)!k!
k 
           n!
   Shapley index can be interpreted as a kind of average value of the
    contribution of element i, individual criteria, alone in all coalitions.

  Summation of these Shapley values for a given set of elements would
            represent the importance of the complete set
                   Calculate the Interaction
                                      Index
        n2
I ij    k            (           ijK     iK   jK   K )
        k 0   k  x \{i , j },| K |  k

With

        (n  k  2)!k!          1
 k                   
           (n  1)!       n  2
                         
                          k (n  1)
                                
                               

  The interaction index Iij can be interpreted as a kind of average value of
   the added value given by putting i and j together, all coalitions being
  considered. When Iij is positive (resp. negative), then the interaction is
                     said to be positive (resp. negative).
                                    Case Study

• Perform on 3 types of WBA
  – Academic
  – E-commerce
  – Museum
• Four quality factor were evaluated
  – Usability,Functionality, Reliability, Efficiency
• Each has different preference, importance
  and interaction
                          Result for academic
                                       website
Table 1. Academic websites data
Univ Websites Usability (U) Functionality(F) Reliability(R)   Efficiency(E)
     A1         76.18           61.84           60.40             69.09
     A2         51.01           50.39           87.62             52.03
     A3         80.08           48.49           90.86             76.11
     A4         57.71           38.99           88.70             53.12
     A5         71.93           82.04           83.05             85.99
     A6         60.94           71.12           63.61             69.47
Table 2. Relative importances for each quality attribute
   Relative          U               F                R                E
 importance
  (Weight)          0.3             0.3               0.2             0.2


 Table 3. Interaction between quality attribute
                       U                          F                        R
     U
     F                 -
     R                 +                          +
     E                   -                        -                        +



Table 4. Interaction preorder constraint matrix
                     U                        F                        R
    U
    F       1  I  (UF )   I

    R
              I  I  (UR )  1      I  I  ( FR )  1
            1  I  (UE )   I    1  I  ( FE )   I    I  I  ( RE )  1
    E

Each row corresponding to a constraint of forma  I  (ij )  b ,a, b   1,1 .
Table 5. The Möbius values
               LP            MV          Threshold
{}             0.000000      0.000000    C= 1,
{U}            0.366704      0.321697    Sh = 0.1,
{F}            0.326805      0.285376    I =0.1,
{R}            0.000001      0.094367
{E}            0.277765      0.298559
{U, F}         -0.226804     -0.100000
{U, R}         0.137865      0.100000
{U, E}         -0.100000     -0.100000
{F, R}         0.217664      0.100000
{F, E}         -0.100000     -0.100000
{R, E}         0.100000      0.100000
 Table 6. The Shapley values

               U               F               R            E
 LP            0.2722348       0.2722348       0.2277652    0.2277652
 MV            0.2716970       0.2353761       0.2443674    0.2485595


Table 7. The interaction indices for LP
                    U                      F                  R
      U
      F       -0.2268038
      R        0.1378647              0.2176637
      E       -0.1000000              -0.1000000           0.1000000


Table 8. The interaction indices for MV
                   U                       F                  R
      U
      F       -0.1000000
      R        0.1000000               0.1000000
      E       -0.1000000              -0.1000000           0.1000000
Table 9. Overall Scores
                               Global evaluation obtained by

 Univ Websites            WA                LP                   MV
      A1              67.304             67.73138              67.32480
      A2              58.350             51.26041              54.75639
      A3              71.965             72.10125              74.05597
      A4              57.374             52.36041              55.79382
      A5              79.999             81.44097              81.17426
      A6              66.234             66.63138              66.32480
                                       Summary(1)
                      LP                        MV
Möbius values         Very extreme value        Lead to unique
                      and does not              solution
                      necessarily lead to a
                      unique solution
Shapley value         Sometimes                 Mostly compatible
                      incompatible with the     with the initial relative
                      initial relative          importance
                      importance
Interaction indices   Satisfy the constraints   Satisfy the
                                                constraints
Global evaluation     Leads to more             Closer to the simple
                      dispersed values          weighted arithmetic
                                                mean
                                Summary(2)
                          Aggregation approaches
                                    Logic
             Arithmetic Weighted Scoring
               Mean Average Preference Choquet Integral
                (AM)     (WA)       (LSP)        (CI)
Importance                 /          /           /
    Veto                              /           /
   Favour                             /           /
 Interaction                                      /
   Additive       /        /          /           /
Non-additive                          /           /
                                Comparison with other
                                         approaches
                                                        Global evaluation obtained by
                                                                                   Choquet Integral Choquet Integral
 Univ Websites           AM                   WA                      LSP            (Interaction)  (No Interaction)
          Rank    Score        Rank    Score       Rank      Score          Rank   Score     Rank     Score        Rank
  A1       3       66.88        3      67.3         3         66.91          3      67.32        3    67.08         3
  A2       6       60.26        5      58.35        5         56.55          5      54.76        6    60.16         5
  A3       2       73.89        2      71.97        2         69.61          2      74.06        2    73.18         2
  A4       5       59.63        6      57.37        6         54.46          6      55.8         5    59.16         6
  A5       1       80.75        1       80          1         79.76          1      81.17        1    80.66         1
  A6       4       66.29        4      66.23        4         66.05          4      66.32        4    66.09         4
MIN-MAX          59.63-80.75          57.37-80             54.46-79.76             54.76-81.17       59.16-80.66
STD DEV          8.141                8.507                9.214                   10.250            8.133
                               Conclusion
• Aggregation by Choquet integral can be
  alternated if there is interaction exist between
  quality factors.
• The proposed approach can be applied for
  non-interactive criteria as well. If there is no
  interaction between the criteria, then the
  fuzzy measure will be additive measures.
• Results show that the global evaluation
  obtained is compatible with the weighted
  average method.
                          Future works

• The evaluation of WBA which cater the
  dynamic changes of the quality factors.
  – Behavior (Preferences,
    importance,interaction, etc.) can be
    change continuously.
• Investigate more than 2 quality attribute
  interactions

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:23
posted:6/26/2012
language:Latin
pages:55