Requirements Volatility Presentation 031010 by 9y9AHs

VIEWS: 13 PAGES: 17

									A study of the Causes of Requirements
 Volatility and its Impact on Systems
           Engineering Effort
                    COSYSMO Workshop
    Center for Software and Systems Engineering, Annual Research
                               Review
                            March 11, 2010

    Mauricio E. Peña – Ph.D. Student, USC Industrial and Systems
                       Engineering Department
  Motivation


“Requirements are the foundation of the project. They
form the basis for design, manufacture, test and
operations….changes in requirements later in the
development cycle can have a significant cost impact,
possibly resulting in cancellation.”

INCOSE Systems Engineering Handbook (2006)




                                                        2
      Objectives

• Provide and overview of requirements volatility
  and its importance to engineering projects
• Introduce the research questions & research plan
• Discuss the state of the art and implications to
  COSYSMO
• Solicit feedback:
  – On a requirements volatility causal model
  – On a requirements volatility survey



                                                     3
           Requirements Volatility Overview
Requirements volatility is the change            Number of Requirements Over Time
                                          1600
in requirements (added, deleted, and
modified) over a given time interval      1200

[MIL-STD-498, 1994, Ferreira, 2002]
                                           800
Also known as:
                                           400

Requirements creep: An increase in                                          CDR
                                            0
scope and number of system                  Dec-09      Mar-10     Jun-10         Sep-10   Jan-11

requirements
                                                  Change in Requirements Over Time
                                          500
Requirements churn: Instability in the                                             New

requirements set – requirements are       400                                      Modified
                                                                                   Deleted
                                          300
modified or re-worked without
                                          200
necessarily resulting in an increase in
                                          100
the total number of requirements
                                            0
                                                   J   F M A M          J    J    A S O N
                                                                                                    4
                                                 [Hammer et al, 1998]
         Importance of Understanding
         Requirements Volatility
• Requirements volatility has been identified by numerous research
  studies as a risk factor and cost-driver of systems engineering
  projects [Ferreira 2002, Boehm 1991]
• Requirements changes are costly, particularly in the later stages of
  the lifecycle process because the change may require rework of the
  design, verification and deployment plans [Kotonya and Sommerville, 1995]
• Requirements volatility trends are considered leading indicators of
  project performance as defined by the Lean Advancement Initiative
  Systems Engineering Leading Indicators Guide (2007)
• The Government Accountability Office (GAO) concluded in a 2004
  report on the DoD’s acquisition of software-intensive weapons
  systems that missing, vague, or changing requirements are a
  major cause of project failure
                                                                              5
             Research Questions

• Research Question 1: What causes
  requirements volatility?

• Research Question 2: How does requirements
  volatility impact systems engineering effort?




                                                  6
                                              Research Plan
                                                                                      Completed   In-progress        Planned
                  Literature
                   Review

                                                               Tested
                                                             Hypothesis
             Observations




                                                              Statistical
                                                                                                  Industry Data
Pilot RV survey                Causal Model                    Analysis




            COSYSMO Workshop
               Discussion                                  Updated Causal
                                                          Model (hypothesis)                          Delphi
                                                                                                      Survey
                                                                                                           Delphi

   RV survey                                                                                                Survey




                    LAI Knowledge               Survey                  Considerations for
                                                                          requirements
                       Exchange                 Results                     volatility



                                                                                                                          7
        Literature Background
• Most of the requirements volatility research to date has been focused
  on software systems
• Various research methods have been utilized to investigate the causes
  and effects of requirement volatility – a methodological breakdown of
  the studies reviewed to date is below

                      1
              2                                Simulation Model
                                     5         Survey / Model
                                               Survey
                                               Project data analysis
               3                               Interviews
                             2


• However, there still a lack of empirical data to determine the
  quantitative impact of requirements volatility on systems engineering
  effort for a broader base of engineering projects
                                                                          8
              Implications to COSYSMO
                                     • Volatility was identified as a size
Reuse             Volatility           driver adjustment factor in
                                       COSYSMO [Valerdi, 2005]
Categories        Factor             • It is currently a placeholder in
                                       the model

# Requirements
# Interfaces
# Scenarios           Size
# Algorithms          Drivers
                                        COSYSMO                              Effort
                      Effort
                      Multipliers
             - Application factors
                 -8 factors              Calibration
             - Team factors
                 -6 factors                     [Fortune, 2009]
                                                                                      9
                       Impact of volatility after the
                    requirements have been baselined
                   PROJECT WITH VOLATILITY                                   PROJECT WITH LOW VOLATILITY
                  Number of Requirements Over Time                           Number of Requirements Over Time
           1800                                                       1800

           1600                                                       1600

           1400                                                       1400
# Req.




                                                             # Req.
           1200                                                       1200
                                                                                                        Planned # Req
           1000                            Planned # Req              1000
                                           Actual # Req                                                 Actual # Req

            800                                                        800

            600                                                        600
                             SRR                                                        SRR
                     Systems Engineering effort                                  Systems Engineering effort
           1600                                                       1600

           1400                                                       1400
SE Hours




                                                           SE Hours
           1200                                                       1200
                           Delta SE                                                              Delta SE
           1000            effort                                     1000                       effort

            800                                                        800

            600                                                        600

            400                                                        400
                             SRR                                                         SRR                            10
         Observations
1.   Requirements volatility is correlated with an increase in the size drivers of systems
     engineering effort [Valerdi, 2005; Houston,2000]
2.   Requirements Volatility is expected during the early stages of a project
     (conceptualize / requirements phase). It becomes a concern when it ocurrs after
     the requirements phase is complete because it is likely to result in re-work of
     engineering products [Kotonya and Sommerville (1995); Rosenberg and Hyatt (1996); Ferreira, (2002)]
3.   Requirements volatility is correlated to increases in cost, schedule duration and re-
     work [Ferreira et al, 2009; Zowghi and Nurmuliani, 2002]
4.   Requirements volatility may have a compounding effect on systems engineering
     effort because the changes may not only cause rework but also trigger increases in
     the number of lower-level requirements [Kulk and Verhoef, 2008]
5.   The use of volatility metrics and thresholds can help determine the adequacy of SE
     processes and staff and trigger corrective actions to mitigate further volatility (Roedler
     and Rhodes, 2007)

6.   Requirements volatility is caused by a number of factors that are both external (e.g.
     environmental, customer priority changes) and internal (process capability,
     organizational policies)

          Based on the review of the literature a causal model was
             developed to investigate both research questions                                          11
                      Causal Model (normative)
                      Case 1 research question: what causes requirements volatility?

      Contextual /                         Changes in org.                  Experienced                         Technology                 Requirements.                             SE Process
     Environmental                          structure and                      staff                             Maturity                     Re-use                                  Maturity
       changes                                 process
                    (Jones 1994)                                                       (Ferreira, 2002)          -       (GAO, 2004)
                                                                                                                                              -      (Ferreira 2002)
                                                                                                                                                                                      -    (Ferreira 2002)
            +       (Curtis et al. 1988)        +       (Kotonya and
                                                        Sommerville 1998)
                                                                                 -     (GAO, 2004)
                    Zowghi et al. (2000)




                       Customer-Driven
                          Changes                         +
                                            (Jones 1994)
                                                                                                                                                                     (Valerdi 2005)
                                            (Curtis et al. 1988)
                                            (Kotonya and
                                            Sommerville 1998)               Requirements                                 +/-               Number of Sys                  +/-                 SE
                                                                              volatility                          (Houston, 2000)          Requirements                                   Productivity
+   (Ferreira, 2002)                                                                                              Zowghi et al. (2000)
    (Ferreira, et al 2009)                                                             (Houston, 2000)            (Ferreira, et al 2009)            (Valerdi 2005)
                                                                             +         (Ferreira 2002)
                                                                                       (Charette et al,2003)
                                                                                                                                              +/-                      +/-   (Valerdi 2005)



                                     (Charette et al,2003)
                   Quality                                                    Re-work
                                                                                                                     +                       SE Project                                     Project
                                                -                                                                                              Effort                    +/-               Schedule
                                                                                                                                                                     (Ferreira, et
                                                                                        (Houston, 2000)                                                                al 2009)
                                                                                        (Ferreira 2002)
                                                                               +        (Charette et al,2003)                                +/-     (Valerdi 2005)




                                    +                                                                                +/-
                                                                            Project Cost                                                      SE Cost
                                                                                                                 (Valerdi 2005)

                                                                                   -
                                                                                                                               -
                                                                              Customer                                                      Dependent                Independent
                                                                                                                                                                                                  12
                                                                             Satisfaction                                                    Variable                  Variable
                      Causal Model (normative)
                      Case 2 research question: how does requirements volatility impact
                      systems engineering effort?
      Contextual /                         Changes in org.                  Experienced                         Technology                 Requirements.                             SE Process
     Environmental                          structure and                      staff                             Maturity                     Re-use                                  Maturity
       changes                                 process
                    (Jones 1994)                                                       (Ferreira, 2002)          -       (GAO, 2004)
                                                                                                                                              -      (Ferreira 2002)
                                                                                                                                                                                      -    (Ferreira 2002)
            +       (Curtis et al. 1988)        +       (Kotonya and
                                                        Sommerville 1998)
                                                                                 -     (GAO, 2004)
                    Zowghi et al. (2000)




                       Customer-Driven
                          Changes                         +
                                            (Jones 1994)
                                                                                                                                                                     (Valerdi 2005)
                                            (Curtis et al. 1988)
                                            (Kotonya and
                                            Sommerville 1998)               Requirements                                 +/-               Number of Sys                  +/-                 SE
                                                                              volatility                          (Houston, 2000)          Requirements                                   Productivity
+   (Ferreira, 2002)                                                                                              Zowghi et al. (2000)
    (Ferreira, et al 2009)                                                             (Houston, 2000)            (Ferreira, et al 2009)            (Valerdi 2005)
                                                                             +         (Ferreira 2002)
                                                                                       (Charette et al,2003)
                                                                                                                                              +/-                      +/-   (Valerdi 2005)



                                     (Charette et al,2003)
                   Quality                                                    Re-work
                                                                                                                     +                      SE Project                                      Project
                                                -                                                                                             Effort                     +/-               Schedule
                                                                                                                                                                     (Ferreira, et
                                                                                        (Houston, 2000)                                                                al 2009)
                                                                                        (Ferreira 2002)
                                                                               +        (Charette et al,2003)                                +/-     (Valerdi 2005)




                                    +                                                                                +/-
                                                                            Project Cost                                                     SE Cost
                                                                                                                 (Valerdi 2005)

                                                                                   -
                                                                                                                               -
                                                                              Customer                                                      Dependent                Independent
                                                                                                                                                                                                  13
                                                                             Satisfaction                                                    Variable                  Variable
Moderating impact of expected volatility / thresholds


                                   Org Improvement
                                        Actions

                          +                                     +                                      +/-

                                                                       SE Process
        Experienced
                                                                        Maturity
           staff
                (Ferreira, 2002)                                               (Ferreira 2002)
           -    (GAO, 2004)                                              -


                                   Requirements                                  +/-                                  Volatility Metrics /
                                     volatility                                                                          Thresholds
                                             (Houston, 2000)
                                    +/-      Zowghi et al. (2000)
                                             (Ferreira, et al 2009)
                                                                              (Kulk and Verhoef 2008
                                                                              Zowghi and Nurmuliani                           (Kulk and
                                                                              (1998)                                          Verhoef 2008)
                                    Number of
                                   Requirements
                                                                                         +/-                                 +/-

                                      +/-   (Valerdi 2005)


                                                              (Ferreira, et
                                                                al 2009)
                                     SE Project                                   Project
                                       Effort                 +/-                Schedule

                                     +/-    (Valerdi 2005)



                                      SE Cost
                                                                                     Dependent          Independent        Moderating
                                                                                                                                              14
                                                                                      Variable            Variable          Variable
                 Questions for discussion
1. Are there other important causes of volatility missing in the model?
2. In what cases is the relationship between requirements volatility and
   # of systems requirements a positive one, and in what cases is it a
   negative one?
3. Should the impact of requirements volatility on the # of system
   requirements be adjusted based on the criticality/coupling of the
   requirements that are added/modified/deleted?
4. Does volatility have a direct impact on productivity?
5. Should volatility thresholds vary depending on the size and duration
   of a project?
6. How does the lifecycle phase affect the expected level of volatility?
7. Feedback on the survey -


                                                                           15
                   Next Steps

• Utilize recommendations/suggestions from this
  workshop to update the causal model and
  survey
• Administer the survey to LAI workshop
  participants
• Need to develop an approach to generate cost
  driver weight factors based on requirements
  volatility
• Start the industry outreach to reach agreement
  on data-sharing

                                                   16
                                               References
•   Boehm, B. (1991). Software Risk Management: Principles and Practices. IEEE Software 8, 1. Pp 32-41.
•   Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009). “Understanding the effects of requirements volatility in
    software engineering by using analytical modeling and software process simulation.” The Journal of Systems and Software.
    Vol. 82, pp 1568-1577.
•   Fortune, J. (2009). Estimating systems engineering reuse with the constructive systems engineering cost model (COSYSMO
    2.0). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department.
•   GAO-04-393 (2004). Report to the Committee on Armed
•   Services, U.S. Senate. Defense Acquisitions. Stronger Management Practices Are Needed to Improve DOD’s Software-
    Intensive Weapon Acquisitions
•   Hammer, T., Huffman, L., and Rosenberg, L. (1998). “Doing requirements right the first time.” Crosstalk, the Journal of
    Defense Software Engineering. Pp 20-25.
•   Houston, Dan X. (2000). A Software Project Simulation Model for Risk Management, Ph.D. Dissertation, Arizona State
    University
•   INCOSE Systems Engineering Handbook, Version 3, INCOSE, June 2006
•   ISO/IEC (2008). ISO/IEC 15288:2008 (E) Systems Engineering - System Life Cycle Processes.
•   Kotonya, G., Sommerville, I., (1998). Requirements Engineering: Processes and Techniques. John Wiley and Sons, Ltd.
•   Kulk, G, and Verhoef, C. (2008). “Quantifying requirements volatility effects”. Science of Computer Programming. Vol 72. pp
    136–175
•   MIL-STD-498. 1994. Software Development and Documentation. U.S. Department of Defense.
•   Roedler, G. and Rhodes, D. (2007). Systems engineering leading indicators guide. Version 1. Massachusetts Institute of
    Technology, INCOSE, and PSM.
•   Rosenberg, L. and Hyatt L., (1996). A Software Quality Model and Metrics for Identifying Project Risks and Assessing
    Software Quality. (retrieved from http://satc.gsfc.nasa.gov/support/STC_APR96/qualtiy/stc_qual.html)
•   Schunn, C. (2008) Engineering Educational Design. Educational Designer, 1(1)
•   Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of
    Southern California, Industrial and Systems Engineering Department.
•   Zowghi, D. and Nurmuliani, N. (2002). A Study of the Impact of Requirements Volatility on Software Project Performance.
    Proceedings of the Ninth Asia-Pacific Software Engineering Conference                                                   17

								
To top