Software Quality Assurance Is it Worth The Effort

Document Sample
Software Quality Assurance Is it Worth The Effort Powered By Docstoc
					Software Quality Assurance.
   Is it Worth The Effort?

           Philip Wain
        19th November 1998
“The views expressed in this talk are my
own and do not in any way represent the
views of any company with which I may
    be associated from time to time”
        Structure and Content
   Is the principle behind Software QA sound?
   Does it have any real value to professional
   Does it work in practice?
   Is ISO 9000 an idea whose time has passed?
   Is all the Bureaucracy necessary?
   Is there a better way?
       Is the Principle Sound?

   What is the principle behind all Quality
    Management Regimes?

   Where did it come from?

   Is Software different enough to
    invalidate the principle?
   What is the Principle?

  “Take care of the means and
       the end will take care of

Mahatma Gandhi
Quality Assurance

   Controlling the process has a
    positive effect on the product.
   Once one quality outcome has been
    achieved using a particular series of
    steps, by repeating those steps and
    controlling variation, the outcome will
    also be repeated
How was this principle established
        and validated?
      Ever since humans began
    producing things, particularly for
    other people, Quality has been
    The principle was established
        through trial and error.
   It appears to make logical sense
              History of Quality

The earliest example of quality
inspection is inscribed on a frieze
at Thebes dated 1450 B.C.
    Development of Independent Quality
   Craft-based quality (up to Industrial Revolution)
    » workmanship = Apprentice, Journeyman, Master Craftsman
    » Personal reputation, Tradesman, self-employed
    » Guilds (Freemasons) 1300 AD - Hallmarks, Weights &
    » volumes low, skill training required - lifetime
   Industrialization
    » Employees
    » Automation of Craft
    » De-skilling, Division of Labour (suggested by Adam Smith)
      (Ford 1920’s)
    » quality is responsibility of employer (technique is inspection but
      consistency from the machinery is also important)
    » Volumes high, much less skills training
  The First Quality Management Strategy

“You can’t achieve quality by inspection, you have
              to build it in at the start”
                Not entirely true!
Quality strategy = Independent Inspection. Set a
 standard then throw away (or rework) anything
             which doesn’t measure up.
Result = quality product but an awful lot of waste.
  Successful Survival Strategy for 100 years +
Quality Control v Quality Assurance

CONTROL                  ASSURANCE
 Product focused         Process focused
 Method is Inspection    Method is Audit
  and Test against a       against Procedures
  Spec.                   Quality Achieved by
 Quality Achieved by      doing things the right
  rework and rejection     way
          Move from Control to Assurance
   1214 AD. William Wrotham appointed Keeper of the King’s
    Ports and Galleys
   1300 AD Edward I introduces Hallmark system by Act of
    Parliament ( forerunner of product quality “kitemark”)
   Circa 1660 AD. Samuel Pepys refers to Government Overseers
    responsible for ship construction
   1916 AD. Inspection of Military equipment production moves to
    Supplier, with Govt. inspectors performing sample checks
   1920 AD. Inspection Approved Firms Service
   1950 AD. Polaris programme
   1963 AD. Mil-Q-9858A - first QA standard (?)
   1979 AD. BS 5750
Did QA give “better” results?
   Theory developed by succession of
    gurus, mostly from the world of
     » Shewhart, Deming, Juran, Taguchi,
       Ishikawa, Feigenbaum
   Aimed at eliminating variation by fine
    tuning the process
     » High focus on machines, tools, SPC
   Success = Japanese economic
    miracle post WWII (?)
    Success for the Purchaser?
   1969, MoD had 16,500 quality
    personnel, mostly inspectors.
   1975 down to 5,000
   1985 3,500
   1989 2,700
   1998 Less than 100
   2000 ?
“Good form is the most efficient manner to
accomplish the purpose of the performance
with a minimum of lost motion and wasted

Zen Master, Circa 1960
      Does it apply to software?

 Most major success stories are outside
  of software
 Most major software development
  organizations apply sound QMS
 Software Projects still fail, sometimes
  spectacularly (Taurus, Ariane 5, London Ambulance
    Service, etc.)
         Software Quality

 The   good news
    Software Production is flawless

 The   bad news
   Software Design is anything but
        (but good news for COTS sellers)
                   Attributes of S/W
           Traditional Eng./ Manufacturing

Software                       Traditional Eng./Manf.
   Design Bound                  Production Bound
   People intensive              Machine intensive
   Difficult to De-skill         De-skilling is possible
   Not stable to small           Tolerant of some variation
    changes                       rejects generate scrap
   Easily re-worked              Tangible product -visible
   Intangible product, hard       progress
    to monitor progress
In 1979, the first NASA probe to Venus missed
the planet and plunged into the Sun because a
  full stop was used instead of a comma in a
                 Fortran routine.
    Result = hundreds of millions of dollars and
              countless manhours lost.
   Wallmüller 1994
          Software Engineering

   1968 - Software Crisis Requirements
    NATO Conference,
    coined term for first time         Design

   Led to the Waterfall Life-
    cycle model                                 Code

   We know now that this
    is an oversimplification,                          Test
    real development
    involves lots of iteration
                                                              Implement &
    and re-working of earlier                                  Maintain
    What lies behind the life-cycle concept?

   Logical serial relationship
    between requiring something,
    designing something, building
    something, testing something,
    then delivering something
   As abstractions get reduced,
    mistakes get more costly to fix
   simplifies process quality
    assurance i.e. life cycle stages
    act as checkpoints for
    deliverables, reviews and so
    Criticisms of document-oriented
        Waterfall Process Model

 “Management need deliverable
  assess project progress. The timing of management
  requirements may not..correspond with [completion]..
  of an activity so artificial documents may be produced
 The need to approve documents tends to constrain
  process iteration at the costs of going back and
  adapting a completed deliverable are high
 The Development Process is not linear. At any stage
  of the process, S/W engineers take account of both
  earlier and later stages”

Sommerville, 1992
“For software systems, where     processes have not
 been standardized, process and product quality
cannot be directly cannot be assumed
      that a high process quality will always
  automatically lead to a high-quality product.
    External factors, such as the novelty of an
 application or commercial pressure for an early
product release might mean that product quality
  is impaired irrespective of the process used”

Sommerville 1992
              Is the Principle behind
               Software QA Sound?

   A little shaky perhaps, but..
   The software process is important, but the most
    critical element in the process is people, which
    process standards tackle only obliquely
   Can’t standardize on a process if variety of ways and
    means employed
    » Same thing is never done twice
   maybe this shakiness is something to do with the
    immaturity of S/W Engineering and our imperfect
    knowledge of, and our inability to define and measure,
    the real software process
The Search for the Holy Grail
    1968 - Waterfall Model, Flowcharts
    1970s - V-model, Spiral Model, Structured Testing,
     Structured Programming, DFD’s, Egoless
     Programming, Fagan Inspection, elimination of GOTO
    Early 1980’s = Entity Relationship modelling, 4GL’s,
     Relational Databases, Unix, GUI’s, JSD, Yourdon
    Mid - Late 80’s =SSADM, Formal Methods, LAN’s
     JAD, CASE, Metrics, Expert Systems, WANs, IPSE,
     Software Tools, Prototyping, Hypertext, One Per Desk
    1990’s Workflow software OOD, RAD, Neural Nets,
     Client- Server, Internet/Intranet, Java, UML, Linux
    “No technique is ever around long enough to mature!”
      The Proof of the Pudding

   Are there examples out there of a QMS
    making a significant difference to the quality of
    software delivered?
   Have things changed in the software industry.
    Do we find less, or a different order of,
    problems now than in the past?
   Has the principle been widely applied
       Does it work in Practice?

“Sadly, there are few published accounts of good quality
    systems or good case studies...While it is clear that
    structured methodologies will have a positive impact
    on software quality, there is little evidence yet of what
    form this impact will take...”
Darryl Ince 1990

   Space Shuttle Onboard Software Project - CMM level
    5 in 1989, last safety-related defect detected Oct
Capability Maturity Model , SEI, Addison Wesley 1997
     Problems found during Software

1998                         1990
   Poor estimation             Incomplete and
                                 ambiguous requirements
   Incomplete and
    ambiguous requirements      Lack of metrics
   changing requirements       Poor Estimation
   planning problems           rapid changes, poor
                                 change control
   Lack of metrics
                                Lack of testing
   poor testing
                                difficulties in progress
   Documentation problems
   Configuration
                                Lack of planning
    But has it really been tried?
   Project managers are rational - they will maximize
    whatever they are being measured on - this is mostly
    on getting it shipped, the quality is irrelevant, and as
    most customers expect software to crash anyway,
    why bother trying too hard?
   People take the path of least resistance. Defining the
    process and appropriate measures is hard, so people
    try to avoid it
   Measure PMs on the right things, who knows what
    might happen
“The need to adhere to standards is often resented by
     software engineers. They see standards as
 bureaucratic and irrelevant to the technical activity
  of software development. Although they usually
   agree about the value of standards in general,
engineers often find good reasons why standards are
      not appropriate to their particular project”
Sommerville 1992
           Return of the Hack
 “Many software developers are uncomfortable with the
 formality of software engineering. They see prototyping
  as an opportunity to evade the structured process and
  continue to... produce code with little or no attempt to
                     document a design...
 In addition, maintaining user commitment and keeping
    track of complex changes over a long development
       timescale is proving impossible. We are being
         overwhelmed by size and complexity while
      simultaneously being asked for ever more rapid
                    responses to IS needs”
Brian Hambling, Managing Software Quality- ISO 9000-3
in an iterative World 1996
            Real Value of QA ?
   Codified company policy and verified good practice in
    a QMS, Management publicly shackled to it
   Plans define what is needed to achieve quality - done
    at the start of the project, free from delivery pressures
    which build up later
   Plans can be reviewed against good practice and
    endorsed by management
   Independent QA staff are not bound by delivery
    constraints - can see wood for trees \ can help PMs
    resist those pressures supported by the QMS
   By having the system perspective, auditors can
    pollinate good practice across projects
             RAD to the Rescue
“Prototyping is necessarily an approach that involves risk.
  Not to address quality issues would be irresponsible in
  a devt. environment designed to avoid the drawbacks of
                   a structured approach.
  RAD is the acceptable face of prototyping - structured
     approaches to rapid delivery which contain the
   management and control necessary to provide some
               confidence in the outcome.”
Brian Hambling 1996
          ISO 9000
   Adopted as the software quality standard
    following Logica and Price Waterhouse
    Reports 1988
   Led to the TickIT scheme
    » Auditors were qualified Software developers
    » Software interpretation was written for ISO 9001
   Written around Manufacturing/Process Control
     » not strong on people element at first
   TickIT written with large scale software
    developing organizations in mind
               ISO 9001 principles
   Control your Inputs
    » Select and train good people, (4.18) give them authority and tell them
      what you want them to do (4.1), select good suppliers, (4.6) buy good
      quality raw materials, (4.6) and test them for quality before use (4.10.1)
    » use good (calibrated) tools, (4.4.2, 4.6, 4.11) Keep documents in line
      and current, (4.5) establish unambiguous achievable requirements. (4.3)
   Control the Process
    » Design, Manufacture, Service, manage the components (4.2, 4.4, 4.7.
       4.9, 4.8, 4.19)
   Control your Outputs
    » Inspect, Test and Rework (4.10, 4.12)
    » Store handle and package to prevent damage (4.15)
   Monitor the System
    » Record activity and Audit the system. ( 4.16, 4.17)
   Learn from the system and improve it
    » Review achievement, change practices to fix and improve things (4.1,
       4.14, 4.20)
      ISO 9000-3, for Software
   Written so developers could understand it
   incorporated accepted wisdom regarding S/w
    Engineering good practice
   Some guidance beyond ISO 9000
     » on Project Management
     » Purchaser’s responsibility,
     very relevant to key software devt. difficulties
   The Holy Grail at last?
          So does ISO 9000 work?
   LRQA commissioned surveys of 400 of its
    customers, 1993 & 1995
     » 9 out of 10 said met or exceeded
     » Large Maj. said increased Market Share,
       profits, efficiency, customer service, ability
       to tender for contracts, improved
       Management control
     » 6% disappointed, too much paper, too
             More LRQA Results
   Profits for Approved companies higher than
    national average, (highest ratio is for small
    firms, ie 6.8% against 1.9%)
   Return on Capital Employed higher, (16%
    against 7.7%)
   Sales per employee higher (62K v 48K)
   Profit per employee higher (3K v 1K)
          Problems with this?
   Company Culture
    » could be correlation with desire for quality
      and business seriousness
    » may have little to do with ISO 9000
      practices per se
   Asking wrong people?
     » what is the level of satisfaction of the
       approved companies’ customers and staff?
   Everyone seems to have a tale of poor
    service from an ISO 9000 company
Problems with ISO 9001 Assessment
     Released (1987) during the term of a
      Government ideologically opposed to public
     BSI - quasi-official
       » now had to be completely self-supporting
     Accreditation of Certification bodies set up via
      DTI /NACCB, (now UKAS)
      » Certification services to be sold to potential
        customers, but regulated and policed
      » not truly third party - customer /supplier
        relationship with candidate company
      ISO 9001 certification
   At first, Certification bodies behaved as
    Policemen, as if regulatory authority
   Customers began to flex their muscles
    » with greater power as the market became
   Certification Bodies became customer
    focused and value-added
   Bad thing?
    » not necessarily
    Effect of customer-supplier Relationship
               in Approval Process

   Customer has the whip hand,
     » can switch bodies if not satisfied, keeps CBs
   assessors urged to find reasons to approve
    companies rather than reasons not to
   To make process affordable and attractive to
    customers, assessment time reduced to the
     » result, shallow audit
             Customer Focus
   What is this in the context of ISO 9001
   Do customers want:
     » a) an easy ride to assessment
     » b) a challenge of their system against the
       highest possible standards because that
       way they will have the best possible
       confidence in the robustness of their
       process and win and retain customers
   Answer is: Depends on the customer
    ISO 9000, victim of its own success?
   Condition of entry into the market
    condition of survival in the market
   1993 - MoD announces will only buy from ISO 9000
    approved suppliers
   Having the standard is now a matter of survival - ITT
    to a certification body with the least onerous route to
   In a competitive market, CBs can do nothing but
   Having ISO 9000 now represents the lowest common
    denominator of quality achievement, cannot rely on
    ISO 9000 to achieving excellence
    Flaws in the Certification Process

   Companies have to be really bad to fail to attain the
    standard and horrendous to have it removed after it’s
    been won-
    » the process of only following up problems in all but
      exceptional cases helps ensure this
    » A larger company has an easier time than a small one,
      because it’s even harder to demonstrate that the system as a
      whole has broken down if you only ever see a small part at a
   Result = experience of poor service and product
    quality from approved companies starts to undermine
    the credibility of the whole approval scheme
    Is there any life left in ISO 9000?
   The value in the standard for improving software quality
    is not in the certification but in the ISO 9001 model itself
   Management must not become complacent because they
    have the certificate and should not rely on external audit
    to improve things radically
   Sanction of losing the standard is not strong, but auditors
    have wealth of experience and are a lot cheaper than
    management consultants
   ISO 9000 does not set the standard of product
    acceptability - this is between management and the
    customer (ISO 9001:1994 clauses 4.1 and 4.3)
Do we need all this Bureaucracy?

 What is Bureaucracy?
 Why do we have so much paper
  generated? What use is it?
 Can we have quality assurance without
  any (or with less) Paper?
 Can we have ISO 9000 and still have
  less paper?
Collins Dictionary defn.

   4. Any Administration in which action
    is impeded by unnecessary official

   Bureaucracy, in this sense, is unnecessary
    by definition
         What Paper is needed?
   Procedures
    » define the process steps which must be carried out (agreed with
      everyone and from historical experience)
   Standards
    » define the characteristics of the output (set by customer)
   Records
    » proof of achievement of above (communicate success to management,
      trail in case things go wrong, proof of process operation for management
      and audit)
   Plans, Specifications
    » development tools, give confidence of adherence to process, improved
      likelihood of good outcome need to be current and available
   All necessary for communication and/or co-ordination
   Guidelines
    » help on any of the above (reduce as organization gets good)
     Why is there too much Paper?

 “Instinct of control seems to always win over
    empowerment“ (Tom Peters on Western Management Style)
   Should only standardize for specific benefits - not
    uniformity for the sake of it
   Claimed Benefits should be justified with evidence
   Burden of proof should lie with those who wish to
    impose standards
   Business process models suggest what is important
    » if the model doesn’t call for it, difficult to make a case for it.
Can we do without so much of it?
   Every piece of paper should have a value in excess
    of the effort required to generate or maintain it
   If you want to remove the paper, Weigh the value
    against the effort. (never really done in practice)
   Records can be much simpler than is generally the
    case (should always only be as complex as
    necessary to meet its purpose)
   file notes are as good as pre-defined forms
   no need to pad plans with impressive sounding
     Is there a better way?

   Other models                                                              Optimizing
    » Process models
       – CMM, SPICE, Bootstrap, Trillium etc.                  defined

    » Business Models                             repeatable

       – BEM, IiP, Baldridge            initial
       – TQM
   Systems Engineering
   Performance based-process
   ISO 9000
    » issue 3.0 (2000)
                 Process Models
   Based on Crosby’s Maturity grid
   Characteristics of organizations/standard processes
    at different levels of “goodness”
   Organization is assessed against this model, (can be
    self assessment)
   allows meaningful comparisons - good for supplier
    selection (particularly CMM)
   Beyond ISO 9000 (theoretically ISO 9000 can only
    take an organization to CMM level 2 ½)
   Effective?
    » CMM paper 1987 - only 2 (?)organizations made it to top to
                 Business Models
   Business Excellence                          People
                                                   9%                              9%
    »   includes results
    »   allows wide comparisons Leadership      Policy &
                                                Strategy          Processes
                                   10%             8%               14%           20%          15%
    »   A lot of self-assessment
    »   awards possible                        Resources                       Impact on
                                                  9%                            Society
   Investors in People                                                           6%

    » most principles covered                Enablers 50%                          Results 50%
      by ISO 9000 though IiP is
      more explicit                                        Plan                      Do

   TQM
    » more a philosophy than a
    » People centred -good for                              Act                    Check
       Business Excellence Model

                People                    People
              Management                Satisfaction
                  9%                         9%

               Policy &                  Customer        Business
Leadership     Strategy     Processes   Satisfaction      Results
   10%            8%           14%          20%            15%

              Resources                 Impact on
                 9%                      Society

             Enablers 50%                       Results 50%
         Systems Engineering

   Holistic approach
   development support systems, i.e. tools,
    organization, not just the product
   strong emphasis on requirements
    engineering, managing change
   interdisciplinary teams, collaborative working
   whole life support,
   Software just a special case
                    Systems Engineering Framework
                            User                               Acquisition                            Installation                Operational
                                                               management                             & validation                 system
                                             Proposed                          Reporting, metrics &                            Supplied
                                             characteristics                   management control                              systems

                   Systems                                              Engineering                       Integration
                                        Architectural                                                           &                     Integrated
                 requirements              design                       management
                   definition                                                                             verification                  system
                    Allocated                        Proposed                     Reporting, metrics &                       Supplied
                    requirements                     characteristics              management control                         sub-systems

“Local” user &          Systems                 Architectural                                         Integration               Integrated
                      requirements                                         Engineering                      &                  configuration
organizational                                     design
 requirements           definition                                         management                 verification                 items
                              Allocated                        Proposed                  Reporting, metrics &               sub-systems
                                                               characteristics           management control
Multiple projects             requirements

   “Local” user &             Systems                 Architectural                                Integration                configuration
   organizational           requirements                 design               management                 &                        items
    requirements              definition                                                           verification
                                     Allocated                         Proposed              Reporting, metrics &        components
            Component                requirements                                            management control

                    “Local” user &             Component                Component                 Component                 Components
                    organizational            requirements                design                  build & test
           Performance-based Process
   Key to success is Measurement of
    process performance
   Getting the right measure which
    affects behaviour in the right way is
    not a simple step
   No model of best practice
   Improvement is incremental,
    Management may get impatient-
    looking for Breakthrough
   Can be used in conjunction with
    standards based methods
      ISO 9000 (our last best hope?)
   New draft - Business orientated
    » Process based
    » de-emphasises testing and
      inspection (still there though and
    » more on improvement
   Good Auditors already
    encouraging this
    » now supported by words in the new
Can any standard keep up with the latest thinking in
any subject? Doesn’t standardizing stifle innovation
                 and creativity?

 The great strength of the ISO 9000 series has been its
 focus on general controls and non-prescriptive style.
 Changes reflect further moves in that direction. Note
 the ability of TickIT to absorb RAD-type development
 approaches without too much problem. Combining a
 principle of agreed good practice with performance-
 based process improvement seems to have the
 potential to overcome the limitations of these
 approaches taken separately
So, is Software Quality Assurance
      worth the effort or not?
   How much effort do we really put into it now?
    » until we try it seriously, we are unable to say for sure

   Has QA practices lagged behind process
    » Most QA regimes are stuck in waterfall when devt. is iterative
    » ISO 9000 is flexible and that should be exploited to the full.

   QA professionals need humility as to the
    difference they can make
    » extravagant claims will alienate S/W engineers who know
    » as much (!) about S/W development than the QA Dept.

   Should QA be renamed “Process Assurance?
    » case for that, but history of renaming things does not have a
      glorious success record
“Set patterns, incapable of pliability only offer
a better cage. Truth is outside of all patterns”

Bruce Lee

Shared By: