Software Engineering
Software Quality Assurance
Objectives
1. To motivate for software quality assurance.
2. To provide a historical context for software
quality assurance.
3. To consider the benefits of formal technical
reviews.
4. To introduce statistical software quality assurance
(SSQA).
Steps in Project Planning
Project Scope
Estimates Software
Risks Project
Schedule Plan
Control strategy
Scope—understand the problem and the work that must be
done.
Estimation—how much effort? how much time?
Risk—what can go wrong? how can we avoid it? what can we
do about it?
Schedule—how do we allocate resources along the timeline?
what are the milestones?
Control strategy—how do we control quality? how do we
control change?
Defining Quality
One goal of software engineering is to produce high-quality
software. But what is “software quality?”
Software quality assurance (SQA) is an umbrella activity
applied throughout the software process.
General engineering objective: reduce the “variation
between samples.” But how does this apply to software?
Quality of Design: Characteristics that designers specify for
an item. Encompasses requirements specification, analysis
and design.
Quality of Conformance: The degree to which the design
specifications are followed during manufacturing. Covers
implementation.
Alternatively: User satisfaction = compliant product + good
quality + delivery within budget and schedule.
McCall’s Triangle of Quality
Maintainability Portability
Flexibility Reusability
Testability Interoperability
PRODUCT REVISION PRODUCT TRANSITION
PRODUCT OPERATION
Correctness Usability Efficiency
Reliability Integrity
Deriving Quality Metrics
Each of McCall’s Quality Metrics combines different
quality factors.
Examples:
Correctness = Completeness + Consistency + Traceability
Portability = Generality + Hardware Independence + Modularity +
Self-Documentation + System Independence
Maintainability = Conciseness + Consistency + Instrumentation +
Modularity + Self-Documentation + Simplicity
“McCall’s quality factors were proposed in the early 1970s.
They are as valid today as they were in that time. It’s likely
that software built to conform to these factors will exhibit
high quality well into the 21st century, even if there are
dramatic changes in technology.”
Quality Concepts
Quality control: A series of inspections, reviews, tests to
ensure that work products meet their requirements. Must
include feedback.
Quality assurance: Analysis, auditing and reporting activities
which communicate information about quality to management.
Cost of quality: all costs incurred in performing quality related
activities.
Prevention Costs (quality planning, formal technical reviews, test
equipment and training)
Appraisal costs (trying to gain insight into product condition = in-
process inspection, equipment calibration and maintenance, testing)
Failure costs (detect an error prior to shipping = rework, repair, failure
mode analysis)
External failure costs (detect an error after shipping = complaint
resolution, product return and replacement, help line support, warranty
work)
Why SQA Activities Pay Off?
Cost to find and fix a defect
100 60.00-100.00
log
scale 10 10.00
3.00
1.50
1 0.75 1.00
design testsystem field
anal. code test use
Software Quality Assurance
“Conformance to explicitly stated functional and performance
requirements, explicitly documented development standards, and implicit
characteristics.”
SQA Process Formal
Definition & Technical
Standards Reviews
Analysis
& Test
Reporting Planning
& Review
Measurement
SQA Definition
Aspects of SQA:
1. Software requirements are the foundation from which quality is
measured. Lack of conformance to requirements is lack of quality.
2. Specified standards must be followed.
3. A set of implicit requirements often goes unmentioned (e.g. ease of
use). If software conforms to explicit requirements but fails to
meet implicit requirements, software quality is suspect.
History:
Prior to 20thC province of the individual craftsman.
First formal quality assurance at Bell labs in 1916.
Software SQA parallels this (1950s and 1960s done by individual
programmers, 1970s military introduced structured SQA)
SQA Group
Prepares an SQA plan for the project. This must identify
evaluations, audits, reviews, standards, procedures for
error reporting and tracking, and feedback.
Participates in the development of the project process.
Reviews software engineering activities to verify
compliance with the defined software process.
Audits designated software work products to verify
compliance with those defined as part of the software
process.
Ensures that deviations in work products are documented
and handled according to set procedures.
Records any non-compliance and reports to senior
management.
Formal Technical Reviews
Formal technical reviews can be up to 75% effective in
uncovering design flaws (which cover 50-65% of all errors)
“There is no particular reason why your friend and
colleague cannot be your sternest critic” - Jerry Weinberg
Objectives:
1. To uncover errors in function, logic, or implementation for any
representation of the software.
2. To verify that the software under review meets its requirements.
3. To ensure that the software has been represented according to
predefined standards.
4. To achieve software that is developed in a uniform manner.
5. To make projects more manageable.
A Case for Reviews
Defect amplification: a model for the generation and
detection of errors during a sequence of tasks.
Errors Defects Detection
Errors
from passed
Errors passed through
previous Percent to next
step efficiency step
Amplified errors 1:x
for error
detection
Newly generated errors
Example:
Each test step uncovers 50% of errors but no reviews conducted in
earlier stages. 10 errors in analysis lead to 12 latent defects. Assume
amplification of 1:1.5.
Formal technical reviews with 50-70% efficiency in analysis design
and coding reduce latent defects to 3.
What Are Reviews?
What reviews are:
a meeting conducted by technical people for technical people
a technical assessment of a work product created during the
software engineering process
a software quality assurance mechanism
a training ground
What reviews are not:
A project budget summary
A scheduling assessment
An overall progress report
A mechanism for reprisal or political intrigue
The Players
review
leader standards bearer (SQA)
producer
maintenance
oracle
recorder reviewer
user rep
Conducting the Review
1. be prepared—evaluate
product before the review
2. review the product, not
the producer
3. keep your tone mild, ask
questions instead of
making accusations
4. stick to the review agenda
5. raise issues, don't resolve them
6. avoid discussions of style—stick to technical
correctness
7. schedule reviews as project tasks
8. record and report all review results
Metrics Derived from Reviews
Inspection time per page of documentation
Inspection time per KLOC or FP
Inspection effort per KLOC or FP
Errors uncovered per reviewer hour
Errors uncovered per preparation hour
Errors uncovered per SE Task (e.g. design)
Number of minor errors (e.g. typos)
Number of major errors (e.g. non-conformance to
design)
Number of errors found during preparation
Statistical SQA
• collect information on all
defects
Product • find the causes of the
& Process defects
• move to provide fixes for
the process
measurement
... an understanding of how
to improve quality ...
Aspects of Quality
Software Reliability:
Important aspect of overall quality
The probability of failure free operation of a computer program in
a specified environment for a specified time.
Where failure is non-conformance to specification.
Example: program X is estimated to have a reliability of 0.96 over
8 elapsed processing hours.
Mean Time Between Failure (MTBF) = Mean Time To Failure
(MTTF) + Mean Time To Repair (MTTR).
Regarded by some as better than num. defects.
Software Availability:
Probability that a program is operating according to requirements
at a given point in time.
Availability = [MTTF / (MTTF + MTTR)] * 100%.
Further Aspects of Quality
Software Safety:
Important in safety critical systems
Unlike software reliability, failure leads to a hazard or
mishap.
Use techniques from Risk Analysis. Hazards are
identified and categorized according to probability and
criticality.
Example: Cruise control in a car.
Hazards: (1) Causes uncontrolled acceleration, (2) does
not respond to depression of break pedal, (3) does not
engage when switch is activated, (4) slowly loses or gains
speed.
Mistake Proofing
Poka-Yoke (Mistake Proofing) developed by Shigeo Shingo
at Toyota in the 1960s.
A quality assurance technique applicable to software.
Automatic error detection and prevention.
Devices:
Prevention of potential quality problems before they occur.
Rapid detection of quality problems if they are introduced.
Everyday examples: seatbelt warning signal (detection) or
Car unable to start when in gear (prevention).
Characteristics:
It is simple and cheap. Otherwise it will not be cost effective.
It is part of the process. Integrated into the process.
It is located near the process task where the mistakes occur. Allows
rapid feedback and error correction.
ISO9000 Quality Standards
A quality assurance system is defined as the organizational
structure, responsibilities, processes and resources for
implementing quality management.
Satisfy customers expectations by meeting their
specifications.
ISO9000:
A generic description of quality assurance activities applicable to
any business
Adopted by many countries
Often required to supply goods and services to the government
Compliance scrutinized by third party auditors and subject to semi-
annual review
ISO9001:
Standards that apply to Software Engineering