Chapter 6.
Making Success Measurable
Through Quality Control
1
Chapter 6. Section 1.
Introduction to Quality Control
2
Introduction
• How do you know the quality and accuracy
of your rent calculations?
– Wait for HUD to tell you?
– Close your eyes and hope for the best?
– Assume that good people lead to good
outcomes?
– Wait for your independent audit results?
• Or …..
3
Introduction
• Are you interested in a better way that
could:
– Enable you to know when to take action on
performance issues
– Help document the improvements in error
reduction you are making
– Enhance the quality of the services you
provide?
4
Introduction
• A good quality control program can be that
better way that helps you:
– Prevent errors
– Document compliance with program
requirements
– Demonstrate performance improvement over
time
5
Introduction
• An effective quality control program:
– Facilitates conversation about expectations
and performance for individuals and the
organization as a whole.
– Enables program managers to be proactive,
rather than reactive
6
Introduction
In this session, we will talk about:
• Three types of quality control
• Illustrate the thought process needed to
develop a QC program
• Provide sample QC tools
• Provide guidance on communicating with
your staff about quality control issues.
7
Introduction
• We will focus on HUD’s expectations
related to income and rent determination
• BUT, the QC concepts we discuss apply
equally well to other parts of a PHA
operation.
8
Why Quality Control?
• Quality control can help to confirm what
you already know. It can:
– Provide documentation that supports
recommendations you want to make
– Generate statistics that document staff
performance
– Provide valuable information on a systematic
basis to share within and outside your
organization.
9
Why Quality Control?
• QC can be even more helpful if the results
are a surprise. Then the results may
highlight:
– Problems of which you were not aware
– Good news about improved performance or
excess capacity.
• Were there aspects of the RIM review for
your agency that were a surprise?
10
Chapter 6. Section 2.
Types of Quality Control
11
Types of Quality Control
Continuous
Types of
Quality
Control
Preventiv
e Periodic
12
Preventive QC
• Preventive QC is checking on the quality before
the transaction is completed. Use Preventive QC
when:
– The consequences of errors are high, or
– The error is not recoverable.
• Examples:
– Supervisors sign termination letters even though
specialists sign other correspondence
– Occupancy supervisor reviews 100% of rent
calculations for new staff (for some period of time).
13
Preventive QC
• More examples:
– Intake specialist presents a “dry run” of the briefing to
staff before doing the real thing in front of participants.
• What examples of preventive QC occur at your
PHA?
• Try to identify at least one additional area
(related to income and rent) where preventive
QC might be helpful.
14
Limits of Preventive QC
• Preventive QC can be an important tool
but what are its limitations?
– PHA’s don’t have time or staffing to double
and triple check every transaction –you need
to prioritize.
– Use common sense: Not everything requires
this level of effort.
15
Periodic Quality Control
• In contrast, periodic quality control happens after
the fact. Work has already been done and
reported.
• It’s purpose is to take a “picture-in-time”
assessment of PHA operations that will help
improve the program and prevent future errors.
• Sound familiar? Yes – HUD’s RIM reviews and
PHAS and SEMAP audits are kinds of periodic
QC.
16
Periodic Quality Control
• During a periodic review HUD systematically
looks at:
– Policies, procedures, and practices and
– Individual files.
• But you don’t have to wait for HUD. You can
also do your own. You could:
– Look at the same things HUD will be
checking, or
– Focus your periodic review differently to learn
even more.
17
Limits of Periodic QC
• Periodic QC can provide an accurate
point-in-time assessment, but:
– It doesn’t help managers see trends
– Results come infrequently so it doesn’t help
managers take corrective action during the
year.
– The level of effort required may seem
overwhelming (but HUD’s RIM Review guide
provides a good model for the review)
18
Continuous Quality Control
• Continuous QC is used to help with day-
to-day management.
• It involves analyzing a few (but very
important) factors on a regular, continuing
basis. For example:
– Tracking on-time reexaminations each month
– Looking at accuracy rates by individual or
transaction
19
Continuous QC
• Continuous QC helps managers:
– Focus more quickly on vulnerable areas.
– Look at trends, rather than point-in-time
information.
• HUD does some continuous QC of 50058
submissions and uses the data to:
– Send error reports to PHAs, and
– Assess in general terms how PHAs are doing.
• But only the PHA can implement an effective
continuous QC program.
20
Limits of Continuous QC
• Continuous QC is a powerful tool.
• But, remember, the scope of the
Continuous QC is limited.
• This means:
– Continuous QC provides only a limited
picture (hopefully of the key areas)
– Continuous QC should be supplemented
by Periodic Reviews that take a broader
view.
21
Continuous QC Example
Page 6-11 and 6-12
• Look at the “Error Analysis” chart
– QC results are shown by quarter & individual
– Caseload = annual caseload
– Errors = things HUD would call findings
– Comments = errors that would not be findings
– Types of errors are defined on page 6-12
22
Continuous QC Example
• What issues does this report raise for you?
• What are your concerns about:
– Productivity?
– Types of errors?
– Error Rates?
• What are the implications of the
information you see for this PHA?
23
Chapter 6. Section 3.
Designing a QC Program
24
Designing a QC Program
• Now that we have discussed each type, it’s time
to think about how to design your program.
• Before you begin it is very important to think
through:
– What you want your QC program to look at.
– What you will do with the information.
25
Designing a QC Program
Where to Start
• To focus your program ask three questions:
– What are the outcomes you want to achieve?
– What would success look like for each
outcome?
– What is the best way to confirm (track,
measure) performance in each area?
• Then decide:
– Which measures are useful and feasible.
– How you are going to record the information.
– Who needs to know the results.
26
A Simple Example- Sleep
• What is my desired outcome?
– A good night’s sleep
• What would success look like?
– Eight hours of uninterrupted sleep
• What is the best way to measure?
– An independent 3rd party could observe my
sleep.
– I could track myself the time I go to bed, time I
wake up, and number of interruptions.
27
Simple Example (Cont)
• Decision:
– Having someone watch me sleep could be
expensive and disruptive
– Instead, I will track sleep patterns myself
– I’ll write this down in a notebook
– No one needs to see it but me (unless I think
maybe my doctor should know)
28
A More Serious Example –
Utility Allowances
• What is my desired outcome?
– Current and correct utility allowances
will always be used in rent calculations.
• What would success look like?
– Utility allowances would be used
correctly in 100% of the files.
• Are there other ways you would describe
success re: utility allowances?
29
A More Serious Example –
Utility Allowances
• What are possible ways to measure the
outcome?
– 100% file review in advance (preventive)
– 100% file review after the fact
– Sample file reviews after the fact
• What else could we do to see if the correct
utility allowances are being used
consistently?
30
A More Serious Example
• Decision:
– 100% review would be too time consuming
(before or after)
– Therefore, review a sample of files (after the
fact) to see how often staff have used the
correct utility allowance
– Generate a report that summarizes errors by
employee, by program type, by transaction
type.
31
Thinking About Outcomes
• Choosing the outcomes you want to use is an
important part of the design process.
• Outcomes can be expressed in very general or
very specific terms.
• Some “big picture” outcomes might include:
– Consistency in data collection, documentation
– Complete and accurate verifications
– Accuracy in rent calculations
– Timeliness
– Complete, accurate reporting
32
Thinking About Outcomes
• Even broader outcomes might include:
– No findings on the next RIM review
– Informed decision making by applicants and
residents
– Fair and equitable treatment
– Equitable workload
– Being a PHAS high performer
– Positive perceptions – we’re a good agency.
33
Thinking About Outcomes
• Caution: Generally more specific
outcomes are better understood and
easier to measure. For example:
– Reduce departmental error rate to less than
one percent
– Complete 100% of annual reexaminations
accurately and on-time
• The more specific the outcome, the easier
it is to track performance.
34
Designing a Program to
Support Outcomes
• Page 6-20 through 6-21 show examples of
turning desired outcomes into a QC plan.
• Follow along on the chart
• We will walk through Row 1 and Row 2
together
35
The End Result
• The end result of this thinking will be a
Quality Control Plan. (see page 6-25).
• The planning itself is more important that
the format for the plan.
– Think through the process before you begin
data collection
– Collect information about the items that have
the most effect on error reduction
– Be realistic about what data you can collect.
36
What’s Next?
• The balance of this chapter discusses 5
important “how to” aspects of QC:
– Conducting file reviews
– Conducting a “general” review of operations
– Using reports to track quality
– Addressing staff skills and skill gaps
– Communicating about error reduction and
quality control
37
Chapter 6. Section 4.
How to Aspects of Quality
Control
38
Conducting File Reviews
• Files reviews are used:
– To determine accuracy and completeness
– To identify potential policy and management
issues
• “Files” for this purpose include the 50058
record as well as paper files.
• Caution: A file review can tell you what
has happened, but not necessarily why.
39
Conducting Tenant File Reviews
• Sample size and characteristics of files selected
depend upon the purpose.
– Generally PHAs use a “directed” sample.
– Directed means selecting the files that contain
the characteristics you want to monitor.
– Page 6-28 shows characteristics that may be
important to consider for the sample.
• Not every characteristic must be represented
each time you review files.
40
Conducting File Reviews
• For Continuous QC
– Purpose is timely feedback on few issues.
– File Selection Priorities would be:
• Each specialist
• Basic transaction types
• Files completed over time
41
Conducting File Reviews
• For Periodic QC
– Purpose is a more comprehensive review.
– File Section Priorities:
• Each specialist (but perhaps more files)
• More transactions types
• Larger cross section of income sources and
household types
42
Conducting File Reviews
• Tools for Selecting Files:
– PIC reports can generate lists for some
transactions. (See Appendix A, A-25)
– PHA vendor software may be able to
generate reports that list files in categories.
• How many files do I need to review?
– QC sample should be sufficient to meet
SEMAP requirements
43
Example File Sampling
• Useful guideline for determining number of files
to review
Universe Minimum Sample Size
50 or less 5
5 plus 1 for each 50 (or part
51-600
of 50) over 50
16 plus 1 for each 100 (or
601-2000
part of 100) over 600
30 plus 1 for each 200 (or
Over 2000
part of 200) over 2000 44
Conducting File Reviews
• Logistics
– Use a file checklist to document your review.
(RIM checklists are a good start.)
– Complete 50058 calculations without looking
at the completed calculations in the files.
– Look for consistency between the paper and
electronic files.
45
What The Reviews
May Show
• Key RIM Review programmatic findings:
– Earned income/pension/public assistance
calculations
– Incomplete/outdated PHA policies
– Data not being transmitted to PIC
– Lack of verification of income and deductions
– File data doesn’t match data used to calculate
rent
– Improper calculation of utility allowances
46
What the Reviews
May Show
• RIM reviews also found errors related to
automation.
• In general, these errors include:
– Programming problems (software vendor)
– Errors related to the way PHAs use
automated systems such as:
• Not attending to error reports and
• Inadequate training for staff on the system
• See page 6-32 for details
47
Conducting the “General”
Portion of a Periodic Review
• Periodic Reviews enable PHAs to take the same
broad look that HUD takes during a RIM Review.
• The agenda:
– Look at policy and procedural documents
– Conduct an expanded file review
• HUD’s RIM checklists are good models.
B. ACOP G. Rent
E. Applications I. Reexaminations
F. Income J. HUD 50058
48
Conducting the “General”
Portion of the Review
• You may need to add to HUD’s checklists:
– Questions covering PHA-specific policies or
performance standards
– Protocols for checking how PHA-specific
software handles income, rent and subsidy
determinations.
49
Using Reports to Track
Quality
• Perspective is important.
– Identifying individual errors is important but
it’s only one aspect of QC.
– Looking at trends (performance over time)
provides important insights that lead to error
reduction.
– Looking at trends can also help set
performance standards.
50
Using QC Reports to
Track Quality
• What should be tracked regularly?
– How often things happen
– How long things take
– Whether improvement is being made
– How individuals are doing.
• Caution: Tracking reports can raise flags
but may not answer the question: “Why?”
51
Analyzing Data
• Pages 6-36 – 6-38.
• Study the three reports
– On-time reexaminations
– Transactions by type
– Error rate trends
• What flags does each report raise?
• What additional information would you like
to have?
52
Addressing Staff
Capacity Issues
• Error reduction will not happen without staff
capacity.
• No one is born knowing the rules of assisted
housing! Most PHAs must grow their own
“experts.”
• Staff development is a joint responsibility of the
organization and the individual.
• Capacity building begins with a mutual
understanding of both the job requirements and
current performance.
53
Addressing Staff Capacity
• A good QC program can support staff
development efforts if it includes an analysis of:
– The knowledge and abilities that each job
requires
– The actual knowledge and abilities of the
staff.
• And, if the PHA uses this information to create:
– A strategy for addressing any gaps in capacity
– A plan for developing staff and monitoring
staff progress in meeting performance goals.
54
Addressing Staff Capacity
• Skills Set is the term used to describe
what people need to know, understand,
and be able to do.
• See pages 6-40 & 6-41 for a list of the skill
sets required by staff who perform income
and rent determinations.
• Review the list briefly. Do you think these
items represent the skills needed by staff
at your PHA?
55
Addressing Staff Capacity
• How to use your analysis of skills sets:
– Your analysis can improve your recruiting
efforts and hiring decisions
– Be aware of skill gaps when considering
changes in staff roles or position
– Use your understanding of required skills vs.
available staff skills to help diagnose
performance problems
– Use your understanding of skill sets to select
the right employees and to develop employee
training plans.
56
Addressing Staff Capacity
• Consider using Individual Development Plans to:
– Help set expectations
– Demonstrate a commitment to staff
development
• See page 6-49 for a sample IDP. Note how it
sets expectations and specifies development
activities that will help the employee succeed.
– Goal #2: Employee will become proficient….
– Goal #2, Activity 3. Attend a 3-day occupancy
training program.
57
Communicating About
Error Reduction and QC
• The Challenge:
– Reducing errors requires staff
commitment.
– A QC program must be seen as a
positive activity to staff as well as
management.
• To succeed, staff need to know:
– What good performance is
– How it will be measured
– How they are doing as performers. 58
Communicating About
Error Reduction and QC
• Performance expectations help staff understand
what is required. They generally relate to:
– Quality and Quantity of Work
– Timeliness
– Customer Service
• Quality control program helps managers to:
– Set expectations
– Share information about actual performance in
comparison to goals
– Recognize and reward good and improving
performance
59
Communicating About
Error Reduction and QC
• Sharing information about performance
helps staff keep focused on:
– What is important
– What needs to be done
– What they have accomplished
• Most organizations have too many reports
and not enough information.
• Finding effective ways to share this
information is challenging.
60
Communicating About
Error Reduction and QC
• Consider using “Scorecards” to help staff
maintain perspective
• Keep them simple (see page 6-45) and
focus on:
– Goals and Progress
– Information that makes paperwork real
(household demographics, neighborhoods)
– Quality and quantity of work
– Progress over time
– Accomplishments – things to brag about
61
Communicating About
Error Reduction and QC
• Scorecards for individuals also can help
reinforce expectations and give feedback
to staff.
62
QC Recap
• A good quality control system should:
– Be comprehensive but not burdensome
– Cover key performance areas
– Fairly and accurately measure performance
– Be perceived as fair by staff and managers
• Few organizations are perfect. QC will find
problems.
• But a good QC program will also show the
pathway to improvements.
63
QC Recap
• The next session talks about how QC data
can help you to develop strategies for:
– Responding to RIM findings
– Solving performance problems, and
– Reducing errors.
64