Get documents about "PowerPoint Slides 1
Document Sample
Professional Development Manual
on NRS Data Monitoring for Program
Improvement
By:
Mary Ann Corley
Principal Research Analyst
AMERICAN INSTITUTES FOR RESEARCH
1000 Thomas Jefferson Street, N.W.
Washington, DC 20007
This manual was prepared for the project:
Promoting the Quality and Use of
National Reporting System (NRS) Data
Contract #ED-01-CO-0026
U.S. DEPARTMENT OF EDUCATION
Office of Vocational and Adult Education
Division of Adult Education and Literacy
Susan Sclafani, Assistant Secretary for
Vocational and Adult Education
Cheryl Keenan, Director
Division of Adult Education and Literacy
Mike Dean, Program Specialist
Division of Adult Education and Literacy
April 2004
Contents
Page
Introduction ................................................................................................................................ iii
Audience ............................................................................................................................... iii
Purpose ................................................................................................................................ iii
Workshop Overview............................................................................................................... iv
Preparation Checklist.............................................................................................................. v
Workshop Outline ...................................................................................................................... vi
Before the Workshop ................................................................................................................viii
Facilitator’s Notes
Facilitator‟s Notes: Day 1 ............................................................................................................ 1
Facilitator‟s Notes: Day 2 ............................................................................................................ 9
Participant’s Handouts
Workshop Objectives .............................................................................................................. H-1
Workshop Agenda ................................................................................................................. H-2
Why Get Engaged with Data? ................................................................................................. H-3
Your Own Personal Motivators................................................................................................ H-4
Questions for Consideration .................................................................................................... H-5
Decision for State Teams: Selecting a Standard-Setting Model .............................................. H-6
Adjusting Local Standards: Sample Scenarios ........................................................................ H-7
Reflection on Success of Past Efforts ..................................................................................... H-8
Variations on a Theme ............................................................................................................ H-9
State Worksheet: Planning for Rewards and Sanctions ....................................................... H-10
Data Carousel ..................................................................................................................... H-11a
Monitoring Performance Using Indicators of Program Quality ............................................... H-12
Steps and Guidelines for Monitoring Local Programs ............................................................ H-13
Planning and Implementing Program Improvement ............................................................. H-14a
Aha! Experiences and Parking Lot Issues ............................................................................. H-15
Workshop Evaluation Form ................................................................................................. H-16a
PowerPoint Slides ..................................................................................................................... 1
Supplement
Possible Questions to Ask When Examining the Data (Answers to H-5) ................................. S-1
Glossary................................................................................................................................ S-2a
Letter to Send to Participants Prior to Training ........................................................................ S-3
Alternative Monitoring Exercise ............................................................................................. S-4a
ii
INTRODUCTION TO PROFESSIONAL DEVELOPMENT ON
NRS DATA MONITORING FOR PROGRAM IMPROVEMENT
Audience
This professional development sequence has three distinct audiences:
1. State-level staff (administrators, professional development coordinators, and data
managers) who are responsible for statewide planning, management, and dissemination of
information and procedures related to the NRS;
2. Professional development specialists who are responsible for rolling out the training
statewide to local programs; and
3. Local program administrators, professional development coordinators, data managers, and
instructors.
Purpose
The purpose of this professional development sequence is to help state and local literacy
program personnel identify and define the interrelationships between data and program
performance, explore ways to monitor local programs to strengthen the connection between
performance and data, and identify and implement program improvement efforts.
This sequence can be adapted to meet the needs of individual states and local programs. It can
be used as a train-the-trainers program in which state-level staff offer the workshop to key
personnel (e.g., professional development specialists and data facilitators) who will then repeat
the training for local program administrative and instructional staff. Information and activities
from this training also can be selected and offered to meet the needs of specific audiences. For
example, professional development for instructors may focus on using data to inform instruction.
Professional development for local program administrators may focus on using data for
enhancing performance or for making program improvements. Users of this training sequence
are encouraged to adapt and augment activities accordingly.
iii
Workshop Overview
Objectives:
By the end of this professional development sequence, participants will be able to:
1. Describe the importance of getting involved with and using data;
2. Identify four models for setting performance standards as well as the policy strategies,
advantages, and disadvantages of each model;
3. Determine when and how to adjust standards for local conditions;
4. Set policy for rewards and sanctions for local programs;
5. Identify programmatic and instructional elements underlying the measures of educational
gain, NRS follow-up, enrollment, and retention;
6. Distinguish between the uses of desk reviews and on-site reviews for monitoring local
programs;
7. Identify steps for monitoring local programs;
8. Identify and apply key elements of a change model; and
9. Work with local programs to plan for and implement changes that will enhance program
performance and quality.
Time:
The total time required for this workshop is approximately 12 hours of instructional time, or 2 full
days of 6 hours each (not including lunch and break times). The 12 hours of training are further
divided into 4 discrete segments of 3 hours each. The entire sequence, therefore, may be
conducted over 2 consecutive days or delivered in individual 3-hour segments over a 2-week
period, thereby affording participants the opportunity to integrate the knowledge and skills
gained into their work activities.
Materials Checklist:
Overhead projector (for use with transparencies) OR
Laptop and LCD projector (for use with CD-ROM)
Copies of participant handouts for each participant
PowerPoint presentation on CD-ROM or transparencies of PowerPoint slides
Facilitator‟s Notes and Supplements
Flipchart, flipchart stand, and marking pens
Blank transparencies and transparency pens
iv
Preparation Checklist
Reserve space for the training.
Duplicate handouts.
Download PowerPoint from the NRS Web site <www.nrsweb.org> and create a CD-
ROM for use during the workshop, or make overhead transparencies from the
PowerPoint slides. Run copies of the PowerPoint slides as handouts, 3 to a page.
Assemble participant packets with copies of handouts and PowerPoint slides.
Make nametags and/or name tents for participants.
Prepare attendance sheet.
Pre-divide attendees into groups (state teams or local program teams or instructor
teams, depending on the audience) for small group activities.
Arrange for food and beverages, as appropriate.
Arrive 1/2 hour before training is scheduled to begin.
Check equipment to ensure that it is working properly.
Pre-label flipchart pages, one of the following headings per page:
Expectations—Setting Performance Standards
Expectations—Monitoring
Parking Lot Issues
The Motivation Continuum—5 or 6 pages labeled with this same heading and
with an arrow leading from Intrinsic to Extrinsic as in the following:
Intrinsic Extrinsic
Matrix of States’ Preferred Standard-setting Models
Reward Structures
Suggested Sanctions
May Produce Unintended Effects
Questions to Ask Local Program About Educational Gain
Questions to Ask Local Program About NRS Follow-up
Questions to Ask Local Program About Retention
Questions to Ask Local Program About Enrollment
Needs/Resources
Participant Feedback
This flipchart should have two columns on the page, one labeled pluses (+), and
one labeled deltas ().
v
WORKSHOP OUTLINE
Materials Activities Times
DAY 1
PPT-1—PPT-5 I. Welcome, Introduction, Objectives, Agenda Review 50 minutes
Flipchart page for Expectations— A. Welcome and Introductions
Setting Performance Standards
B. Professional Development Objectives/Agenda/
Flipchart page for Expectations—
Expectations
Monitoring
C. Parking Lot Issues
Flipchart page for Parking Lot Issues
D. Terms and Definitions
S-2a, b, and c; H-16
E. Workshop Evaluation Form
PPT-6; PPT-7 II. The Power of Data
Flipchart page for The Motivation A. Why Get Engaged with Data? 30 minutes
Continuum
H-3; H-4
BREAK 15 minutes
PPT-9—PPT-24 B. The Data-driven Program Improvement Model 25 minutes
H-5; H-6 C. Setting Performance Standards for Program Quality 60 minutes
S-1 (answers to H-5)
Flipchart page for Matrix of States’
Preferred Standard-setting Models
LUNCH 60 minutes
PPT-25—PPT-33; H-7—H-10 D. Adjusting Standards for Local Conditions 30 minutes
Flipchart page for Reward Structures E. Shared Accountability with Appropriate Rewards and 60 minutes
Flipchart page for Suggested Sanctions
Sanctions
Flipchart page for May Produce
Unintended Effects
BREAK 15 minutes
PPT-34—PPT-40 III. Getting Under the Data: Performance Measures and 60 minutes
H-11a, b, c, and d Program Processes
4 flipchart pages
(1) Questions to Ask Local Program
about Educational Gain
(2) Questions to Ask Local Program
about NRS Follow-up Measures
(3) Questions to Ask Local Program
about Retention
(4) Questions to Ask Local Program
about Enrollment
Flipchart page for Parking Lot Issues IV. Day 1 Evaluation and Wrap-up 15 minutes
Flipchart page of pluses and deltas,
+ and
vi
WORKSHOP OUTLINE (CONTINUED)
Materials Activities Times
DAY 2
Flipchart page for Parking Lot Issues V. Agenda Review for Day 2 30 minutes
PPT-4; PPT-5
VI. Planning for and Implementing Program Monitoring
PPT-41—PPT-54 A. Presentation and Discussion 20 minutes
PPT-55; H-12 B. Small Group Work on Data Sources 60 minutes
BREAK 15 minutes
C. Small Group Reports 30 minutes
PPT-56; H-13 D. Steps and Guidelines for Monitoring Local Programs 25 minutes
LUNCH 60 minutes
VII. Planning for and Implementing Program Improvement
PPT-57—PPT-66 A. A Model of the Program Improvement Process 20 minutes
H-14 B. Bringing it Home: The Start of a State Action Plan 60 minutes
BREAK 15 minutes
C. Sharing Action Plans 45 minutes
Flipchart page for Needs/Resources VIII. Closing and Evaluation 30 minutes
H-16a, b, and c A. Review Parking Lot Issues
B. Identify Additional Resources
C. Reflection
D. Workshop Evaluation
vii
BEFORE THE WORKSHOP
The following tasks should be completed before the workshop:
Send out flyers announcing the workshop and the dates.
Send out confirmation letters to those who have registered for the workshop. Tell
them that, in preparation for the workshop, they should meet with other persons
(from their state or local program) who also will be attending the workshop. Ask
them to come prepared to give a 5-minute report that highlights their state or
program data. (See S-2a and b for sample letter.)
Duplicate all handouts for the session (H-1 through H-16) and arrange them into
participant packets. By providing a packet of materials to each participant, you can
avoid constant distribution and handling of materials during the workshop.
Download the PowerPoint slides from the NRS Web site (www.nrsweb.org) and
create a CD-ROM for use during the workshop or make overhead transparencies
from the PowerPoint slides (PPT-1 through PPT-69).
Pre-label flipchart pages for activities, as indicated in the Preparation Checklist and
in the Facilitator‟s Notes.
Order all equipment (overhead projector, screen, flipcharts). If you plan to use a
CD-ROM instead of overhead transparencies, be sure that you will have a laptop
computer and LCD projector available for the session. Check the equipment to
ensure that it is working properly. Also check the size of the screen and the clarity of
print from the back of the room.
Prepare nametags or name tents for participants.
Make signs or folded cards for each group (state names if this session is for
national training, local program names if this is for state training, class names [e.g.,
ESL, ABE, GED, Workplace Literacy, Family Literacy, EL Civics] if this is for
instructor training).
Arrange for a place to hold the workshop session and ensure that it has sufficient
space and moveable chairs for break-out activities. Consider the room arrangement
that will best facilitate your activities. For this workshop, it is suggested that, if
possible, the room arrangement consist of table rounds that each seat from 5 to 8
persons.
Prepare a participant sign-in sheet to verify attendance. Include spaces for
participants‟ names, program names, addresses, phone and fax numbers, and e-mail
addresses. This will be useful if you need to make future contact with participants.
Arrange for refreshments and lunch, as appropriate.
Read the Facilitator’s Notes for the workshop, pages 1-12.
Review the handouts (H-1 through H-16), the PowerPoint slides (PPT-1 through
PPT-70), and the Supplements (S-1 through S-3).
viii
FACILITATOR’S NOTES
FACILITATOR’S NOTES: DAY 1
Materials Activities Times
I. Welcome, Introduction, Objectives, Agenda Review for Day 1 50 minutes
A. Welcome and Introduction (30 min.)
PPT-1 Welcome participants to this professional development workshop on NRS Data
Monitoring for Program Improvement (PPT-1). Have each of the facilitators
introduce themselves and make a brief statement about their background and
expertise in either professional development or the management of data collection
and reporting. Then ask participants to introduce themselves. If participants are
few in number, they can introduce themselves one by one to the large group,
stating their names, programs, and positions. Move the activity along, allowing
each person to speak for only a minute. If the group is large, ask participants to
pair up and share background information (name, program, position). As an
optional activity, ask to see a show of hands for those who are local program
directors, instructors, and professional development coordinators. Ask whether
there are other roles represented among the group and what those roles are.
B. Professional Development Objectives/Agenda (15 min.)
PPT-2 Show participants PPT-2 and outline for them the workshop objectives for Day 1.
PPT-3 Then show PPT-3, Workshop Agenda. Quickly summarize the activities that will be
part of this workshop and state their relationship to the expected outcomes. Also
PPT-4 show them PPT-4 and PPT-5, the objectives and agenda for Day 2, but do not
PPT-5 spend as much time on Day 2 objectives and agenda as you did on Day 1. You will
show these again on Day 2; the reason for showing them now is to give
participants a sense of the objectives of the full 2-day workshop.
Flipchart page titled Ask participants to consider one question they want answered about setting
Expectations—Setting performance standards and one question they want answered about monitoring
Performance Standards before this workshop is over. After about 5 minutes, sample responses from the
Flipchart page titled group and record them on the flipcharts. Continue listing expectations until there
Expectations—Monitoring are no more responses.
NOTE: It is not necessary that every participant respond to this question; it’s likely
that some participants will have expectations that have already been listed.
Refer to the flipchart list and identify for participants those topics that have been
planned for in this workshop, those that have not been planned for but that can be
addressed easily during the workshop, and those, if any, that are outside the realm
of this workshop. To the extent possible, identify resources that participants can
access for information about those content issues that will not be covered in this
workshop.
C. Parking Lot Issues (2 min.)
Flipchart page titled Tell participants that they will keep a “Parking Lot” of issues and questions that arise
Parking Lot Issues that are related to the NRS and assessment but not directly related to this workshop on
NRS data monitoring for program improvement. To the extent possible, those questions
and issues will be addressed at the end of this workshop.
1
Materials Activities Times
H-15 NOTE: Post on the wall a flipchart page marked “Parking Lot Issues.” Also place a
Post-It pad on each table. Ask participants, throughout the workshop, to write their
questions on the Post-It Notes and place them on the flipchart. They may also use
H-15 to keep notes of their issues for the parking lot as well as to take notes of any
revelations they have, or any light bulbs that go off for them during the workshop.
D. Terms and Definitions (2 min.)
Point out that this professional development sequence, at times, uses technical
S-2a, b, and c terms common to the National Reporting System. A Glossary (S-2a, b, and c) of
these terms has been provided to make the reading as clear as possible. The
Glossary can be found in the Supplement.
E. Evaluation Form (1 min.)
H-16 Call participants’ attention to the evaluation form H-16. Remind them that they will
be asked to complete the evaluation form at the end of the workshop.
II. The Power of Data
A. Why Get Engaged with Data? [Warm-up Exercise] 30 minutes
PPT-6, PPT-7 Divide participants into teams of 3 to 5 people. Show PPT-6 and PPT-7 and ask
Post-It Notes each team to consider the question, “Why is it important to be able to produce
evidence of what your state (or local) adult education program achieves for its
students?” Provide a supply of Post-It Notes to each team and ask the team to
record one reason that it identifies on each of the Post-It Notes. Allow (10 min.)
approximately 10 minutes for this activity.
Now show PPT-8. Refer to H-3 and to the large wall charts titled The Motivation
PPT-8, H-3
Continuum. Ask each team to arrange the factors they have identified on one of
5-6 flipchart pages of H-3, the large wall charts, ranging in order from those factors that are internally
The Motivation motivated to those factors that are driven by external forces. (5 min.)
Continuum, posted about
Ask each team to briefly report to the whole group any problems or questions they
the room
had to resolve or consider before placing one or more factors on the Motivation
Continuum. Allow time for discussion. (10 min.)
H-3 Refer to H-3 again and ask participants individually to record on the arrow those
factors that they believe are most meaningful. Tell them that these are the factors
H-4 that are their own personal motivators for getting engaged with data. Refer them to
H-4 and ask them to complete the sentence, “I can be motivated to work with our
data if I remember that…” Point out that, when state or local program team
members share their motivating factors with one another, this can be a powerful,
unifying activity for the team in determining next steps in setting policy, monitoring
programs, and initiating program improvement efforts. (5 min.)
BREAK 15 minutes
2
Materials Activities Times
B. Overview of the Data-driven Program Improvement Model 25 minutes
PPT-9 Tell participants that the relationship between data and program quality is dynamic
PPT-10 because data, in the form of performance standards and other goals, not only
measure program performance but can change it as well. Referring to PPT-9, -10,
PPT-11 and -11, describe the steps of a model for data-driven program improvement. The
process begins with the setting of standards that define acceptable levels of
performance. Underlying the performance measures (or data) are the Powerful
Ps, or the program elements of policies, procedures, processes, and products. It
is these program elements underlying the data that can be observed and (5 min.)
monitored with the aim of improving performance. State and local teams, acting
collaboratively, can plan and implement program improvements by making
changes to policies, procedures, processes, and products.
PPT-12 Now show PPT-12, which displays educational gains for ESL levels and
performance standards for one adult education program. Point out that the
program exceeded its targets for three of the ESL levels, failed to meet them for
two levels, and did not serve any students at the high-advanced ESL level. Ask
H-5 participants, working in pairs, to use H-5 to identify questions that they, as program
monitors, would want to ask of local program staff. Allow 5 minutes for this activity (15 min.)
PPT-13 and then sample responses from the whole group. Accept all answers. After
Supplemental Handout, responses from the group seem exhausted, show PPT-13 and refer participants to
S-1 the Supplemental Handout, S-1, which lists possible questions and invite them to
review this list to see if it includes their questions. Tell them that this list
represents only a subset of all possible questions that a visiting team might ask in
monitoring program data and performance.
Tell participants that we now are ready to examine more closely the 4 areas of this
workshop, namely, (1) setting performance standards, (2) examining the elements
underlying the data, (3) program monitoring, and (4) program change. Before (5 min.)
proceeding, ask if participants have any questions or comments. After responding
to any questions, ask participants to get ready to begin their journey into
PPT-14 harnessing the power of data… (Show PPT-14.)
C. Setting Performance Standards for Program Quality 60 minutes
PPT-15 Show PPT-15 and tell participants that an accountability system can measure (20 min.)
quality accurately only when it contains the following four elements:
An underlying set of goals that the program is to achieve;
A common set of measures (either qualitative or quantitative) that reflect
the goals;
Performance standards tied to the measures; and
Sanctions or rewards for programs, tied to performance.
PPT-16 Now show PPT-16 and explain that the goals of the federally funded adult
education program (e.g., literacy skills development, lifelong learning, employment)
are reflected in the NRS core outcome measures of
educational gain,
GED credential attainment,
entry into postsecondary education, and
employment.
3
Materials Activities Times
PPT-17 Show PPT-17, while reminding participants that each state sets its performance
standards in collaboration with the U.S. Department of Education. However, a
state’s performance is a reflection of the aggregate performance of all the
States’ prior programs it funds. At this point, distribute to each state team a copy of its prior
performance performance. Be sure that you give each state only its own data, and not data
from any other state or program. Suggest that each state team review and
consider its negotiated performance standards throughout this workshop. Tell them
that states may soon be required to set standards for their local programs and to
monitor local programs with the aim of meeting or exceeding its performance
standards and improving program quality.
PPT-18 through PPT-22 Show PPT-18 through PPT-22, which outline four different models for setting
performance standards (continuous improvement, relative ranking, external criteria,
and return on investment). Discuss each slide, inviting comments from
participants. The slides are self-explanatory; however, it is not advisable that the
facilitator simply read the slides. Participants will be more engaged in the
presentation if you elaborate on some of the slides with comments and anecdotes
and also invite comments from participants. Survey the group to determine
whether any state currently uses a specific standard-setting model for local
programs and ask for comments on the successes and the challenges faced in the
standards-setting process.
PPT-23 and PPT-24 Now show PPT-23 and PPT-24 and refer them to H-6. Ask participants, in their
state teams, to consider the questions on H-6. Allow 20 minutes for this. (20 min.)
H-6
Then ask each state to report on the performance standard-setting model(s) it is
Flipchart page for leaning toward and to state the reason for this decision. Record the states’
Matrix of States’ responses by checking the selected model(s) on a flipchart matrix similar to the
Preferred Standard- following and post the chart on a wall for the duration of the workshop. Ensure that
setting Models participants know that they are not committing to using this performance standard-
setting model; that this activity is just for the purpose of getting them familiar with
the different models and their uses. (15 min.)
Matrix of States’ Preferred Standard-setting Models
State Continuous Relative External Return on
Improvement Ranking Criteria Investment
Ask participants in what way the standard-setting model(s) they selected represent
a policy statement on the relationship between performance and quality that they
want to instill in local programs. (Answer: Continuous improvement model means
the state wants every program to improve; relative ranking model implies a more
uniform level of quality across programs, etc.) Ask whether there were any AHAs!
or light bulbs that went off during this process. (5 min.)
Conduct a brief discussion. Tell participants that, after lunch, they will look
at ways to adjust standards for local conditions and they will set rewards and
sanctions.
4
Materials Activities Times
LUNCH 60 minutes
D. Adjusting Standards for Local Conditions 30 minutes
Welcome participants back following the lunch break. Tell them that they now need
to consider whether the standard-setting models they have selected will have the (10 min.)
PPT-25 desired effect on all programs. Show PPT-25 and tell them that research on the
effective use of performance standards suggests that standards often need to be
adjusted for local conditions before locals can work to improve program quality.
Ask if anyone can tell you why this is so. (Answer: Standards that are set at the
wrong level will not work—they will be either too easy or too difficult for the program
to meet, and they will not result in program improvement.)
PPT-26 Show PPT-26 and explain that there are three main factors that affect program
performance and that may require them to adjust standards for local conditions.
These factors are:
student characteristics,
local program elements, and
external conditions.
For example, the state standard may be too high for the local program that
serves predominantly lower-level students, or
experiences a sudden influx of refugees, or
sees a dramatic increase in student enrollment when the community’s
largest employer closes its doors and moves out-of-state.
The state may find it helpful to adjust literacy standards for the program that
emphasizes special content, such as workplace skills. Likewise, programs in areas
of high unemployment may need to have lowered standards for “entered and
retained employment.” And when natural disasters affect student attendance or
availability of services, standards may need to be adjusted.
H-7 Now refer participants to H-7, sample scenarios. Tell them that each scenario (20 min.)
represents a local program’s claim that it cannot meet the state-set performance
standards. Divide the group into teams of 4-5 people and assign one scenario to
each team. Ask each team to consider its scenario and to propose (1) a strategy
for verifying the accuracy of the local program’s claim, and (2) a suggested solution
or way to respond to the local program. Have each team select a recorder and a
reporter. Allow 10 minutes for the team work and then ask each team to report to
the whole group its strategy and suggested solution for its assigned scenario.
E. Shared Accountability with Appropriate Rewards and Sanctions 60 minutes
Tell participants that the section of the training that they are now beginning may be
one of the most critical in terms of the ultimate success (or failure) of their program
improvement and reform efforts, because, without local involvement and
cooperation, every initiative launched by the state will be met with resistance and
will be doomed to failure.
H-8 Refer state teams to H-8 and ask them to consider past efforts that their state has (15 min.)
initiated and to identify those that have been successful and those that the locals
resisted. Ask if they can identify elements that led to the success or failure of these
initiatives. Allow 10 minutes for the state teams to work and then sample
responses from the total group. It is not necessary for each state team to report
here.
5
Materials Activities Times
PPT-27 through PPT-30 Show PPT- 27 through PPT-30 on Shared Accountability. The table on PPT-30 has a (10 min.)
horizontal and a vertical dimension (or axis), each indicating movement from low to
high. The horizontal axis represents state control and the vertical axis represents local
involvement. The table cells representing the intersection of state guidance and support
with local involvement show possible effects. In other words, low state control coupled
with low degree of local involvement will likely result in stagnation; and high state control
coupled with low degree of local involvement will likely result in local animosity and
resistance, etc. If state control is delivered in the spirit of providing guidance and
support, and if local involvement means that locals truly have a hand in identifying and
designing program improvement efforts, then the local program will yield ever-increasing
quality performance. In this scenario, everybody wins: students achieve their goals,
programs win recognition and increased funding, and states increase overall
performance.
PPT-31 Tell participants that states must consider the use of appropriate rewards and sanctions
for local programs. Show PPT-31 and ask them which they think is the more powerful
motivator—rewards or sanctions? (Answer: Research clearly indicates that rewards are
more effective than sanctions in promoting program improvement.) Ask them how
sanctioning might be counter-productive? (Answer: The pressure created from
sanctions such as partial loss of funding may prompt undesirable behavior by locals,
such as limiting enrollment to higher-level students or placing students in inappropriately
low levels.) Explain that such questionable tactics designed to yield high performance
results are what is known as unintended effects—that is, the action does not benefit
students; it is put in place for the sole purpose of avoiding harsh sanctions on the
program.)
H-9 Refer participants to H-9, Variations on a Theme, and show PPT-32. Ask them to work
PPT-32 in groups of 3 or 4 to brainstorm possible reward structures and possible sanctions for
local programs that meet or fail to meet their performance standards. Each group
should select a recorder and a reporter. Ensure that there is an ample supply of Post-It
Notes and marking pens on each table. Instructions are for the recorder to write one
response per Post-It note, using one color Post-It Note for the reward structures and a
different color for the suggested sanctions. Allow 10 to 15 minutes for this activity and
monitor the groups to determine when to call “Time.” (15 min.)
Invite one group to read one of its rewards Post-It Notes and to place the note on the
Flipchart (Reward
Reward Structures flipchart. Ask other groups if they also had “variations on the theme”
Structures)
(e.g., monetary rewards) and to post these on the Reward Structures flipchart in a
column under the first note. When all groups have posted their rewards related to
monetary incentives, invite another group to read one of its remaining rewards (e.g.,
Published Honor Roll of programs that met or exceeded its performance standards) and
repeat the process. Continue until there are no remaining rewards to be posted. (15 min.)
Flipchart (Suggested Then repeat the process for the sanctions, inviting groups to place their Post-It Notes on
Sanctions) the Suggested Sanctions flipchart. For each sanction that is read, ask the whole group
to serve as an applause-o-meter, clapping for the gentle sanctions and gonging for
Flipchart (May Produce
sanctions that may be too harsh. Place those sanctions that the group considers too
Unintended Effects)
harsh on the flipchart labeled May Produce Unintended Effects.
(5 min.)
H-10 Conduct a discussion of the various sanctions and rewards and tell each state team
that, following the break, it is to complete the planning worksheet on H-10 in which the
state makes preliminary decisions about the reward structures and sanctions it will put
in place and identifies stakeholders it will include as well as the process for making final
decisions on rewards and sanctions and turning the decisions into policy.
6
Materials Activities Times
BREAK 15 minutes
H-10 Allow 15 minutes for state teams to complete the activity described before the 20 minutes
break, using H-10. Then ask if any team wants to share its strategies and perhaps
help light bulbs to go off for other states.
PPT-33 In summary, show PPT-33, telling participants that the state process of setting 5 minutes
local performance standards consists of the following five steps:
1. Select standard-setting model;
2. Set rewards and sanctions policy;
3. Review performance levels for local adjustment;
4. Provide technical assistance to locals in an atmosphere of shared
accountability; and
5. Monitor performance often.
III. Getting Under the Data: Performance Measures and Program Processes 60 minutes
PPT-34, PPT-35 Show PPT-34 and PPT-35. Tell participants that, for the purpose of this workshop,
we will consider four sets of measures:
1. Educational gain;
2. The NRS follow-up measures of obtained a secondary credential,
entered and retained employment, and entered postsecondary
education;
3. Retention; and
4. Enrollment.
PPT-36 through PPT-39 Under each set of measures lie the programmatic and instructional decisions and
PPT-40 procedures that affect program performance and quality. Show the corresponding
data pyramids on PPT-36 through PPT-39.
4 flipchart pages posted
about the room:
(1) Questions to Ask Local Post the flipchart pages indicated at the left at various places in the room. Next to
Program about each flipchart, post a copy of one of the data displays found on H-11 (a through d).
Educational Gain Now show PPT-40 and refer participants to the Data Carousel activity (H-11a, b, c,
(2) Questions to Ask Local d, and e). Have participants, in groups of 4 or 5, visit each of the carousel displays
Program about NRS and develop questions about the data and the underlying elements. Each group
Follow-up Measures should identify as many underlying elements as possible that affect this data and
the program’s performance and write their questions on the flipchart provided.
(3) Questions to Ask Local
Then each group should make recommendations for improving the program’s
Program about
performance. Inform participants that they should spend approximately 15
Retention
minutes at each display (for a total of 1 hour) and that each group should be
(4) Questions to Ask Local prepared in the morning to report on their findings. As each group rotates on the
Program about carousel to different stops in the room, it should read the questions generated by
Enrollment other groups and add their own questions. Note: Keep these carousel stops with
Next to each flipchart, post the questions posted for the duration of the workshop.
one of the data displays
from H-11a, b, c, d, and e.
7
Materials Activities Times
IV. Day 1 Evaluation and Wrap-up 15 minutes
Tell participants that they have now reached the mid-point in the workshop
sequence on NRS Data Monitoring for Program Improvement. Review for them the
content that they covered and the activities that they engaged in during this first
day of the workshop on getting to understand and appreciate the value of data:
Warming up to data through the Why Get Engaged with Data Exercise;
Selecting standard-setting models (from the four models of continuous
improvement, relative ranking, external criteria, and return on investment;
Adjusting performance levels to meet local circumstances of student
characteristics, local program elements, and external conditions;
Determining ways to share accountability with locals;
Setting policy for rewards and sanctions; and
Examining the programmatic and instructional decisions and procedures
underlying the data.
Flipchart (Parking Lot Ask if there are any questions about the day’s workshop. Respond to questions
Issues) that you can answer on the spot. If there are questions that will take some
research before you can answer or policy questions that you must refer to another
source, be sure to add the questions to the “Parking Lot Issues.” These will be
addressed at the end of the workshop sequence.
Flipchart Now tell participants that you would like to “take the temperature” of the group
page of pluses and deltas, concerning Day 1 activities and content by doing an informal pluses-and-deltas
+ and exercise. On a flipchart page, make a page of two columns, one with a plus sign
[+] and one with a delta []. Ask them to call out those things that they liked about
today’s workshop. Accept all comments and write them under the [+] column.
When there are no more responses, ask them to identify those things that they felt
could have been improved about today’s workshop. Again, accept all comments
and write them under the [] column. Tell them that you appreciate and take their
comments seriously and that, to the extent possible, you will attempt to address
those items in the [] column that are under your control throughout the remainder
of the workshop sequence.
Thank them for their participation and enthusiasm and tell them that you look
forward to seeing them tomorrow (or at the next scheduled workshop) and give
them the date and location for the next workshop.
8
FACILITATOR’S NOTES: DAY 2
Materials Activities Times
V. Agenda Review for Day 2 30 minutes
Flipchart (Parking Lot Welcome participants back and ask if there are any questions or residual issues (5 min.)
Issues) from Day 1 for which participants would like clarification before moving on to Day
2 activities. Respond to those questions and issues that relate to the content of
this workshop on NRS Data Monitoring for Program Improvement. Use the
“Parking Lot Issues” page to list those issues that are outside the realm of this
workshop. Tell participants that the parking lot issues will be addressed at the end
of this workshop.
PPT-4 Show PPT-4, Workshop Objectives for Day 2. Tell them that this handout reviews (5 min.)
the same objectives that were stated at the beginning of this workshop sequence,
but that you are specifically highlighting the objectives for today’s activities. Also
PPT- 5 show PPT-5, Workshop Agenda for Day 2. Quickly summarize the activities that
will be part of this workshop and state their relationship to the expected
outcomes.
Ask for feedback on yesterday’s Carousel Activity. Have one team report its (20 min.)
findings and invite other teams to comment or to add questions about the data
and suggestions for program improvement. Continue until each of the four
Carousel stops has been discussed, approximately 5 minutes per stop. Now tell
participants that all they have accomplished so far in this workshop will serve as
background information for the next important responsibility of states: local
program monitoring.
VI. Planning for and Implementing Program Monitoring
A. Presentation and Discussion 20 minutes
PPT-41 Show PPT-41 and ask participants how/why they think local programs might
PPT-42 benefit from a state’s policy of conducting regular monitoring of local program
performance and measuring it against performance standards. Sample
responses and then show PPT-42, discussing each of the points. Encourage
PPT-43 participants to share stories/experiences they have had related to these points in
conducting local monitoring. Show PPT-43 and PPT-44. Tell them that you
PPT-44
understand that the idea of monitoring may seem overwhelming to states with
already overburdened staff, but that monitoring can be manageable and can
provide an excellent opportunity to work with locals on program improvement.
PPT-45 When showing PPT-45, ask whether any states currently include in their
monitoring visits a discussion of whether locals are meeting performance
standards and whether the monitoring is structured to encourage program
improvement. If no states currently measure performance against performance
standards, ask whether and how they currently monitor any aspect of local
programs. Sample responses, and then ask whether adding the process of
measuring performance against performance standards can be fit into their
existing monitoring structures.
NOTE: Be prepared to hear that some states conduct no monitoring at all.
Accept all responses without making judgmental statements. The purpose of this
workshop is to provide tools for states to collaboratively set performance
standards with locals, to measure local performance against performance
standards, and to plan and initiate programmatic improvement.
9
Materials Activities Times
PPT-46 Show PPT-46 and discuss the difference between desk reviews and on-site
PPT-47, PPT-48 monitoring, and then the advantages/disadvantages of each (PPT-47 and PPT-
48). Then, showing PPT-49 through PPT-54, discuss the various data collection
PPT-49 through PPT-54 strategies for monitoring (Program Self-Reviews, Document Reviews,
PPT-52 Observations, and Interviews). PPT-52 shows the difference between
quantitative (collected primarily via desk reviews) and qualitative data (collected
primarily via on-site reviews).
B. Small Group Work on Data Sources 60 minutes
PPT-55 Now show PPT-55 and ask participants, in groups of 4-5, to review H-12 and to
H-12 fill in the data sources that they would use for each indicator and the strategies
they would use if they were conducting (1) a desk review, and (2) an on-site
review. Ask all groups to consider all nine indicators, but assign each group only
one or two indicators to report on to the whole group. Ask each group to select a
recorder and a reporter. Allow 1 hour for this activity before reconvening the
group for the reports. Tell them that they also have the scheduled break time (an
additional 15 minutes) to use if they need it. Each group will have 2 minutes to
report on one indicator (or, if the groups each have been assigned two indicators
to report on, they will have a total of 4 minutes to report).
BREAK 15 minutes
C. Small Group Reports
Ask each group to report on its indicator(s) and the data sources and monitoring 30 minutes
strategies for on-site and desk reviews. Allow only 2 minutes per report on each
indicator. Following the report on each indicator, ask the total group to comment
or, if they wish, suggest additional NRS data sources and other vehicles for
conducting the desk and on-site reviews.
D. Steps and Guidelines for Monitoring Local Programs 25 minutes
PPT-56 Show PPT-56 and conclude this section on Planning and Implementing Program
Monitoring with a review of steps and guidelines for monitoring local programs, H-
H-13 13. The steps are as follows:
1. Identify state policy for monitoring and gather support from those who
have a stake in the monitoring results;
2. Consider past practices when specifying scope of work for monitoring;
3. Identify persons to lead and participate in monitoring;
4. Identify resources available for monitoring locals;
5. Determine process for collecting data with clearly defined criteria for
rating;
6. Conduct monitoring;
7. Report findings and recommendations; and
8. Follow-up on results.
NRS Data Monitoring for More information about each of these steps can be found on pages 50 through 54 of
Program Improvement the guide, NRS Data Monitoring for Program Improvement (2004). Point out other
Guide, pp. 50-55 states’ models and procedures for monitoring (e.g., Pennsylvania, Tennessee)
beginning on page 55 in the guide.
Tell participants that, following lunch, state teams will have time to consider these
steps and the guidelines listed on H-13 and begin to plan a process for local
monitoring. Use this time before lunch to clarify any issues related to the workshop
content presented thus far and to allow participants to share their concerns about, as
well as their past experiences, in monitoring local programs. Now is also a good time
to review any of the issues or questions posted on the Parking Lot.
LUNCH 60 minutes
10
Materials Activities Times
VII. Planning for and Implementing Program Improvement
A. A Model of the Program Improvement Process 20 minutes
Tell participants that they now are entering the final phase of this workshop
series: planning and implementing a program improvement based on what they
learn through program monitoring. Remind participants that, so far, they have
reviewed strategies for setting performance standards for program quality,
adjusting standards for local conditions, and setting policy for appropriate rewards
and sanctions. They also have examined the programmatic components and
policy decisions underlying the measures of educational gain, NRS follow-up,
retention, and enrollment, and they have considered strategies for conducting
monitoring through both desk reviews and on-site reviews, as well as the
stakeholders they need to include in setting policy related to data monitoring.
PPT-57 Show PPT- 57 and remind participants once again that data can be of
considerable use to state and local programs. However, as indicated in PPT-58,
PPT-58 data are useful only if the data are valid and reliable, if the state and locals ask
appropriate questions after reviewing the data, and if data analysis leads to
making wise decisions.
PPT-59 Before moving on, it would be wise to remind participants about the change
PPT-60 process—that it is a process, not an event. This means that change does not
happen overnight. Show PPT-59, the factors that allow us to accept change;
PPT-61 PPT-60, the stages of change; and PPT-61, a word of caution from the State
Superintendent of Schools in Spokane, Washington.
PPT-62 Now show PPT-62 and describe the four steps of a program improvement
process:
1. Planning;
2. Implementing;
3. Evaluating; and
4. Documenting lessons learned and making adjustments, as needed.
B. Bringing it Home: The Start of a State Action Plan
60 minutes
PPT-63 through PPT-66 Show PPT-63 through PPT-66 and refer participants to H-14a and b. Ask them to
work in their state teams to consider the questions on H-14a and b in beginning
H-14a and b to plan a model for monitoring local programs. Tell them that they will have 1
hour for planning. They also may use the 15 minutes of scheduled break time, if
they need it. When the group reconvenes following the break, each state is to
report on the team’s plans as well as the potential problems it anticipates and the
strategies it plans to use to mitigate potential problems. They are to be prepared
to make 5-minute reports on their planned changes to the whole group.
BREAK 15 minutes
C. Sharing Action Plans
Ask representatives from each team to report on the team’s plans for 45 minutes
implementing monitoring and program improvement processes. Allow 5 minutes
for each report and encourage questions from the other teams. After all teams
have reported their plans, ask if anyone would like to make any general
observations about the reports—for example, Do states anticipate similar
obstacles in setting policy for implementing a program monitoring and
improvement process? Have states come up with vastly different models for
involving local programs? Ask participants if the various state reports have
generated ideas that they can use. Ask whether state teams will modify their
plans based on the reports made by other states.
11
Materials Activities Times
X. Closing and Evaluation 30 minutes
A. Review Parking Lot Issues
Collect the flipchart pages with the Parking Lot Issues that were posted at the
beginning of the workshop. Review lists to determine if these questions have
been answered during the workshop. Provide answers to unanswered questions
or, if the questions need to be referred to others or if they need research, give
participants an approximate date by which they can expect to receive either the
answers or referrals to other information sources. Also ask if participants have
any questions/items/issues that still need to be clarified.
B. Identify Additional Resources
Flipchart Mark a flipchart page “Needs/Resources.” Ask participants to name additional
(Needs/Resources) resources that they need to implement the changes they identified in their action
plans. It could be additional training, online resources, policy issues and
changes, etc. The purpose of this activity is to provide information to the drivers of
change and of policy at the federal and state levels to help them in their planning.
Tell participants that their brainstormed list of needs and additional resources will
be compiled and mailed to all workshop participants. Allow approximately 20
minutes for this activity.
C. Reflection
Provide closure to the workshop by asking participants to reflect on what they
have learned and how they can apply the information they have discussed or
acquired.
Refer participants to the workshop objectives:
Describe the importance of getting involved with and using data;
Identify four models for setting performance standards as well as the policy
strategies, advantages, and disadvantages of each model;
Determine when and how to adjust standards for local conditions;
Set policy for rewards and sanctions for local programs;
Identify programmatic and instructional elements underlying the measures of
educational gain, NRS follow-up, enrollment, and retention;
Distinguish between the uses of desk reviews and on-site reviews for
monitoring local programs;
Identify steps for monitoring local programs;
Identify and apply key elements of a change model; and
Work with local programs to plan for and implement changes that will
enhance program performance and quality.
D. Workshop Evaluation
H-16a, b, and c Direct participants’ attention to H-16a, b, and c (Workshop Evaluation). Ask
participants to complete the evaluation. Thank them for attending and
participating, and tell them that you look forward to seeing them at the next
workshop.
12
PARTICIPANT’S HANDOUTS
WORKSHOP OBJECTIVES
By the end of this professional development sequence, participants will be able to:
Day 1
1. Describe the importance of getting involved with and using data;
2. Identify four models for setting performance standards as well as the policy
strategies, advantages, and disadvantages of each model;
3. Determine when and how to adjust standards for local conditions;
4. Set policy for rewards and sanctions for local programs; and
5. Identify programmatic and instructional elements underlying the measures of
educational gain, NRS follow-up, enrollment, and retention.
Day 2
1. Distinguish between the uses of desk reviews and on-site reviews for monitoring
local programs;
2. Identify steps for monitoring local programs;
3. Identify and apply key elements of a change model; and
4. Work with local programs to plan for and implement changes that will enhance
program performance and quality.
H-1
WORKSHOP AGENDA
Day 1
I. Introductions, Objectives, Agenda Review for Day 1
II. The Power of Data
Why Get Engaged with Data? (exercise and the motivation continuum)
Overview of the Data-drive Program Improvement Model (questioning data
exercise)
Setting Performance Standards for Program Quality (presentation on four
models for setting standards and exercise)
Adjusting Standards for Local Conditions (scenarios)
Shared Accountability with Appropriate Rewards and Sanctions (variations on
a theme exercise)
III. Getting Under the Data: Performance Measures and Program Processes (data
pyramids and the underlying program components, decisions, and processes)
IV. Day 1 Evaluation and Wrap-up
Day 2
V. Agenda Review for Day 2
VI. Planning for and Implementing Program Monitoring
Differences between Desk Reviews and On-site Reviews
Data Sources for Each (exercise)
Steps and Guidelines for Local Program Monitoring
VII. Planning for and Implementing Program Improvement
The Change Process
A Model of the Program Improvement Process
State Action Planning
Sharing Reports
VIII. Closing and Evaluation
H-2
WHY GET ENGAGED WITH DATA?
Directions: Form a team of 3 to 5 people and consider the following question:
Why is it important to be able to produce evidence of what your
state (or local) adult education program achieves for its students?
Jot down as many responses as you can think of, writing one response on each Post-It Note.
Your workshop facilitator will provide you with a flipchart page titled The Motivation Continuum,
similar to the chart that appears below. When your team can think of no more responses, take
all the Post-It Notes your team has created and place them on The Motivation Continuum
flipchart, ranging in order from those factors that are internally motivated to those factors that
are driven by external forces.
The Motivation Continuum
Intrinsic Extrinsic
H-3
YOUR OWN PERSONAL MOTIVATORS
Directions: Complete the following sentence with as many things as apply to you personally.
I can be motivated to work with our data if I remember that…
1. ______________________________________________________________________
______________________________________________________________________
2. ______________________________________________________________________
______________________________________________________________________
3. ______________________________________________________________________
______________________________________________________________________
4. ______________________________________________________________________
______________________________________________________________________
5. ______________________________________________________________________
______________________________________________________________________
6. ______________________________________________________________________
______________________________________________________________________
When state or local program team members share their motivating factors with one
another, it can be a powerful, unifying activity for the team in determining next steps in
setting policy, monitoring programs, and initiating program improvement efforts.
H-4
QUESTIONS FOR CONSIDERATION
Directions: Examine the data in the following graph.
100%
91%
80%
80%
60%
50%
40% 33% 36% 34%
31% 26%
27% 26%
22%
20% Program
Performance Standards
0%
Beg. Lit Beg. Low Int. High Int. Low Adv. High Adv.
List all the questions you can think of to ask about this local program‟s data and what the
underlying reasons might be for the results.
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
H-5
DECISION FOR STATE TEAMS: SELECTING A STANDARD-SETTING MODEL
Directions: In your state teams, consider the following questions:
1. Which model do you favor for setting standards for/with local programs? Why?
_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________
2. Is it appropriate for your state to use one statewide model or will you need to use
different models for different programs?
_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________
3. How will you involve the locals in setting the standards to which they will be held
accountable?
_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________
Consider question #4, but do not include it in your state report. We will discuss this
with the entire group following the state reports.
4. How do the standard-setting models(s) that states select represent a policy statement on
the relationship between performance and quality that states want to instill in local
programs?
_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________
_________________________________________________________________________
H-6
ADJUSTING LOCAL STANDARDS: SAMPLE SCENARIOS
Directions: Read the following scenarios. Each represents a local program‟s claim that it cannot meet
the state-set performance standards. In your small group, discuss how you would handle each claim by
(1) proposing a strategy for verifying the accuracy of the claim, and (2) proposing a solution to the
problem. Be prepared to report your team‟s strategy and proposed solution to the whole group.
1. Continuous Improvement Model
Using a continuous improvement model, one state set performance standards for GED attainment for
each local program at levels slightly higher than the previous year‟s. However, in the previous year,
several local programs had received grants to offer an extensive amount of “fast-track GED” instruction
prior to the release of GED 2002 and, consequently, their secondary completion and GED rates soared.
The “fast-track” grant is now over and the programs think the current levels set by the state are too high
and should be lowered, based on levels they attained before the grant.
Your Strategy for Verifying the Accuracy of the Local Program‟s Claim:
____________________________________________________________________________________
____________________________________________________________________________________
Your Solution to the Problem and Response to the Local Program:
____________________________________________________________________________________
____________________________________________________________________________________
2. Relative Ranking Model
Another state uses a relative ranking model to set local performance standards. In reviewing its student
demographic data, one local program that fails to meet its educational gain performance standards found
that it serves a high proportion of older learners. The state average age of ABE learners is 33 years old,
but the local program‟s average student age is 49. The program requests that the state adjust standards
lower for them, based on the common belief that older learners do not make gains as quickly.
Your Strategy for Verifying the Accuracy of the Local Program‟s Claim:
____________________________________________________________________________________
____________________________________________________________________________________
Your Solution to the Problem and Response to the Local Program:
____________________________________________________________________________________
____________________________________________________________________________________
3. External Criteria Model
The state legislature requires all adult education programs to show at least a 20 percent increase in the
percentage of participants who get jobs. In response, the state increases the standard for adult education
programs by 25 percent over previous years for „entered employment.‟ Several adult education programs
claim that they cannot meet this standard because they serve significant numbers of learners who are
already working, and the number of students with the goal of „obtain employment‟ is low. If a program
does not focus on employment skills, it cannot substantially increase its „entered employment‟ rate.
Your Strategy for Verifying the Accuracy of the Local Program‟s Claim:
____________________________________________________________________________________
____________________________________________________________________________________
Your Solution to the Problem and Response to the Local Program:
____________________________________________________________________________________
____________________________________________________________________________________
H-7
REFLECTION ON SUCCESS OF PAST EFFORTS
Directions: In your state teams, consider past policy changes that your state adult education
office has initiated and asked local programs to comply with. Select one policy change that was
well received by the locals, and one policy change that was met with resistance from the locals.
Then identify some the factors that you think contributed to the success or failure of these
initiatives. You have approximately 10 minutes for this exercise. Be prepared to share your
responses with the whole group.
1. List a policy change (imposed by the state office on local programs) that was successful
and well received by local programs.
______________________________________________________________________
______________________________________________________________________
What factors contributed to the success of this effort?
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
2. List a policy change (imposed by the state office on local programs) that was not
successful and was met with resistance from local programs.
______________________________________________________________________
______________________________________________________________________
What factors contributed to the poor reception of this effort?
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
H-8
VARIATIONS ON A THEME
Directions: In a small group of 3 or 4 people, brainstorm as many possible rewards or
incentives as you can for recognizing local programs that meet their performance standards.
Write these in the left column below. Then brainstorm sanctions that the state might impose on
local programs that do not meet their performance standards. Select a recorder for your group
to write one reward per Post-It Note and one sanction per Post-It Note. When you have
finished, wait for further instructions from the facilitator.
Reward Structures Sanctions
H-9
STATE WORKSHEET: PLANNING FOR REWARDS AND SANCTIONS
Directions: In your state team, make some preliminary decisions about the rewards and
sanctions that you might use to reward and motivate local programs to meet their performance
standards. Consider the following questions.
1. What reward structures are you thinking of putting in place for local programs that meet
their performance standards?
_________________________________________________________________________
_________________________________________________________________________
2. What sanctions are you thinking of putting in place for local programs that fail to meet their
performance standards?
_________________________________________________________________________
_________________________________________________________________________
3. What timeline are you thinking of for putting the rewards and sanctions policy in place?
_________________________________________________________________________
_________________________________________________________________________
4. What stakeholders will you include in the decision-making process about rewards and
sanctions when you get back to your home state?
_________________________________________________________________________
_________________________________________________________________________
5. Who in the state office will have primary responsibility for the following:
a. Announcing the new policy _____________________________________________
b. Reviewing local programs‟ data _________________________________________
c. Determining the reward or sanction for each program ________________________
d. Providing support and technical assistance to the programs that need to improve
___________________________________________________________________
e. What will be the nature of the support and technical assistance provided by the
state?
___________________________________________________________________
___________________________________________________________________
6. What obstacles or resistance do you foresee in putting this policy in place? How might you
plan in advance to lessen these obstacles? ______________________________________
_________________________________________________________________________
7. What other factors, if any, do you need to consider that are specific to your state?
_________________________________________________________________________
_________________________________________________________________________
H-10
DATA CAROUSEL
Directions: In teams of 4-5, visit each of the four carousel stops around the room. Each of
these stops represents one of your local programs. Review the data table or chart displayed at
each stop (and also displayed on H-11b, c, d, and e). As you review the data at the first stop,
ask yourselves what, if anything, in this data is a cause for concern? What questions do you
want to ask the local program about these results? Note: Your questions should target and try
to get at the underlying elements that may be causing the performance problem(s). Write your
questions on the flipchart provided at this stop. Then proceed to the next carousel stop and
repeat the process. At each stop, you will see questions that other review teams have written
as they revolve on the data carousel. Add your questions, if they are different from the ones
that already appear on the flipchart. You may spend up to 15 minutes at each stop.
H-11a
DATA CAROUSEL—STOP #1
EDUCATIONAL GAIN
ABE LEVELS
Definition: Students who advance an NRS Level
76%
80% 69% 71%
70%
60%
Program
50%
40% 35% 35%
26%29% 29%30% 29% State
30% 22% 19% 20%19% 19%21%
20% 14%
Performance
10% NA Standards
0%
ABE Beg. Lit ABE Beg. ABE Int. Low ABE Int High ASE Low ASE High
Number Advancing Level And Total Enrolled % Completing Level
State
NRS Level Program State Performance
Program State Standards
Advanced Total Advanced Total
ABE Beg. Lit. 2 14 169 761 14% 22% 35%
ABE Beg. 9 26 339 1,284 35% 26% 29%
ABE Int. Low 6 31 589 2,060 19% 29% 30%
ABE Int. High 40 58 683 3,339 69% 20% 19%
ASE Low 35 46 385 2,044 76% 19% 21%
ASE High 24 34 307 1,062 71% 29% —
TOTAL 116 209 2,470 10,550 55% 23%
H-11b
DATA CAROUSEL—STOP #2
FOLLOW-UP
RECEIPT OF SECONDARY SCHOOL DIPLOMA OR GED (2001-2003)
Definition: The number of students who received a secondary school diploma or
GED divided by the number of students who had that as a goal.
100
80
70 70
60
60
50 Percent Achieved Goal
33 36 Performance Standard
40
20
0
2001 2002 2003
Number Achieved Percent Achieved Performance
Year Number with Goal Goal Goal Standard
2001 150 75 50% 60%
2002 120 40 33% 70%
2003 110 40 36% 70%
H-11c
DATA CAROUSEL—STOP #3
RETENTION
AVERAGE HOURS ATTENDED
Definition: Total attended hours divided by number of enrolled students
180
160
140
120
Program
100
80 State
60
Performance
40 Standards
20
0
ABE ABE ABE Int. ABE Int. ASE ASE
Beg. Lit. Beg. Low High Low High
Performance
Average Attended Hours
Standards
NRS Level
Performance
Program State
Standards
ABE Beg. Lit. 40 90 70
ABE Beg. 120 120 100
ABE Int. Low 60 120 80
ABE Int. High 161 110 100
ASE Low 50 130 60
ASE High 150 140 100
TOTAL 80 120 —
H-11d
DATA CAROUSEL—STOP #4
Enrollment of Student Sub-Populations
100
90
80
70
60
50
40
30
19 20 21 18
20
12 13 8 7
10 5 6
0
ABE Beginning ESL Beginning On Public Immigrant/Refugee Aged 16-24
Literacy Literacy Assistance
Actual % Performance Standard %
Target Population Actual Performance Standard
Number % of Total Number % of Total
ABE Beginning Literacy 80 19% 90 20%
ESL Beginning Literacy 52 12% 60 13%
On Public Assistance 90 21% 80 18%
Immigrant/Refugee 21 5% 35 8%
Aged 16-24 25 6% 30 7%
Total Enrollment in
Program 430 — 450 —
H-11e
MONITORING PERFORMANCE USING INDICATORS OF PROGRAM QUALITY
Directions: In groups of 4-5, for each indicator in the table below, fill in the data sources that you would
use as well as the questions you would ask in monitoring local programs and the strategies you would
use to conduct (1) a desk review, and (2) an on-site review. Complete all 9 indicators in the table below.
Select a recorder and a reporter for your group and be prepared to report to the whole group on the
indicator(s) assigned to your group by the facilitator. You have 1 hour for this activity, plus 15 minutes for
the scheduled break. Your group will have 2 minutes to report on each indicator assigned.
Program Area in NRS Data Questions to Strategies for Strategies for
Indicators of Source Pose of Locals Desk Review On-site
Program Quality re: this Indicator Monitoring
Program
Management
(Data Reporting)
Recruitment
Goal Setting
(Intake and
Orientation)
Educational
Gains
Assessment
Curriculum and
Instruction
Persistence
Support Services
Professional
Development
H-12
STEPS AND GUIDELINES FOR MONITORING LOCAL PROGRAMS
Monitoring Steps Implementation Guidelines Examples
1. Identify state policy for monitoring. Provide clear written guidelines to all State plan should be open to the public
Gather support from those who levels of stakeholders on the scope of the and shared at all levels. State plans often
have a stake in the results. monitoring activities (including process specify:
and timelines). Outcome measures, and
Frequency of evaluation
2. Specifying the scope of work for Uses quantitative and qualitative data for Quantitative = look at outcome
monitoring. effective monitoring. measurements
Qualitative = look for evidence using
program quality indicators
3. Identify individuals to lead and to Consider the unique program features Local staff: practitioners, administrators,
participate in monitoring activities. when identifying who should be involved partners
from the local program and who should External team members: content
be part of monitoring team. Consider specialists, other educators, and staff from
strength in diversity. partnering agencies
4. Identify resources available for With competing demands for resources Desk reviews look at program data from a
monitoring local programs. (staff, time, and money for monitoring), distance.
consider formalizing a two-stage Onsite reviews look at data in context—to
monitoring approach. see first-hand how the process and
operations lead to positive outcome
measures.
5. Determine process for collecting Create and use standard tools for data Desk Reviews can include data,
data with clearly stated criteria for collection and analysis. Monitors (state proposals, plans, reports, and program
rating. Conduct monitoring staff and team) need to fully understand self-review.
activities. the tools, their use, and the rating criteria. Onsite reviews can include discussion of
self-review, observations, interviews, and
a review of files and documents.
6. Report on the findings, including Conclude onsite monitoring visits with a Report might include a short description of
recommendations. verbal debriefing followed by a written the monitoring activities with supporting:
report. Qualitative description
Quantitative information.
7. Follow up on the results. Given that the major purpose of Follow-up activities might include
monitoring is program improvement, is reviewing performance standards and
essential, and should include an ongoing program improvement, rewarding or
exchange between the state office and sanctioning, and the beginning of technical
the local program. assistance.
H-13
PLANNING AND IMPLEMENTING PROGRAM IMPROVEMENT
Directions: With your state team members, consider the following questions and begin to plan
a program improvement policy and process. Be prepared to report on your plans to the whole
group.
1. Who should be included on your program improvement team? List the positions from
the state office as well as the local program. Anyone else, such as community
members, learners, etc.?
2. How will you prioritize areas needing improvement when you review a local program‟s
data and find several areas that may need to be addressed?
3. How will you gain cooperation from locals in this process?
4. What type of training or professional development will be needed to get local buy-in?
5. How will you identify and select strategies for effecting improvement?
6. Who will be responsible for taking the lead on ensuring that the change is implemented?
7. How will expectations for the change be promoted and nurtured?
8. How will the change be monitored?
H-14a
9. How will the changes that are implemented be evaluated?
10. Who will interpret the results?
11. Who will be on the lookout for unintended consequences?
12. Who will document the process of what worked, what didn‟t, and lessons learned?
13. What problems do you anticipate facing as you plan for and implement policy related to
data monitoring for program improvement?
14. What solutions or precautions can you suggest to avoid having these problems become
major ones?
15. What is your timeline or expected completion date for the following activities:
A. Setting performance standards?__________________________________________
B. Announcing the standards and asking locals to comply? ______________________
C. Developing a policy for adjusting standards for local conditions?________________
D. Developing a policy for rewards and sanctions?_____________________________
E. Developing a policy and process for monitoring local programs?________________
F. Developing a policy and process for effecting program improvements?___________
G. Other ______________________________________________________________
H-14b
Aha! Experiences Parking Lot Issues
H-15
NRS DATA MONITORING FOR PROGRAM IMPROVEMENT WORKSHOP
EVALUATION FORM
Date ______________________ Location of Workshop ____________________________________
State or Local Program Name ________________________________________________________________
Your Position (Check all that apply)
Instructor Local Administrator
Data Facilitator Professional Development Coordinator
State Director or State Staff Other (identify)___________________________________
1. The objectives of the NRS professional development packet were met
(not at all) 1 2 3 4 (completely)
The Power of Data
2. The “Why Get Engaged with Data?” exercise was
(not effective) 1 2 3 4 (highly effective)
3. The concepts and information presented in “The Data-driven Program Improvement Model” were
(not useful) 1 2 3 4 (highly useful)
4. The “Setting Performance Standards for Program Quality” exercise was useful in identifying problem
standard-setting models we might use in our state.
(not at all) 1 2 3 4 (extremely)
5. The concepts and information presented in “Adjusting Standards for Local Conditions” were
(not helpful) 1 2 3 4 (extremely helpful)
6. The “Shared Accountability with Appropriate Rewards and Sanctions” exercise was useful for making
decisions about the rewards and sanctions my state might put in place.
(not at all) 1 2 3 4 (extremely)
General comments about the Power of Data Section:
H-16a
Getting Under the Data: Performance Measures and Program Processes
7. The directions for the “Data Carousel” exercise were
(confusing) 1 2 3 4 (clear)
8. The concepts and information presented in the data pyramids were
(not helpful) 1 2 3 4 (extremely helpful)
General comments about the Getting Under the Data Section:
Planning for and Implementing Program Monitoring
9. The concepts and information presented in the Planning and Implementing Program Monitoring section
were
(not useful) 1 2 3 4 (very useful)
10. The small group work on Data Sources was helpful in understanding the differences in monitoring local
program using desk reviews versus on-site reviews.
(not at all) 1 2 3 4 (extremely)
General comments about the Planning for and Implementing Program Monitoring Section:
Planning for and Implementing Program Improvement
11. The concepts and information presented in the “Model for Change” presentation and discussion were
(not useful) 1 2 3 4 (very useful)
12. The state action planning was helpful in getting my state started toward data monitoring for program
improvement.
(not at all) 1 2 3 4 (extremely)
General comments about the Planning for and Implementing Program Improvement Section:
H-16b
Overall Comments
1. What were the most helpful features of the workshop? Please be specific.
2. What do you think were the least helpful features of the workshop?
3. What suggestions do you have for improving this professional development activity?
POWERPOINT SLIDES
NRS DATA MONITORING FOR PROGRAM IMPROVEMENT
(PowerPoint Slides)
___________________________________
Slide 1
___________________________________
NRS Data Monitoring for ___________________________________
Program Improvement ___________________________________
Unlocking Your Data ___________________________________
___________________________________
4/6/2004
4/6/2004 M. Corley
M. Corley 1
___________________________________
Slide 2 ___________________________________
Objectives—Day 1
1. Describe the importance of getting involved with and
___________________________________
using data;
2. Identify four models for setting performance standards
as well as the policy strategies, advantages,and
___________________________________
disadvantages of each model;
3. Determine when and how to adjust standards for local
conditions; ___________________________________
4. Set policy for rewards and sanctions for local programs;
5. Identify programmatic and instructional elements
underlying the measures of educational gain, NRS
follow-up, enrollment, and retention.
___________________________________
___________________________________
4/6/2004 M. Corley 2
___________________________________
Slide 3 ___________________________________
Agenda—Day 1
Welcome, Introduction, Objectives, Agenda Review
___________________________________
The Power of Data
–
–
Why Get Engaged with Data? Exercise
The Data-driven Program Improvement Model
___________________________________
– Setting Performance Standards
–
–
Adjusting Standards for Local Conditions
Establishing a Policy for Rewards and Sanctions
___________________________________
Getting Under the Data
– Data Pyramids
– Data Carousel
___________________________________
Evaluation and Wrap-up for Day 1
___________________________________
4/6/2004 M. Corley 3
___________________________________
1
Slide 4 ___________________________________
Objectives—Day 2
___________________________________
1. Distinguish between the uses of desk reviews
and on-site monitoring of local programs;
2. Identify steps for monitoring local programs;
___________________________________
3. Identify and apply key elements of a change
model; and ___________________________________
4. Work with local programs to plan for and
implement changes that will enhance program
performance and quality.
___________________________________
___________________________________
4/6/2004 M. Corley 4
___________________________________
Slide 5 ___________________________________
Agenda—Day 2
___________________________________
Agenda Review
Planning for and Implementing Program Monitoring
– Desk Reviews Versus On-site Reviews
___________________________________
– Data Sources (small group work)
– Steps and Guidelines for Monitoring Local Programs
Planning for and Implementing Program Improvement ___________________________________
– A Model of the Program Improvement Process
– State Action Planning
Closing and Evaluation ___________________________________
___________________________________
4/6/2004 M. Corley 5
___________________________________
Slide 6 ___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
2
Slide 7 ___________________________________
Question for Consideration
___________________________________
Why is it important to be able to ___________________________________
produce evidence of what your state
(or local) adult education program ___________________________________
achieves for its students?
___________________________________
___________________________________
4/6/2004 M. Corley 7
___________________________________
Slide 8 ___________________________________
The Motivation Continuum
___________________________________
Intrinsic Extrinsic
___________________________________
___________________________________
Which is the more powerful force for change?
___________________________________
___________________________________
4/6/2004 M. Corley 8
___________________________________
Slide 9 NRS Data-driven Program
___________________________________
Improvement (Cyclical Model)
___________________________________
STEPS
– Set performance standards
– Examine program elements underlying the
___________________________________
data
– Monitor program data, policy, and
procedures
___________________________________
– Plan and implement program improvement
– Evaluate progress and revise, as necessary, ___________________________________
and recycle
___________________________________
4/6/2004 M. Corley 9
___________________________________
3
Slide 10 What’s Under Your Data? ___________________________________
The Powerful Ps ___________________________________
__Performance_(Data)_
Program Policies ___________________________________
Procedures ___________________________________
Processes
Products
___________________________________
___________________________________
4/6/2004 M. Corley 10
___________________________________
Slide 11 NRS Data-driven Program ___________________________________
Improvement Model
Set Performance ___________________________________
Standards
___________________________________
Plan and Implement
Program
NRS
DATA
Examine Program
Elements Underlying
___________________________________
Improvement; the Data
Evaluate
Improvement
___________________________________
Monitor Program
Data, Policy,
Procedures
___________________________________
4/6/2004 M. Corley 11
___________________________________
Slide 12 Educational Gains for ESL Levels ___________________________________
and Performance Standards
___________________________________
100% Exhibit 1-2
91% 80%
80% ___________________________________
60% 50% Program
31% 33% 36% 34%
40%
22% 27% 26% 26%
Performance Standards
___________________________________
20%
0%
Beg. Beg. Low High Low High
___________________________________
Lit Int. Int. Adv. Adv.
___________________________________
4/6/2004 M. Corley 12
___________________________________
4
Slide 13 ___________________________________
Questions Raised by Exhibit 1-2
___________________________________
How were performance standards set? Based on past
performance?
Are standards too low at the higher levels? ___________________________________
Is performance pattern similar to that of previous
years? If not, why not?
What are program’s assessment and placement
___________________________________
procedures? Same assessments for high and low ESL?
How do curriculum and instruction differ by level? ___________________________________
What are student retention patterns by level?
___________________________________
4/6/2004 M. Corley 13
___________________________________
Slide 14 The Power of Data: Setting ___________________________________
Performance Standards ___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
4/6/2004 M. Corley 14
___________________________________
Slide 15 Essential Elements of ___________________________________
Accountability Systems ___________________________________
• Goals
___________________________________
• Measures
___________________________________
• Performance Standards
___________________________________
• Sanctions and Rewards
___________________________________
4/6/2004 M. Corley 15
___________________________________
5
Slide 16 National Adult Education ___________________________________
Goals ___________________________________
Reflected in NRS Outcome Measures of
___________________________________
educational gain,
GED credential attainment, ___________________________________
entry into postsecondary
education, and ___________________________________
employment.
___________________________________
4/6/2004 M. Corley 16
___________________________________
Slide 17 ___________________________________
Performance Standards
Similar to a “sales quota”: how well are you
___________________________________
going to perform this year?
– Should be realistic and attainable, but ___________________________________
– Should stretch you toward improvement
Set by each state in collaboration with ED ___________________________________
Each state’s performance is a reflection of
the aggregate performance of all the
programs it funds
___________________________________
___________________________________
4/6/2004 M. Corley 17
___________________________________
Slide 18 ___________________________________
Standards-setting Models
___________________________________
Continuous Improvement ___________________________________
Relative Ranking
External Criteria
___________________________________
Return on Investment (ROI) ___________________________________
___________________________________
4/6/2004 M. Corley 18
___________________________________
6
Slide 19 ___________________________________
Continuous Improvement
Standard based on past performance
___________________________________
Designed to make all programs ___________________________________
improve compared to themselves
Works well when there is stability and
a history of performance on which to
___________________________________
base standard
Ceiling reached over time, resulting in ___________________________________
little additional improvement
___________________________________
4/6/2004 M. Corley 19
___________________________________
Slide 20 ___________________________________
Relative Ranking
___________________________________
Standard is mean or median performance of
all programs
Programs ranked relative to each other ___________________________________
Works for stable systems where median
performance is acceptable ___________________________________
Improvement focus mainly on low-
performing programs
Little incentive for high-performing
___________________________________
programs to improve
___________________________________
4/6/2004 M. Corley 20
___________________________________
Slide 21 ___________________________________
External Criteria
Set by formula or external policy
___________________________________
Promotes a policy goal to achieve a ___________________________________
higher standard
Used when large-scale improvements are ___________________________________
called for, over the long term
No consideration of past performance: ___________________________________
unrealistic, unattainable
___________________________________
4/6/2004 M. Corley 21
___________________________________
7
Slide 22 ___________________________________
Return on Investment
Value of program :: Cost of program
___________________________________
A business model; answers question, Are
services or program worth the
___________________________________
investment?
Can be a powerful tool for garnering ___________________________________
funding (high ROI) or for losing funding
(low ROI) ___________________________________
May ignore other benefits of program
___________________________________
4/6/2004 M. Corley 22
___________________________________
Slide 23 ___________________________________
Decision Time for State Teams
___________________________________
1. Which model(s) do you favor for setting
standards for/with locals?
___________________________________
2. Is it appropriate to use one statewide
model or different models for different
programs? ___________________________________
3. How will you involve the locals in
setting the standards they will be held
___________________________________
to?
___________________________________
4/6/2004 M. Corley 23
___________________________________
Slide 24 ___________________________________
Question for Consideration
___________________________________
How do the standard-setting ___________________________________
model(s) that states select represent
a policy statement on the ___________________________________
relationship between performance
and quality that states want to
instill in local programs?
___________________________________
___________________________________
4/6/2004 M. Corley 24
___________________________________
8
Slide 25 Adjusting Standards for ___________________________________
Local Conditions ___________________________________
Research suggests that standards
often need to be adjusted for local ___________________________________
conditions before locals can work to
improve program quality. ___________________________________
WHY IS THIS SO?
___________________________________
___________________________________
4/6/2004 M. Corley 25
___________________________________
Slide 26 Factors that May Require
___________________________________
Adjustment of Standards ___________________________________
Student Characteristics
– An especially challenging group ___________________________________
– Students at lower end of level
– Influx of different types of students ___________________________________
Local Program Elements
External Conditions ___________________________________
___________________________________
4/6/2004 M. Corley 26
___________________________________
Slide 27 ___________________________________
Shared Accountability
___________________________________
State and locals share responsibility
to meet accountability requirements
___________________________________
– State provides tools and environment
for improved performance
___________________________________
– Locals agree to work toward
improving performance
___________________________________
___________________________________
4/6/2004 M. Corley 27
___________________________________
9
Slide 28 ___________________________________
Locals should know…
___________________________________
The purpose of the performance standards;
The policy and programmatic goals the ___________________________________
standards are meant to accomplish;
The standard-setting model that the state
adopts; and
___________________________________
That State guidance and support is available
to locals in effecting change.
___________________________________
___________________________________
4/6/2004 M. Corley 28
___________________________________
Slide 29 ___________________________________
Shared Accountability
___________________________________
Which state-initiated efforts have been
easy to implement at the local level?
Which have not?
___________________________________
What factors contributed to locals’
successfully and willingly embracing the ___________________________________
effort?
What factors contributed to a failed ___________________________________
effort?
___________________________________
4/6/2004 M. Corley 29
___________________________________
Slide 30 ___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
10
Slide 31 What About Setting
___________________________________
Rewards and Sanctions?
Which is the more powerful motivator: rewards or
___________________________________
sanctions?
List all the different possible reward structures you
___________________________________
can think of for local programs.
How might sanctioning be counter-productive? ___________________________________
List sanctioning methods that will not destroy locals’
motivation to improve or adversely affect ___________________________________
relationships with the state office.
___________________________________
4/6/2004 M. Corley 31
___________________________________
Slide 32 ___________________________________
Variations on a Theme Exercise
(Refer to H-10). Brainstorm as many possible rewards
___________________________________
or incentives as you can for recognizing local programs
that meet their performance standards.
___________________________________
Then brainstorm sanctions that the state might impose
on local programs that do not meet their performance
standards. ___________________________________
Select a recorder for your group to write one reward
per Post-It Note and one sanction per Post-It Note.
When you have finished, wait for further instructions
___________________________________
from the facilitator.
___________________________________
4/6/2004 M. Corley 32
___________________________________
Slide 33 Summary of Local Performance ___________________________________
Standard-setting Process
Procedure Goal
___________________________________
Select standard- Reflect state policies;
setting model Promote program improvement ___________________________________
Set rewards and Create incentives;
sanctions policy Avoid unintended effects
Make local Ensure standards are fair &
___________________________________
adjustments realistic for all programs
Provide T/A Create atmosphere of shared
accountability
___________________________________
Monitor often Identify and avoid potential
problems ___________________________________
4/6/2004 M. Corley 33
___________________________________
11
Slide 34 ___________________________________
Getting Under the Data
___________________________________
NRS data, as measured and ___________________________________
reported by states, represent the
product of underlying ___________________________________
programmatic and instructional
decisions and procedures. ___________________________________
___________________________________
4/6/2004 M. Corley 34
___________________________________
Slide 35 ___________________________________
Four Sets of Measures
___________________________________
1. Educational gain
2. NRS Follow-up Measures ___________________________________
– Obtained a secondary credential
– Entered and retained employment
– Entered postsecondary education
___________________________________
3. Retention
___________________________________
4. Enrollment
___________________________________
4/6/2004 M. Corley 35
___________________________________
Slide 36 ___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
12
Slide 37 ___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
Slide 38 ___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
Slide 39 ___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
13
Slide 40 ___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
Slide 41 ___________________________________
Question for Consideration
___________________________________
How might it benefit local programs
if the State office were to initiate ___________________________________
and maintain a regular monitoring
schedule to compare local program ___________________________________
performance against performance
standards? ___________________________________
___________________________________
4/6/2004 M. Corley 41
___________________________________
Slide 42 Regular Monitoring of Performance ___________________________________
Compared with Standards
Keeps locals focused on outcomes and
___________________________________
processes;
Highlights issues of importance; ___________________________________
Increases staff involvement in the process;
Helps refine data collection processes and
products; ___________________________________
Identifies areas for program improvement;
Identifies promising practices; ___________________________________
Yields information for decision-making;
Enhances program accountability.
___________________________________
4/6/2004 M. Corley 42
___________________________________
14
Slide 43 ___________________________________
BUT…
___________________________________
How can states possibly monitor
performance of all local programs? ___________________________________
Don’t we have enough to do already??
Where will we find staff to conduct the
___________________________________
reviews?
You’re kidding, right??
___________________________________
___________________________________
4/6/2004 M. Corley 43
___________________________________
Slide 44 ___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
Slide 45 ___________________________________
So….Let’s Find Some Answers
How can you monitor performance of
___________________________________
locals without overburdening state staff?
What successful models are already out
___________________________________
there??
How does your state office currently
___________________________________
ensure local compliance with state
requirements? ___________________________________
Can you build on existing structures?
___________________________________
4/6/2004 M. Corley 45
___________________________________
15
Slide 46 ___________________________________
Approaches to Monitoring
On-site Reviews
___________________________________
Desk Reviews
– Ongoing process – Single event,
– Useful for
lasting 1-3 days ___________________________________
quantitative data – Useful for
qualitative data
• Proposals
• Performance – Review of ___________________________________
measures processes &
• Program program quality
improvement plans
– Input from diverse
___________________________________
• Staffing patterns
• Budgets
stakeholders
___________________________________
4/6/2004 M. Corley 46
___________________________________
Slide 47 Advantages and Disadvantages ___________________________________
of Desk Reviews
Advantages Disadvantages
___________________________________
Data, reports, proposals, Assumes accurate data
etc., already in state office that reflect reality ___________________________________
Review can be built into Local staff and
staff’s regular workload stakeholders not heard
Data is quantitative; can Static view of data; no
___________________________________
be compared to previous interaction in context
years
No travel time or costs No team perspective
___________________________________
required
___________________________________
4/6/2004 M. Corley 47
___________________________________
Slide 48 Advantages and Disadvantages ___________________________________
of On-site Reviews
Advantages Disadvantages
___________________________________
Data is qualitative; review of Stressful for local program and
processes & program quality team
___________________________________
Input from perspectives of Arranging site visits and team is
diverse stakeholders time-intensive for both locals
State works with locals to
and state
Requires time out-of-office
___________________________________
explore options for
improvement; provides T/A
___________________________________
Opportunity to recognize Incurs travel costs
strengths; offer praise; identify
best practices
___________________________________
4/6/2004 M. Corley 48
___________________________________
16
Slide 49 Data Collection Strategies
___________________________________
for Monitoring
___________________________________
1. Program Self-Reviews (PSRs)
2. Document Reviews
___________________________________
3. Observations ___________________________________
4. Interviews
___________________________________
___________________________________
4/6/2004 M. Corley 49
___________________________________
Slide 50 ___________________________________
Program Self-Reviews
___________________________________
Conducted by local program staff
Review indicators of program quality ___________________________________
Completed in advance of monitoring
visit and can help focus the on-site ___________________________________
review
Results can guide the program ___________________________________
improvement process
___________________________________
4/6/2004 M. Corley 50
___________________________________
Slide 51 ___________________________________
Document Reviews
Can review from a distance:
___________________________________
– Proposals
– Qualitative and quantitative reports ___________________________________
– Improvement plans
Can review on-site: ___________________________________
– Student files
– Attendance records
– Entry and update records ___________________________________
– Course evaluations
___________________________________
4/6/2004 M. Corley 51
___________________________________
17
Slide 52 Qualitative and Quantitative Data ___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
___________________________________
4/6/2004 M. Corley 52
___________________________________
Slide 53 ___________________________________
Observations
Interactions
___________________________________
– during meetings
– At intake and orientation ___________________________________
– In hallways and on grounds
– In the classroom ___________________________________
Link what is observed to
– Indicators of quality
– Activities in the program plan ___________________________________
– Professional development workshops
___________________________________
4/6/2004 M. Corley 53
___________________________________
Slide 54 ___________________________________
Interviews
___________________________________
Help clarify or explore ambiguous
findings
Provide information re: stakeholders’
___________________________________
opinions, knowledge, and needs
– Administrative, instructional, and support staff ___________________________________
– Community partners
– Community agencies (e.g., employment, social
services) ___________________________________
– Learners
___________________________________
4/6/2004 M. Corley 54
___________________________________
18
Slide 55 Fill in the Boxes: Monitoring
___________________________________
with Indicators of Program Quality
___________________________________
In teams of 4-5 and using H-12, fill
in the data sources you would ___________________________________
expect to use, the questions you
would ask locals, and the strategies ___________________________________
you would use in conducting a desk
review versus an on-site review. ___________________________________
___________________________________
4/6/2004 M. Corley 55
___________________________________
Slide 56 Steps for Monitoring ___________________________________
Local Programs
___________________________________
1. Identify state policy for monitoring; gather support
from stakeholders.
2. Consider past practices when specifying scope of
work for monitoring.
___________________________________
3. Identify persons to lead and participate in
4.
monitoring.
Identify resources available for monitoring locals.
___________________________________
5. Determine process for collecting data with clearly
6.
defined criteria for rating; conduct monitoring.
Report findings and recommendations.
___________________________________
7. Follow-up on results.
___________________________________
4/6/2004 M. Corley 56
___________________________________
Slide 57 ___________________________________
Data Help…
Measure student progress
___________________________________
Measure program effectiveness
Assess instructional effectiveness
___________________________________
Guide curriculum development
Allocate resources wisely ___________________________________
Promote accountability
Report to funders and to the community
Meet state and federal reporting requirements
___________________________________
Show trends
___________________________________
4/6/2004 M. Corley 57
___________________________________
19
Slide 58 ___________________________________
BUT…
___________________________________
Data do not help:
If the data are not valid and reliable; ___________________________________
If the appropriate questions are not
asked after reviewing the data; or ___________________________________
If data analysis is not used for making
wise decisions. ___________________________________
___________________________________
4/6/2004 M. Corley 58
___________________________________
Slide 59 ___________________________________
A Word about the Change Process
___________________________________
Factors that allow us to accept change:
1. There is a compelling reason to do so;
2. We have a sense of ownership of the change;
___________________________________
3. Our leaders model they are serious about
supporting the change; ___________________________________
4. We have a clear picture of what the change
will look like; and
5. We have organizational support for lasting
___________________________________
systemic change.
___________________________________
4/6/2004 M. Corley 59
___________________________________
Slide 60 ___________________________________
Stages of Change
1. Maintenance of the old system
___________________________________
2. Awareness of new possibilities
3. Exploration of those new possibilities
___________________________________
4. Transition to some of those possibilities
or changes
___________________________________
5. Emergence of a new infrastructure
6. Predominance of the new system
___________________________________
___________________________________
4/6/2004 M. Corley 60
___________________________________
20
Slide 61 ___________________________________
A Word of Caution
Start small; don’t overwhelm locals with a “data
___________________________________
dump.”
Begin with the core issues, such as educational gain.
Listen to what the data tell about the big picture; don’t
___________________________________
get lost in too many details.
Work to create trust and build support by laying data
on the table without fear of recrimination. ___________________________________
Provide training opportunities for staff on how to use
data.
Be patient, working with what is possible in the local ___________________________________
program.
Source: Spokane, WA School Superintendent Brian Benzel
___________________________________
4/6/2004 M. Corley 61
___________________________________
Slide 62 Planning and Implementing ___________________________________
Program Improvement
___________________________________
Stages of the Program Improvement Process
1. Planning;
___________________________________
2. Implementing;
3. Evaluating; and
___________________________________
4. Documenting Lessons Learned
and Making Adjustments, as
needed ___________________________________
___________________________________
4/6/2004 M. Corley 62
___________________________________
Slide 63 ___________________________________
Planning Questions
___________________________________
Who should be included on your
program improvement team?
___________________________________
How will you prioritize areas needing
improvement?
___________________________________
How will you identify and select
strategies for effecting improvement? ___________________________________
___________________________________
4/6/2004 M. Corley 63
___________________________________
21
Slide 64 ___________________________________
Guiding Questions for Strategies
Is the strategy:
___________________________________
Clear and understandable to all users?
One specific action or activity, or dependent on other ___________________________________
activities? (If so, describe the sequence of actions.)
An activity that will lead to accomplishing the goal?
Observable and measurable? ___________________________________
Assignable to specific persons?
Based on best practices?
One that all team members endorse?
___________________________________
Doable—one that can be implemented?
___________________________________
4/6/2004 M. Corley 64
___________________________________
Slide 65 ___________________________________
Implementation Questions
Who will be responsible for taking the
___________________________________
lead on ensuring that the change is
implemented? ___________________________________
Who will be members of the “change”
team and what will be their roles? ___________________________________
How will expectations for the change be
promoted and nurtured? ___________________________________
How will the change be monitored?
___________________________________
4/6/2004 M. Corley 65
___________________________________
Slide 66 ___________________________________
Evaluation Questions
How will the changes that are
___________________________________
implemented be evaluated?
How will the team ensure that both
___________________________________
short- and long-term effects are
measured? ___________________________________
Who will interpret the results?
Who will be on the look-out for ___________________________________
unintended consequences?
___________________________________
4/6/2004 M. Corley 66
___________________________________
22
Slide 67 ___________________________________
Possible Evaluation Results
___________________________________
Significant improvement with no
significant unintended ___________________________________
consequences: Stay the course.
Little or no improvement: Stay the ___________________________________
course OR scrap the changes?
A deterioration in outcomes: Scrap ___________________________________
the changes.
___________________________________
4/6/2004 M. Corley 67
___________________________________
Slide 68 ___________________________________
Documenting the Process
___________________________________
Document
what worked and what didn’t; ___________________________________
lessons learned; and
logical next steps or changes to the ___________________________________
plan.
Use as guide for future action.
___________________________________
___________________________________
4/6/2004 M. Corley 68
___________________________________
Slide 69 ___________________________________
State Planning Time
In your state teams, consider the questions on
___________________________________
H-14 and begin planning.
Consider the stakeholders you want to include ___________________________________
in your planning for data monitoring and
program improvement.
Consider the problems you anticipate facing
___________________________________
and propose solutions to those problems.
Complete H-14 to the best of your ability and ___________________________________
be prepared to report on your plan in one hour.
___________________________________
4/6/2004 M. Corley 69
___________________________________
23
Slide 70 ___________________________________
Thank you
___________________________________
Great Audience!
Great Participation! ___________________________________
Great Ideas!
Live Long and Prosper! ___________________________________
Good Luck!!
___________________________________
___________________________________
4/6/2004 M. Corley 70
___________________________________
24
SUPPLEMENT
POSSIBLE QUESTIONS TO ASK WHEN EXAMINING THE DATA ON H-5
1. How were the performance standards set for this program? Were they based on past performance or some
other criteria?
2. Are these standards appropriate given the pattern of performance the program has shown? For example,
are the standards too low at the higher levels where performance greatly exceeded targets?
3. Is this performance pattern similar to that observed in previous years? If not, what has caused it to change?
Will this affect setting of performance standards in the future?
4. What are the program’s assessment and placement procedures? What assessments are used for pre- and
posttesting?
5. s the program using the same assessment methods for high and low level ESL? If so, is this appropriate
given the performance pattern?
6. What type of curriculum and instruction is the program offering? How does it differ by instructional level?
7. What are student retention patterns by level? Is retention affecting the differences in performance among
students at different levels?
8. Could the program’s recruitment practices have had an influence on performance? How many students is it
serving at each level?
9. Why are no students enrolled at the highest ESL level? Is this a result of recruitment, type of classes the
program offers, or placement procedures? Does the program need to change is recruitment practices?
S-1
GLOSSARY
Advancement Learner advances from one educational functioning level to the next, based on the learner’s
performance on state designated assessments.
Aggregation, or Data The process of combining reports from one level of administration into a single report at the
aggregation next (e.g., combining local program reports into one statewide report).
Alternative assessments Procedures and techniques used as an alternative to traditional testing. The focus tends to be
on individual student growth over time, rather than comparing students with one another.
Assessment Measures of student progress, including standardized testing, teacher assessment, portfolios,
checklists, etc.
Class level The educational functioning level in which students are placed.
Contact hours Hours of instruction or instructional activity the learner receives from the program. Instructional
activity includes any program-sponsored activity designed to promote student learning in the
program curriculum such as classroom instruction, assessment, tutoring, or participation in a
learning lab.
Continuous improvement Model that uses past performance to set standards for the future and to plan for program
improvement.
Data forms A written or electronic document for collecting student information.
Data items Individual questions or pieces of information contained on data forms.
Data quality All states use the same definitions and coding categories for every data element in the NRS.
States follow the same step-by-step instructions on how and when to collect each data
element.
Desk review A structured way to look at program information related to outcomes. May include review of
data, proposals, reports, budget, etc.
Descriptive measures For the purposes of the NRS, descriptive measures may include student: demographics,
educational status, and goals
Earn a high school diploma Obtaining a state accredited secondary diploma/credential or passing the General Educational
or achieve a GED Development (GED) Tests.
Educational gain Learner completes or advances one or more educational functioning levels from starting level
as measured at program entry or beginning of an instructional cycle.
Employed Learners who work as paid employees, work in their own business or farm, or who work 15
hours or more per week as unpaid workers on a farm or in a business operated by a member
of the family. Also included are learners who are not currently working, but who have jobs or
businesses from which they are temporarily absent.
English as a Second Programs for limited English proficient students that focus on improving English
Language programs communication skills such as speaking, reading, writing, and listening.
Enters employment The learner obtains full- or part-time paid employment before the end of the first quarter after
the program exit quarter.
S-2a
Enters post-secondary The learner enters another education or training program, such as community college, trade
education or training school, a four-year college or university, etc.
program
Evidence Data and documentation to support findings.
External Criteria Model for setting standards with a formula based on factors not directly related to a program’s
performance in the past or in relation to others.
Family literacy programs A program with a literacy component for parents and children or other intergenerational literacy
components.
GED Certificate given to learners who attain passing scores on the General Educational
Development (GED) Tests.
Generalizable The extent to which a finding can be generalized to other populations or situations.
Goals Information collected at intake about the main reason(s) a student is enrolling in the adult
education program. Consider both long- and short-term goals. For NRS purposes, report goals
that can be reached within the fiscal year.
Indicators of program Measures that define policies and practices for effective adult education programs.
quality
Improved employment The learner maintains his or her current employment, and receives an increase in pay,
additional responsibilities, or improved job-related skills.
Level benchmarks Guidelines for determining students’ educational functioning levels based on performance on
standardized tests.
Longitudinal data Data measured consistently from year to year in order to track learner progress over time.
Mandatory program A local, state, or federal program that requires a student to attend adult education classes, for
example welfare, NAFTA, or probation.
Mandatory students Students who are required to attend adult education classes because of their participation in
some other local, state, or federal program, including welfare, NAFTA, job training, or
probation. Mandatory students do not include students required to attend classes by their
employer.
Mean The arithmetic average of a set of scores, or the sum of observations divided by the number of
observations.
Median The middle score of a set of scores.
Mode The most frequently occurring score in a set of scores.
NAFTA program A federal program to assist workers displaced by the North American Free Trade Agreement
(NAFTA).
Norm-referenced tests Tests on which the performance is interpreted in the context of the performance of a group with
whom it is reasonable to compare the individual (for example, achieving at a 3.4 grade level).
On-site review Monitoring local programs on site to verify data by looking at the processes and procedures
being used. May include a review of files and recruiting materials, observations, interviews,
etc.
Outcome measures For the purposes of the NRS, core and secondary outcomes of adult education include
learning gains, entry into post-secondary education and training, obtaining high school
credentials obtained, entering or advancing in employment, and other gains related to family,
education and community participation.
Performance standards (for Numeric levels established for outcome measures in the state plan indicating what proportion
states) of students at each level will achieve each outcome.
Performance Standards (for Statement that indicates how well or to what extent a student will demonstrate knowledge or
students) skills.
Persistence Student’s ability to continue learning over time; the length of time student remains engaged
with learning.
Post-test A test administered to a student at designated intervals during a program. It is usually used to
measure progress or advancement in the program.
Pre-test A test administered to a student upon entry into a program. It is usually used for determine
level for placement.
S-2b
Probation A situation in which a student is under the supervision of a court and may be required to attend
classes.
Program (or program area) The main emphasis of instruction for a set of classes. Examples of program areas are ABE,
GED, workplace literacy, ESL, family literacy, etc.
P.I.P. or P.I.T Program Improvement Plan or Team
Qualitative data Detailed data collected in the form of words or images that is analyzed for description and
themes.
Quantitative data Data used to describe trends and relationships among variables. Analysis of the data entails
the use of statistics.
Relative ranking Model for setting standards based on a program’s rank relative to the state mean or median
rating or score.
Reliability The extent to which others would arrive at similar results if they studied the same case using
the same procedures; evidence of consistency of a measure.
Researchable question A research question that can realistically be answered with the skills and resources available
Retain employment The learner remains employed in the third quarter after the exit quarter.
Return on Investment (ROI) Net value in relation to cost.
Rubric A guide to evaluate a program (or student performance) on a scale with clearly defined criteria.
Scales may be numeric (1 to 5) or descriptive (not evident to exemplary).
Standard deviation A measure of the variability or spread of scores; the square root of the average of the squared
deviations of the scores from the means of the set of scores.
Student performance Student attainment formally measured by some assessment method.
Student record system A computerized or paper-based system for tracking student information related to intake
information, goals, educational levels, attendance, achievements, and outcomes.
Student retention Student attends program long enough (persists) to show learning gains.
TANF Temporary Assistance for Needy Families. A federal public assistance program.
Uniform system for All states and programs use the same methodology for collecting data on the measures.
collecting measures States certify validity through “data quality checklists”
Validity The extent to which a research instrument measures what it purports to measure.
Variance A measure of the variability of the scores in a frequency distribution; more specifically, the
square of its standard deviation.
Voluntary students Students who attend adult education classes of their own free will; they are not required to
attend by any state agency.
Work-based project learner A short-term course (at least 12 hours but no more than 30 hours) in which instruction is
activity designed to teach work-based skills and in which the educational outcomes and standards for
achievement are specified.
Workplace literacy A program designed to improve the literacy skills needed to perform a job and at least partly
programs under the auspices of an employer.
S-2c
LETTER TO SEND TO PARTICIPANTS PRIOR TO TRAINING
The Department of Education invites adult education staff to attend NRS Data Monitoring for
Program Improvement. This workshop is one in a series of trainings designed to promote the quality
and use of data collected under the National Reporting System (NRS). The goal of the workshop is
to provide training to staff on how to meet requirements to set local program performance standards,
better monitor and continuously improve their local programs using NRS data.
TOPICS
Adult education is facing a time of change, as new legislation replacing the Workforce Investment
Act is under consideration. As part of these changes, states may soon be developing new state
plans and setting performance standards for program accountability. States will be facing new
challenges to monitor and continuously improve local program performance. To assist in these
efforts, this workshop will cover the following topics:
Setting local performance standards – learn about performance standard setting models
and how to select the model that best meets your state policies and promotes local program
improvement.
Local monitoring – explore how standards can reflect program instruction, assessment,
retention and other procedures and how to use performance standards as indicators of local
performance.
Making change – the workshop will cover change models and how to make real changes in
your program to improve program performance and student outcomes.
WHO SHOULD ATTEND
The intended audience for the training is adult education staff who are or will be responsible for
setting performance standards, monitoring local programs and providing technical assistance for
program improvement. Staff responsible for conducting professional development on these topics,
and staff who perform these functions may also wish to attend. To provide a rich training experience,
we encourage you to send a team of up to three persons to this training.
COST
The cost of this training is $ , which covers…
LOCATION AND DATES
Locations and dates of the trainings are:
Attendance is limited to xx persons, so please register as soon as possible.
To register for the training, please return the attached registration form and please return the
registration form no later than … For questions about the training, please contact:
S-3
Alternative Monitoring Exercise: Facilitator’s Notes
GENERAL NOTES FOR FACILITATORS:
This alternative session design allows participants to actually use the spiral bound guide, Data
Monitoring for Program Improvement, to find information related to the purposes and strategies for
monitoring. Participants can work in state teams or across state teams during Part One – using the
guide to find information on different monitoring strategies that will link data collection with the
Indicators of Program Quality. Later in Part Two, participants will return to work in state teams to
look at the monitoring steps that the state already has in place and the steps that might require
updating in the future.
Part One:
Time:
Materials Notes 90 minutes
PPT- 41-45 A. Introduce local program monitoring for program (10 min)
improvement.
SH-12a B. Small Group Work - use the monitoring guide for (60 min)
questions one and two on SH-12a followed by more
SH- 12b or H-12
intensive practice using either SH-12b (an in depth
focus on three Indicators of Program Quality in more
depth) or H –12 (a broader focus on nine Indicators of
Program Quality)
C. Small Group Report - share perspectives (5 min.) on (20 min)
the benefits of monitoring and to identify the
advantages and disadvantages of both desk and on-
site reviews. Allow each group to report how they
might monitor one indicator (15 min.).
Part Two:
Times
Materials Notes 25 minutes
PPT- 56 A. Review steps and guidelines for a state monitoring (5 min)
system.
SH-12c B. State Team Work to identify what the state already has (15 min)
in place and what steps need to be developed or
revised.
C. Whole Group wrap up monitoring section by asking (5 min)
participants to share any salient ideas, strategies, or
challenges that came to the surface during this
segment of training.
S-4a
Alternative Monitoring Exercise: Monitoring Local Programs
Directions for Search and Find in the spiral guidebook - NRS Data Monitoring for Program
Improvement:
Work in state teams. Scan Chapter 4 for information to help complete the following exercises.
1. Identify 2-3 benefits for monitoring local programs (pages 37-38; suggested time 10
minutes).
FROM A STATE PERSPECTIVE… FROM A LOCAL PROGRAM PERSPECTIVE
2. The Monitoring Guide suggests that states can use a two-prong approach to monitoring –
desk reviews and on-site reviews (pages 38-40, suggested time 10 minutes).
Scan the information about the approaches and the advantages/disadvantages for each
approach.
a. Brainstorm ways that your state could use desk reviews and on-site reviews.
b. Brainstorm any obstacles that your state might encounter with each approach.
Approach Useful in our state Possible obstacles in our state
Desk Reviews
On-Site Reviews
S-4b
Alternative Monitoring Exercise: Using Indicators of Program Quality to
Monitor Programs
Refer to the NRS Guide to Data Monitoring, Table 4-4 on pages 46-48 and the Pennsylvania
sample on pages 55-56. (Suggested time 40 minutes)
a. Select several of your state‟s Indicators of Program Quality - exact wording is not necessary.
Alternatively, select an indicator from one of the sample states included in the Appendix pages
73-78.
b. Complete the chart below for monitoring the outcomes and processes by identifying your data
sources, questions to be answered, and then outlining effective strategies for desk reviews and
on-site reviews.
1. Indicator of Program Quality
Data Sources Questions to Pose Strategies Strategies
(NRS and Local) with Local Staff Desk Review On-site Review
2. Indicator of Program Quality
Data Sources Questions to Pose Strategies Strategies
(NRS and Local) with Local Staff Desk Review On-site Review
3. Indicator of Program Quality
Data Sources Questions to Pose Strategies Strategies
(NRS and Local) with Local Staff Desk Review On-site Review
S-4c
Alternative Monitoring Exercise: Steps and Guidelines for Monitoring
Local Programs
Directions: Scan the steps and identify what your state has in place and what needs to be
done.
Column 4: outline the process and products in place
Column 5: identify what needs to be developed
2. Implementation 4. Process/Products 5. To Be
1. Monitoring Steps 3. Examples
Guidelines in place Developed
1. Identify state policy for Provide clear written State plan should be open
monitoring. Gather guidelines to all levels of to the public and shared at
support from those who stakeholders on the all levels. State plans often
have a stake in the scope of monitoring specify:
results. activities (including 1. Outcome measures
process and timelines). 2. Frequency
2. Specifying the scope of Uses quantitative and Quantitative = look at
work for monitoring. qualitative data for outcome measurements
effective monitoring. Qualitative = look for
evidence using program
quality indicators
3. Identify individuals to Consider the unique Local staff: practitioners,
lead and to participate in program features when administrators, partners
monitoring activities. identifying who should be External team members:
involved from the local content specialists, other
program and who should educators, and staff from
be part of monitoring partnering agencies
team.
Consider strength in
diversity.
4. Identify resources With competing demands Desk reviews look at
available for monitoring for resources (staff, time, program data from a
local programs. and money for distance.
monitoring), consider Onsite reviews look at data
formalizing a two-stage in context—to see first-hand
monitoring approach. how the process and
operations lead to positive
outcome measures.
5. Determine process for Create and use standard Desk Reviews can include
collecting data with tools for data collection data, proposals, plans,
clearly stated criteria for and analysis. Monitors reports, and program self-
rating. Conduct (state staff and team) review.
monitoring activities. need to fully understand Onsite reviews can include
the tools, their use, and discussion of self-review,
the rating criteria. observations, interviews,
and a review of files and
documents.
6. Report on the findings, Conclude onsite Report might include a short
including monitoring visits with a description of the monitoring
recommendations. verbal debriefing followed activities with supporting:
by a written report. A. Qualitative description
B. Quantitative
7. Follow up on the results. Given that the major Follow-up activities might
purpose of monitoring is include reviewing
program improvement, it performance standards and
should include an program improvement,
ongoing exchange rewarding or sanctioning,
between the state office and technical assistance.
and the local program.
S-4d
Related docs
