4-Step Worksheet

Document Sample
4-Step Worksheet Powered By Docstoc
					                                                                                   Using the 4-Step Process To Build And Sustain Change
GENERAL INSTRUCTIONS: First, choose one of the four levels of the pyramid (e.g., “Improve Data Quality”) to begin. Each of the levels is listed in yellow, below. Once you choose a level, pick any question listed in that section. If the question is hyperlinked, click on it. You will be taken to a data table. Complete the data
table, look at the graph on the following tab in that Excel file, and write up what you learned from looking at the graph in the column below. Then, identify a related goal, if one is needed, in the column labeled "Step 2" and develop a related strategy and evaluation under Steps 3 and 4. Feel free to create your own
questions, graphs, or data to examine; the information below including the subsections and graphs are just meant to get you started.


Step 1: Examine Data to Assess Where You Are                                                                                                                            Step 2: Identify Goal                                 Step 3: Plan Strategy                                                Step 4: Implement & Evaluate
1a. Choose a question or make up a new one.                  1b. Ask questions about the data presented 1c. Put the answers to those questions, based If your observations lead to more                                       What will you do next? What activities will you do to                How (and when) will you evaluate whether you have
Follow the link, fill in the data, and examine the           in the graph, such as the questions listed on your examination of data, or other         questions or potential goals, write them                                reach your goals?                                                    made progress toward your goal?
graph on the next tab.                                       in this column or make up new ones.        observations of your data here.               here. Be specific about what you will
                                                                                                                                                      achieve by when.


1. Improve Data Quality: Data Systems and Error Checks
EXAMPLE: Are the numbers of students setting goals stable How do each of our programs vary year to year?      Four programs are responsible for most of the variation   Increase amount of accurate data entered, focusing    Re-train data entry staff and program administrators during     Assess progress in August. Identify lowest two programs for
from year to year?                                        How do the numbers of students setting goals differ [based on graph showing last three years for every        on the programs identified in Step #1. Target: 95%    summer training institute on reporting requirements to improve  intensive site visit and on-site training in September. December data
                                                          from the number achieving goals?                    program]. In two programs, a greater number of            accuracy in reporting by the end of this year, with   accuracy. Provide graphs of performance at training and monthly will be used to assess whether 95% target was met.
                                                                                                              students are achieving the goal than setting the goal.    additional training on goal setting (see separate     thereafter.
                                                                                                                                                                        goal).



Stability and Reliability of Data

How has the percentage of students with pre and Does the number show significant (e.g., more than [Write observations based on the graph(s) or data                     [If needed based on the review of the data, write     [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
posttest data changed over time?                10%) swings from year to year?                    here.]                                                                goals here.]

Are the percentages of completers relatively                 Are goal numbers plausible? Does the number        [Write observations based on the graph(s) or data       [If needed based on the review of the data, write     [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
stable?                                                      show significant (e.g., more than 10%) swings from here.]                                                  goals here.]
Which goals are students setting and how do                  year to year?
                                                             Does the number show significant (e.g., more than [Write observations based on the graph(s) or data        [If needed based on the review of the data, write     [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
they compare over time?                                      10%) swings from year to year?                     here.]                                                  goals here.]

Are the percentages of students obtaining follow- Does the number show significant (e.g., more than [Write observations based on the graph(s) or data                   [If needed based on the review of the data, write     [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
up outcomes relatively stable?                    10%) swings from year to year?                    here.]                                                              goals here.]

What is the overall and by variable percentage of data         Which data input variables (e.g., student age) have [Write observations based on the graph(s) or data    [If needed based on the review of the data, write     [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
errors? [For this question, ask your data administrator to run the greatest percentage of mistakes? What are the here.]                                                 goals here.]
frequency distributions or histograms for each variable.]      errors being made?


[Add your own question here]                                 [Do your own graph]                                  [Write observations based on the graph(s) or data     [If needed based on the review of the data, write     [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                                                                                  here.]                                                goals here.]

Validity of Data
How high are response rates? How do response Are response rates reasonably high and meet basic [Write observations based on the graph(s) or data                        [If needed based on the review of the data, write     [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
rates compare across programs and to the state standards?                                      here.]                                                                   goals here.]
average or standard?

How do response rates differ by subgroup?                    Are any groups disproportionately not represented? [Write observations based on the graph(s) or data       [If needed based on the review of the data, write     [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                                                                                here.]                                                  goals here.]
                                                                                 Using the 4-Step Process To Build And Sustain Change
GENERAL INSTRUCTIONS: First, choose one of the four levels of the pyramid (e.g., “Improve Data Quality”) to begin. Each of the levels is listed in yellow, below. Once you choose a level, pick any question listed in that section. If the question is hyperlinked, click on it. You will be taken to a data table. Complete the data
table, look at the graph on the following tab in that Excel file, and write up what you learned from looking at the graph in the column below. Then, identify a related goal, if one is needed, in the column labeled "Step 2" and develop a related strategy and evaluation under Steps 3 and 4. Feel free to create your own
questions, graphs, or data to examine; the information below including the subsections and graphs are just meant to get you started.


 Step providers leave cells blank, which to Assess Where You Are
When data 1: Examine Dataones are they Are there variables that are consistently left blank [Write observations based on the graph(s) or data                         [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
leaving blank? [For this question, ask your data             across programs? Which are the top three missing here.]                                                  goals here.]
administrator to run a report of missing values or frequency data points in each program? How much would
distributions that include missing values.]                  your data improve by addressing each of these?

Which data variables do you and your staff mistrust the    On what evidence is this based? Do you all agree? [Write observations based on the graph(s) or data        [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
most?                                                                                                        here.]                                                   goals here.]
Where do your data providers report mistrust of the data? (of On what evidence is this based? Do you agree?    [Write observations based on the graph(s) or data      [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
their own and others)                                                                                          here.]                                                 goals here.]
[Add your own question here]                               [Do your own graph]                                 [Write observations based on the graph(s) or data      [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                                                                               here.]                                                 goals here.]
Step 1: Examine Data to Assess Where You Are                                                                                                                          Step 2: Identify Goal                               Step 3: Plan Strategy                                                Step 4: Implement & Evaluate
1a. Choose a question or make up a new one.                1b. Ask questions about the data presented 1c. Put the answers to those questions, based If your observations lead to more                                     What will you do next? What activities will you do to                How (and when) will you evaluate whether you have
Follow the link, fill in the data, and examine the         in the graph, such as the questions listed on your examination of data, or other         questions or potential goals, write them                              reach your goals?                                                    made progress toward your goal?
graph on the next tab.                                     in this column or make up new ones.        observations of your data here.               here. Be specific about what you will
                                                                                                                                                    achieve by when.



2. Improve Data Quality: Policies and Procedures
EXAMPLE: Are programs following the new policies of pre    Is each program following the policy? When they     The graph showed that two programs are not             By the end of next year, all programs will have     For the two programs out of compliance, we will show them these At the end of the first year, we will need to be within 10 percentage
and posttesting students within the prescribed time?       are not, where are they off?                        administering pre and posttests within the timeframe   fewer than 2% of students who fall outside of the   graphs and develop with them a plan for improvement by the end points of the goal. If we are not, we will need to add strategies in
                                                                                                               set by state assessment policy.                        NRS timeframe.                                      of this year. Each month we will review progress. For the three  Year 2.
                                                                                                                                                                                                                          programs doing well, we will share this graph, congratulate them
                                                                                                                                                                                                                          and let them know that we will be tracking this information
                                                                                                                                                                                                                          quarterly.



Goal Setting
Which goals are students setting and how do                Are any goals not well represented? Are any over- [Write observations based on the graph(s) or data        [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
they compare over time?                                    represented? How have these shifted?              here.]                                                   goals here.]

How does goal setting in Program A compare to              Is Program A reporting a similar percentage in each [Write observations based on the graph(s) or data      [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
the state level?                                           area? Do any differences in the percentages make here.]                                                    goals here.]
                                                           sense?


Are the percentages of students setting            Do goals appear well-aligned with the NRS levels of [Write observations based on the graph(s) or data              [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
educational attainment goals consistent with their the students setting them?                          here.]                                                         goals here.]
NRS level and program goals?
                                                                               Using the 4-Step Process To Build And Sustain Change
GENERAL INSTRUCTIONS: First, choose one of the four levels of the pyramid (e.g., “Improve Data Quality”) to begin. Each of the levels is listed in yellow, below. Once you choose a level, pick any question listed in that section. If the question is hyperlinked, click on it. You will be taken to a data table. Complete the data
table, look at the graph on the following tab in that Excel file, and write up what you learned from looking at the graph in the column below. Then, identify a related goal, if one is needed, in the column labeled "Step 2" and develop a related strategy and evaluation under Steps 3 and 4. Feel free to create your own
questions, graphs, or data to examine; the information below including the subsections and graphs are just meant to get you started.


Step 1: number of students who set Assesspercentage credible and appear, on its face,
How does theExamine Data to the                Is the Where You Are                                           [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
goal compare to the number who achieved it (or to have been set prior to achievement (e.g., 100%              here.]                                              goals here.]
who are in the program)?                       goal achievement is suspect on its face).

Does the percentage of students setting the goal Is the goal-setting reflecting the level of need in the [Write observations based on the graph(s) or data        [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
of entering employment reflect the percentage of community and if not are there reasons why not?         here.]                                                   goals here.]
students who are unemployed?

How does goal setting differ by subgroup?                Are any subgroups disproportionately represented     [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                         in any goal?                                         here.]                                              goals here.]
[Add your own question here]                             [Do your own graph]                                  [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                                                                              here.]                                              goals here.]

Assessment
How many students have pre and posttest data? Are all students getting pre and posttested?                    [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                                                                              here.]                                              goals here.]
How has the percentage of students with pre and Is the percentage increasing? If not, reasons why             [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
posttest data changed over time?                not?                                                          here.]                                              goals here.]

Which students are not tested?                           Is there any pattern to the students not tested?     [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                                                                              here.]                                              goals here.]

Are pre-tests and posttests given at the right           Are guidelines on timing of tests being followed?    [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
time?                                                                                                         here.]                                              goals here.]
Are the right tests given?                               Are the right versions of tests being given?         [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                                                                              here.]                                              goals here.]
Are the percentages of completers relatively             If not, are completers being defined consistently?   [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
stable?                                                                                                       here.]                                              goals here.]

[Add your own question here]                             [Do your own graph]                                  [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                                                                              here.]                                              goals here.]

Follow-up
How do response rates compare across                     Are response rates consistent with any state         [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
programs and to the state average or standard?           standards?                                           here.]                                              goals here.]


Were the times for collecting entered and                Were guidelines for follow-up followed?              [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
retained employment data consistent with NRS                                                                  here.]                                              goals here.]
requirements?

Are the percentages of students obtaining follow- If not, are follow-up procedures being followed?            [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
up outcomes relatively stable?                                                                                here.]                                              goals here.]

[Add your own question here]                             [Do your own graph]                                  [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                                                                              here.]                                              goals here.]
                                                                                Using the 4-Step Process To Build And Sustain Change
GENERAL INSTRUCTIONS: First, choose one of the four levels of the pyramid (e.g., “Improve Data Quality”) to begin. Each of the levels is listed in yellow, below. Once you choose a level, pick any question listed in that section. If the question is hyperlinked, click on it. You will be taken to a data table. Complete the data
table, look at the graph on the following tab in that Excel file, and write up what you learned from looking at the graph in the column below. Then, identify a related goal, if one is needed, in the column labeled "Step 2" and develop a related strategy and evaluation under Steps 3 and 4. Feel free to create your own
questions, graphs, or data to examine; the information below including the subsections and graphs are just meant to get you started.


Step 1: Examine Data to Assess Where You Are                                                                                                                       Step 2: Identify Goal                               Step 3: Plan Strategy                                                Step 4: Implement & Evaluate
1a. Choose a question or make up a new one.                 1b. Ask questions about the data presented 1c. Put the answers to those questions, based If your observations lead to more                                 What will you do next? What activities will you do to                How (and when) will you evaluate whether you have
Follow the link, fill in the data, and examine the          in the graph, such as the questions listed on your examination of data, or other         questions or potential goals, write them                          reach your goals?                                                    made progress toward your goal?
graph on the next tab.                                      in this column or make up new ones.        observations of your data here.               here. Be specific about what you will
                                                                                                                                                     achieve by when.



3. Manage Performance
EXAMPLE: Are programs showing continuous improvement How do completion rates compare over the last           Graphs of ABE level completion rates for the last 3   Improve performance on ABE completions              Provide struggling programs with intensive professional             Collect multi-year data and look at trends by program and compare
in completion rates?                                 three years?                                            year indicate that most programs in county Y have                                                         development and link them up with a high performing program for to state average.
                                                                                                             shown little improvement in completion rates.                                                             assistance. The PD will focus on increasing correct student level
                                                                                                                                                                                                                       placement, ensuring that appropriate curriculum is in use, and
                                                                                                                                                                                                                       sharing models for high achievement. Reward programs that show
                                                                                                                                                                                                                       improved attainment rates with certificates of recognition (Learn &
                                                                                                                                                                                                                       Control; Belong & Compete).



Best Practices
Which site has the highest completion (or                   Does one program stand out or a few programs?    [Write observations based on the graph(s) or data     [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
retention, etc) rate?                                       Does one program do one of the outcomes          here.]                                                goals here.]
                                                            especially well?
What are the trends in goal attainment rates and Which program is showing the most improvement? [Write observations based on the graph(s) or data                  [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
how do they compare across programs?             Are there reasons outside the program that caused here.]                                                          goals here.]
                                                 this?

How do goal attainment rates compare among                  Does one program stand out or a few programs?    [Write observations based on the graph(s) or data     [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
programs?                                                   Does one program do one of the outcomes          here.]                                                goals here.]
                                                            especially well?

Which teachers have the highest percentage of Which teachers show the best results? Are there [Write observations based on the graph(s) or data                    [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
completers? [Use program graph and simply put teachers who are showing improvement? Are there here.]                                                               goals here.]
in teacher's names in place of programs.]     teachers who show the best results with growing
                                              subpopulations?
[Add your own question here]                  [Do your own graph]                             [Write observations based on the graph(s) or data                    [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                                                              here.]                                                               goals here.]

Professional Development (PD)
On which measures is state average                          Are there a few areas where targeted PD could    [Write observations based on the graph(s) or data     [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
performance more than 10 percentage points                  significantly improve statewide data?            here.]                                                goals here.]
lower than the national average?
What errors are most frequently reported from the data or   What do people report (through accessing help)   [Write observations based on the graph(s) or data     [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
NRS system? What do people most often access the help       having the most difficulty with?                 here.]                                                goals here.]
menu to answer?
                                                                                Using the 4-Step Process To Build And Sustain Change
GENERAL INSTRUCTIONS: First, choose one of the four levels of the pyramid (e.g., “Improve Data Quality”) to begin. Each of the levels is listed in yellow, below. Once you choose a level, pick any question listed in that section. If the question is hyperlinked, click on it. You will be taken to a data table. Complete the data
table, look at the graph on the following tab in that Excel file, and write up what you learned from looking at the graph in the column below. Then, identify a related goal, if one is needed, in the column labeled "Step 2" and develop a related strategy and evaluation under Steps 3 and 4. Feel free to create your own
questions, graphs, or data to examine; the information below including the subsections and graphs are just meant to get you started.


 Step 1: Examine Data (see graphs Where Where provide Are
Which of the core measures is the least reliableto Assess might PD help Yougreater reliability?              [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
in level 1)?                                                                                                 here.]                                              goals here.]

How often do your programs use data to manage? [May be Are the data available and used in ways to promote [Write observations based on the graph(s) or data      [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
able to assess usage from system]                      improved program management? How could the here.]                                                         goals here.]
                                                       data system and use be improved?

[Add your own question here]                              [Do your own graph]                                [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                                                                             here.]                                              goals here.]

Efficiency
What is the investment per completer (efficiency) Are there programs that are significantly more             [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
and how does it compare by program?               efficient or inefficient? What is the range?               here.]                                              goals here.]

What is the investment per computer (efficiency) Are there programs that are significantly more              [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
and how does it compare by program?              efficient or inefficient? What is the range?                here.]                                              goals here.]


Which programs show the greatest efficiency               Are there programs that are significantly more     [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
(greatest number of outcomes per dollar)?                 efficient or inefficient? What is the range?       here.]                                              goals here.]

What is the investment per goal attained                  Are there programs that are significantly more     [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
(program efficiency)?                                     efficient or inefficient? What is the range?       here.]                                              goals here.]

[Add your own question here]                              [Do your own graph]                                [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                                                                             here.]                                              goals here.]

Projections & Populations
What are the trends for the numbers of people     Are programs prepared to meet demand?                      [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
who speak a language other than English at                                                                   here.]                                              goals here.]
home within my state or city and how do they
compare?
How do the percentages of national origins within Are programs prepared to meet demand?                      [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
my city compare to those in my program?                                                                      here.]                                              goals here.]


What are the levels of educational attainment             Are programs meeting these needs?                  [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
within my counties and state?                                                                                here.]                                              goals here.]
Where, on a map of the state, are the clusters of Where are the clusters of need? Are programs               [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
those adults without a diploma?                   located accordingly?                                       here.]                                              goals here.]
Where, on a map of my state, are the                      Where are the clusters? Are programs located       [Write observations based on the graph(s) or data   [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
percentages of foreign born highest?                      accordingly?                                       here.]                                              goals here.]
What are the percentages of unemployed                    Do these percentages match the populations in the [Write observations based on the graph(s) or data    [If needed based on the review of the data, write   [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
persons in the counties and state?                        programs and their goals?                         here.]                                               goals here.]
                                                                                    Using the 4-Step Process To Build And Sustain Change
GENERAL INSTRUCTIONS: First, choose one of the four levels of the pyramid (e.g., “Improve Data Quality”) to begin. Each of the levels is listed in yellow, below. Once you choose a level, pick any question listed in that section. If the question is hyperlinked, click on it. You will be taken to a data table. Complete the data
table, look at the graph on the following tab in that Excel file, and write up what you learned from looking at the graph in the column below. Then, identify a related goal, if one is needed, in the column labeled "Step 2" and develop a related strategy and evaluation under Steps 3 and 4. Feel free to create your own
questions, graphs, or data to examine; the information below including the subsections and graphs are just meant to get you started.


Step the projected percentages of the
What are1: Examine Data to                               Do these percentages match Are
                                                      Assess Where Youthe populations in the                        [Write observations based on the graph(s) or data      [If needed based on the review of the data, write    [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
population over age 65?                                      programs?                                              here.]                                                 goals here.]


What are the trends in, for example, reported                Do these percentages match the populations in the [Write observations based on the graph(s) or data           [If needed based on the review of the data, write    [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
ancestry?                                                    programs?                                         here.]                                                      goals here.]


Is teacher turnover likely to decline or increase?           How is retention among teachers now and is it likely   [Write observations based on the graph(s) or data      [If needed based on the review of the data, write    [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                             to change in the immediate future due to market or     here.]                                                 goals here.]
                                                             other forces outside?
How long is it taking programs to fill teacher openings?     What are the current trends in filling teacher         [Write observations based on the graph(s) or data      [If needed based on the review of the data, write    [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                             positions? Are they likely to continue or worsen?      here.]                                                 goals here.]
                                                             How might this impact programs?
What are employment projections for the state and regions    Is employment likely to go one way or the other?       [Write observations based on the graph(s) or data      [If needed based on the review of the data, write    [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
within the state?                                                                                                   here.]                                                 goals here.]

[Add your own question here]                                 [Do your own graph]                                    [Write observations based on the graph(s) or data      [If needed based on the review of the data, write    [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                                                                                    here.]                                                 goals here.]
Step 1: Examine Data to Assess Where You Are                                                                                                                               Step 2: Identify Goal                                Step 3: Plan Strategy                                                Step 4: Implement & Evaluate
1a. Choose a question or make up a new one.                  1b. Ask questions about the data presented 1c. Put the answers to those questions, based If your observations lead to more                                         What will you do next? What activities will you do to                How (and when) will you evaluate whether you have
Follow the link, fill in the data, and examine the           in the graph, such as the questions listed on your examination of data, or other         questions or potential goals, write them                                  reach your goals?                                                    made progress toward your goal?
graph on the next tab.                                       in this column or make up new ones.        observations of your data here.               here. Be specific about what you will
                                                                                                                                                      achieve by when.



4. Innovate and Evaluate
EXAMPLE: How can we improve completion rates and             Does attendance have any relationship to               Data showed that students who attend 100 hours or      Improve completion rates by 10% in target or pilot   Begin a pilot program at one or two sites that tries to improve      Measure the intensity of instruction by looking at frequency of
make it more likely that students will complete a level?     completion?                                            more have lower completion rates than those who        program.                                             completion by increasing the intensity of instruction- how often     attendance six months into the program and at the end of the year.
                                                                                                                    attend less than 100 hours, the opposite of what was                                                        students attend not just the total number of hours they attend.      Compare completion rates in this program to non-participating
                                                                                                                    anticipated.                                                                                                                                                                     programs.
Long-term: Since the new policy or procedure                 How have the core measures changed since               [Write observations based on the graph(s) or data      [If needed based on the review of the data, write    [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
has been put into place, have the outcomes                   implementing the policy?                               here.]                                                 goals here.]
(core NRS) measures changed?
Short-term: Since the new policy or procedure                How have the completion rates changed since            [Write observations based on the graph(s) or data      [If needed based on the review of the data, write    [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
has been put into place, is there an improved                implementing the policy?                               here.]                                                 goals here.]
completion (or attendance, etc) rate?
How are participants in our adult education program different [Impact evaluation]                                   [Write observations based on the graph(s) or data      [If needed based on the review of the data, write    [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
from those like them who do not participate?                                                                        here.]                                                 goals here.]

Is the benefit worth the state investment in my              How do the costs compare to the income as              [Write observations based on the graph(s) or data      [If needed based on the review of the data, write    [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
program?                                                     measured by estimated increased tax revenue?           here.]                                                 goals here.]

[Add your own question here]                                 [Do your own graph]                                    [Write observations based on the graph(s) or data      [If needed based on the review of the data, write    [List the activities that you will undertake to meet those goals.]   [Write how and when you will assess your progress.]
                                                                                                                    here.]                                                 goals here.]
                            Using the 4-Step Process To Build And Sustain Change
Comprehensive List of Questions to Assess Where You Are
1. Improve Data Quality: Data Systems and Error Checks
How has the percentage of students with pre and posttest data changed over time?
Are the percentages of completers relatively stable?
Which goals are students setting and how do they compare over time?
Are the percentages of students obtaining follow-up outcomes relatively stable?
What is the overall and by variable percentage of data errors? [For this question, ask your data administrator to run frequency distributions or histograms for each variable.]
How high are response rates? How do response rates compare across programs and to the state average or standard?
How do response rates differ by subgroup?
When data providers leave cells blank, which ones are they leaving blank? [For this question, ask your data administrator to run a report of missing values or frequency ditributions that include missing
values.]
Which data variables do you and your staff mistrust the most?
Where do your data providers report mistrust of the data? (Of their own and others)
[Add your own question here]



2. Improve Data Quality: Policies and Procedures
Which goals are students setting and how do they compare over time?
How does goal setting in Program A compare to the state level?
Are the percentages of students setting educational attainment goals consistent with their NRS level and program goals?
How does the number of students who set the goal compare to the number who achieved it (or who are in the program)?
Does the percentage of students setting the goal of entering employment reflect the percentage of students who are unemployed?
How does goal setting differ by subgroup?
How many students have pre and posttest data?
How has the percentage of students with pre and posttest data changed over time?
Which students are not tested?
Are pre-tests and posttests given at the right time?
Are the right tests given?
Are the percentages of completers relatively stable?
How do response rates compare across programs and to the state average or standard and how have they changed over time?
Were the times for collecting entered and retained employment data consistent with NRS requirements?
Are the percentages of students obtaining follow-up outcomes relatively stable?
See also the NRS Desk MonitoringTool
[Add your own question here]




3. Manage Performance
                           Using the 4-Step Process To Build And Sustain Change
Comprehensive List of Questions to Assess Where You Are
Which site has the highest completion (or retention, etc) rate?
What are the trends in goal attainment rates and how do they compare across programs?
How do goal attainment rates compare among programs?
Which teachers have the highest percentage of completers? [Use program graph and simply put in teacher's names in place of programs.]
On which measures is state average performance more than 10 percentage points lower than the national average?
What errors are most frequently reported from the data or NRS system? What do people most often access the help menu to answer?
Which of the core measures is the least reliable (see graphs in level 1)?
How often do your programs use data to manage? [May be able to assess uasage from system]
What is the investment per completer (efficiency) and how does it compare by program?
How has efficiency changed over time?
Which programs show the greatest efficiency (greatest number of outcomes per dollar)?
What is the investment per goal attained (program efficiency)?
What are the trends for the numbers of people who speak a language other than English at home within my state or city and how do they compare?
How do the percentages of national origins within my city compare to those in my program?
What are the levels of educational attainment within my counties and state?
Where, on a map of the state, are the clusters of those adults without a diploma?
Where, on a map of my state, are the percentages of foreign born highest?
What are the percentages of unemployed persons in the counties and state?
What are the projected percentages of the population over age 65?
What are the trends in, for example, reported ancestry?
Is teacher turnover likely to decline or increase?
How long is it taking programs to fill teacher openings?
What are employment projections for the state and regions within the state?
What is the trend in enrollment?
What are the state and national enrollment percentages?
What is the enrollment by gender?
What is the enrollment by age?
What is the enrollment by ethnicity?
What is the census vs. enrollment by age?
What is the trend in employment rates?


4. Innovate and Evaluate
Long-term: Since the new policy or procedure has been put into place, have the outcomes (core NRS) measures changed?
Short-term: Since the new policy or procedure has been put into place, is there an improved completion (or attendance, etc) rate?
How are participants in our adult education program different from those like them who do not participate?
Is the benefit worth the state investment in my program?