Program Evaluation

Document Sample
Program Evaluation Powered By Docstoc
					 Program Evaluation

It’s Not Just for OMB Anymore…
                   Questions

Think about your state
• What has your public investment been in workforce
  development in your state in the last five years?
• How many people participated in what services?
• What has been the net impact of participation in the
  services?
• What has been the return on this investment?



                                                     2
         It’s Not Just A Good Idea,
                 It’s the Law


[PL 105-220] WIA Chapter 6, Section 136(e)(1)
• “Using funds made available under [WIA], the
  State, in coordination with local boards, shall
  conduct ongoing evaluation … of workforce
  investment activities …”




                                                    3
               Why Evaluate?
• “ … in order to promote, establish, implement, and
  utilize methods for continuously improving the
  activities … to achieve high-level performance …
• Evaluation ensures that states are getting the
  highest possible return on the investment of public
  funds
• If you don’t know what is working and what isn’t,
  your performance will be driven solely by
  happenstance and not by design
  – The best defense IS NOT accidental offense


                                                        4
   What Should Evaluations of the
Workforce Investment System Include?

• Analysis of Customer Feedback
  – Not simply the collection of the data for WIA reporting
    purposes
• Outcome Measures
• Process Measures in the Workforce Investment
  System




                                                              5
Fundamental Evaluation Questions

• What is the program doing; what are the activities?
• What are the customers’ reactions?
  – Participants and Employers
• What are the results?
• How are the results achieved?
• What is the return on investment?



                                                        6
       Isn’t monitoring enough?


• Monitoring determines the level of compliance with
  administrative policies and procedures designed to
  interpret the legislative intent of the program.
• Monitoring does not measure impact, process or
  outcomes.




                                                       7
      Okay, how about reviews?

• Review determines the extent to which program
  plans are consistent with legislative intent of the
  program
• Compliance Reviews do not measure impact,
  process or outcomes
• Reviews are primarily designed to ensure
  compliance with laws and guidance and are not an
  evaluation of the program



                                                        8
      Performance Reporting on 17
      Measures Must be Sufficient…

• Performance Measurement determines the extent to which
  immediate program outputs and short-term outcomes meet
  performance expectations established and consistent with
  the legislative intent.
• Just because a state reports high performance levels, this
  does not mean that they are running a program worth
  emulating. Only an evaluation can make that determination.
• Success on performance measures under WIA could also
  reflect negotiation skills and not strong performance design



                                                                 9
      Evaluation is the Only Way

• Evaluation determines through longitudinal-type
  studies the extent to which a program is achieving
  its broader legislative intent in terms of producing
  the expected effects on the individuals or other
  entities for whom the program was created.
• None of the WIA measures provide information on
  UI savings, the return on investment or business
  services.



                                                         10
            Types of Evaluation
There are seven (7) types of evaluations that satisfy
the legislative intent of the Workforce Investment Act:
1. Program Evaluation
2. Assessment of Need
3. Assessment of Program Theory and Design
4. Process/Implementation Evaluation
5. Outcome Evaluations
6. Impact Evaluations Defined by GAO
7. Cost-Benefit and Cost-Effectiveness Analysis


                                                          11
          1. Program Evaluation


• The systematic collection, analysis, and
  interpretation of information that answers questions
  about the efficiency and effectiveness of program
  implementation and outcomes, using the principles
  and methods of social science research.




                                                     12
         2. Assessment of Need


• Considers the assumptions that Employment and
  Training programs utilize in regards to problems
  they address for their target population. Problems
  would be determined through existing data
  sources, surveys, and forecasts.




                                                       13
            3. Assessment of
       Program Theory and Design


• Compares intervention specified in the program
  with the needs the program is expected to address,
  and determines whether the program plan can
  reasonably expect to improve the problem.




                                                   14
4. Process/Implementation Evaluation


• Assesses the extent to which a program is
  operating as it was intended. This type of
  evaluation can provide a linkage between the
  process and program outcomes (i.e. impact
  evaluations).




                                                 15
        5. Outcome Evaluations


• Focuses on intended and unintended outputs and
  outcomes to judge program effectiveness; may also
  employ a process evaluation to establish a
  complete picture of how the outcomes are
  produced.




                                                  16
           6. Impact Evaluations:
               Defined by GAO

• One that assesses the net effect of a program by
  comparing program outcomes with an estimate of
  what outcomes would have happened in the
  absence of the program.
• The “gold” standard is a random assignment study
  to assign individuals (or …) to one of two groups: a
  control and an experimental group.



                                                         17
           7. Cost-Benefit and
       Cost-Effectiveness Analysis
• Compares program outputs and outcomes (i.e.
  outcome evaluation) with the “costs” to produce
  them.
• Both measure efficiency of a program using dollars
  expended; however, Cost Benefit Analysis aims at
  identifying all relevant costs and benefits of a
  program.
• Cost Effectiveness Analysis compares programs
  with similar goals and compares their costs.


                                                       18
                  Examples



• Massachusetts
• Washington




                             19
20
21
22
23
24
25
Washington




             26
27
                 Take Away Points

•   Start with a question
•   Let question(s) drive the choice of evaluation design/study
•   Remember that often mixed methods will be needed
•   Avail yourself of resources out there
•   Start small to make your point (“proof of concept” studies)
•   Maximize secondary analyses of available administrative
    data
          It can be done – It is not an unfunded mandate




                                                                  28
           What are you doing?




• Audience discussion on evaluation activities




                                                 29
           Websites/Resources
• Government Accountability Office (GAO)
  http://www.gao.gov/special.pubs/erm.html
• Social Policy and Research Associates (SPRA)
  http://www.spra.com/publications.shtml
• Mathematica
  http://www.mathematica-mpr.com
• Environmental Protection Agency (EPA)
  http://www.epa.gov/evaluate/workshop/index.htm
• Sage Associates
  http://www.sageways.com/home/et_types.cfm
• Commonwealth Corporation – Center for Research and
  Evaluation
  http://www.commcorp.org/cre/index.html



                                                       30
          Presenter Information


                           Johan Uvin
Timothy Theberge           Director of Adult Literacy
ETA - Region 1             RI Dept. of Elementary and
617-788-0139                  Secondary Education
theberge.timothy@dol.gov   401-222-8950
                           johan.uvin@ride.ri.gov




                                                        31