Evaluating success of 'research to action

Document Sample
Evaluating success of 'research to action Powered By Docstoc
					          Evaluating success of
           ‘research to action’
How to evaluate whether research was translated into
action in the community, practice and policy environment.
How to measure
impact,
direct and indirect outcomes,
success or failure,
short term or long term benefits, etc..
How can we learn from our and others’ mistakes,
communicating successes and failures.
                   Jerry Spiegel MA MSc PhD
                          Director, Global Health
              Director, Liu Institute for Global Issues,
                  Centre for International Health
                     University of British Columbia
                           jerry.spiegel@ubc.ca
          Learning objectives
Be familiar with program evaluation approaches
& techniques

Be familiar with resources that can be used

Consider particular viewpoints for KT evaluation

Develop a network and approach for mutual
learning in an emerging “community of practice
Benefit of a team approach
Transdisciplinary   Intersectoral
            KT Cycle: Framework

                       Publications                    3

Researchers        Questions       Research       Global
Knowledge      1               2   Findings       Knowledge
                   Methods
Users
                                                   4
       6
                                        Contextualization
     Impacts                             of Knowledge

                       Application of         5
                        Knowledge
Knowledge Translation
        INDIVIDUAL




Knowledge       Practice




       SYSTEM
Knowledge Translation
            INDIVIDUAL



Knowledge                Practice



              Policy




            SYSTEM
A framework for research / action interaction

                 Action Into Research
 “Reality” t                    Knowledge t           RESEARCH
                                               Review (incl. relevance)
               Stakeholders
                                               Research question
               -Individuals /
                                               Hypothesis
               communities
                                      ACTION
                                               Methodology
               -Practitioners
                                               Results / Analysis
                  -Policy
                                               Discussion
                                               - implications
 “Reality” t+1              Knowledget1

           Research Into Action
University & Community Expectations of
         Academic Researchers
Functional         Valued Academy             Community
Activity Area      Expectations               Expectations
Research           •Scholarly publications    •Manuals and working
                   (peer reviewed)            papers
                                              •Project evaluation
Applied research   •External grants           •Funded projects
agenda                                        •Problem-solving initiatives
Teaching           •Supervision of student    •Facilitate workshops
                   research                   •Training & development of
                   •Professional mentoring    paraprofessionals
Service            •University & collegiate   •Engage in collaborative &
                   service                    social action research
                   •Professional              •Participate on civic &
                   associations               community boards
        Program Evaluation
•   Overview
•   Steps in evaluation
•   Evaluability assessment
•   Developing a logic model
•   Specifying indicators & outcomes
   What Is Program Evaluation?
“Evaluation is a process of determining the merit,
 worth and value of things, and evaluations are the
 products of that process.“ Scriven (1981)

Comparison of an object of interest against a
standard of acceptability.” Green (1974)
               Why evaluate?
•   Determine achievement of objectives
•   Improve program implementation
•   Answer to stakeholders
•   Increase support for a program
•   Contribute to scientific base of knowledge
•   Inform policy decisions

              Why evaluate KT?
    • Integration of KT in all research
    • Evaluation of specific KT techniques
   Types of Evaluation Activities

• Needs assessment - conditions a program is
  intended to address
• Assessment of program theory -
  conceptualization and design
• Assessment of program process - operations,
  implementation, delivery
• Impact assessment - outcomes, impact
• Efficiency assessment - cost, cost effectiveness
“Social programs are inherently
 inhospitable environments for
     research purposes.”
                    Weiss, 1972
   Evaluation: A rational
enterprise in a political context
  •   Policies/programs are creatures of
      political decisions
  •   Evaluation reports are part of the
      political arena
  •   Evaluation itself represents a
      political stance
Steps For Conducting an Evaluation

 Step 1:    Engage stakeholders
 Step 2:    Describe the program
 Step 3:    Focus the evaluation design
 Step 4:   Gather credible evidence
 Step 5:   Justify conclusions
 Step 6:   Ensure use and share lessons learned

       CDC (US) Evaluation Working Group
                 http://www.cdc.gov/eval
What is evaluability assessment?
•   Process for clarifying program design
•   Determines if a program can be evaluated
•   Useful for planning an evaluation
•   Two products:
      1. Program description
      2. Program logic model
   Step 1: Program Description
• To evaluate must have a clear description
  of program
• Review available documents
• Interview staff, managers
• Concise description with focus on key
  elements
Elements of Program Description
•   Need or problem being addressed
•   Purpose and rationale for program
•   Programs organizational structure
•   Program’s goals and objectives
•   Major activities and components
•   Target population
 Step 2: Develop a Logic Model
• Logic model is a diagram that shows how
  program works
• Typically diagramed on a flow chart with
  program components, activities, goals,
  outputs, and outcomes
• Describes cause and effect relationships
    Step 2: Logic Model cont.,
• Can help identify problems in design of
  programs
• Useful framework to examine outcomes
• Identifies program components that need
  to be tracked
• Prepares evaluator for designing the
  evaluation
    Advantages of Logic Models
• Clarifies program assumptions
• Clarifies goals and objectives
• Relates activities to goals/objectives
• Facilitates agreement about indicators of
  success
• Determines if program has potential to
  achieve outcomes
• Prepares staff/evaluator for evaluation
    Disadvantages of Logic Models

• Can be tedious
• Cannot replace good program planning
• Sometimes difficult to get consensus
• Programs often have little documentation
  about program
• Requires skill in specifying indicators and
  outcomes
    How to Construct a Logic Model

•   Talk to key stakeholders
•   Clarify program rationale
•   Describe program components and activities
•   Define goals of components
•   Determine program outcomes
•   Specify short term and long term outcomes
•   Verify logic model
Basic Elements of a Logic Model

• Components – groups of closely related
  program activities
• Activities – identifiable program tasks
• Goals – General statements about desired
  program direction
• Outputs – products of program activities
Basic Elements of a Logic Model
            cont.
• Outcomes – the results program hopes to
  achieve
• Outcome indicators – each outcome
  should have clearly defined markers of
  success (specific, measurable, relevant)
          What is an indicator?
• Indicators are the specific items of information
  that describe observable, measurable
  characteristics or changes in corresponding
  outcomes
• It is specific, observable, and linked to desired
  outcome
• Examples:
   – 60% of program participants report they know where
     to go for vision care
   – 25% of program participants report they used vision
     care in the past year
  Example: Outcome with Single Indicator
          Outcome
                                   Improved Academic Performance




           Indicator                        Improved Grades




            Potential                       Official Statistics:
         Method for Data                    Improvement in
           Collection                   participant GPA overtime


                    Is this a valid and reliable indicator?
Not everything that can be COUNTED necessarily COUNTS (i.e. is important)!
                                                                                                                      For Discussion
                                                                                                   General Evaluation Logic Model:
                                                                                                    UBC Undergraduate Program

                                                                                                   Inputs                    Process                        Outputs
                                                                                                   •Curriculum           •   Formal Curriculum              Completion of
                              •Lack of access to care




                                                                                                                         I. Orientation                     UGP
                                                                              •Expansion UBC UGP


                                                                                                   •Faculty              II. Human biology
                                                        •Health disparities
          •Shortage of MD’s




                                                                                                                             Foundations medicine           •Competencies
                                                                                                   •Students
                                                                                                                             DPAS
Context




                                                                                                                                                            •Qualifying
                                                                                                   •Facilities               Clinical skills
                                                                                                                             Family practice                 exam
                                                                                                   •Funding
                                                                                                                         III.Rural & underserved            •Degree
                                                                                                   •Other?                   community practice
                                                                                                                                                            •Other?
                                                                                                                         IV.Clinical Clerkship
                                                                                                                         V. Advanced clinical electives/
                                                                                                                             selectives
                                                                                                                         VI. Learning skills for practice

                                                                                                                          • Informal Curriculum

                                                                                                                          • Hidden Curriculum

                                                                                                              Process & Implementation                      Immediate
                                                                                                              Evaluation                                    Outputs
                                                    For Discussion
                                     General Evaluation Logic Model:
                                      UBC Undergraduate Program

                                          Products

                                                            Working in
                                          Completion of     academic/         Academic
                                          residency          research         Physicians
                                          placement          settings
                                          •Generalists
                              Immediate
                               Outcomes




                                                           Working in                         Reduced
Context




                    Process




                                          •Specialists
           Inputs




                                                           clinical         Access to          Health
                                          •Peer review                                       Disparities
                                                           practice         clinical care
                                          •Certification                                        in BC
                                                           •Setting
                                          •Licensing       •BC/Canada
                                          •Research        •Rural/urban         Public        Enhanced
                                          training         •Aboriginal          Health       Community
                                                                            Infrastructure    Capacity
                                          •Other?          •Other?

                                          Intermediate                Long-term               Goals
                                          Outcomes                    Outcomes

      2 February 2004
 Model to evaluate the effectiveness
   of the Cayo Hueso intervention
Inputs              Outputs          Results                      Impacts
                                      Needs reduced
 MATERIALS                                                       Averted costs
  - ministries     HOUSES REPAIRED
   - residents                         Improved housing
                   STREET REPAIRS         conditions
    LABOUR
   - ministries        LIGHTS
    - residents                       Improved cultural life         Improved
                      CULTURAL           in community                 Health
                      ACTIVITES
ADMINISTRATIVE
TIME, TRAVEL etc    WATER SUPPLY      Improved safety /
                    CONNECTIONS       feeling of security             Improved
                                                                   satisfaction and
   Community
                     SOLID WASTE                                    quality of life
    Support
                      COLLECTED
                                       Improved living
                                        conditions in
                                         community              At individual and
                      Community                                 community level
                      Mobilization      Community
                                        integration
                                       maintained and
                                         enhanced
                                     Social cohesion enhanced
         The Results Chain
Inputs   Activities   Outputs   Outcomes   Impact
                     A result can be…

• An improvement (e.g. in the health conditions)

• An increase (e.g. of the revenues of a given community)

• A strengthening (e.g. of the capacities of local partners)

• A reduction (e.g. in the infant mortality rate)

• A transformation in the attitudes, the practices of
  behaviour of a given group (e.g. adoption of
  contraceptive methods by the women of a region and
  acceptance of birth-control methods by the men)


                   A KT result can be…
                     Impacts

• Impacts are long-term socio-economic results
  that usually happen beyond the life of the project
  and are attributable to the achievements of
  outcomes.

• Impacts involve changes in living conditions of
  populations in developing countries.

• Impacts are the changes necessary to achieve
  the project goal.
                        Outcomes

• Outcomes are specific measurable institutional and
  community level results that will be produced by the end-
  date of the project.

• Outcomes, which are the logical results of project
  outputs, include new programs or processes that will be
  sustained after the project completion.

• Outcomes are the changes necessary to achieve the
  project purpose.
                             Review
Goal/Impact         WHAT change in society do Changes that
                    we want to contribute to? occur beyond the
                                              life of the project.
Purpose/Outcomes WHAT changes will be                 Measurable
                 realized on institutions and         changes that occur
                 communities?                         during the life of
                 Outcomes contribute to the           the project.
                 project impact.
Outputs             WHAT first level of change        Measurable
                    will follow project activities?   changes that occur
                    Outputs lead to project           during the life of
                    outcomes.                         the project.
                      Review (continued)

Unexpected WHAT changes have            Measurable changes that
Results    been noticed that were       occur during or beyond the
           not planned for at the       life of the project.
           outset of the project.
Activities   HOW will the change        Actions that will lead to
             happen?                    planned and unexpected
                                        changes.

Inputs       MEANS necessary to         Financial resources
             carry out the activities   Human resources
             that will engender         Organizational resources
             changes
                                        Material and equipment
                                        Time
              Outputs

• Outputs are the measurable results
  necessary to produce the outcomes.

• Outputs are short-term consequences
  of completed activities.
                      Role Play

     • Study Advisory Committee
        -   Member of Government Ministry
        -   Member of local community NGO
        -   Practitioner
        -   Researcher (PI)


               DISCUSS / DEBATE THE SELECTION
OF AT LEAST 3 INDICATORS OF SUCCESSFUL KT FOR YOUR PROJECT
  Group Development Sequence
• Forming
  – Testing and dependence
  – Orientation to the task “Ground Rules”
• Storming
  – Intragroup Conflict
  – Lack of unity
• Norming
  – Development of group cohesion
  – Open exchange
• Performing
  – Functional role relatedness
  – Emergence of solutions