Program Evaluation Webinar Series Part 1 by hcj

VIEWS: 0 PAGES: 159

									Program Evaluation Webinar Series Part 2:
           “Getting Started and
        Engaging Your Stakeholders”
 Presented by: Leslie Fierro and Carlyn Orians
Getting Started and Engaging
      Your Stakeholders
      Leslie A. Fierro, MPH
      TKCIS Contractor NCEH/CDC
      Let6@cdc.gov



      Carlyn Orians, MA,
      Battelle Centers for Public Health
      Research and Evaluation
      orians@battelle.org


      Presented November 20, 2008
               Agenda

Definition of evaluation.

Difference between evaluation & research.
CDC’s Evaluation Framework.

The who, why, when & how of
stakeholders.

Tangible examples of engaging
stakeholders.
         What is the Purpose of
             Evaluation?

The purpose of evaluation can be:


           social betterment

          program planning and decision making

          empowerment of individuals
      Evaluation: Definition 1

Evaluation is the systematic investigation of
the merit, worth, or significance of an object.
                                -- Michael Scriven
      Evaluation: Definition 2

Evaluation is the systematic assessment of the
operation and/or the outcomes of a program or
policy, compared to a set of explicit or implicit
standards as a means of contributing to the
improvement of the program or policy.
                             -- Carol Weiss
      Evaluation: Definition 3

Evaluation is the systematic collection of
information about the activities, characteristics,
and outcomes of programs to make judgments
about the program, improve program
effectiveness, and/or inform decisions about
future programming.
                                -- Michael Patton
      The Common Element
Note that all 3 of these statements share a
common element. ..
                      …they all define evaluation as
            a systematic and formalized endeavor.
       Summative Evaluations

Summative evaluations seek to judge a program by
asking, “Should this project be…”
       cancelled?
       continued?
       expanded?
         Formative Evaluations

Formative evaluations seek to use the evaluation
findings.
They ask the question:
       “Is the program being conducted
       as planned?”
       “Is the program doing well?”

Formative evaluations seek to
improve programs or policies.
         The CDC Definition

Evaluation is the systematic collection of
  information about the activities, characteristics
  and outcomes of the program to make
  judgments about the program, improve
  program effectiveness and/or inform decisions
  about future program development.
    Research vs. Evaluation

Research and evaluation share methodologies
         but ask different questions.

 Specific to   Commonalities   Specific to
  Research                     Evaluation
      The Research Context

               To add to an existing body of
Purpose
                  theoretical knowledge.

            Does the theory hold in this situation?
 Typical
Questions   How can we improve the theory to make
             better predictions in every situation?
         Research Findings


Research is conducted to:
       test and improve theories.
       develop generalizable theories.
       theories apply across different
       settings, people, and times.
      The Evaluation Context

             Judge the merit or worth of a program.
Purpose
            Provide information for decision making.

             Is the program doing what was
 Typical       intended?
Questions
             What steps might be taken to improve it?
           Evaluation Findings


Evaluation results:

     are not usually generalizable.
     focus on a specific situation.
     evaluation of a single program in
     a particular context.
     Research vs. Evaluation

Evaluation asks:
   “Is this program working?”

Research asks:
  “Will this program work across multiple settings?”
     The Role of Researchers

Researchers:
     Play a single role as content experts.
     The Role of Researchers

Researchers:
     Play a single role as content experts.
     Identify gaps in current knowledge.
     The Role of Researchers

Researchers:
     Play a single role as content experts.
     Identify gaps in current knowledge.
     Derive their own questions.
     The Role of Researchers

Researchers:
     Play a single role as content experts.
     Identify gaps in current knowledge.
     Derive their own questions.
     Perform their own research.
       The Role of Evaluators

Evaluators:
       Play multiple roles—facilitator,
       educator, scientific expert, etc.
       The Role of Evaluators

Evaluators:
       Play multiple roles—facilitator,
       educator, scientific expert, etc.
       Involve stakeholders.
       The Role of Evaluators

Evaluators:
       Play multiple roles—facilitator,
       educator, scientific expert, etc.
       Involve stakeholders.
       Collaborate to identify and prioritize
       questions.
            In a Nutshell…


“Research seeks to prove,
     evaluation seeks to improve.”
                       M.Q. Patton
    The Findings Must be Useful

To improve a program, the findings must be useful!
How?
       The stakeholders must be involved.
       The questions must be relevant to the
       program.
       The findings must be credible to key
       stakeholders.
Getting Started and Engaging
      Your Stakeholders



  The Who, When, Why, and How
     of Stakeholder Involvement
  in CDC’s Evaluation Framework
       “Who” are the Stakeholders?

Stakeholders are:
      people and/or organizations that are
      interested in the program, are
      interested in the results of the evaluation and/or
      have a stake in what will be done with the
      results of the evaluation.
     Examples of Stakeholders

Persons involved in program operations
• Example: asthma program staff and partners

Persons served or affected by the program
• Example: participants in asthma interventions,
  asthma patients, families, caregivers, etc.
Intended users of evaluation findings
• Example: policy makers, managers,
  administrators, advocates, funders, and others
       Each Program is Different

Develop a list of stakeholders at the start of any
  evaluation activity.
 Which Stakeholders Matter Most?

Review your list of stakeholders and think strategically
  about these questions:
   “Who do we need to…”
 Which Stakeholders Matter Most?

Review your list of stakeholders and think strategically
  about these questions:
   “Who do we need to…”
      enhance credibility?
 Which Stakeholders Matter Most?

Review your list of stakeholders and think strategically
  about these questions:
   “Who do we need to…”
      enhance credibility?
      implement program changes?
 Which Stakeholders Matter Most?

Review your list of stakeholders and think strategically
  about these questions:
   “Who do we need to…”
      enhance credibility?
      implement program changes?
      advocate for changes?
 Which Stakeholders Matter Most?

Review your list of stakeholders and think strategically
  about these questions:
   “Who do we need to…”
      enhance credibility?
      implement program changes?
      advocate for changes?
      fund, authorize, or expand the program?
  “Why” Engage Stakeholders?

Engaging stakeholders is an important part of
the CDC Evaluation Framework.
 Involving stakeholders may be a requirement of
  your program.
  “Why” Engage Stakeholders?

Engaging stakeholders is an important part of
the CDC Evaluation Framework

Stakeholders will add credibility.
 If you want the results to be acted upon, they
  must be credible.
  “Why” Engage Stakeholders?

Engaging stakeholders is an important part of
the CDC Evaluation Framework.
Stakeholders will add credibility.

Stakeholders may have resources to help.
 Stakeholders may be able to contribute data,
   skills, analytical skills, etc.
  “Why” Engage Stakeholders?

Engaging stakeholders is an important part of
the CDC Evaluation Framework.
Stakeholders will add credibility.
Stakeholders may have resources to help.

Stakeholders may be critical to implementing or
advocating for action based on the results.
  “Why” Engage Stakeholders?

Engaging stakeholders is an important part of
the CDC Evaluation Framework.
Stakeholders will add credibility.
Stakeholders may have resources to help.
Stakeholders may be critical to implementing or
advocating for action based on the results.
You will build trust and understanding among
program constituents.
 Involving stakeholders helps to reduce
  fear of the evaluation process.
  “Why” Would Stakeholders
    Want to be Involved?

Get answers to their questions.
  “Why” Would Stakeholders
    Want to be Involved?

Get answers to their questions.
Learn about evaluation.
  “Why” Would Stakeholders
    Want to be Involved?

Get answers to their questions.
Learn about evaluation.
Influence the design and methods.
  “Why” Would Stakeholders
    Want to be Involved?

Get answers to their questions.
Learn about evaluation.
Influence the design and methods.
Protect their constituents.
  “Why” Would Stakeholders
    Want to be Involved?

Get answers to their questions.
Learn about evaluation.
Influence the design and methods.
Protect their constituents.
Motivated to help program succeed.
CDC’s Evaluation Framework
                    STEPS
                    Engage
                 stakeholders
    Ensure use                    Describe
    and share                       the
                   Standards
     lessons                      program
                     Utility
     learned
                   Feasibility
                    Propriety
                    Accuracy
                                 Focus the
       Justify                   evaluation
     conclusions                   design
                    Gather
                   credible
                   evidence
     “When” and “How” Can You
       Engage Stakeholders?
You can use the CDC Evaluation Framework to
  engage stakeholders in:
    identifying and prioritizing evaluation
     questions,
    selecting credible sources, and
    developing a plan for action based on
     evaluation results.
      “When” and “How” Can You
        Engage Stakeholders?
You can engage stakeholders in every step of
  the evaluation process.

But… you don’t have to engage stakeholders
     in all these ways in every evaluation.
  “How” Depends on…

Evaluator preference.
  “How” Depends on…

Evaluator preference.
Stakeholder preference.
  “How” Depends on…

Evaluator preference.
Stakeholder preference.
Resources.
   “How” Depends on…

Evaluator preference.
Stakeholder preference.
Resources.
Degree of trust or threats to credibility.
       “How” Depends on…

    Evaluator preference.
    Stakeholder preference.
    Resources.
    Degree of trust or threats to credibility.

If there is a high degree of mistrust, engage
stakeholders early in the evaluation
process. This helps ensure that the results
are viewed as credible and are acted upon.
CDC’s Evaluation Framework
                    STEPS
                    Engage
                 stakeholders
    Ensure use                    Describe
    and share                       the
                   Standards
     lessons                      program
                     Utility
     learned
                   Feasibility
                    Propriety
                    Accuracy
                                 Focus the
       Justify                   evaluation
     conclusions                   design
                    Gather
                   credible
                   evidence
               CDC’s Framework Step 1
               STEP 1                     Who should be involved?
                Engage
                 Engage
             stakeholders                    Develop list of
              stakeholders
                                               potential
Ensure use
and share
                              Describe         stakeholders.
              Standards         the
 lessons
                Utility       program        Decide which
 learned
              Feasibility                      stakeholders are
               Propriety
               Accuracy
                                               the most important
                             Focus the
  Justify                    evaluation
                                               to include.
conclusions                    design
               Gather
              credible
              evidence
              CDC’s Framework Step 2
               STEP 1                    Do stakeholders share a vision
                Engage                   of what the program does and
             stakeholders                its intended outcomes?
Ensure use                   Describe
                             Describe
and share
              Standards         the
                               the          Diverse views?
 lessons
 learned
                Utility      program
                             program        Similar views?
              Feasibility
               Propriety
                                            Engage them early in the
               Accuracy
                            Focus the
                                             process.
  Justify                   evaluation
conclusions                   design
               Gather
              credible
              evidence
              CDC’s Framework Step 3
               STEP 1                  What are the most pressing and
                Engage                 important evaluation questions
             stakeholders              for stakeholders?
Ensure use
and share
                              Describe      What questions do they
              Standards         the
 lessons                      program         need answered to be able
                Utility
 learned
              Feasibility                     to take action?
               Propriety                    What methods are
               Accuracy                       available and preferred?
                            Focus the
                             Focus the
  Justify
conclusions
                            evaluation
                             evaluation     What will be considered
                              design
               Gather          design         “credible evidence”?
              credible
              evidence
              CDC’s Framework Step 4
               STEP 1                    Gather evidence
                Engage
             stakeholders
                                         stakeholders will find
Ensure use                   Describe
                                         credible.
and share                      the
              Standards
 lessons                     program
                Utility
 learned
              Feasibility
               Propriety
               Accuracy
                            Focus the
  Justify                   evaluation
conclusions    Gather         design
               Gather
               credible
              credible
              evidence
              evidence
               CDC’s Framework Step 5
                STEP 1
                                          How do diverse stakeholders
                Engage                    interpret the findings?
             stakeholders                     May engage
Ensure use                    Describe          stakeholders in the
and share
               Standards        the             analysis.
 lessons
 learned
                 Utility      program
                                              Perhaps solicit their
               Feasibility
                Propriety
                                                interpretation before
                Accuracy
                             Focus the
                                                results are finalized.
   Justify
   Justify                   evaluation
conclusions
 conclusions                   design
                Gather
               credible
               evidence
              CDC’s Framework Step 6
              STEP 1                    Which stakeholders can play
               Engage                   a role in disseminating
 Ensure    stakeholders                 results or acting on findings?
use and
Ensure use
  share
 and share
                            Describe         This is a critical step
            Standards         the
lessons
  lessons                   program             for stakeholder
               Utility
learned
  learned
             Feasibility                        involvement.
              Propriety
              Accuracy
                           Focus the
  Justify                  evaluation
conclusions                  design
               Gather
              credible
              evidence
Getting Started and Engaging
      Your Stakeholders


      Tangible Examples of
     Stakeholder Involvement
    in the Evaluation Process
    Example #1 – Asthma and Home
    Environment in Low-Income Apts
Intervention to improve indoor environment for
   children with asthma via following activities:
    Example #1 – Asthma and Home
    Environment in Low-Income Apts
Intervention to improve indoor environment for
   children with asthma via following activities:
       Provide education and training for apartment
       owners, building inspectors, maintenance
       vendors, and tenants regarding asthma
       triggers and housing codes.
   Example #1 – Asthma and Home
   Environment in Low-Income Apts
Intervention to improve indoor environment for
   children with asthma via following activities:
        Provide education and training for apartment
        owners, building inspectors, maintenance
        vendors, and tenants regarding asthma
        triggers and housing codes.
        Work with city officials to enhance existing
        housing code.
   Example #1 – Asthma and Home
   Environment in Low-Income Apts
Intervention to improve indoor environment for
   children with asthma via following activities:
        Provide education and training for apartment
        owners, building inspectors, maintenance
        vendors, and tenants regarding asthma
        triggers and housing codes.
        Work with city officials to enhance existing
        housing code.
        Promote smoke-free housing.
CDC’s Evaluation Framework
                    STEPS
                    Engage
                 stakeholders
    Ensure use                    Describe
    and share                       the
                   Standards
     lessons                      program
                     Utility
     learned
                   Feasibility
                    Propriety
                    Accuracy
                                 Focus the
       Justify                   evaluation
     conclusions                   design
                    Gather
                   credible
                   evidence
Step 1: Engage the Stakeholders

         Low-Income Housing Scenario

Who are the stakeholders?
Step 1: Engage the Stakeholders

         Low-Income Housing Scenario

Who are the stakeholders?
 affected families
Step 1: Engage the Stakeholders

         Low-Income Housing Scenario

Who are the stakeholders?
 affected families
 those who receive training (e.g., owners,
  inspectors, maintenance personnel, tenants)
Step 1: Engage the Stakeholders

         Low-Income Housing Scenario

Who are the stakeholders?
 affected families
 those who receive training (e.g., owners,
  inspectors, maintenance personnel, tenants)
 trainers
Step 1: Engage the Stakeholders

          Low-Income Housing Scenario

Who are the stakeholders?
 affected families
 those who receive training (e.g., owners,
  inspectors, maintenance personnel, tenants)
 trainers
 city officials responsible for the housing code
Step 1: Engage the Stakeholders

          Low-Income Housing Scenario

Who are the stakeholders?
 affected families
 those who receive training (e.g., owners,
  inspectors, maintenance personnel, tenants)
 trainers
 city officials responsible for the housing code
 smoke-free advocates
Step 1: Engage the Stakeholders

            Low-Income Housing Scenario

Who are the stakeholders?
 affected families
 those who receive training (e.g., owners,
  inspectors, maintenance personnel, tenants)
 trainers
 city officials responsible for the housing code
 smoke-free advocates

Stakeholders may include advocates as well as those
resistant to change . It’s important to include a mix of
views so you don’t get surprised at the end.
 Step 2: Describe the Program

         Low-Income Housing Scenario

Obtain stakeholders’ views on:
 Step 2: Describe the Program

         Low-Income Housing Scenario

Obtain stakeholders’ views on:

 What is the current indoor environment?
 Step 2: Describe the Program

         Low-Income Housing Scenario

Obtain stakeholders’ views on:

 What is the current indoor environment?
 What changes are both desired and seen as
  realistic?
 Step 2: Describe the Program

         Low-Income Housing Scenario

Obtain stakeholders’ views on:

 What is the current indoor environment?
 What changes are both desired and seen as
  realistic?
 What outcomes do they think are most
  important?
 Step 2: Describe the Program

         Low-Income Housing Scenario

Obtain stakeholders’ views on:

 What is the current indoor environment?
 What changes are both desired and seen as
  realistic?
 What outcomes do they think are most
  important?
 Which activities contribute to which outcomes?
 Step 2: Describe the Program

       Low-Income Housing Scenario
Obtain stakeholders’ views on:

 What is the current indoor environment?
 What changes are both desired and seen as
  realistic?
 Which outcomes do they think are most
  important?
 Which activities contribute to which outcomes?
   (You could use a logic model here… Stay tuned for
   the next Webinar in this series…)
            Step 3: Focus
        the Evaluation Design
       Low-Income Housing Scenario
What questions do the stakeholders think are the
  most important to ask?
            Step 3: Focus
        the Evaluation Design
       Low-Income Housing Scenario
What questions do the stakeholders think are the
  most important to ask?

 Should the evaluation focus on changes to the
  housing code?
            Step 3: Focus
        the Evaluation Design
       Low-Income Housing Scenario
What questions do the stakeholders think are the
  most important to ask?

 Should the evaluation focus on changes to the
  housing code?
 Focus on trigger reduction? Which triggers?
            Step 3: Focus
        the Evaluation Design
       Low-Income Housing Scenario
What questions do the stakeholders think are the
  most important to ask?

 Should the evaluation focus on changes to the
  housing code?
 Focus on trigger reduction? Which triggers?
 What is the best design?
             Step 3: Focus
         the Evaluation Design
        Low-Income Housing Scenario
What questions do the stakeholders think are the
  most important to ask?

   Should the evaluation focus on changes to the
   housing code?
   Focus on trigger reduction? Which triggers?
   What is the best design?
   Is a pre-post design best?
            Step 3: Focus
        the Evaluation Design
       Low-Income Housing Scenario
What questions do the stakeholders think are the
  most important to ask?

 Should the evaluation focus on changes to the
  housing code?
 Focus on trigger reduction? Which triggers?
 What is the best design?
 Is a pre-post design best?
 Should there be a control group?
            Step 3: Focus
        the Evaluation Design
       Low-Income Housing Scenario
What questions do the stakeholders think are the
  most important to ask?

 Should the evaluation focus on changes to the
  housing code?
 Focus on trigger reduction? Which triggers?
 What is the best design?
 Is a pre-post design best?
 Should there be a control group?
 Are observational methods appropriate?
   Step 4: Gather Credible Evidence
            Low-Income Housing Scenario

What role can stakeholders play?
   Step 4: Gather Credible Evidence
             Low-Income Housing Scenario

What role can stakeholders play?
 Are there stakeholders that can facilitate access to
  residences and/or translate if necessary?
   Step 4: Gather Credible Evidence
             Low-Income Housing Scenario

What role can stakeholders play?
 Are there stakeholders that can facilitate access to
  residences and/or translate if necessary?
 Can the trainers help collect data before, during, or after
  each training? Other opportunities for data collection?
   Step 4: Gather Credible Evidence
             Low-Income Housing Scenario

What role can stakeholders play?
 Are there stakeholders that can facilitate access to
  residences and/or translate if necessary?
 Can the trainers help collect data before, during, or after
  each training? Other opportunities for data collection?
 What information will the stakeholders need to act? For
  example, will policy makers be more influenced by
  information on environmental conditions, asthma
  symptoms, or medical care usage?
   Step 4: Gather Credible Evidence
             Low-Income Housing Scenario

What role can stakeholders play?
 Are there stakeholders that can facilitate access to
  residences and/or translate if necessary?
 Can the trainers help collect data before, during, or after
  each training? Other opportunities for data collection?
 What information will the stakeholders need to act? For
  example, will policy makers be more influenced by
  information on environmental conditions, asthma
  symptoms, or medical care usage?
 Data in what form? Pictures? Stories? Numerical data?
 Step 5: Justify Conclusions
       Low-Income Housing Scenario


How do stakeholders view success? What does
  “success” look like to them?
  Step 5: Justify Conclusions
         Low-Income Housing Scenario


How do stakeholders view success? What does
  “success” look like to them?
 Do stakeholders have alternative
  interpretations of the initial findings?
  Step 5: Justify Conclusions
         Low-Income Housing Scenario


How do stakeholders view success? What does
  “success” look like to them?
 Do stakeholders have alternative
  interpretations of the initial findings?
 Do they perceive barriers you might have
  missed?
    Step 6: Ensure Use and
    Share Lessons Learned
        Low-Income Housing Scenario


Can stakeholders help disseminate evaluation
  findings?
    Step 6: Ensure Use and
    Share Lessons Learned
        Low-Income Housing Scenario


Can stakeholders help disseminate evaluation
  findings?
 Is there an owner’s newsletter? A tenant
  meeting? A neighborhood association?
     Step 6: Ensure Use and
     Share Lessons Learned
        Low-Income Housing Scenario


Can stakeholders help disseminate evaluation
  findings?
 Is there an owner’s newsletter? A tenant
  meeting? A neighborhood association?
 Is information needed in a different form to
  prompt action? A report? An executive brief?
  An oral presentation?
     Step 6: Ensure Use and
     Share Lessons Learned
        Low-Income Housing Scenario


Can stakeholders help disseminate evaluation
  findings?
 Is there an owner’s newsletter? A tenant
  meeting? A neighborhood association?
 Is information needed in a different form to
  prompt action? A report? An executive brief?
  An oral presentation?
 Is there interest beyond the community
  involved in the intervention?
What if you ignored stakeholders?

If you ignore the stakeholders, potential pitfalls include…
What if you ignored stakeholders?

If you ignore the stakeholders, potential pitfalls include…


 Failure to ask important questions.

    • For example, it may be important to know if
      tenants have access to quit-smoking services.
What if you ignored stakeholders?

If you ignore the stakeholders, potential pitfalls include…


 Difficulty gaining access to or trust of program
 constituents.

    • For example, will you gain access to data or be
      received into residents’ homes?
What if you ignored stakeholders?

If you ignore the stakeholders, potential pitfalls include…


 Failure to uncover unanticipated
 consequences.
     • For example, if improvements are made, will
       rents increase?
What if you ignored stakeholders?

If you ignore the stakeholders, potential pitfalls include…


 Failure to communicate results in ways that are
 understood and/or accessible.

   • For example, are there literacy or language
     issues? Cultural preferences?
What if you ignored stakeholders?

  If you fail to involve the stakeholders…

      …your evaluation may not lead to action!
   Example #2 – Care Coordination
      Across Health Systems
Intervention to provide and integrate care coordination
   and case management for high-risk children with
   asthma.




                                                          105
   Example #2 – Care Coordination
      Across Health Systems
Intervention to provide and integrate care coordination
   and case management for high-risk children with
   asthma.
Involves standardizing protocols across care systems,
   including:
      Medicaid HMOs
      Home nursing agencies
      Health departments




                                                          106
   Example #2 – Care Coordination
      Across Health Systems
Intervention to provide and integrate care coordination
   and case management for high-risk children with
   asthma.
Involves standardizing protocols across care systems,
   including:
      Medicaid HMOs
      Home nursing agencies
      Health departments

The goal: to prove success and thus convince
   insurers to continue reimbursing case
   management services.



                                                          107
    Engage the Stakeholders
  Housing Scenario    Healthcare Scenario


 affected families   Who are the
 those who receive   stakeholders
                      in this scenario?
  training
 trainers
 city officials
 smoke-free
  advocates
    Engage the Stakeholders
  Housing Scenario    Healthcare Scenario


 affected families    insurers
 those who receive
  training
 trainers
 city officials
 smoke-free
  advocates
    Engage the Stakeholders
  Housing Scenario    Healthcare Scenario


 affected families    insurers
 those who receive    case managers
  training
 trainers
 city officials
 smoke-free
  advocates
    Engage the Stakeholders
  Housing Scenario    Healthcare Scenario


 affected families    insurers
 those who receive    case managers
  training             families
 trainers
 city officials
 smoke-free
  advocates
    Engage the Stakeholders
  Housing Scenario    Healthcare Scenario


 affected families    insurers
 those who receive    case managers
  training             families
 trainers             health system
 city officials        administrators
 smoke-free
  advocates
  Focus the Evaluation Design
   Housing Scenario     Healthcare Scenario


 What changes are      What outcome
  both desired and      questions?
  seen as realistic?
 What outcomes do
  they think are most
  important?
 Which activities
  contribute to which
  outcomes?
  Focus the Evaluation Design
   Housing Scenario     Healthcare Scenario


 What changes are       Is this intervention
  both desired and        sustainable?
  seen as realistic?
 What outcomes do
  they think are most
  important?
 Which activities
  contribute to which
  outcomes?
  Focus the Evaluation Design
   Housing Scenario     Healthcare Scenario


 What changes are       Is this intervention
  both desired and        sustainable?
  seen as realistic?     What is the cost of
 What outcomes do        doing this kind of
  they think are most     case management?
  important?
 Which activities
  contribute to which
  outcomes?
   Gather Credible Evidence
 Housing Scenario      Healthcare Scenario


 Data in what form?   What type of data do
  Pictures? Stories?   they need to make a
  Numerical data?      decision?
 Stakeholders that
  can facilitate
  access ?
 Trainers help
  collect data ?
   Gather Credible Evidence
 Housing Scenario      Healthcare Scenario


 Data in what form?    Is quantitative data
  Pictures? Stories?     more highly valued
  Numerical data?        than qualitative
 Stakeholders that      data?
  can facilitate
  access ?
 Trainers help
  collect data ?
   Gather Credible Evidence
 Housing Scenario      Healthcare Scenario


 Data in what form?    Is quantitative data
  Pictures? Stories?     more highly valued
  Numerical data?        than qualitative
 Stakeholders that      data?
  can facilitate        Do the data need to
  access ?               show cost savings?
 Trainers help
  collect data ?
   Gather Credible Evidence
 Housing Scenario       Healthcare Scenario


 Data in what form?    Be sensitive to the
  Pictures? Stories?     stakeholders’
  Numerical data?        interests and time
 Stakeholders that      constraints.
  can facilitate
  access ?
 Trainers help
  collect data ?
   Gather Credible Evidence
 Housing Scenario       Healthcare Scenario


 Data in what form?    Be sensitive to the
  Pictures? Stories?     stakeholders’
  Numerical data?        interests and time
 Stakeholders that      constraints.
  can facilitate        Be aware of and
  access ?               consider the
 Trainers help          political
  collect data ?         environment.
   Example #3 – Daycare Education
Intervention to train childcare providers to identify
triggers and to manage children with asthma.
   Example #3 – Daycare Education
Intervention to train childcare providers to identify
triggers and to manage children with asthma.

Intervention reaches out to:
       Large daycare centers
       Licensed home daycare programs
       Participation is encouraged through
       continuing education credits
   Example #3 – Daycare Education
Intervention to train childcare providers to identify
triggers and to manage children with asthma.

Intervention reaches out to:
       Large daycare centers
       Licensed home daycare programs
       Participation is encouraged through
       continuing education credits

Long-term goal: to expand beyond pilot
providers to larger community.
    Engage the Stakeholders
  Housing Scenario     Daycare Scenario


 affected families   Who would be the
 those who receive   stakeholders in this
                      scenario?
  training
 trainers
 city officials
 smoke-free
  advocates
    Engage the Stakeholders
  Housing Scenario        Daycare Scenario


 affected families      trainers
 those who receive      daycare providers
  training               licensing boards
 trainers               families
 city officials
 smoke-free
  advocates
  Focus the Evaluation Design
   Housing Scenario      Daycare Scenario


 What changes are      What outcome
  both desired and      questions?
  seen as realistic?
 What outcomes do
  they think are most
  important?
 Which activities
  contribute to which
  outcomes?
  Focus the Evaluation Design
   Housing Scenario      Daycare Scenario


 What changes are       behavior change
  both desired and       environmental
  seen as realistic?      outcomes
 What outcomes do       evidence to support
  they think are most     decision to expand
  important?              beyond pilot
 Which activities
  contribute to which
  outcomes?
 Focus the Evaluation Design
  Housing Scenario      Healthcare Scenario


 Focus on changes to   What design mix?
  the housing code?
 Trigger reduction?
 Should there be a
  control group?
 Observational
  methods?
 Focus the Evaluation Design
  Housing Scenario      Healthcare Scenario


 Focus on changes to    Consider pre-post
  the housing code?       design.
 Trigger reduction?     Consider control
 Should there be a       group design.
  control group?
 Observational
  methods?
   Gather Credible Evidence
 Housing Scenario      Healthcare Scenario


 Data in what form?   What data collection
  Pictures? Stories?   methods?
  Numerical data?
 Stakeholders that
  can facilitate
  access ?
 Trainers help
  collect data ?
   Gather Credible Evidence
 Housing Scenario      Healthcare Scenario


 Data in what form?    Success stories
  Pictures? Stories?     may be important .
  Numerical data?
 Stakeholders that
  can facilitate
  access ?
 Trainers help
  collect data ?
Getting Started and Engaging
      Your Stakeholders


       Tangible Examples of
    Stakeholder Involvement in
 Applying the Evaluation Standards
Evaluation Standards
 Apply to Every Step
                 STEPS
                 Engage
              stakeholders
 Ensure use                    Describe
 and share                       the
                Standards
  lessons                      program
                  Utility
  learned
                Feasibility
                 Propriety
                 Accuracy
                              Focus the
    Justify                   evaluation
  conclusions                   design
                 Gather
                credible
                evidence
The Evaluation Standards


         Standards

         •   Utility
         •   Feasibility
         •   Propriety
         •   Accuracy
   The Evaluation Standards


There are actually 30 evaluation standards
grouped into four categories.

A complete list of the standards is published
in CDC’s Framework for Program Evaluation
in Public Health. (See link under “Learning
Aids”.)
          Why Use Standards?
Standards provide a way to:
      Make difficult decisions when
      designing and implementing an
      evaluation.
      Judge the quality of an evaluation.
      Determine where an evaluation can be
      better balanced.
        The “Utility” Standard
 Who needs the evaluation information and
     what information do they need?

 Identify affected stakeholders.
 Collect data that addresses important
  questions.
 Ensure the data is responsive to the needs of
  the stakeholders.
    The “Feasibility” Standard
         How much money, time, and
         effort can we put into this?

 Keep the burden of data collection as low as
  possible.
 Consider the political interests of the
  stakeholders.
 Ensure that the evaluation is cost effective.
     The “Propriety” Standard
       What steps need to be taken for
        the evaluation to be ethical?

 Ensure the ethical treatment of evaluation
  participants.
 Ensure the ethical treatment of those who
  commissioned the evaluation.
 Ensure the ethical treatment of those affected
  by the findings of the evaluation.
    The “Accuracy” Standard
           What design will lead to
           accurate information?

 Clearly and accurately document the program.
 Gather data in a way that produces reliable
  information.
 Analyze and report the data in a systematic and
  unbiased way.
 Assure that the source of the information is
  defensible.
     CDC’s Asthma Control Program

CDC’s Asthma Control Program funds 35
states and territories to implement statewide
asthma control programs using a public
health perspective.

Their approach involves:
     Engaging, enhancing, and maintaining
     relationships with partners.
     Developing, improving, and conducting
     asthma surveillance activities.
     Designing and implementing
     interventions with partners.
      Example #4 - Asthma Program
          Monitoring System

This example involves collecting information
for the purpose of evaluation across multiple
sites.
     Example #4 - Asthma Program
         Monitoring System

Purpose:

    Develop a strategic, systematic
    approach to collecting information
    about our program activities, progress,
    and accomplishments across all funded
    states.
Employing the Framework Model
 Step 1: Involve the stakeholders
   CDC Air Pollution and Respiratory
    Health Branch.
   Representatives of funded State
    Asthma Programs.
 Step 2: Describe the Program
   Develop logic models.
 Step 3: Focus the evaluation design
   Identify and prioritize evaluation
    questions.
 Step 4: Gather credible evidence
          Developing the
     Data Collection Instrument

In this example, developing the data collection
   instrument was a highly collaborative
   endeavor aided by the program evaluation
   standards.
      The Iterative Process
CDC internal workgroup identifies core
information needs and drafts survey
instrument.
State workgroup members review
instrument, pilot test, provide pilot data
and comments.
Create mock report to illustrate type of
information obtained from instrument.
CDC internal workgroup discusses
comments and mock report.
    Applying the Standards
How were the evaluation standards
employed in this example?

How was “engaging the stakeholders”
important in applying each of the
standards?
           The “Utility” Standard
Who are the intended   Why do they need the
users of the           information?
information?
 CDC Air Pollution     Examine progress.
  and Respiratory       Identify promising
  Health Branch          patterns.
                        Share successful practices
 State partners
                        Identify areas for
                         improvement.
                        Compare and contrast
                         program designs and
                         performance.
The Utility Standard in Action

Determined what information the CDC
needed to do their job and how they will
use the information.
Engaged states in discussions about how
to make the information useful for them.
Developed mock reports to facilitate
conversations with stakeholders about
how the data could be used.
    The “Feasibility” Standard
         How much money, time, and
         effort can we put into this?

 Remained conscious of how much time this
  data collection will require of the states.
 Identified areas where processes could be
  streamlined.
 Recognized that there will be less time for other
  programmatic activities.
The Feasibility Standard in Action

  Carefully selected “need-to-know”
  questions.
  Asked states:
    • What information do you already
      have?
    • How feasible is it for you to obtain?
    • How long does it take to locate
      information?
    • How long does it take to fill out the
      data collection instrument?
  Always kept the burden of data collection
The Feasibility Standard in Action

The content of the data collection
instrument was a careful balance between
feasibility and utility.

High priority data requiring higher burden
was balanced by eliminating lower priority
questions.
     The “Propriety” Standard
       What steps need to be taken for
        the evaluation to be ethical?

 Program monitoring systems can be used to
  generate performance measures.
 The data collected could have significant
  implications.
 Understand that the information collected can
  affect the stakeholders’ programs.
The Propriety Standard in Action


   Used a highly collaborative process.
   Fostered open, honest dialogue
   about expectations and concerns.
   Remained highly responsive to
   stakeholder concerns.
   Modified instrument to address
   stakeholder concerns.
     The “Accuracy” Standard
            What design will lead to
            accurate information?

 Be certain that the information collected is valid
  and reliable.
 If the information is inconsistent or irrelevant
  to the concept, reconsider whether to collect it.
 Balancing the Standards

When balancing feasibility vs. accuracy
  consider the effect on propriety.
 The Accuracy Standard in Action

Involved states in:
      Pilot tests and reviews.
      Discussions about “guessing”.
      Analyzing the quality of the
      information provided.
      Revising questions or collecting
      information that would help to improve
      accuracy.
      Decisions to discard some questions.
What We Have Reviewed Today
  The goal of evaluation and role of
  stakeholders.
  Who to engage as stakeholders.
  Why it is important to engage
  stakeholders.
  When and how to engage
  stakeholders.
  Stakeholder involvement across all
  steps of CDC framework.
  Stakeholder involvement in fulfilling
Thank You for Joining Us!


     Leslie Fierro: Let6@cdc.gov
  Carlyn Orians: Orians@Battelle.org

								
To top