Audience Worksheet - PowerPoint

							Communicating Evaluation Data
     Workshop Description
This session will provide an overview of
how evaluation questions and data
collection can be implemented using a
communications for sustainability
framework.

Participants will learn how to use marketing
techniques and strategies to promote
evaluation data effectively as well as to
discover how to use this data to allow for
the greatest impact with various target
audiences in their communities.
        Session Objectives

At the end of the session, participants will
be able to
    1. Identify organizations within their
       community for potential sustainability
       partnerships
    2. Understand how to work with their
       evaluator to collect and disseminate data
       based on communications needs
    3. Create meaningful communications using
       evaluation data that is developed and
       gathered for specific audiences
 Sustainability Communications
             Mindset
Look down the road not only for sustaining
SS/HS initiatives but also for creating an
environment that integrates the goals and
mission of the initiatives into the fabric of
your community
    1. Who needs to be at the table?
    2. What do we need to say to get them
       there?
    3. What data do we need to provide to
       make the information relevant to a
       particular audience?
 Who Needs To Be at the Table?

• Organizations with similar priorities
  as yours

  – Think outside of the typical audiences
     •   Current grant partners
     •   Other community agencies/organizations
     •   Civic organizations
     •   Others?
 Who Needs To Be at the Table?

• Things to think about
  – What are the top three priorities of our
    current partner agencies?
     • Where is there natural alignment without money
       as a motivator?
  – Within the community, what other
    organizations/agencies have
    priorities/vision/mission that align with one or
    more initiatives of SS/HS?
  – Follow the money
     • How does funding funnel into your community?
     • How are existing youth initiatives funded?
   What Do We Need To Say to
       Potential Partners?

• Messages must be relevant to your
  audience

• Messages must be in a format that is
  easy for your audience to read and
  understand

• Messages must be delivered in a
  timely way by a trusted source
    What Do We Need To Say to
        Potential Partners?
Questions that you can use to help decide how to
present your evaluation data to potential partners
   – What’s their definition of success?
   – What are their priorities?
   – What are they interested in?
   – What information do they trust?
   – What level of technical detail do they
     demand?
   – What data do you already have?
   – What data do you need to tell your story?
Primary and Secondary Audiences for
   Communicating Evaluation Data

• Primary Users are the individuals that
  you want to use your results; these
  people are your chief audience and
  often provide funding or other support

• Secondary Users are individuals who
  may be associated with your program
  or have an interest in what you are
  doing (e.g., city council, neighborhoods,
  and service recipients)
What Information Is Most Relevant to
    Your Audience’s Priorities?

• Description of       • Change in
  –   Programs           – Expenditures
  –   Cost               – Student referrals
  –   Students           – School climate
  –   School climate     – Academic
  –   Services             performance
                         – Consumer
                           satisfaction
 What Information Do They Trust?
• Caregiver-reported data

• Teacher-reported data

• Student-reported data

• Community member-reported data

• Administrator-reported data

• Agency staff-reported data

• Management information system data
        What Format Do We Use?

Once you have completed your evaluation and analysis,
you should have information that provides an accurate
picture of your program as well as information to make
decisions about future program implementation. So, how
do you share your findings? Results can be reported not
only in a text document, but also in

   –   Oral reports and presentations
   –   Videos
   –   Posters
   –   Press releases
   –   Newsletters
   –   Other forms of communication
    Exercise:

Understanding Your
    Audience’s
   Data Needs
                      Understanding Your Audience
TO USE: Decide if each audience would prefer quantitative or hard data (QT); qualitative, self-reported, or anecdotal
data (QL); or both (B). Then, for each audience, enter a QT, QL, or B in each of the boxes that represent the types of
data your target audiences will most want to see.

   Audience Segments                                                                    Types of Data
  Use space under each
  segment to specifically                            Descriptive                                                       Outcome
identify (e.g., does district   Students   Program       Service    Satis-   Climate   Student    Academic   Service    Costs     Satis-   School    Academic
    leadership include                                             faction             Behavior   Perform.    Array              faction   Climate   Indicators
 superintendent? School
   board? Principals?)

District Leadership


Law Enforcement


Mental Health Providers


Policymakers


Business Leaders


Faith-Based Leaders


Community Members


Other
       Inventory Your Data

• What type(s) of data do you have?

• Where did your data come from?

• What is the quality of your data?

• What message does the data convey?
   What Data Do You Need?

• What do you currently have? What are
  you currently gathering?
• Depending on the status of your program
  evaluation you can
  – Work with your evaluator to ensure that
    evaluation instruments that will provide you
    with relevant data for communications goals
    are developed
  – Use existing evaluation data in a way that is
    relevant to your audience
  – Some of both
 What Type(s) of Data Do You Have?

• Qualitative Data              • Quantitative Data
- Provides answers to open-     - Provides answers to close-
  ended questions                 ended questions
- Derives from notes from       - Uses numbers not words
  observations
                                - Is obtained through
- Uses words not numbers
                                  structured surveys and
- Is obtained through focus       interviews, existing records
  groups and unstructured
  interviews                    - Often includes a large
- Includes contextual detail      number of participants
  about feelings and            - Provides less contextual
  perceptions in an effort to     detail
  gain a holistic               - Is amenable to common
  understanding
                                  statistical procedures
- Includes relatively few
  participants
What Type(s) of Data Do You Have?
• Descriptive Data       • Outcome Data
  – Is cross sectional     – Is longitudinal
  – Describes              – Tracks people,
    characteristics of       organizations, or
    people,                  programs over time
    organizations, or        by using the same
    programs at a            measures
    single point in        – Assesses change in
    time                     people, programs,
                             or organizations
                           – Has an established
                             baseline
        Exercise:

Do You Have What You Need?


Data Inventory Worksheet
                    Data Inventory Worksheet

  TO USE: Check off the boxes that best represent the types of data you currently are collecting.
  Note where you might need to collect additional data, based on your findings from the Audience
  Data Preferences Worksheet. Discuss with your evaluator how you might fill in those gaps.


    Data Source                                            Type and Quality of Data

                      Qualitative Descriptive    Qualitative Outcome           Quantitative          Quantitative Outcome
                                                                               Descriptive

                      Good     Fair     Poor    Good    Fair    Poor    Good       Fair       Poor   Good    Fair    Poor

Agencies

Parents

Students

Teachers

Systems Generated

Community Members

Academic Indicators

External Comparison

Other_____________
           So Far We’ve -
• Identified potential partners/audiences

• Analyzed the type of data and
  information that is relevant to them

• Determined the data that you currently
  have and/or are collecting and identified
  where there are data needs

• So how do we take all of this information
  and put pen to paper?
Planning Your Project: Things To Decide
  Before Anyone Writes a Single Word

• Who is going to read this?

• What do they have in common?

• When they’re done reading, what do you
  want this audience to do or know?

• What materials do you have to work
  from? (See Data Inventory Worksheet)
Planning Your Project: Things To Decide
  Before Anyone Writes a Single Word

• What kind of research and/or interviews
  are necessary? (See Audience
  Worksheet)
• Will this piece be printed only, or will it
  be posted on the Web?
• Do you need to design the piece, or must
  you fit the text into one that’s already
  designed?
• What ―extras‖ do we need, such as
  sidebars, charts, or photos?
Planning Your Project: Things To Decide
  Before Anyone Writes a Single Word

• Internal factors

• Who will have input, and who has the final say?

• If a lot of people will have input, who will
  coordinate their comments and resolve any
  disputes?

• Does everyone agree on the number of drafts
  allowed and when the draft will become final,
  with no more changes?

• What resources are available to help you
  complete this project?
Planning Your Project: Things To Decide
  Before Anyone Writes a Single Word

• Internal factors

• Do you have someone who can put in
  the time necessary for organizing,
  researching, thinking, drafting, re-
  writing, and polishing over the life of the
  project?

• What is the budget for this project?

• How will your materials be distributed?
What Information Should Be Included
  When Sharing Evaluation Data?

1. State your program objectives

  a. This is important because,
     eventually, you will discuss how your
     evaluation data contributes to your
     stated objective.
What Information Should Be Included
  When Sharing Evaluation Data?

2. Describe progress toward achieving
   your objectives

  a. Describe your activities as they relate to
     your objectives
  b. Identify your beneficiaries—the number
     and characteristics of people you serve
  c. Describe your desired results—talk about
     the indicators of success that you have
     chosen. These are the factors that are most
     important to your program
What Information Should Be Included
  When Sharing Evaluation Data?

3. Describe your evaluation activities

  a. Identify the types of instruments you used
  b. Report on the information you collected
  c. Detail who administered your instruments,
     how they were administered (e.g., by mail,
     in person, or by telephone), and who, as
     well as how many people, completed them
  d. Note how you measured your desired result
  e. Discuss any problems encountered during
     the evaluation
What Information Should Be Included
  When Sharing Evaluation Data?

4. Describe the results of your evaluation

  This is the most important section!

  a. Report your findings—statistics and
     qualitative information
  b. Employ graphs, charts, and tables to help
     convey your message
  c. Share stories from people who are directly
     involved in your program and/or your
     evaluation
What Information Should Be Included
  When Sharing Evaluation Data?

5. Compare the results of your evaluation
   with your objectives

  a. Describe how your services addressed the
     existing need. You can relate the need to
     baseline data if you have it

  b. Refer to previous evaluation findings (such
     as those reported in earlier reports) to
     provide a more detailed picture of the
     program progress
What Information Should Be Included
  When Sharing Evaluation Data?

6. State ideas for partner support and ask
   for commitment

  This is your “ask”

  a. What do you want the audience to do?

  b. Show them a clear path to doing it

  c. Incorporate message development work to
     ensure success
    Pulling It All Together

      Harrisburg, Pennsylvania
        Student Survey Data


Goal: To create meaningful
communications using evaluation data
that is developed and gathered for
specific audiences
              Harrisburg, PA
           Student Survey Data
• Purpose
  – Evaluate prevention programs at the
    secondary level
     •   Life Skills Training
     •   TimeWise
     •   HealthWise
     •   School Transitional Environment Project (STEP)
     •   Teen Outreach Program
  – Establish a monitoring system for social,
    emotional, and behavioral outcomes
           Harrisburg, PA
        Student Survey Data
• Methods
  – Conducted annually (beginning in October
    2003)
  – All students in 6th–12th grade
  – Paper/pencil administration during a required
    class
  – 77 percent response rate
          Harrisburg, PA
       Student Survey Data
• Communications Goals
  – Give the data back to students before
    the 2004 survey administration
  – Integrate data into agency
    accountability systems
  – Drive program decisions during grant
    period
            Harrisburg, PA
         Student Survey Data
• Target audience: Project director
  – Communications goal
     • Drive program decisions during grant period
  – Messaging
     • Straightforward report of all data
     • Not bad vs. good, but pointing how to improve
       quality
  – Channel
     • Bound report
            Harrisburg, PA
         Student Survey Data
• Target audience: Students
  – Communications goal
     • Give data back to students before the 2004 survey
       to increase participation
  – Message
     • Show students that survey data is used and
       relevant
  – Channels
     • Student newsletter
     • Article from student perspective in school
       newspaper
            Harrisburg, PA
         Student Survey Data
• Target audience: Steering committee
  – Communications goal
     • Drive program decisions during grant period
  – Messaging
     • How do our students compare to national and
       State averages?
     • These students are all our responsibility, how can
       we collaborate to provide the best mix of programs
       given these data?
  – Channel
     • PowerPoint presentation
            Harrisburg, PA
         Student Survey Data
• Target audience: Superintendent
  – Communications goal
     • Integrate data into agency accountability systems
  – Messaging
     • The climate of schools is important; we care about
       the whole student (socially, emotionally,
       behaviorally, and academically)
  – Channel
     • Talking points in PowerPoint of survey highlights
       for Superintendent’s annual ―state of the union‖
         Other Audiences

• How would you use this same data for
  other audiences?
   – Businesses?
  – Community organizations?
  – Parents?
  – Others?


• What different information might you
  need for these audiences?
Questions?