Basic Elements of a Final Evaluation Report (worksheet), 104 by uzr27298


									I N D E X

A                                             C

Action plans                                  California Endowment, 5
  using evaluation findings to develop,        Cao Yu, H., 5
  monitoring implementation of, 111
  stakeholder role in developing, 110
                                              Capacity builder role, 4
                                              CCL (Center for Creative Leadership)
                                                how change is measured by, 46e
  See also Communicating results                on retrospective pretest and posttest
Adefuin, J., 5                                     assessments, 44

Advice for Reporting Data (worksheet),          on 360-degree assessment retests, 45
     105e–107e                                Center for Problem-Oriented Policing, 42
Advocate role, 4                              Change

Archival documents/data, 93–96                  based on evaluation findings
Assessor evaluator role, 3                         determining needed, 107–108
                                                designing evaluation to observe, 38–39

B                                               evaluation as tool for enhancing, 5
Basic Elements of a Final Evaluation            using evaluation finding to facilitate,

     Report (worksheet), 104e–105e                 111–113
Behavioral observation                          leadership development impact of

  evaluation using, 75–79                          social/systems, 16fig, 19
  qualitative data example of, 78e              methods and issues of measuring,
  quantitative data example of, 79e                43–48
Bias                                          Change measurement
  definition of, 42e                             climate survey retests, 45–46
  response-shift, 43, 44                        culture survey retests as, 46–47
Boundary spanner role, 4                        degree-of-change ratings as, 44–45
Building Evaluation Capacity: 72 Activities     how CCL approaches, 46e
     for Teaching and Training (Preskill &      pre and post-initiative assessments
     Russ-Eft), 3                                  as, 44

      Change measurement (continued)             Content validity, 42e
        response-shift bias problem of, 43, 44   Control groups, 39
        retrospective pretest and posttest       Criterion validity, 42e
           assessments as, 44                    Culture survey retests, 46–47
        return on investment (ROI) as, 47–48
        shifting perspectives challenges         D
           individual, 43e                       Daily evaluation, 57–59
        360-degree assessment retests as, 45     Data
      Change survey, 71–74                         appropriate use of evaluation, 112e
      Change Survey (worksheet examples),          archival documents and, 93–96
           72e–74e                                 confidentiality of individual, 112
      Checklists                                   qualitative, 32, 78e, 106e–107e
        Communicating and Using Evaluation         quantitative, 32, 79e, 105e–106e
           Findings, 114e                          storage and treatment of, 96–97
        Designing and Conducting the Evalua-     Data triangulation, 323
           tion, 99e                             Data-collection methods
        Focusing the Evaluation, 35e               archival documents and data, 93–96
      Climate survey retests, 45–46                behavioral observation, 75–79e
      Communicating and Using Evaluation           change survey, 71–75
           Findings: Checklist, 114e               considering goal of evaluation for
      Communicating results                           selecting, 33
        Advice for Reporting Data worksheet,       considering purpose and use of
           105e–107e                                  evaluation for selecting, 32–33
        Basic Elements of a Final Evaluation       daily evaluation, 57–59
           Report worksheet, 104e–105e             end-of-initiative evaluation, 59–64
        checklist on, 114e                         expectations survey, 55–57e
        identifying specific courses of action,     focus group, 79–87
           107–109                                 group dialogue, 87–90
        mechanisms for, 102–103                    guideline for using multiple, 38
        what information and when to, 103          interview, 64–69
        who receives information on, 103           learning survey, 69–71
        See also Action plans                      matching evaluation questions to,
      Community                                       31–32
        data collection through statistics on,     practical and feasibility issues of, 33–34
           90–93                                   response rates to, 97e–98e
        determining leadership development         workplace or community statistics,
           expectations of, 16fig, 18                 90–93
      Concurrent evidence, 42e                     See also Samples/sampling
      Confidentiality issue, 112                  Day 1 Evaluation (worksheet), 58e–59e
      Construct validity, 42e                    Degree-of-change ratings, 44–45

130   Index
Designer evaluator role, 3–4                   use time-series designs for compari-
Designing and Conducting the Evaluation           son, 42
     Checklist, 99e                          Evaluation findings
Developing Evaluation Questions                Advice for Reporting Data,
     (worksheet), 24e–25e                         105e–107e
Dialogue data-collection, 87–90                appropriate use of, 112e
Documents                                      Basic Elements of a Final Evaluation
  data-collection using archival,                 Report, 104e–105e
     93–94, 96                                 checklist on communicating, 114e
  example of archival, 94e–95e                 used to create broader change and
                                                  learning, 111–113
E                                              as data-collection method selection
End-of-initiative evaluation, 59–64               factor, 33
End-of-Program (worksheet), 81e–84e            developing detailed action plan
Evaluation                                        following, 109–110
  advantages of using both external/           effectively communicating, 102–107e
     internal, 2–3                             four critical activities to complete, 101
  common reasons for leadership                identifying specific courses of action
     development, 1                               using, 107–109
  context for, 5                               monitoring action plan’s
  cyclical nature of the process, 4–5             implementation, 111
  daily, 57–59                                 planning how to use, 34
  end-of-initiative, 59–64                   Evaluation focus
  as enhancing change, 5                       checklist for, 35e
  See also Evaluation findings; Leadership      defining purpose to establish, 10–15
     development                               determining impact over time, 19
Evaluation design guidelines                   determining resources for, 15
  for assessing change at different            determining surface expectations,
     levels, 39                                   19–20, 21e–22e
  assessing multiple changes that can be       determining types and amount of
     observed, 38                                 impact, 15–19
  use control groups for comparison, 39        determining/prioritizing evaluation
  for designing evaluation plan, 48–55            questions for, 23–31
  examining impact from multiple               identify and engage stakeholders for,
     perspectives, 37–38                          8–10e
  to look at change over time, 38–39           importance of establishing, 7–8
  for measuring change, 43–48                Evaluation plan design
  use multiple-data-collection methods, 38     on data storage and treatment, 96–99e
  use pilot study, 40e                         on data-collection methods, 55–96
  for reliability and validity, 41e–42e        elements of, 49

                                                                                   Index   131
      Evaluation plan design (continued)               End-of-Program worksheet, 81e–84e
        example 1: ABC Program Longitudinal            Focus Group (Several Methods After
           Outcome Evaluation, 50t–51t                   the Program) worksheet, 85e–86e
        example 2: MNQ Program Pilot and             Focus. See Evaluation focus
           Evaluation, 52t–54t                       Focusing the Evaluation Checklist
        importance of creating a good, 48–49             (worksheet), 35e
      Evaluation questions
        creating focus through, 23                   G
        Developing Evaluation Questions              Gerger, S. R., 44
           (worksheet) for, 24e–25e                  Greenbaum, T. L., 80
        examples of matching development             Ground Rules for the Dialogue Method
           initiatives to, 26                            (worksheet), 88e
        Focus Group (Several Months After the        Group dialogue, 87–90
           Program), 85e–86e                         Group Dialogue Questions (worksheet
        Group Dialogue Questions (worksheet              example), 89e
           example), 89e                             Gulanick, N. A., 44
        guidelines for writing survey, 76e
        matching data-collection strategies to,      H
           31–34                                     The Handbook of Leadership Development
        prioritizing, 31                                 Evaluation (Hannum, Martineau, &
        target population for, 26–30e                    Reinelt), 5, 14, 20, 39, 47
      Evaluators                                     Hannum, K. M., 1, 3, 5, 14, 18, 20, 39, 47
        external, 2–3                                Howard, G. S., 44
        internal, 2–3
        multiple roles played by, 3–4                I
      EVALULead model, 20                            Inouye, T., 5
      Executive Sponsor Interview (worksheet), 68e   Internal consistency, 41e
      Executive summary, 104                         Internal evaluators, 2–3
      Expectations survey, 55–57e                    Interrater agreement, 41e
      External evaluators, 2–3                       Interviews
                                                       Executive Sponsor Interview
      F                                                   worksheet, 68e
      Fetterman, D. M., 15                             guidelines on data-collection using,
      Final Program Evaluation (worksheet),               64–65, 67, 69
           60e–63e                                     Participant Interview worksheet,
      Findings. See Evaluation findings                    66e–67e
      Focus Group (Several Methods After the
           Program) [worksheet], 85e–86e             J
      Focus groups                                   Joint Committee on Standards for
        data-collection using, 79–80, 86–87               Educational Evaluation, 37, 113

132   Index
K                                          McGuigan, W., 44
Katzev, A., 44                             McGuire, J. B., 87
Kraemer, H. C., 40                         Measuring change. See Change
Krueger, R. A., 80                             measurement
                                           Morgan, D. L., 80
Langone, C. A., 44                         N
Leadership development                     Nance, D. W., 44
  determining types and amount of          Network building, 18
     impact, 15–19
  expectations for, 16fig, 19–20, 21e–22e   O
  list of possible outcomes for, 10        Organizations
  See also Evaluation                        climate survey retests of, 45–46
Leadership development initiatives           culture survey retests of, 46–47
  addressing surface expectations in,        determining leadership development
     19–20                                      expectations of, 16fig, 19–20, 21e–22e
  evaluation findings used to broaden,        determining types and amount of
     108, 111–113                               impact on, 15–19
  exploring need for follow-up activity
     to, 109                               P
  matching evaluation questions to, 26     Palus, C. J., 87
  pre- and post-initiative assessments     Participant Interview (worksheet),
     of, 44                                      66e–67e
  retrospective pretest and posttest       Participants
     assessments of, 44                       confidentiality of individual, 112
Learning                                      interviewing, 66e–67e
  using evaluation finding to facilitate,      response rates of, 97e–98e
     111–113                                  samples of, 31–32
  surface expectations on, 19–20           Patton, M. Q., 14
Learning Opportunities                     Peterson, D. B., 45
     (worksheet), 11e                      Phillips, J. J., 90
Learning survey, 69–71                     Pilot Study Is a Good Evaluation Tool
Learning Survey (worksheet), 70e                 (worksheet), 40e
                                           Plan. See Evaluation plan design
M                                          Planner evaluator role, 3–4
Managers of and Those Working with the     Post-initiative assessment, 44
    Target Population (worksheet), 30e     Pratt, C., 44
Martineau, J. W., 1, 3, 5, 14, 18, 20,     Pre-initiative assessment, 44
    39, 47                                 Preprogram Expectations Survey Example
Maxwell, S. E., 44                               (worksheet), 56e–57e

                                                                               Index    133
      Preskill, H., 3, 5, 101                       deciding who receives evaluation
      Purpose                                          findings, 103
        evaluation focus by establishing, 10–12     determining leadership development
        Purpose Definition (worksheet),                 impact on, 15–19
           12e–14e                                  gathering and consulting on evaluation
                                                       findings, 109
      Q                                             identifying and engaging, 8–10e
      Qualitative data                            Surface Expectations (worksheet),
       advice for reporting, 106e–107e                 21e–22e
       behavioral observation, 78e                Surveys
       data-collection method using, 32             change, 71–75
      Quantitative data                             expectations, 55–57e
       advice for reporting, 105e–106e              guidelines for writing questions, 76e
       behavioral observation, 79e                  learning, 69–71
       data-collection method using, 32             workplace, 91e–92e
      Questions. See Evaluation questions
      R                                           Target population
      Ralph, K. M., 44                              importance of understanding your, 26
      Reflective practitioner, 4                     managers of and those working with,
      Reinelt, C., 1, 3, 5, 14, 18, 20, 39, 48         29, 30e
      Reliability and Validity (worksheet),         Target Population (worksheet) on,
           41e–42e                                     27e–28e
      Resource determination, 15                  Teams/groups, 16fig, 17
      Response rates, 97e–98e                     Thiemann, S., 40
      Response-shift bias, 43, 44                 360-degree assessment retests, 45
      Retrospective pretest assessment, 44        Time-series designs, 42
      Return on investment (ROI), 47–48           Torres, R. T., 5, 101
      Rohs, F. R., 44                             Trainer evaluator role, 4
      Russ-Eft, D., 3                             Translator role, 4

      S                                           V
      Samples/sampling, 31–32                     Validity, 41e–42e
        See also Data-collection methods
      Sawyer, D., 42                              W
      Social/systems change, 16fig, 19             Wandersman, A., 15
      Stakeholder Identification (worksheet),      Websites
           9e–10e                                  California Endowment, 5
      Stakeholders                                 Center for Problem-Oriented
        action plan development role of, 110          Policing, 42

134   Index
  on leadership development initia-          Executive Sponsor Interview, 68e
     tives, 20                               Final Program Evaluation, 60e–63e
  list of possible outcomes for leadership   Focus Group (Several Methods After
     development, 10                            the Program), 85e–86e
  time-series designs, 42                    Focusing the Evaluation Checklist, 35e
  W.K. Kellogg Foundation, 14                Ground Rules for the Dialogue
W.K. Kellogg Foundation, 14                     Method, 88e
Workplace statistics, 90–93                  Group Dialogue Questions
Workplace Statistics Survey (worksheet          (example), 89e
     example), 91e–92e                       Learning Opportunities, 11e
Worksheets                                   Learning Survey, 70e
  Advice for Reporting Data, 105e–107e       Managers of and Those Working with
  Basic Elements of a Final Evaluation          the Target Population, 30e
     Report, 104e–105e                       Participant Interview, 66e–67e
  Behavioral Observation: Qualitative        Pilot Study Is a Good Evaluation
     Data, 78e                                  Tool, 40e
  Behavioral Observation Rating Table:       Preprogram Expectations Survey
     Quantitative Data, 79e                     Example, 56e–57e
  Change Survey examples, 72e–74e            Purpose Definition, 12e–14e
  Communicating and Using Evaluation         Reliability and Validity, 41e–42e
     Findings: Checklist, 114e               Stakeholder Identification, 9e–10e
  Day 1 Evaluation, 58e–59e                  Surface Expectations, 21e–22e
  Developing Evaluation Questions,           Target Population, 27e–28e
     24e–25e                                 Workplace Statistics Survey (example),
  End-of-Program, 81e–84e                       91e–92e

                                                                              Index   135

To top