"Basic Elements of a Final Evaluation Report (worksheet), 104"
I N D E X AL RI A C TE Action plans California Endowment, 5 using evaluation ﬁndings to develop, Cao Yu, H., 5 109–111 monitoring implementation of, 111 stakeholder role in developing, 110 MA Capacity builder role, 4 CCL (Center for Creative Leadership) how change is measured by, 46e See also Communicating results on retrospective pretest and posttest D Adefuin, J., 5 assessments, 44 TE Advice for Reporting Data (worksheet), on 360-degree assessment retests, 45 105e–107e Center for Problem-Oriented Policing, 42 Advocate role, 4 Change GH Archival documents/data, 93–96 based on evaluation ﬁndings Assessor evaluator role, 3 determining needed, 107–108 designing evaluation to observe, 38–39 RI B evaluation as tool for enhancing, 5 Basic Elements of a Final Evaluation using evaluation ﬁnding to facilitate, PY Report (worksheet), 104e–105e 111–113 Behavioral observation leadership development impact of CO evaluation using, 75–79 social/systems, 16ﬁg, 19 qualitative data example of, 78e methods and issues of measuring, quantitative data example of, 79e 43–48 Bias Change measurement deﬁnition of, 42e climate survey retests, 45–46 response-shift, 43, 44 culture survey retests as, 46–47 Boundary spanner role, 4 degree-of-change ratings as, 44–45 Building Evaluation Capacity: 72 Activities how CCL approaches, 46e for Teaching and Training (Preskill & pre and post-initiative assessments Russ-Eft), 3 as, 44 129 Change measurement (continued) Content validity, 42e response-shift bias problem of, 43, 44 Control groups, 39 retrospective pretest and posttest Criterion validity, 42e assessments as, 44 Culture survey retests, 46–47 return on investment (ROI) as, 47–48 shifting perspectives challenges D individual, 43e Daily evaluation, 57–59 360-degree assessment retests as, 45 Data Change survey, 71–74 appropriate use of evaluation, 112e Change Survey (worksheet examples), archival documents and, 93–96 72e–74e conﬁdentiality of individual, 112 Checklists qualitative, 32, 78e, 106e–107e Communicating and Using Evaluation quantitative, 32, 79e, 105e–106e Findings, 114e storage and treatment of, 96–97 Designing and Conducting the Evalua- Data triangulation, 323 tion, 99e Data-collection methods Focusing the Evaluation, 35e archival documents and data, 93–96 Climate survey retests, 45–46 behavioral observation, 75–79e Communicating and Using Evaluation change survey, 71–75 Findings: Checklist, 114e considering goal of evaluation for Communicating results selecting, 33 Advice for Reporting Data worksheet, considering purpose and use of 105e–107e evaluation for selecting, 32–33 Basic Elements of a Final Evaluation daily evaluation, 57–59 Report worksheet, 104e–105e end-of-initiative evaluation, 59–64 checklist on, 114e expectations survey, 55–57e identifying speciﬁc courses of action, focus group, 79–87 107–109 group dialogue, 87–90 mechanisms for, 102–103 guideline for using multiple, 38 what information and when to, 103 interview, 64–69 who receives information on, 103 learning survey, 69–71 See also Action plans matching evaluation questions to, Community 31–32 data collection through statistics on, practical and feasibility issues of, 33–34 90–93 response rates to, 97e–98e determining leadership development workplace or community statistics, expectations of, 16fig, 18 90–93 Concurrent evidence, 42e See also Samples/sampling Conﬁdentiality issue, 112 Day 1 Evaluation (worksheet), 58e–59e Construct validity, 42e Degree-of-change ratings, 44–45 130 Index Designer evaluator role, 3–4 use time-series designs for compari- Designing and Conducting the Evaluation son, 42 Checklist, 99e Evaluation ﬁndings Developing Evaluation Questions Advice for Reporting Data, (worksheet), 24e–25e 105e–107e Dialogue data-collection, 87–90 appropriate use of, 112e Documents Basic Elements of a Final Evaluation data-collection using archival, Report, 104e–105e 93–94, 96 checklist on communicating, 114e example of archival, 94e–95e used to create broader change and learning, 111–113 E as data-collection method selection End-of-initiative evaluation, 59–64 factor, 33 End-of-Program (worksheet), 81e–84e developing detailed action plan Evaluation following, 109–110 advantages of using both external/ effectively communicating, 102–107e internal, 2–3 four critical activities to complete, 101 common reasons for leadership identifying speciﬁc courses of action development, 1 using, 107–109 context for, 5 monitoring action plan’s cyclical nature of the process, 4–5 implementation, 111 daily, 57–59 planning how to use, 34 end-of-initiative, 59–64 Evaluation focus as enhancing change, 5 checklist for, 35e See also Evaluation ﬁndings; Leadership deﬁning purpose to establish, 10–15 development determining impact over time, 19 Evaluation design guidelines determining resources for, 15 for assessing change at different determining surface expectations, levels, 39 19–20, 21e–22e assessing multiple changes that can be determining types and amount of observed, 38 impact, 15–19 use control groups for comparison, 39 determining/prioritizing evaluation for designing evaluation plan, 48–55 questions for, 23–31 examining impact from multiple identify and engage stakeholders for, perspectives, 37–38 8–10e to look at change over time, 38–39 importance of establishing, 7–8 for measuring change, 43–48 Evaluation plan design use multiple-data-collection methods, 38 on data storage and treatment, 96–99e use pilot study, 40e on data-collection methods, 55–96 for reliability and validity, 41e–42e elements of, 49 Index 131 Evaluation plan design (continued) End-of-Program worksheet, 81e–84e example 1: ABC Program Longitudinal Focus Group (Several Methods After Outcome Evaluation, 50t–51t the Program) worksheet, 85e–86e example 2: MNQ Program Pilot and Focus. See Evaluation focus Evaluation, 52t–54t Focusing the Evaluation Checklist importance of creating a good, 48–49 (worksheet), 35e Evaluation questions creating focus through, 23 G Developing Evaluation Questions Gerger, S. R., 44 (worksheet) for, 24e–25e Greenbaum, T. L., 80 examples of matching development Ground Rules for the Dialogue Method initiatives to, 26 (worksheet), 88e Focus Group (Several Months After the Group dialogue, 87–90 Program), 85e–86e Group Dialogue Questions (worksheet Group Dialogue Questions (worksheet example), 89e example), 89e Gulanick, N. A., 44 guidelines for writing survey, 76e matching data-collection strategies to, H 31–34 The Handbook of Leadership Development prioritizing, 31 Evaluation (Hannum, Martineau, & target population for, 26–30e Reinelt), 5, 14, 20, 39, 47 Evaluators Hannum, K. M., 1, 3, 5, 14, 18, 20, 39, 47 external, 2–3 Howard, G. S., 44 internal, 2–3 multiple roles played by, 3–4 I EVALULead model, 20 Inouye, T., 5 Executive Sponsor Interview (worksheet), 68e Internal consistency, 41e Executive summary, 104 Internal evaluators, 2–3 Expectations survey, 55–57e Interrater agreement, 41e External evaluators, 2–3 Interviews Executive Sponsor Interview F worksheet, 68e Fetterman, D. M., 15 guidelines on data-collection using, Final Program Evaluation (worksheet), 64–65, 67, 69 60e–63e Participant Interview worksheet, Findings. See Evaluation ﬁndings 66e–67e Focus Group (Several Methods After the Program) [worksheet], 85e–86e J Focus groups Joint Committee on Standards for data-collection using, 79–80, 86–87 Educational Evaluation, 37, 113 132 Index K McGuigan, W., 44 Katzev, A., 44 McGuire, J. B., 87 Kraemer, H. C., 40 Measuring change. See Change Krueger, R. A., 80 measurement Morgan, D. L., 80 L Langone, C. A., 44 N Leadership development Nance, D. W., 44 determining types and amount of Network building, 18 impact, 15–19 expectations for, 16ﬁg, 19–20, 21e–22e O list of possible outcomes for, 10 Organizations See also Evaluation climate survey retests of, 45–46 Leadership development initiatives culture survey retests of, 46–47 addressing surface expectations in, determining leadership development 19–20 expectations of, 16ﬁg, 19–20, 21e–22e evaluation ﬁndings used to broaden, determining types and amount of 108, 111–113 impact on, 15–19 exploring need for follow-up activity to, 109 P matching evaluation questions to, 26 Palus, C. J., 87 pre- and post-initiative assessments Participant Interview (worksheet), of, 44 66e–67e retrospective pretest and posttest Participants assessments of, 44 conﬁdentiality of individual, 112 Learning interviewing, 66e–67e using evaluation ﬁnding to facilitate, response rates of, 97e–98e 111–113 samples of, 31–32 surface expectations on, 19–20 Patton, M. Q., 14 Learning Opportunities Peterson, D. B., 45 (worksheet), 11e Phillips, J. J., 90 Learning survey, 69–71 Pilot Study Is a Good Evaluation Tool Learning Survey (worksheet), 70e (worksheet), 40e Plan. See Evaluation plan design M Planner evaluator role, 3–4 Managers of and Those Working with the Post-initiative assessment, 44 Target Population (worksheet), 30e Pratt, C., 44 Martineau, J. W., 1, 3, 5, 14, 18, 20, Pre-initiative assessment, 44 39, 47 Preprogram Expectations Survey Example Maxwell, S. E., 44 (worksheet), 56e–57e Index 133 Preskill, H., 3, 5, 101 deciding who receives evaluation Purpose ﬁndings, 103 evaluation focus by establishing, 10–12 determining leadership development Purpose Deﬁnition (worksheet), impact on, 15–19 12e–14e gathering and consulting on evaluation ﬁndings, 109 Q identifying and engaging, 8–10e Qualitative data Surface Expectations (worksheet), advice for reporting, 106e–107e 21e–22e behavioral observation, 78e Surveys data-collection method using, 32 change, 71–75 Quantitative data expectations, 55–57e advice for reporting, 105e–106e guidelines for writing questions, 76e behavioral observation, 79e learning, 69–71 data-collection method using, 32 workplace, 91e–92e Questions. See Evaluation questions T R Target population Ralph, K. M., 44 importance of understanding your, 26 Reﬂective practitioner, 4 managers of and those working with, Reinelt, C., 1, 3, 5, 14, 18, 20, 39, 48 29, 30e Reliability and Validity (worksheet), Target Population (worksheet) on, 41e–42e 27e–28e Resource determination, 15 Teams/groups, 16ﬁg, 17 Response rates, 97e–98e Thiemann, S., 40 Response-shift bias, 43, 44 360-degree assessment retests, 45 Retrospective pretest assessment, 44 Time-series designs, 42 Return on investment (ROI), 47–48 Torres, R. T., 5, 101 Rohs, F. R., 44 Trainer evaluator role, 4 Russ-Eft, D., 3 Translator role, 4 S V Samples/sampling, 31–32 Validity, 41e–42e See also Data-collection methods Sawyer, D., 42 W Social/systems change, 16ﬁg, 19 Wandersman, A., 15 Stakeholder Identiﬁcation (worksheet), Websites 9e–10e California Endowment, 5 Stakeholders Center for Problem-Oriented action plan development role of, 110 Policing, 42 134 Index on leadership development initia- Executive Sponsor Interview, 68e tives, 20 Final Program Evaluation, 60e–63e list of possible outcomes for leadership Focus Group (Several Methods After development, 10 the Program), 85e–86e time-series designs, 42 Focusing the Evaluation Checklist, 35e W.K. Kellogg Foundation, 14 Ground Rules for the Dialogue W.K. Kellogg Foundation, 14 Method, 88e Workplace statistics, 90–93 Group Dialogue Questions Workplace Statistics Survey (worksheet (example), 89e example), 91e–92e Learning Opportunities, 11e Worksheets Learning Survey, 70e Advice for Reporting Data, 105e–107e Managers of and Those Working with Basic Elements of a Final Evaluation the Target Population, 30e Report, 104e–105e Participant Interview, 66e–67e Behavioral Observation: Qualitative Pilot Study Is a Good Evaluation Data, 78e Tool, 40e Behavioral Observation Rating Table: Preprogram Expectations Survey Quantitative Data, 79e Example, 56e–57e Change Survey examples, 72e–74e Purpose Deﬁnition, 12e–14e Communicating and Using Evaluation Reliability and Validity, 41e–42e Findings: Checklist, 114e Stakeholder Identiﬁcation, 9e–10e Day 1 Evaluation, 58e–59e Surface Expectations, 21e–22e Developing Evaluation Questions, Target Population, 27e–28e 24e–25e Workplace Statistics Survey (example), End-of-Program, 81e–84e 91e–92e Index 135