Telling Training's Story

Telling Training’s Story This page intentionally left blank Telling Training’s Story Evaluation Made Simple, Credible, and Effective Robert O. Brinkerhoff Telling Training’s Story Copyright © 2006 by Robert O. Brinkerhoff All rights reserved. No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, including photocopying, recording, or other electronic or mechanical methods, without the prior written permission of the publisher, except in the case of brief quotations embodied in critical reviews and certain other noncommercial uses permitted by copyright law. For permission requests, write to the publisher, addressed “Attention: Permissions Coordinator,” at the address below. Berrett-Koehler Publishers, Inc. 235 Montgomery Street, Suite 650 San Francisco, California 94104-2916 Tel: (415) 288-0260, Fax: (415) 362-2512 www.bkconnection.com Ordering information for print editions Quantity sales. Special discounts are available on quantity purchases by corporations, associations, and others. For details, contact the “Special Sales Department” at the Berrett-Koehler address above. Individual sales. Berrett-Koehler publications are available through most bookstores. They can also be ordered directly from Berrett-Koehler: Tel: (800) 929-2929; Fax: (802) 864-7626; www.bkconnection.com Orders for college textbook/course adoption use. Please contact Berrett-Koehler: Tel: (800) 929-2929; Fax: (802) 864-7626. Orders by U.S. trade bookstores and wholesalers. Please contact Ingram Publisher Services, Tel: (800) 509-4887; Fax: (800) 838-1149; E-mail: customer .service@ingrampublisherservices.com; or visit www.ingrampublisherservices.com/ Ordering for details about electronic ordering. Berrett-Koehler and the BK logo are registered trademarks of Berrett-Koehler Publishers, Inc. First Edition Paperback print edition ISBN 978-1-57675-186-2 PDF e-book ISBN 978-1-60509-387-1 2009-1 Book Design & Production: Jimmie Young. Copy Editing: Cecile Kaufman. Cover Design: Ark Stein, The Visual Group. To my wife, Stevie, my children, Jory, Darcy, Susanna, and Allie, my love for whom is my beginning and my end. I want to make a special note also to the memory of Malcolm Provus, my dear friend and doctoral advisor, who started me on my convoluted career path. His untimely death at the age of 46 robbed too many of us of his inspiration and caring energy. This page intentionally left blank Contents Preface Acknowledgements xi xvii Part 1 Chapter One 1 3 6 Getting to the Heart of Training Impact Can We “Prove” Training Impact? Chapter Two How the Success Case Method Works: Two Basic Steps Training Evaluation Realities The Success Case Method: Step by Step Chapter Three 17 18 30 Success Case Method Strategy—Building Organizational Learning Capacity The Determinants of Training Impact Learning Alone Is Insufficient Risks of the Common Training Evaluation Strategy An Evaluation Strategy for Building Organizational Learning Competence A Fundamental Shift in Focus 35 36 37 40 43 44 vii viii Telling Training’s Story Redefining the Evaluation Process Implementing the Strategy to Drive Change Chapter Four 47 50 Focusing and Planning a Success Case Method Study 1. Clarify ing the Purpose of the SCM Study 2. Meet with and Discuss the Study with Key Stakeholders 3. Define the Program That Is to Be Studied 4. Define the Population of Participants to Be Studied, and Identify Any Needed Sampling Parameters 5. How Soon After the Training to Conduct the Survey 6. Establishing a Schedule for the Study 7. Specify and Confirm the Resources Available for the Study 8. Finalize the Success Case Strategy 57 58 60 61 63 65 66 67 68 Chapter Five Constructing a Simple Training Impact Model Elements of the Impact Model Constructing an Impact Model Chapter Six 71 73 77 Fishing for Success Conducting the SCM Survey Defining the Survey Step Conducting the SCM Survey Chapter Seven 83 84 85 Sorting the “Catch”—Analyzing Survey Results Scoring and Sorting the Survey Response Choosing Interviewee Candidates Analyzing Data to Estimate Nature and Scope of Impact Chapter Eight 101 102 104 107 Digging Out and Telling the Stories— the SCM Interviews Resolving Causal Questions The SCM Interview Structure Preparing for the Interviews 111 112 118 121 Contents ix Chapter Nine Drawing Compelling Conclusions The Eight Major SCM Conclusions Conclusion Type One: What, If Any, Impact Was Achieved? Conclusion Type Two: How Widespread Is Success? Summary of Final Conclusions Conclusion Type Three: Did the Training Work Better in Some Parts of the Organization, or with Some Types of Participants, Better than Others? Conclusion Type Four: Were Some Parts of the Training More Successfully Applied than Others? Conclusion Type 5: What Systemic Factors Were Associated with Success and a Lack of Success? Conclusion Type Six: What Is the Value of Outcomes Achieved? Conclusion Type Seven: What Is the Unrealized Value of the Training? Conclusion Type Eight: How Do the Benefits of the Training Compare with the Costs? 133 133 136 137 140 141 143 145 148 149 154 Part II Chapter Ten Sales Training at Grundfos The Setting Purposes of the SCM Study Organizing the Study Challenges and Constraints The SCM Chronology The Impact Model Survey Actions After the SCM Study Chapter Eleven 159 159 160 161 162 163 163 166 176 Service Technician Training at the Compaq Computer Corporation Other Important Factors Preparing for the Evaluation Evaluation Results The Rest of the Story 181 182 187 192 194 x Telling Training’s Story Chapter Twelve Coaching and Training at Coffee Bean and Tea Leaf ® 199 200 201 202 203 208 212 215 216 The Business Scenario The Training Intervention The Evaluation Purpose SCM Study Procedures Survey Results Interview Results Recommendations Challenges and Lessons Learned Chapter Thirteen Executive Development at Allstate Insurance Evaluation Purposes The Evaluation Process Challenges and Constraints Results Conclusions Recommendations Lessons Learned 219 220 221 227 229 233 235 236 References Success Case Me