VIEWS: 2 PAGES: 41 POSTED ON: 9/9/2012
RETURN ON INVESTMENT (ROI) – An Appreciation Susan Ng MA IDT 1 Presentation Objectives Identify a feasible ROI Process Use ROI to measure the effectiveness of training Explain the reasons for conducting ROI Relate ROI to E-learning Identify the reasons for measuring ROI for e-learning 2 Global Training Trends Source: Performance Resources Organisation Importance : 5 – Critically important and 1 is Unimportant Level of Level of Agreement Importance Agreement with Trend: 5 – Very much progress and 1 – No progress 1. Training Costs are monitored more accurately to manage 3.92 4.83 resources and demonstrate accountability 2. Measuring the return on investment in training is growing in 4.02 4.71 use 3. Systematic evaluation processes measure the success of 4.57 4.69 training 4. Needs assessment and analysis is receiving more emphasis 3.76 4.64 5. Training staff and line management are forming partnerships 3.95 4.57 to achieve common goals 6. Training is linked to strategic direction of the organisation 3.96 4.48 7. The learning organisation concept is being adopted. 4.09 4.47 8. Training delivery is changing rapidly 4.26 4.39 9. Training is shifting to a performance improvement role 4.25 4.37 10. The technology of training is developing rapidly. 4.68 4.32 3 ROI ROI measures the contribution of a program/solution designed to improve or retain intellectual capital ROI = Value of Benefits – Cost of Training Cost of Training 4 More popular Types of Evaluation Balanced Scorecard (Drs Kaplan and Norton), Kirkpatrick Four-Level Framework (Kirkpatrick, 1975), Jack Phillips’s Five Levels of Evaluation 5 Balanced Scorecard Balanced Scorecard (BSC) is a management framework used for implementing organisational strategy by linking the objectives, initiatives and measures of a business. It integrates traditional financial measures with other key performance indicators including customer perspectives, internal business processes and organisational development and innovation. 6 Donald Kirkpatrick’s Model of Evaluation (1979) Level 1 Reaction Evaluation Level 2 Learning Evaluation Level 3 Transfer of Learning Evaluation Level 4 Results Evaluation This model is also adopted by Singapore Skills Development Fund (SDF) for companies’ Training Effectiveness Reports to SDF for funding purposes. 7 Dr Jack Phillips’ Evaluation Levels Level Measurement Focus 1. Reaction & Measures participant satisfaction with Planned Action the program and captures planned actions. 2. Learning Measures changes in knowledge, skills and attitudes. 3. Application Measures changes in on-the-job behavior. 4. Business Impact Measures changes in business impact variables – any measurable results? 5. Return on Compares program benefits to the costs Investment 8 Who are Using the ROI Process? Singapore Airlines Singapore Technology Motorola Compaq Companies who submit to SDF for funding 9 What Companies Do With ROI Data? Improve program/Process Discontinue/Expand Programs Approve projects (If Pilot) Like a front end analysis – to decide whether worthwhile to proceed on the training project. Develop Data Base of Program Results Inform/Educate Management Inform/Educate target Groups Build Skills with Staff 10 ROI Process Evaluation Purposes Tabulating Evaluation Program Instruments Costs Converting Collecting Isolating the Data to Calculating Post Program Effects of Monetary the Return on Data the Program Value Investment Significant Influences Evaluating Policy Statement Identifying Timing Procedures & Guidelines Intangible Staff Skills Benefits Management Support Evaluating Technical Support Levels Organisational Culture 11 Case Study Interactive Selling Skills – 3 days course, 48 participants Retail Merchandise Company You may want to pay attention – you will be tested on your understanding! – Level 2 Evaluation 12 ROI Process Surveys, Performance records If management don’t Evaluation believe in it – don’t do it Instruments Evaluation Purposes Tabulating Program Costs *Performance Monitoring *Questionnaire Increased Sales/Profits *Follow-up Session Converting Collecting Isolating the Data to Calculating Post Program Effects of Monetary the Return on Data the Program Value Investment Control Groups Significant Influences Evaluating Policy Statement Identifying Timing Procedures & Guidelines Intangible Staff Skills Benefits Questionnaire Management Support -3 mths later Evaluating Technical Support Levels Organisational Culture 13 Level 1 Data Selected Data: Success with Objectives 4.3 Relevance of Material 4.4 Usefulness of Program 4.5 Exercises/Skill Practices 3.9 Overall Instructor Skill 4.1 14 Level 2 Data All Participants Demonstrated That They Could Use The Skills Successfully 15 Level 3 Data Selected Data: Strongly Agree Neither Disagree Strongly Agree Agree/ Disagree Disagree I utilise the 78% 22% 0% 0% 0% Skills taught In the program With Each Every Several At Least At Least customer 3rd Times Once Daily Once customer each day Weekly Frequency of Use of skills 52% 26% 18% 4% 0% 16 Level 4 Data Post Training Data Weeks After Training Training Groups Control Groups 1 $9,723 $9,698 2 $9,978 $9,720 3 $10,424 $9,812 13 $13,690 $11,572 14 $11,491 $9,683 15 $11,044 $10,092 Average for Weeks $12,075 $10,449 13, 14, 15 17 Annualised Program Benefits 46 participants were still in job after 3 months Ave Wkly Sales Trained Groups $12,075 Untrained Groups $10,449 Increase $1,626 Profit Contribution 2% $32.50 Total Weekly Improvement (x46) $1,495 Total Annual benefits (x48wks) $71,760 18 Cost Summary 48 participants in 3 courses Facilitation Fees: 3 courses @ $3750 $11,250 Program Materials: 48 @ $35/ppt $1,680 Meals/Refreshments: 3 days @$28/ppt $4,032 Facilities: 9 days @ $120 $1,080 Participants Salaries Plus Benefits’ (35%) $12,442 Coordination/Evaluation $2,500 Total Costs $32,984 19 Level 5 Data ROI =(Value of benefits –cost of training)/Cost of training Benefit Cost Ratio 71,760 = 2.2:1 32,984 ROI (%) = 38,776 = 118% 32,984 20 In view of the above example, what implementation problems/issues do you foresee? – Group Discussions Evaluation Purposes Tabulating Evaluation Program Instruments Costs Converting Collecting Isolating the Data to Calculating Post Program Effects of Monetary the Return on Data the Program Value Investment Evaluating Significant Influences Identifying Timing Policy Statement Intangible Procedures & Guidelines Benefits Evaluating Staff Skills Levels Management Support Technical Support Organisational Culture 21 Some Key ROI Implementation Issues Time spent on ROI Cost of conducting the measurement Complexity of variables in ROI Accuracy in measurements Credibility Lack of Skills to measure If staff does not see the need for ROI, it will usually fail Without support from management, ROI process will usually fail. 22 Evaluation Targets No need to evaluate all courses. Egs of courses to be evaluated: Level % Courses 1. Participant satisfaction 100% 2. Learning 70% 3. On-the-job Applications 50% 4. Results 10% 5. Return on Investment 5% 23 Examples of Business Results Coca Cola – 8 half-day on supervisory skills workshops – 1447% ROI, Benefit/Cost Ratio 15:1 Yellow Freight System – Performance Appraisal Course – 1115% ROI, Benefit/Coast ratio 12:1 Litton Industries (Avionics) – Self Directed Work Team course – Productivity increased 30%, Scrap rate reduction 50%, 700% ROI Multi-Marques, Inc (Bakery) – 15 hr Supervisory Skills Training – 215% ROI, Benefit/Cost Ratio 3.2:1 24 Now you know what ROI is, so why are companies measuring it? Let’s discuss 25 ROI Measurement – WHY? Training budgets are increasing ROI is the ultimate level of evaluation Competitive pressures on costs and productivity Top executives are requiring ROI information 26 ROI Measurement – WHY? Commitment of Training Expenditure as a Percentage of Payroll: US – 1% - 4% of Payroll Europe – 2.5% to 3% of Payroll Asia – 4% to 8% of Payroll 27 ROI Measurement – WHY? Management Budget Cuts Wants to see (No Results) ..And the results Cycle Continues Budget Levels Off Minimum Budget Training Renewed increase Level interest Renewed Interest in Training 28 ROI & E-Learning 29 Is There Any Difference in Evaluating E-learning & Classroom Learning? Please discuss 30 Difference in Evaluating E-learning & Classroom Learning Findings: by Jack Phillips,PhD, Patricia Phillips, Lizette Zuniga, PhD Measuring ROI is driven by clients of e-learning (those who are funding) & not the designers, developers & implementers. Traditional classroom instruction yields more favourable responses than e-learning solutions (Level 1) E-learning is as effective as face to face learning – Level 1 evaluation shows more satisfaction for face to face learning. But learning outcomes are not different (Level 2 evaluation) 31 Difference in Evaluating E-learning & Classroom training Same evaluation strategies used in other types of evaluations can be applied to e- learning programs. Building ROI evaluation into computerised training process can save time & money. 32 Exercise A Training Executive met up with an Instructional Designer to discuss about the design of an e-learning programme for 5000 participants worldwide. This is the first time that the company would be using e-learning. The initial investment in the training is huge. The CEO is very excited about the programme and has high expectations of this training. The Training Executive suggested to forecast/measure the ROI. As an instructional designer, would you consider the suggestion by the Training Executive? Why? As the Training Executive, what would you like to achieve from measuring ROI? 33 Why Measure ROI for E- Learning? Cost of Technology - Initial cost of implementing e-learning Newness of e-learning (is it effective & efficient as F2F learning?) E-learning is not a proven process in many organisation – need to show value now than later 34 Why Measure ROI for E- Learning In many e-learning projects, the client wants to know the projected payback from the project. To venture into the expensive development process without having some sense of the payback is undesired by many clients. Consequently, there is tremendous pressure to forecast ROI even if it is not very accurate. 35 Research on ROI & Technology-based Training – Hall (1997) A major consulting firm CBT course for 7000 consultants in 50 countries. Cost of CBT training program = $106/student. Previous F2F program cost is $760/student. Life of program was 5 years & savings = $4.5m What does this means to U? • When requesting monetary investment for e-learning, be prepared to evaluate effectiveness • Conduct cost-benefits analysis to assess cost savings for organisation 36 Research Study on Perception of Managers Towards CBT Based on interviews with 300 managers in UK Managers discouraged staff from CBT Managers recognised potential benefits of CBT 44% were unsure where CBT was less expensive than classroom training 66% felt CBT would isolate staff What does this means to U? • Evaluate e-learning to get management support • Determine whether e-learning meet organisational efforts & the identified needs 37 Research Study on Investment in Learning Technologies Based on Yr 2000 ASTD State if the Industry Report (USA) Projections for 2001 – significant majority of organisations expect to be using multimedia (91%), CD-ROM (87%), CBT (81%) and intranets (77%) What does this mean to U? • ROI will provide data to help you decide to continue to stop existing training programmes • ROI will help you to obtain support from the “bean counters” 38 E-Learning - Suggest to Measure: Effectiveness of e-learning programs by measuring enrollment, no-shows, and retention rates of participants. The intangible data, such as reduced conflict, over time The initial comfort level (Level 1) of the participant in an online learning program & implement sequential measurements of participant's comfort level. The impact of your efforts in producing effective e-learning programs. 39 return on investment MediaPro, Inc. Instructional Technology & Services CALCULATOR (800)726-6951 Cost of Development Data Sheet Resource: Download Development costs include all expenses associated with the analysis, design, and development of the training course. These cost estimates may be drawn from your company's past experience, vendor proposals, or published industry studies. GENERAL BACKGROUND ROI Calculator: What is the title of the training course? Course Title What is your current method of training (i.e., ILT, CBT, WBT…)? Method #1 www.mediapro.com What is the new method being considered (i.e., ILT, CBT, WBT…)? Method #2 Current Method New Method Method #1 Method #2 COST OF DEVELOPMENT PERSONNEL A1 What is the estimated internal labor cost for developing the course? $ - $ - A2 What are the estimated contractor costs for developing the course? $ - $ - Contractor #1: Describ e here $ - $ - Contractor #2: Describ e here $ - $ - Contractor #3: Describ e here $ - $ - PRODUCTION & MATERIALS A3 What are the production and material expenses for developing the $ - $ - course? (i.e., equipment rental, masters, printing, etc.) Expense #1: Describ e here $ - $ - Expense #2: Describ e here $ - $ - Expense #3: Describ e here $ - $ - Expense #4: Describ e here $ - $ - MISCELLANEOUS A4 Include any additional expenses related to the training course. $ - $ - Misc. #1: Describ e here $ - $ - Misc. #2: Describ e here $ - $ - Misc. #3: Describ e here $ - $ - Misc. #4: Describ e here $ - $ - REVISION A5 How much revision to the course content is expected per year? 12.0% 12.0% [Enter as a percentage of initial development cost: i.e .12 = 12%] Once you have completed this page, proceed to "Delivery Costs" 40 return on investment MediaPro, Inc. Instructional Technology & Services CALCULATOR (800)726-6951 Return On Investment Summary Report This analysis compares annualized training costs of Method #1 and Method #2 for the production of the Course Title course. Current Method New Method Method #1 Method #2 COST OF DELIVERY A6 Total cost of development personnel: $ - $ - A7 Total cost of production and materials: $ - $ - A8 Total cost of miscellaneous expenses: $ - $ - A9 Total cost of course revision: $ - $ - A11 Total cost of development over life of course: $ - $ - A12 Total Cost Per Year (Course Development) = $ - $ - COURSE DELIVERY COSTS B7 Total number of students per year: 0 0 B8 Total number of students that require travel per year: 0 0 B9 Total number of students trained over the life of the course: 0 0 B10 Total number of instructors per year: 0 0 B11 Total number of instructors required over the life of the course: 0 0 STUDENT COSTS C6 Average student cost per class (salary, benefits, & materials): $ - $ - C7 Average student travel cost per class (travel & per diem): $ - $ - C8 Average total student cost per class (all costs): $ - $ - C9 Total student cost per class (salary, benefits, & materials): $ - $ - C10 Total student travel cost per class (travel & per diem): $ - $ - C11 Total student cost per class (all costs): $ - $ - C12 Total Cost Per Year (Student) = $ - $ - INSTRUCTOR COSTS D6 Average instructor cost per class (salary, benefits, & materials): $ - $ - D7 Average instructor traveling costs per class (travel & per diem): $ - $ - D8 Total instructor cost per course: $ - $ - D9 Total Cost Per Year (Instructor) = $ - $ - EQUIPMENT COSTS E4 Total equipment purchase requirement: $ - $ - E5 Cost of equipment per year over the life of the course: $ - E6 Cost of maintenance per year: $ - $ - E7 Total Cost Per Year (Equipment) = $ - $ - FACILITY COSTS F3 Total Cost Per Year (Facilities) = $ - $ - T1 TOTAL COST OF TRAINING PER YEAR = $ - $ - T2 TOTAL COST OF TRAINING OVER LIFE OF THE COURSE = $ - $ - T3 TOTAL SAVINGS PER YEAR = T4 SAVINGS OVER LIFE OF COURSE = T5 PERCENT SAVINGS OVER LIFE OF COURSE = DELIVERY METHOD OFFERING THE BEST VALUE = T6 T7 RETURN ON INVESTMENT (total savings / total cost) = MONTHS TO BREAK EVEN = 41
"RETURN ON INVESTMENT _ROI_ -"