RETURN ON INVESTMENT (ROI) – An Appreciation
Susan Ng MA IDT
1
Presentation Objectives
Identify a feasible ROI Process Use ROI to measure the effectiveness of
training
Explain the reasons for conducting ROI
Relate ROI to E-learning
Identify the reasons for measuring ROI
for e-learning
2
Global Training Trends
Importance : 5 – Critically important and 1 is Unimportant
Source: Performance Resources Organisation
Agreement with Trend: 5 – Very much progress and 1 – No progress
1. Training Costs are monitored more accurately to manage resources and demonstrate accountability 2. Measuring the return on investment in training is growing in use 3. Systematic evaluation processes measure the success of training Level of Agreement 3.92 4.02 4.57 Level of Importance 4.83 4.71 4.69
4. Needs assessment and analysis is receiving more emphasis
5. Training staff and line management are forming partnerships to achieve common goals 6. Training is linked to strategic direction of the organisation 7. The learning organisation concept is being adopted.
3.76
3.95 3.96 4.09
4.64
4.57 4.48 4.47
8. Training delivery is changing rapidly
9. Training is shifting to a performance improvement role 10. The technology of training is developing rapidly.
4.26
4.25 4.68
4.39
4.37 4.32 3
ROI
ROI measures the contribution of a
program/solution designed to improve or retain intellectual capital
ROI = Value of Benefits – Cost of Training Cost of Training
4
More popular Types of Evaluation
Balanced Scorecard (Drs Kaplan and
Norton),
Kirkpatrick Four-Level Framework
(Kirkpatrick, 1975),
Jack Phillips’s Five Levels of
Evaluation
5
Balanced Scorecard
Balanced Scorecard (BSC) is a management
framework used for implementing organisational strategy by linking the objectives, initiatives and measures of a business. other key performance indicators including customer perspectives, internal business processes and organisational development and innovation.
It integrates traditional financial measures with
6
Donald Kirkpatrick’s Model of Evaluation (1979)
Level 1 Level 2
Reaction Evaluation Learning Evaluation
Level 3 Level 4
Transfer of Learning Evaluation Results Evaluation
This model is also adopted by Singapore Skills Development Fund (SDF) for companies’ Training Effectiveness Reports to SDF for funding purposes.
7
Dr Jack Phillips’ Evaluation Levels
Level 1. Reaction & Planned Action 2. Learning
3. Application 4. Business Impact
Measurement Focus Measures participant satisfaction with the program and captures planned actions. Measures changes in knowledge, skills and attitudes.
Measures changes in on-the-job behavior. Measures changes in business impact variables – any measurable results?
5. Return on Investment
Compares program benefits to the costs
8
Who are Using the ROI Process?
Singapore Airlines Singapore Technology
Motorola
Compaq Companies who submit to SDF for
funding
9
What Companies Do With ROI Data?
Improve program/Process
Discontinue/Expand Programs
Approve projects (If Pilot) Like a front end analysis – to decide whether Develop Data Base of Program Results
worthwhile to proceed on the training project.
Inform/Educate Management Inform/Educate target Groups
Build Skills with Staff
10
ROI Process
Evaluation Instruments Evaluation Purposes Tabulating Program Costs
Collecting Post Program Data
Isolating the Effects of the Program
Converting Data to Monetary Value
Calculating the Return on Investment
Evaluating Timing
Evaluating Levels
Significant Influences Policy Statement Procedures & Guidelines Staff Skills Management Support Technical Support Organisational Culture
Identifying Intangible Benefits
11
Case Study
Interactive Selling Skills – 3 days
course, 48 participants
Retail Merchandise Company
You may want to pay attention – you
will be tested on your understanding! – Level 2 Evaluation
12
ROI Process
Surveys, Performance records
Evaluation Instruments
Evaluation Purposes
If management don’t believe in it – don’t do it
Tabulating Program Costs
*Performance Monitoring *Questionnaire *Follow-up Session
Increased Sales/Profits
Converting Data to Monetary Value
Calculating the Return on Investment
Collecting Post Program Data
Isolating the Effects of the Program
Control Groups
Evaluating Timing
Questionnaire -3 mths later
Evaluating Levels
Significant Influences Policy Statement Procedures & Guidelines Staff Skills Management Support Technical Support Organisational Culture
Identifying Intangible Benefits
13
Level 1 Data
Selected Data:
Success with Objectives Relevance of Material Usefulness of Program Exercises/Skill Practices Overall Instructor Skill 4.3 4.4 4.5 3.9 4.1
14
Level 2 Data
All Participants Demonstrated That
They Could Use The Skills Successfully
15
Level 3 Data
Selected Data:
Strongly Agree I utilise the Skills taught In the program 78%
Agree
Neither Agree/ Disagree 0%
Disagree Strongly Disagree 0% 0%
22%
With Each customer Frequency of Use of skills
52%
Every Several At Least At Least 3rd Times Once Daily Once customer each day Weekly
26% 18% 4% 0%
16
Level 4 Data
Post Training Data Training Groups $9,723 $9,978 $10,424 Control Groups $9,698 $9,720 $9,812
Weeks After Training 1 2 3
13 14
15 Average for Weeks
$13,690 $11,491
$11,044 $12,075
$11,572 $9,683
$10,092 $10,449
13, 14, 15
17
Annualised Program Benefits
46 participants were still in job after 3 months Ave Wkly Sales
Trained Groups $12,075
Untrained Groups
$10,449
Increase Profit Contribution 2%
$1,626 $32.50
$1,495
Total Weekly Improvement (x46)
Total Annual benefits (x48wks)
$71,760
18
Cost Summary
48 participants in 3 courses
Facilitation Fees: 3 courses @ $3750
Program Materials: 48 @ $35/ppt Meals/Refreshments: 3 days @$28/ppt Facilities: 9 days @ $120 Participants Salaries Plus Benefits’ (35%) Coordination/Evaluation Total Costs
$11,250
$1,680 $4,032 $1,080 $12,442 $2,500
$32,984
19
Level 5 Data
ROI =(Value of benefits –cost of training)/Cost of training
Benefit Cost Ratio71,760 = 2.2:1 32,984 ROI (%) = 38,776 32,984 = 118%
20
In view of the above example, what implementation problems/issues do you foresee? – Group Discussions
Evaluation Instruments Collecting Post Program Data Evaluation Purposes
Tabulating Program Costs
Converting Data to Monetary Value Calculating the Return on Investment
Isolating the Effects of the Program
Evaluating Timing Evaluating Levels
Significant Influences Policy Statement Procedures & Guidelines Staff Skills Management Support Technical Support Organisational Culture
Identifying Intangible Benefits
21
Some Key ROI Implementation Issues
Time spent on ROI
Cost of conducting the measurement
Complexity of variables in ROI Accuracy in measurements
Credibility Lack of Skills to measure
If staff does not see the need for ROI, it will
usually fail
Without support from management, ROI process
will usually fail.
22
Evaluation Targets
No need to evaluate all courses. Egs of courses to be evaluated:
Level
1.
2. 3.
% Courses
100%
70%
Participant satisfaction
Learning
On-the-job Applications Results Return on Investment
50% 10% 5%
23
4. 5.
Examples of Business Results
Coca Cola – 8 half-day on supervisory skills
Course – 1115% ROI, Benefit/Coast ratio 12:1 workshops – 1447% ROI, Benefit/Cost Ratio 15:1
Yellow Freight System – Performance Appraisal Litton Industries (Avionics) – Self Directed Work
Team course – Productivity increased 30%, Scrap rate reduction 50%, 700% ROI
Multi-Marques, Inc (Bakery) – 15 hr Supervisory
Skills Training – 215% ROI, Benefit/Cost Ratio 3.2:1
24
Now you know what ROI is, so why are companies measuring it?
Let’s discuss
25
ROI Measurement – WHY?
Training budgets are increasing ROI is the ultimate level of evaluation
Competitive pressures on costs and
productivity information
Top executives are requiring ROI
26
ROI Measurement – WHY?
Commitment of Training Expenditure
as a Percentage of Payroll:
US – 1% - 4% of Payroll
Europe – 2.5% to 3% of Payroll Asia – 4% to 8% of Payroll
27
ROI Measurement – WHY?
Management Wants to see results Budget Levels Off Minimum Training Level Budget Cuts (No Results) ..And the Cycle Continues
Renewed interest
Budget increase
Renewed Interest in Training
28
ROI & E-Learning
29
Is There Any Difference in Evaluating E-learning & Classroom Learning?
Please discuss
30
Difference in Evaluating E-learning & Classroom Learning
Findings:
by Jack Phillips,PhD, Patricia Phillips, Lizette Zuniga, PhD
Measuring ROI is driven by clients of e-learning
(those who are funding) & not the designers, developers & implementers.
Traditional classroom instruction yields more
favourable responses than e-learning solutions (Level 1) – Level 1 evaluation shows more satisfaction for
E-learning is as effective as face to face learning
face to face learning. But learning outcomes are not different (Level 2 evaluation)
31
Difference in Evaluating E-learning & Classroom training
Same evaluation strategies used in other
types of evaluations can be applied to elearning programs.
Building ROI evaluation into computerised
training process can save time & money.
32
Exercise
A Training Executive met up with an Instructional Designer
to discuss about the design of an e-learning programme for 5000 participants worldwide. This is the first time that the company would be using e-learning. The initial investment in the training is huge. The CEO is very excited about the programme and has high expectations of this training. The Training Executive suggested to forecast/measure the ROI.
suggestion by the Training Executive? Why?
As an instructional designer, would you consider the
As the Training Executive, what would you like to achieve
from measuring ROI?
33
Why Measure ROI for ELearning?
implementing e-learning
Cost of Technology - Initial cost of Newness of e-learning (is it effective
& efficient as F2F learning?)
E-learning is not a proven process in
many organisation – need to show value now than later
34
Why Measure ROI for ELearning
In many e-learning projects, the client wants
to know the projected payback from the project.
To venture into the expensive development
process without having some sense of the payback is undesired by many clients.
Consequently, there is tremendous pressure
to forecast ROI even if it is not very accurate.
35
Research on ROI & Technology-based Training – Hall (1997)
A major consulting firm
CBT course for 7000 consultants in 50 countries. Cost of CBT training program = $106/student. Previous F2F program cost is $760/student. Life of program was 5 years & savings = $4.5m
What does this means to U?
• When requesting monetary investment for e-learning, be prepared to evaluate effectiveness
• Conduct cost-benefits analysis to assess cost savings for organisation
36
Research Study on Perception of Mangers Towards CBT
Based on interviews with 300 managers in
UK
Managers discouraged staff from CBT
Managers recognised potential benefits of CBT
44% were unsure where CBT was less expensive than classroom training 66% felt CBT would isolate staff What does this means to U?
• Evaluate e-learning to get management support • Determine whether e-learning meet organisational efforts & the identified needs
37
Research Study on Investment in Learning Technologies
Based on Yr 2000 ASTD State if the
Industry Report (USA)
Projections for 2001 – significant majority of organisations expect to be using multimedia (91%), CD-ROM (87%), CBT (81%) and intranets (77%) What does this mean to U?
• ROI will provide data to help you decide to continue to stop existing training programmes
• ROI will help you to obtain support from the “bean counters”
38
E-Learning - Suggest to Measure:
Effectiveness of e-learning programs by
measuring enrollment, no-shows, and retention rates of participants.
The intangible data, such as reduced conflict,
over time
The initial comfort level (Level 1) of the
participant in an online learning program & implement sequential measurements of participant's comfort level.
The impact of your efforts in producing
effective e-learning programs.
39
return on investment
MediaPro, Inc.
Instructional Technology & Services (800)726-6951
CALCULATOR
Cost of Development Data Sheet
Developm ent cos ts include all expens es as s ociated with the analys is , des ign, and developm ent the of training cours e. Thes e cos t es tim ates m ay be drawn from your com pany's pas t experience, vendor propos als , or publis hed indus try s tudies .
GENERAL BACKGROUND What is the title of the training cours e? What is your current m ethod of training (i.e., ILT, CBT, WBT…)? Course Title Method #1
What is the new m ethod being cons idered (i.e., ILT, CBT, WBT…)? Method #2
Resource: Download ROI Calculator: www.mediapro.com
Current Method COST OF DEVELOPMENT PERSONNEL A1 What is the es tim atedinternal labor cos tfor developing the cours e? A2 What are the es tim atedcontractor cos ts for developing the cours e? Contractor #1: Describ e here Contractor #2: Describ e here Contractor #3: Describ e here PRODUCTION & MATERIALS A3 What are the production and m aterial expens es developing the for cours e? (i.e., equipm ent rental, m as ters , printing, etc.) Expens e #1: Expens e #2: Expens e #3: Expens e #4: Describ e here Describ e here Describ e here Describ e here
$ $ $ $ $ -
New Method Method #2
Method #1
$ $ $ $ $
-
$ $ $ $ $
-
$ $ $ $ $
-
MISCELLANEOUS A4 Include anyadditional expens esrelated to the training cours e. Mis c. #1: Mis c. #2: Mis c. #3: Mis c. #4: Describ e here Describ e here Describ e here Describ e here
$ $ $ $ $
-
$ $ $ $ $
-
REVISION A5 How much revision to the course content is expected per year? [Enter as a percentage of initial development cost: i.e .12 = 12%]
12.0%
12.0%
Once you have completed this page, proceed to "Delivery Costs"
40
return on investment
MediaPro, Inc.
Instructional Technology & Services (800)726-6951
CALCULATOR
Return On Investment Summary Report
This analysis compares annualized training costs of Method #1 and Method #2 for the production of the Course Title course.
Current Method COST OF DELIVERY
A6 A7 A8 A9 A11 A12 Total cost of development personnel: Total cost of production and materials: Total cost of miscellaneous expenses: Total cost of course revision: Total cost of development over life of course:
New Method Method #2
$ $ $ $ $ $ -
Method #1
$ $ $ $ $ $ -
Total Cost Per Year (Course Development) =
COURSE DELIVERY COSTS
B7 B8 B9 B10 B11 Total number Total number Total number Total number Total number of of of of of students per year: students that require travel per year: students trained over the life of the course: instructors per year: instructors required over the life of the course: 0 0 0 0 0 0 0 0 0 0
STUDENT COSTS C6 Average student cost per class (salary, benefits, & materials): C7 Average student travel cost per class (travel & per diem): C8 Average total student cost per class (all costs): C9 Total student cost per class (salary, benefits, & materials): C10 Total student travel cost per class (travel & per diem): C11 Total student cost per class (all costs): C12 Total Cost Per Year (Student) = INSTRUCTOR COSTS D6 Average instructor cost per class (salary, benefits, & materials): D7 Average instructor traveling costs per class (travel & per diem): D8 Total instructor cost per course: D9 Total Cost Per Year (Instructor) = EQUIPMENT COSTS E4 Total equipment purchase requirement: E5 Cost of equipment per year over the life of the course: E6 Cost of maintenance per year: E7 Total Cost Per Year (Equipment) = FACILITY COSTS F3 T1 T2 T3 T4 T5 T6 T7
$ $ $ $ $ $ $
-
$ $ $ $ $ $ $
-
$ $ $ $ $ $ $ $ $ $ $
-
$ $ $ $ $ $ $ $ $ $
-
Total Cost Per Year (Facilities) =
TOTAL COST OF TRAINING PER YEAR =
TOTAL COST OF TRAINING OVER LIFE OF THE COURSE = TOTAL SAVINGS PER YEAR = SAVINGS OVER LIFE OF COURSE = PERCENT SAVINGS OVER LIFE OF COURSE = DELIVERY METHOD OFFERING THE BEST VALUE = RETURN ON INVESTMENT (total savings / total cost) = MONTHS TO BREAK EVEN =
41