Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

RETURN ON INVESTMENT _ROI_ -

VIEWS: 2 PAGES: 41

									RETURN ON
INVESTMENT (ROI) –
An Appreciation
Susan Ng
MA IDT


                     1
Presentation Objectives
 Identify a feasible ROI Process

 Use ROI to measure the effectiveness of
  training
 Explain the reasons for conducting ROI

 Relate ROI to E-learning

 Identify the reasons for measuring ROI
  for e-learning


                                            2
Global Training Trends
Source: Performance Resources Organisation
Importance : 5 – Critically important and 1 is Unimportant         Level of    Level of
                                                                   Agreement   Importance
Agreement with Trend: 5 – Very much progress and
                      1 – No progress
1. Training Costs are monitored more accurately to manage          3.92        4.83
   resources and demonstrate accountability
2. Measuring the return on investment in training is growing in    4.02        4.71
   use
3. Systematic evaluation processes measure the success of          4.57        4.69
   training
4. Needs assessment and analysis is receiving more emphasis        3.76        4.64
5. Training staff and line management are forming partnerships     3.95        4.57
   to achieve common goals
6. Training is linked to strategic direction of the organisation   3.96        4.48
7. The learning organisation concept is being adopted.             4.09        4.47
8. Training delivery is changing rapidly                           4.26        4.39
9. Training is shifting to a performance improvement role          4.25        4.37
10. The technology of training is developing rapidly.              4.68        4.32

                                                                                        3
ROI
 ROI measures the contribution of a
  program/solution designed to improve
  or retain intellectual capital
 ROI = Value of Benefits – Cost of Training
                 Cost of Training




                                               4
More popular Types of
Evaluation
 Balanced Scorecard (Drs Kaplan and
 Norton),
 Kirkpatrick Four-Level Framework
 (Kirkpatrick, 1975),
 Jack Phillips’s Five Levels of
 Evaluation


                                       5
Balanced Scorecard
 Balanced Scorecard (BSC) is a management
  framework used for implementing organisational
  strategy by linking the objectives, initiatives and
  measures of a business.

 It integrates traditional financial measures with
  other key performance indicators including
  customer perspectives, internal business
  processes and organisational development and
  innovation.




                                                        6
Donald Kirkpatrick’s Model
of Evaluation (1979)
 Level 1          Reaction Evaluation
 Level 2          Learning Evaluation
 Level 3          Transfer of Learning
                   Evaluation
 Level 4          Results Evaluation
 This model is also adopted by Singapore Skills
 Development Fund (SDF) for companies’ Training
 Effectiveness Reports to SDF for funding purposes.

                                                      7
Dr Jack Phillips’ Evaluation
Levels
Level                Measurement Focus
1. Reaction &        Measures participant satisfaction with
   Planned Action    the program and captures planned
                     actions.
2. Learning          Measures changes in knowledge, skills
                     and attitudes.
3. Application       Measures changes in on-the-job
                     behavior.
4. Business Impact   Measures changes in business impact
                     variables – any measurable results?
5. Return on         Compares program benefits to the costs
   Investment

                                                              8
Who are Using the ROI
Process?
 Singapore Airlines

 Singapore Technology

 Motorola

 Compaq

 Companies who submit to SDF for
 funding

                                    9
What Companies Do With
ROI Data?
 Improve program/Process

 Discontinue/Expand Programs
 Approve projects (If Pilot)
 Like a front end analysis – to decide whether
  worthwhile to proceed on the training project.
 Develop Data Base of Program Results
 Inform/Educate Management

 Inform/Educate target Groups
 Build Skills with Staff

                                                   10
ROI Process
                 Evaluation
                  Purposes                              Tabulating
 Evaluation
                                                         Program
Instruments
                                                           Costs


                                       Converting
 Collecting        Isolating the        Data to          Calculating
Post Program        Effects of         Monetary         the Return on
    Data            the Program          Value           Investment


                              Significant Influences
Evaluating                    Policy Statement           Identifying
  Timing                      Procedures & Guidelines    Intangible
                              Staff Skills                Benefits
                              Management Support
               Evaluating     Technical Support
                 Levels       Organisational Culture

                                                                        11
Case Study
 Interactive Selling Skills – 3 days
 course, 48 participants
 Retail Merchandise Company

 You may want to pay attention – you
 will be tested on your understanding!
 – Level 2 Evaluation



                                         12
  ROI Process
  Surveys,
  Performance records
                                   If management don’t
   Evaluation                      believe in it – don’t do it
  Instruments       Evaluation
                     Purposes                                    Tabulating
                                                                  Program
                                                                    Costs
*Performance Monitoring
*Questionnaire                          Increased Sales/Profits
*Follow-up Session
                                             Converting
    Collecting          Isolating the         Data to             Calculating
   Post Program          Effects of          Monetary            the Return on
       Data              the Program           Value              Investment
                    Control Groups
                                  Significant Influences
   Evaluating                     Policy Statement                Identifying
     Timing                       Procedures & Guidelines         Intangible
                                  Staff Skills                     Benefits
Questionnaire                     Management Support
-3 mths later     Evaluating      Technical Support
                    Levels        Organisational Culture

                                                                                 13
Level 1 Data
 Selected Data:
  Success with Objectives     4.3
  Relevance of Material       4.4
  Usefulness of Program       4.5
  Exercises/Skill Practices   3.9
  Overall Instructor Skill    4.1


                                     14
Level 2 Data
 All Participants Demonstrated That
 They Could Use The Skills
 Successfully




                                       15
Level 3 Data
 Selected Data:

                 Strongly    Agree   Neither    Disagree Strongly
                 Agree               Agree/              Disagree
                                     Disagree

I utilise the      78%        22%      0%          0%      0%
Skills taught
In the program


                 With Each    Every   Several At Least At Least
                 customer      3rd     Times    Once Daily Once
                              customer each day            Weekly
Frequency of
Use of skills      52%        26%       18%        4%        0%


                                                                    16
Level 4 Data
    Post Training Data


Weeks After Training        Training Groups   Control Groups
1                            $9,723                 $9,698
2                            $9,978                 $9,720
3                            $10,424                $9,812
13                           $13,690                $11,572
14                           $11,491                $9,683
15                           $11,044                $10,092


Average for Weeks         $12,075                   $10,449
13, 14, 15



                                                               17
Annualised Program Benefits
 46 participants were still in job after 3 months

 Ave Wkly Sales
    Trained Groups             $12,075
    Untrained Groups           $10,449

 Increase                      $1,626
 Profit Contribution 2%        $32.50
 Total Weekly Improvement (x46)      $1,495
 Total Annual benefits (x48wks)      $71,760


                                                     18
Cost Summary
 48 participants in 3 courses

 Facilitation Fees: 3 courses @ $3750         $11,250
 Program Materials: 48 @ $35/ppt              $1,680
 Meals/Refreshments: 3 days @$28/ppt          $4,032
 Facilities: 9 days @ $120                    $1,080
 Participants Salaries Plus Benefits’ (35%)   $12,442
 Coordination/Evaluation                      $2,500
 Total Costs                                  $32,984




                                                         19
 Level 5 Data
ROI =(Value of benefits –cost of training)/Cost of training



Benefit Cost Ratio 71,760 = 2.2:1
                   32,984


ROI (%) = 38,776              = 118%
         32,984

                                                         20
In view of the above example, what
implementation problems/issues do
you foresee? – Group Discussions
                  Evaluation
                   Purposes                              Tabulating
  Evaluation
                                                          Program
 Instruments
                                                            Costs

                                       Converting
  Collecting       Isolating the        Data to           Calculating
 Post Program       Effects of         Monetary          the Return on
     Data           the Program          Value            Investment


 Evaluating                    Significant Influences     Identifying
   Timing                      Policy Statement           Intangible
                               Procedures & Guidelines     Benefits
                Evaluating     Staff Skills
                  Levels       Management Support
                               Technical Support
                               Organisational Culture

                                                                         21
Some Key ROI Implementation
Issues
 Time spent on ROI

 Cost of conducting the measurement

 Complexity of variables in ROI
 Accuracy in measurements

 Credibility
 Lack of Skills to measure
 If staff does not see the need for ROI, it will
  usually fail
 Without support from management, ROI process
  will usually fail.
                                                    22
Evaluation Targets
No need to evaluate all courses.
Egs of courses to be evaluated:
Level                           % Courses
1.   Participant satisfaction   100%
2.   Learning                   70%
3.   On-the-job Applications    50%
4.   Results                    10%
5.   Return on Investment       5%

                                            23
Examples of Business Results
 Coca Cola – 8 half-day on supervisory skills
  workshops – 1447% ROI, Benefit/Cost Ratio 15:1
 Yellow Freight System – Performance Appraisal
  Course – 1115% ROI, Benefit/Coast ratio 12:1
 Litton Industries (Avionics) – Self Directed Work
  Team course – Productivity increased 30%, Scrap
  rate reduction 50%, 700% ROI
 Multi-Marques, Inc (Bakery) – 15 hr Supervisory
  Skills Training – 215% ROI, Benefit/Cost Ratio
  3.2:1


                                                      24
Now you know what ROI is, so
why are companies measuring it?
 Let’s discuss




                              25
ROI Measurement – WHY?
 Training budgets are increasing

 ROI is the ultimate level of evaluation

 Competitive pressures on costs and
 productivity
 Top executives are requiring ROI
 information


                                        26
ROI Measurement – WHY?
 Commitment of Training Expenditure
 as a Percentage of Payroll:
  US – 1% - 4% of Payroll
  Europe – 2.5% to 3% of Payroll
  Asia – 4% to 8% of Payroll




                                       27
ROI Measurement – WHY?

        Management         Budget Cuts
        Wants to see       (No Results)                 ..And the
        results                                         Cycle Continues


                  Budget
                  Levels
                  Off


                              Minimum
              Budget          Training
  Renewed     increase        Level
  interest                                Renewed
                                          Interest in
                                          Training




                                                                          28
ROI & E-Learning




                   29
Is There Any Difference in Evaluating
E-learning & Classroom Learning?
 Please discuss




                                        30
Difference in Evaluating E-learning &
Classroom Learning
 Findings:
 by Jack Phillips,PhD, Patricia Phillips, Lizette Zuniga, PhD

 Measuring ROI is driven by clients of e-learning
  (those who are funding) & not the designers,
  developers & implementers.
 Traditional classroom instruction yields more
  favourable responses than e-learning solutions
  (Level 1)
 E-learning is as effective as face to face learning
  – Level 1 evaluation shows more satisfaction for
  face to face learning. But learning outcomes are
  not different (Level 2 evaluation)
                                                                31
Difference in Evaluating E-learning &
Classroom training
 Same evaluation strategies used in other
  types of evaluations can be applied to e-
  learning programs.
 Building ROI evaluation into computerised
  training process can save time & money.




                                              32
Exercise
 A Training Executive met up with an Instructional Designer
  to discuss about the design of an e-learning programme for
  5000 participants worldwide. This is the first time that the
  company would be using e-learning. The initial investment in
  the training is huge. The CEO is very excited about the
  programme and has high expectations of this training. The
  Training Executive suggested to forecast/measure the ROI.
 As an instructional designer, would you consider the
  suggestion by the Training Executive? Why?
 As the Training Executive, what would you like to achieve
  from measuring ROI?




                                                                 33
Why Measure ROI for E-
Learning?
 Cost of Technology - Initial cost of
 implementing e-learning
 Newness of e-learning (is it effective
 & efficient as F2F learning?)
 E-learning is not a proven process in
 many organisation – need to show
 value now than later


                                           34
Why Measure ROI for E-
Learning
 In many e-learning projects, the client wants
  to know the projected payback from the
  project.
 To venture into the expensive development
  process without having some sense of the
  payback is undesired by many clients.
 Consequently, there is tremendous pressure
  to forecast ROI even if it is not very accurate.


                                                     35
Research on ROI & Technology-based
Training – Hall (1997)
 A major consulting firm
   CBT course for 7000 consultants in 50
    countries. Cost of CBT training program =
    $106/student. Previous F2F program cost is
    $760/student. Life of program was 5 years &
    savings = $4.5m
   What does this means to U?
     • When requesting monetary investment for e-learning,
       be prepared to evaluate effectiveness
     • Conduct cost-benefits analysis to assess cost savings
       for organisation

                                                               36
Research Study on Perception
of Managers Towards CBT
 Based on interviews with 300 managers in
 UK
   Managers discouraged staff from CBT
   Managers recognised potential benefits of CBT
   44% were unsure where CBT was less expensive
    than classroom training
   66% felt CBT would isolate staff
   What does this means to U?
      • Evaluate e-learning to get management support
      • Determine whether e-learning meet organisational
        efforts & the identified needs
                                                           37
Research Study on Investment
in Learning Technologies
 Based on Yr 2000 ASTD State if the
 Industry Report (USA)
   Projections for 2001 – significant majority of
    organisations expect to be using multimedia
    (91%), CD-ROM (87%), CBT (81%) and intranets
    (77%)
   What does this mean to U?
     • ROI will provide data to help you decide to continue
       to stop existing training programmes
     • ROI will help you to obtain support from the “bean
       counters”


                                                              38
E-Learning - Suggest to
Measure:
 Effectiveness of e-learning programs by
  measuring enrollment, no-shows, and
  retention rates of participants.
 The intangible data, such as reduced conflict,
  over time
 The initial comfort level (Level 1) of the
  participant in an online learning program &
  implement sequential measurements of
  participant's comfort level.
 The impact of your efforts in producing
  effective e-learning programs.
                                                   39
return on investment                                                            MediaPro, Inc.
                                                                                Instructional Technology & Services
CALCULATOR                                                                                 (800)726-6951




         Cost of Development Data Sheet                                                                               Resource:
                                                                                                                      Download
     Development costs include all expenses associated with the analysis, design, and development of the
training course. These cost estimates may be drawn from your company's past experience, vendor proposals,
                                        or published industry studies.



GENERAL BACKGROUND
                                                                                                                      ROI
                                                                                                                      Calculator:
What is the title of the training course?                            Course Title
What is your current method of training (i.e., ILT, CBT, WBT…)?      Method #1



                                                                                                                      www.mediapro.com
What is the new method being considered (i.e., ILT, CBT, WBT…)? Method #2


                                                                        Current Method           New Method
                                                                          Method #1               Method #2
COST OF DEVELOPMENT

PERSONNEL
A1 What is the estimated internal labor cost for developing the course?     $       -            $       -
A2 What are the estimated contractor costs for developing the course?       $       -            $       -
    Contractor #1:    Describ e here                                        $       -            $       -
    Contractor #2:    Describ e here                                        $       -            $       -
    Contractor #3:    Describ e here                                        $       -            $       -

PRODUCTION & MATERIALS
A3 What are the production and material expenses for developing the         $       -            $       -
   course? (i.e., equipment rental, masters, printing, etc.)
    Expense #1:       Describ e here                                        $       -            $       -
    Expense #2:       Describ e here                                        $       -            $       -
    Expense #3:       Describ e here                                        $       -            $       -
    Expense #4:       Describ e here                                        $       -            $       -

MISCELLANEOUS
A4 Include any additional expenses related to the training course.          $       -            $       -
    Misc. #1:         Describ e here                                        $       -            $       -
    Misc. #2:         Describ e here                                        $       -            $       -
    Misc. #3:         Describ e here                                        $       -            $       -
    Misc. #4:         Describ e here                                        $       -            $       -

REVISION
A5 How much revision to the course content is expected per year?                  12.0%                12.0%
   [Enter as a percentage of initial development cost: i.e .12 = 12%]

                     Once you have completed this page, proceed to "Delivery Costs"
                                                                                                                                    40
return on investment                                                                     MediaPro, Inc.
                                                                                         Instructional Technology & Services
CALCULATOR                                                                                          (800)726-6951


      Return On Investment Summary Report
      This analysis compares annualized training costs of Method #1 and Method #2 for the production of the
                                                Course Title course.

                                                                                Current Method           New Method
                                                                                  Method #1               Method #2
COST OF DELIVERY

 A6   Total cost of   development personnel:                                     $       -                $       -
 A7   Total cost of   production and materials:                                  $       -                $       -
 A8   Total cost of   miscellaneous expenses:                                    $       -                $       -
 A9   Total cost of   course revision:                                           $       -                $       -
A11   Total cost of   development over life of course:                           $       -                $       -
A12                          Total Cost Per Year (Course Development) =          $       -                $       -

COURSE DELIVERY COSTS
 B7   Total number    of   students per year:                                                0                        0
 B8   Total number    of   students that require travel per year:                            0                        0
 B9   Total number    of   students trained over the life of the course:                     0                        0
B10   Total number    of   instructors per year:                                             0                        0
B11   Total number    of   instructors required over the life of the course:                 0                        0

STUDENT COSTS
 C6 Average student cost per class (salary, benefits, & materials):              $       -                $       -
 C7 Average student travel cost per class (travel & per diem):                   $       -                $       -
 C8 Average total student cost per class (all costs):                            $       -                $       -
 C9 Total student cost per class (salary, benefits, & materials):                $       -                $       -
C10 Total student travel cost per class (travel & per diem):                     $       -                $       -
C11 Total student cost per class (all costs):                                    $       -                $       -
C12                                    Total Cost Per Year (Student) =           $       -                $       -

INSTRUCTOR COSTS
  D6 Average instructor cost per class (salary, benefits, & materials):          $       -                $       -
  D7 Average instructor traveling costs per class (travel & per diem):           $       -                $       -
  D8 Total instructor cost per course:                                           $       -                $       -
  D9                                 Total Cost Per Year (Instructor) =          $       -                $       -
EQUIPMENT COSTS
 E4 Total equipment purchase requirement:                                        $       -                $       -
 E5 Cost of equipment per year over the life of the course:                      $       -
 E6 Cost of maintenance per year:                                                $       -                $       -
 E7                               Total Cost Per Year (Equipment) =              $       -                $       -
FACILITY COSTS
 F3                                        Total Cost Per Year (Facilities) =    $       -                $       -


 T1                                 TOTAL COST OF TRAINING PER YEAR =            $       -                $       -
 T2          TOTAL COST OF TRAINING OVER LIFE OF THE COURSE =                    $       -                $       -
 T3                                            TOTAL SAVINGS PER YEAR =
 T4                                       SAVINGS OVER LIFE OF COURSE =
 T5                           PERCENT SAVINGS OVER LIFE OF COURSE =
                       DELIVERY METHOD OFFERING THE BEST VALUE =
 T6
 T7
               RETURN ON INVESTMENT (total savings / total cost) =
                                                 MONTHS TO BREAK EVEN =
                                                                                                                               41

								
To top