Framework for Evaluating Return on Investment in Management Development Programs
Presented by Kara Lynch Prepared for 2005 World Food and Agribusiness Symposium, IAMA June 25-26, 2006
Problem
Return on investment from training programs are unknown
Results of training programs are not evaluated
Objectives
Develop an evaluation process using ROI calculations to determine financial benefits Apply this process to a case example
Kirkpatrick’s Four Level Framework
Level 1: Reaction
Measures
participant’s satisfaction with the training program
“smile sheets” “happy sheets”
Completed
immediately following the program
Kirkpatrick’s Four Level Framework
Level 2: Learning
Knowledge, Tested
skills, and attitudes
with
Posttest Observation Self-reporting
Kirkpatrick’s Four Level Framework
Level 3: Behavior/Transfer of Knowledge
Measures
the transfer of skills from the training program to behavior on the job
Before and after comparisons Observations from supervisors and peers Statistical comparisons
Kirkpatrick’s Four Level Framework
Level 4: Results/ ROI
Measures
the results and impacts of the training program Others separate this into two categories
Results Return on investment
Return on Investment
Simple ratio that expresses the relationship of performance dollars to expended dollars
ROI = (Benefits – Costs) / Costs
Model
Phase I: Assessment Planning Define Program Objectives
State Purpose of Evaluation
Determine Type of Benefits
Determine Method of Data Collection
Establish Timing of Evaluation
Model
Phase II Data Collection Costs of Program
Collection of Benefits Data
Model
Phase III Data Analysis
Assess Data
Report Results
General Management Development for a Diverse Audience
AMP
Agribusiness Management Program Leadership and management development for a major agribusiness Six seminars over 21 months 30 participants
AMP
Program objectives from promotional literature Purpose of evaluation
ROI for company Using transfer of knowledge to assess ROI
Determine
Method of data collection
questionnaire
AMP
Questionnaire administered with Zoomerang online survey tool
months after final session 24 months after initial session
6
22 out of 30 participants responded
AMP
Overall Mean
6 5 4 3 2 1 0 Future Position Current Postion
Fi na M nc ar i a ke l tin O g H R Su M pp ly Ec on IT om Bu ic sin s Ag e Is ss su e Po s lic y
AMP
Policy Ag Issues Business Economics IT Supply OHRM Marketing Financial 0 1 2 Mean 3 4 5
Knowledge Importance
AMP
Value for self
more value than time invested 4.8% less value than time invested
90.5%
Value for company
90.5%
more value than cost to company
AMP
ROI
Adjusted
by
Confidence intervals Return on sales for the company
Increases
in revenues /decreases in costs
Range 0.6% to 324%
ROI
of 398%
Conclusions
Case Example
Shows
positive changes Creates value Produce results which affect revenues and costs
Limitations
Types of questions which can be asked
Confidentiality
Results are biased
only report on responses received No mechanism to measure benefits from nonrespondents
Can
Responses may be geared toward what the respondent feels is expected of them, not what actually occurred No mechanism in model to adjust for overlap of answers
Concluding Remarks
Provides feedback for participants, sponsors, and program administrators Determining how well the organization met its business goals is dependent upon perspective and interpretation of program goals
Concluding Remarks
This is intended to provide a template for ROI that can be modified to fit the needs of specific training and development programs.