1􀀛 Evaluating Training ROI with Learning Intelligence by Mark Place with Kurt Crisman, John Zonneveld Training is a critical component in any organization’s strategy for innovation and/or continuous improvement. For example, corporate health and safety programs train personnel in the hope that they will reduce workplace accideents whether mandated by OSHA regulations or by the company. Yet, training is an area where the actual return-on-investment (ROI) is uncertaain Is the training program effective? How can the program be improoved Did the program achieve the desiire results at the lowest possible cost? Given the large U.S. expenditure in training, it is important to develop tools that will help companies answer these questions and improve the measureemen of training effectiveness. 􀂃􀂃􀂃 These tools need to provide both a methodology to measure, evaluate, and continuously improve training, as well as the organizational and technicca infrastructure (systems) to implemeen the methodology. Measuring Organizational Results Training ROI calculation requires the measurement of organizational resullts Those results may be directly related to training operations, such as course enrollments and completions, assessment scores, and the results of feedback forms and surveys. Other results may be tied to performance data on an individual, department, or business unit level. Unless training program exists simply for the sake of training, measurements should incllud non-training performance data. Selected metrics, such as sales, cus-© Latitude Consulting Group 􀀛 888-577-2797 􀀛 www.latitudecg.com EXECUTIVE SUMMARY Training ROI calculation requires the measurement of organizational results. Unless training progrram exist simply for the sake of training, measurements should include non-training performaanc data. When evaluating training impact on individual behavior and business results, data collection requirements extend beyond course delivery. Individual performance data exists in performmanc management systems, and organizational data exists in marketing, sales, and fi nancial systems. A learning intelligence system can act as a broker between learning and business systems and make training and performance data more manageable for ROI analyses.White Paper 􀀚 􀀵􀁉􀁆􀀁􀀦􀁙􀁕􀁆􀁏􀁅􀁆􀁅􀀁􀀦􀁏􀁕􀁆􀁓􀁑􀁓􀁊􀁔􀁆􀀁􀀥􀁆􀁍􀁊􀁗􀁆􀁓􀁆􀁅􀀚 2 © Latitude Consulting Group 􀀛 888-577-2797 􀀛 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence tomer satisfaction, workplace safety, productivity, and others, should help demonstrate where training has eithhe increased revenue or decreased costs. The stated goal of a workplace safety training program, for example, is to reduce workplace accidents. Implicit in that goal is the reduction of costs, both tangible and intangible. While tangible costs of workplace accidents are fairly straightforward, the compaan also incurs other less tangible costs in lost productivity that are even more diffi cult to measure and account for. An accident in the workplace can adversely affect the behavior of uninjuure workers in other ways. They may approach a piece of machinery more tentatively or otherwise work with extreme care to avoid an accident themselves. Rough ROI measurements that consiide performance improvements can provide a benchmark for training effectiveness. After implementing a training initiative or changing an existing program, an organization can observe and record a change in performance. Reduced incidents of workplace accidents, for example. The value of the performance change (X dollars saved per reduced incident) can then be compared to the program cost to arrive at an ROI measurement. While this approach appears to answwe the question of training program effectiveness, it provides no insight into improving training from a results or cost perspective. It also does not provide a means to calculate ROI that may occur as a result of routine training programs. (How often should workers take the workplace safety program to ensure that accidents remain at the lower level of incidence?) Evaluating Training and Performance Many learning operations evaluate success based on the ubiquitous Kirkpattric model or some variation of it. Most organizations that follow this model are unable to evaluate their programs beyond the fi rst two levels. In part, the systems used make lower level evaluations easy and don’t proviid any mechanism for higher level evaluation. Most learning management systems will automatically track and include templates to report the information required to do level one and level two analyses. They include assessment tools that can capture each learner’s reaction to the course. For online and blended learning, the assessment can be completely integrated into the course. Likewise, training programs can inexpensively and easily administte pre-and post-tests that evaluate learning results (level two). Evaluating levels three and four becoome more diffi cult and costly to impleemen and administer. What makes these levels so diffi cult to evaluate? In the case of the fi rst two levels, data collection occurred during course delivvery When evaluating changes in student behavior and training impact on business results, data collection requirements extend beyond course delivery. Different evaluation methods can help answer whether student’s behaviio actually changed after completing a course or attaining a certifi cation. Data can come from many different 􀀚 3 © Latitude Consulting Group 􀀛 888-577-2797 􀀛 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence sources, including performance evaluatiion by the student’s supervisor or customer satisfaction survey data. Training or other designated staff can also observe students as they perform their jobs and record behavior. To evaluate retention rates, there should be a lag between the training and these behavior measurements. Rarely do organizations evaluating the business impact of a program, which is much more diffi cult and costlly One might think, “What is so dif-fi cult about evaluating the impact of learning and knowledge transfer on business performance? If organizatioona results improved, you will know that the training was effective.” One problem with this assertion is that you don’t really know if those results are due to training or some intervening factor. Large organizations can use a control group to isolate training from other intervening variables. One group does not get training, while another group does. The business results from each group can then be compared to see if the trained group performed better. Clearly, control groups present a few challenges. Performance must be measured over the same time periods. Both groups must be nearly identical in make-up, performing similar job functions. Sales groups in operating in different sales regions or manufacturiin groups working in different shifts will confound the results. With these challenges, structuring a valid control group requires great commitment on the part of the organization. But to perform a good ROI analysiis an organization must really be evaluating its training at levels three and four. Without these levels ROI analysis becomes more an exercise of cost justifi cation for expenditures. Implementing blended learning, for example, may appear to have a high ROI. After all, it’s less expensive than instructor-led, and level two assessmeen scores have improved. However, without an understanding of the impaac on individual and organizational performance, what value is the trainiin really delivering to the organizatiion Actually, much of the data exists at many organizations to bridge the gap between training and performance. Individual performance data exists in performance management systems. Organizational data exists in marketting sales, and fi nancial systems. Bridging this gap requires a technical infrastructure that minimizes the administtrativ effort needed to collect and analyze the training and performaanc data together. However, learniin management systems, the most common repository for training data and mechanism to deliver training, cannot bridge the gap easily. From a functional standpoint, each new LMS release adds more robust reporting and data analysis capabilitty as well as human resource system integration. In addition, many LMS vendors have added talent and performmanc and management to their human capital suites of solution. To some degree, these evolutionary changes address the gap between training its impact on individual performmance They don’t even begin to address the gap with organizational performance. Why are organizations still unlikely to evaluate training at Kirkpatrick’s level 􀀚 4 © Latitude Consulting Group 􀀛 888-577-2797 􀀛 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence three? What path can lead to level four evaluations? System integration, one common point of failure, is criticaal Many LMS vendors with a history as product companies have limited expertise in system integration that extends beyond learning systems and databases. Successfully managing performance-based training evaluatiion however, requires expertise in data management and warehousing, a variety of corporate systems and databases, analytics, and web-based application development. Evaluating training ROI most effectivvel requires the right technical infrastrructur and a model of learning improvement. Learning Intelligence System A reporting and data management strategy that focuses on the LMS as the foundation only compounds the system integration challenges that make performance-based training evaluation unmanageable. Instead, the organization should adopt a cross-functional corporate reporting and data management strategy. The technical foundation for this strategy is not the LMS, but a learning intelliggenc system that acts as a broker between an LMS and other corporate systems. The characteristics of a learning intelliggenc system include: Independent of LMS Cross-functional system integration Alignment to individual and organizattiona performance Reporting and analytical tools 􀂃􀂃􀂃􀂃 LMS Independence A learning intelligence system should not be locked into a single LMS platfoorm By utilizing a generic framewoork common LMS data should map to variables in the learning intelliggenc system. LMS independence ensures more stability over time. It minimizes the extent of system integrratio that is required if the LMS should be upgraded or replaced by another system. Typically, an organization will feed performance, job code, certifi cation, and other corporate data into the LMS’s reporting system. By adding a learning intelligence system between the LMS and other corporate systems, only one data connection must be updaate if the LMS changes. Cross-Functional System Integration As a broker for business intelligence throughout the organization, a learning intelligence system needs to aggregate the data from multiple corporate systems. If assembling informattio is too cumbersome and time consuming and the data is outdated or not even correct, the system cannot enhance ROI evaluations by combiniin training with other business data. Cross-functional system integration allows the organization to leverage training and business data together in a context-sensitive manner. Technical or political requirements may dictate that decision-makers in different corporrat domains access data through different systems. Cross-functional system integration allows the learning intelligence system to push data to the portal or reporting system used by a particular decision-maker.􀀚 5 © Latitude Consulting Group 􀀛 888-577-2797 􀀛 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence One of the primary challenges when implementing cross-functional system integration is the migration of a diveers range of existing data sources. Data feeds, often in a fl at fi le format, may be processed on different systeems including those of third-party vendors. Creating reports and correcctin mistakes can be diffi cult, especiiall when it involves many people exchanging fl at fi les. If the organization has a corporaat data warehouse, the learning management system can push the learning management data into this consolidated data source. Any corporaat reporting system can then access this learning data, combine it with other business data, and make more advanced ROI calculations. Different data owners maintain data integrity in the consolidated data source, which provides a unifi ed data access point. If the organization does not have a centralized data warehouse, the learning intelligence system becomes even more critical to cross-functional reporting. With a single data warehouuse the system pulls data from two locations (the LMS and the data warehouuse and sends data to one other location (the data warehouse). If the data is contained in many different systems, the system must now send data and receive data through many connections. To manage these differren data sources dynamically, the learning management system should send and receive data through a commmo data format understood by the different systems. Although the multiipl data sources scenario requires a greater system integration effort to accomplish, it provides the organizaatio with greater control over its learning and business data. Automatiin the collection of the training data and consolidating it with business data reduces sources of error and ensures accurate and up-to-date informaation which can be shared more extensively with a minimal degree of administrative effort. Individual and Organizational Performance What differentiates a cross-functioona learning intelligence system from most LMSs is its ability to align training with performance objectiive that have been set for the entire extended enterprise, which includes individuals, the organization, and its business partners. The learning intelligence system can combine the course completion, certifi cation, and assessment scores of the LMS with the evaluation and competency data in a performance management system. Likewise, the system can combine LMS data with business results from other corporate systems. Historical training and business data provides a good starting point for develoopin a statistical training model that will identify what training progrram had the greatest impact on individual and organizational performannce After developing this initial model, the organization can apply the model to current data to quantify how training impacted performance. Reporting and Analytical Tools Reporting fl exibility and intuitive user interfaces are also key to making the data available and easily accessible. Portal applications can provide reportiin tools that training professionals 􀀚 6 © Latitude Consulting Group 􀀛 888-577-2797 􀀛 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence can use to leverage their statistical training models to make more inforrme decisions when designing and implementing a training program. Some possible tools include: A training scorecard that evaluates training programs on ROI and other performance metrics. Sales, manufacturing, distribution, customer service, and other scorecaard that provide performance metrics specifi c to that domain, incluudin training Ad hoc reporting for sophisticated and quick information retrieval to meet pressing business requiremeent Predictive analytical tool that alloowe organizations to allocate training resources to achieve desirabbl organizational performance For example, a training scorecard appliccatio can track ROI metrics, such as , and maintain performance accountaability The training scorecard becomes a much more powerful tool to manage interdependent activities and performance if it has the followiin features: Drills down to supporting data detaai for ROI scores Provides a breakdown of performaanc scores for a wide range of training attributes, including curricullum location, instructor, delivery method, and others Account for regional economic differeence and other environmental factors beyond a the organization’s direct control 􀂃􀂃􀂃􀂃􀂃􀂃􀂃 With an easy-to-use “drill-down” capability, training professionals can identify how cost and performance results contribute to a training program’s ROI score. Providing this type of analysis, however, depends on a statistically validated model of cause-and-effect. Developing a Model for Learning Improvement Physicians study the signs and symptoom revealed by medical tests and prescribe medicines, diets, surgical procedures, or exercise programs to their patients. Taking the medicines in the prescribed dosages, following the recommended diet and practicing the exercises are the critical behaviors that help patients improve their health. If the medical tests were faulty the signs and symptoms would be wrong, the doctors would be unable to make proper recommendations. In medicine, physicians can properly read the signs and symptoms uncoverre by tests and other diagnostic tools. The tests and diagnostic tools work because they embody scientiif c theories of human anatomy and physiology that explain biological interdepenndencies To develop the tools that will help training professionals achieve higher ROI, they need a modee of training for their organization similar to the diagnostic tools used by physicians. A model of training would identify the critical learning variables in all areas of interest and how those variables impact business results. A training model based on multivariaat statistical analysis provides the necessary—and often missing—basis 􀀚 7 © Latitude Consulting Group 􀀛 888-577-2797 􀀛 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence to reduce variation and improve trainiin processes. Evaluating Eff ectiveness Tracking training and business performaanc results are critical to achieving an expected ROI. However, when an organization measures without an understtandin of interdependent causeanndeffect relationships, it does not accurately evaluate training effectiveneess Often, people make inferences about simple causal relationships, focussin on a single cause and effect. For example, good or bad sales may be the result of general economic condittion and other factors. A company may achieve better sales numbers followwin a sales training initiative even if the training itself was defi cient. Tracking results does not necessarily evaluate how training modifi ed sale staff behavior or ability. There can be a distressing independency between training metrics and the information that actually helps an organization leverage training to improve business results. Multivariate analyses can provide a tool for organizations to evaluate and quantify training effectiveness. Specialized techniques such as structuura equations modeling (SEM) serve to merge a decision makers intuitive understanding of the world with multivaariat analyses in order to propose and test broader models of the world (Kline, 2005; MacCallum & Austin, 2000). This statistical understanding can become a predictive model that identiif es how training allocations and expendiiture should be invested to have the greatest impact on training results (competency demonstrated by assessmeen scores or job performance evaluatiions or business results (reductions in assembly line rework or increased customer service satisfaction) for ROI calculations. Additionally, training does not always produce the desired organizational results and improving outcomes depeend on recognizing failed training and its causes. This mathematical method not only helps ROI measuremeent but can identify remedies for lower than expected ROI by identify what intervening factors that disrupt training effectiveness. Simplify Interdependencies by Reducing Complexity A complete ROI analysis depends on the causal connections between trainiin and nontraining data. Developiin a robust training model that can make these connections can require signifi cant data collection and analysis effort. Many training and nontraining activities contribute to performance. Which activities does the organization attempt to manage and to what degrree Identifying what factors actually affect desired performance narrows the scope of activities that must be managed by the enterprise, reducing both the complexity of the relationshhi and the administrative effort required to infl uence the partner’s performance. A statistical analysis of performance and training data and other informatiio can help map the interdependenncie of an enterprise’s training programs with other factors and their impact on business performance. Usiin structured equation modeling, for 􀀚 8 © Latitude Consulting Group 􀀛 888-577-2797 􀀛 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence example, results in a causal mapping that quantifi es how the many-tomaan relationships of a set of activitiie will impact overall organizational performance. These models can then help predict the way in which infl uenccin training in one way or the othee will impact desired behavior (such as key performance indicators). For example, suppose that an automotiiv manufacturer fi nds that differeen types of training yield different customer satisfaction results. One approoac would be to create incentive systems and certifi cation programs that encouraged all dealerships (reduccin variability across the dealer channel) to allocate resources and effoor in those areas that will positively affect customer service. For example, the manufacturer may mandate that dealership service managers complete specialized training that maximizes customer satisfaction more than other types of training. The training model provides both a justifi cation for serviic manager training and identifi es an area for improvement. The organizattio develops a greater confi dence in how training will impact performannce One automotive dealership study contained a detailed analysis of an evaluation system used by an automotiiv manufacturer to measure dealer compliance to the manufacturer’s standards. The manufacturer’s dealer evaluation system measured over a hundred items, including cleanliness, size of signs, whether the waiting room had fresh coffee, and a wide range of other factors. Independent Figure 1 – Dealership Service Training. How specialized service manager and technical infrastructure training can affect customer satisfaction.􀀚 9 © Latitude Consulting Group 􀀛 888-577-2797 􀀛 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence of the compliance scores, the study obtained eight dealer performance measures, including unit sales, part sales, market share for two different vehicles, and customer satisfaction with the sales, parts and service departmments After applying statistical analysis to this data, only seventeen of the measuure items in the dealer evaluations system showed consistent relationshhip with business results. Of those items, certifi ed training was an item that had a positive impact on the dealer performance measures; howevver non-certifi ed training did not appear to affect dealership performannce In this case, certifi cation could demonstrate a measurable ROI based on the business result improvement, quantifi ed by the statistical analysis, and the cost of the certifi ed training programs. Structural Equation Modeling Structural equations modeling (SEM) provides a statistical method to develop a causal network of exceptioona service and quantify how each relevant variable affects service. It is an advanced statistical technique to study the simultaneous impact of severra independent variables on a speciif c outcome variable. Each outcome variable of signifi cant interest would have its own model. For example, if the goal of training is improving customer service, a model might be created for the customer satisfaction index outcome variable. The way structural modeling works is as follows: Set initial model according to expeer judgment. Collect data for all variables. Data may already exist in company databaase or may be commercially availabble Run the statistical software to estimaat the model parameters. Review and revise model according to data results. The resulting output of SEM – a path diagram (see fi gure 1) showing which variables cause changes in other variables – could represent a dealer channel causal network that describes what interrelated factors affect custoome service. Not only do training professionals get an idea of what variabble directly or indirectly affect custoome service, they also get coeffi cient values that quantify how the variables affect one another. For example, an initial model for automootiv customer satisfaction may include a wide range of possible relevvan data values. Large amounts of data already can be collected from third-party dealer management systeem and evaluation systems used by an automotive manufacturer’s fi eld organization to score dealerships on a wide range of metrics. Using statistical software the initial model is tested. Some variables will have an impact on the customer satisfaction outcome and others will not. Variables that 1. 2. 3. 4.􀀚 10 © Latitude Consulting Group 􀀛 888-577-2797 􀀛 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence appear relevant are retained and the model parameters are run again. The emerging body of knowledge on transfer of training (Salas and CannoonBowers, 2001) suggests a number of factors that can affect training effectiveness. For example, (a) the transfer “climate” can have a powerffu impact on the extent to which newly acquired competencies are used back on the job; (b) delays between training and actual use on the job are directly related to skill decay; (c) social, peer, subordinate, and supervisso support all play a central role in transfer; and (d) intervention strategiie can be designed to improve the probability of transfer. These factors would be a good starting point for inclusion in the initial iteration of a structural equation model for training programs. Continuous Improvement In the education and training industriies measuring ROI has been criticiize as an instrument of justifi cation, rather than performance improvemeent However, by using a robust learning management platform and applying statistical methodologies, ROI calculations can be used for continuous improvement as it has in other industries. In manufacturing, fi nal inspection was the dominant model for quality control until it was demonstrated that it made more sense to focus on the processes. It is easier to locate the true cause of a failure, or potential failure, in the process rather than in the fi nal product. Similarly, evaluating training ROI should not use on results helps rationalize what metrics to measure. By selecting those measurements that learning and training professionaal can use to make valid inferences about the effectiveness of programs, they can know where to improve and how to allocate resources and effort. In the manufacturing industry, Six Sigma and Just-in-Time (JIT) productiio are two well-developed concepts that improve quality (Six Sigma) and reduce costs (JIT). Six Sigma is a measureemen of process quality using statistical procedures to continually improve manufacturing processes. Both Six Sigma and JIT continue to signifi cantly impact the automotive industry and can be applied to trainiin effectiveness. Six Sigma improves the quality of each product that rolls off the manufacturrin line and reduces variability in quality from one manufactured produuc to the next. A Six Sigma for trainiin effectiveness would improve each program’s impact on business results.