Evaluating Training ROI with Learning Intelligence

White Paper Evaluating Training ROI with Learning Intelligence by Mark Place with Kurt Crisman, John Zonneveld EXECUTIVE SUMMARY Training ROI calculation requires the measurement of organizational results. Unless training programs exist simply for the sake of training, measurements should include non-training performance data. When evaluating training impact on individual behavior and business results, data collection requirements extend beyond course delivery. Individual performance data exists in performance management systems, and organizational data exists in marketing, sales, and financial systems. A learning intelligence system can act as a broker between learning and business systems and make training and performance data more manageable for ROI analyses. 1 Training is a critical component in any organization’s strategy for innovation and/or continuous improvement. For example, corporate health and safety programs train personnel in the hope that they will reduce workplace accidents, whether mandated by OSHA regulations or by the company. Yet, training is an area where the actual return-on-investment (ROI) is uncertain. Is the training program effective? How can the program be improved? Did the program achieve the desired results at the lowest possible cost? Given the large U.S. expenditure in training, it is important to develop tools that will help companies answer these questions and improve the measurement of training effectiveness. These tools need to provide both a methodology to measure, evaluate, and continuously improve training, as well as the organizational and technical infrastructure (systems) to implement the methodology. Measuring Organizational Results Training ROI calculation requires the measurement of organizational results. Those results may be directly related to training operations, such as course enrollments and completions, assessment scores, and the results of feedback forms and surveys. Other results may be tied to performance data on an individual, department, or business unit level. Unless training program exists simply for the sake of training, measurements should include non-training performance data. Selected metrics, such as sales, cus- © Latitude Consulting Group 888-577-2797 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence tomer satisfaction, workplace safety, productivity, and others, should help demonstrate where training has either increased revenue or decreased costs. The stated goal of a workplace safety training program, for example, is to reduce workplace accidents. Implicit in that goal is the reduction of costs, both tangible and intangible. While tangible costs of workplace accidents are fairly straightforward, the company also incurs other less tangible costs in lost productivity that are even more difficult to measure and account for. An accident in the workplace can adversely affect the behavior of uninjured workers in other ways. They may approach a piece of machinery more tentatively or otherwise work with extreme care to avoid an accident themselves. Rough ROI measurements that consider performance improvements can provide a benchmark for training effectiveness. After implementing a training initiative or changing an existing program, an organization can observe and record a change in performance. Reduced incidents of workplace accidents, for example. The value of the performance change (X dollars saved per reduced incident) can then be compared to the program cost to arrive at an ROI measurement. While this approach appears to answer the question of training program effectiveness, it provides no insight into improving training from a results or cost perspective. It also does not provide a means to calculate ROI that may occur as a result of routine training programs. (How often should workers take the workplace safety program to ensure that accidents remain at the lower level of incidence?) Evaluating Training and Performance Many learning operations evaluate success based on the ubiquitous Kirkpatrick model or some variation of it. Most organizations that follow this model are unable to evaluate their programs beyond the first two levels. In part, the systems used make lower level evaluations easy and don’t provide any mechanism for higher level evaluation. Most learning management systems will automatically track and include templates to report the information required to do level one and level two analyses. They include assessment tools that can capture each learner’s reaction to the course. For online and blended learning, the assessment can be completely integrated into the course. Likewise, training programs can inexpensively and easily administer pre- and post-tests that evaluate learning results (level two). Evaluating levels three and four becomes more difficult and costly to implement and administer. What makes these levels so difficult to evaluate? In the case of the first two levels, data collection occurred during course delivery. When evaluating changes in student behavior and training impact on business results, data collection requirements extend beyond course delivery. Different evaluation methods can help answer whether student’s behavior actually changed after completing a course or attaining a certification. Data can come from many different 2 © Latitude Consulting Group 888-577-2797 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence sources, including performance evaluations by the student’s supervisor or customer satisfaction survey data. Training or other designated staff can also observe students as they perform their jobs and record behavior. To evaluate retention rates, there should be a lag between the training and these behavior measurements. Rarely do organizations evaluating the business impact of a program, which is much more difficult and costly. One might think, “What is so difficult about evaluating the impact of learning and knowledge transfer on business performance? If organizational results improved, you will know that the training was effective.” One problem with this assertion is that you don’t really know if those results are due to training or some intervening factor. Large organizations can use a control group to isolate training from other intervening variables. One group does not get training, while another group does. The business results from each group can then be compared to see if the trained group performed better. Clearly, control groups present a few challenges. Performance must be measured over the same time periods. Both groups must be nearly identical in make-up, performing similar job functions. Sales groups in operating in different sales regions or manufacturing groups working in different shifts will confound the results. With these challenges, structuring a valid control group requires great commitment on the part of the organization. But to perform a good ROI analysis, an organization must really be evaluating its training at levels three and four. Without these levels ROI analysis becomes more an exercise of cost justification for expenditures. Implementing blended learning, for example, may appear to have a high ROI. After all, it’s less expensive than instructor-led, and level two assessment scores have improved. However, without an understanding of the impact on individual and organizational performance, what value is the training really delivering to the organization. Actually, much of the data exists at many organizations to bridge the gap between training and performance. Individual performance data exists in performance management systems. Organizational data exists in marketing, sales, and financial systems. Bridging this gap requires a technical infrastructure that minimizes the administrative effort needed to collect and analyze the training and performance data together. However, learning management systems, the most common repository for training data and mechanism to deliver training, cannot bridge the gap easily. From a functional standpoint, each new LMS release adds more robust reporting and data analysis capability, as well as human resource system integration. In addition, many LMS vendors have added talent and performance and management to their human capital suites of solution. To some degree, these evolutionary changes address the gap between training its impact on individual performance. They don’t even begin to address the gap with organizational performance. Why are organizations still unlikely to evaluate training at Kirkpatrick’s level 3 © Latitude Consulting Group 888-577-2797 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence three? What path can lead to level four evaluations? System integration, one common point of failure, is critical. Many LMS vendors with a history as product companies have limited expertise in system integration that extends beyond learning systems and databases. Successfully managing performance-based training evaluation, however, requires expertise in data management and warehousing, a variety of corporate systems and databases, analytics, and web-based application development. Evaluating training ROI most effectively requires the right technical infrastructure and a model of learning improvement. LMS Independence A learning intelligence system should not be locked into a single LMS platform. By utilizing a generic framework, common LMS data should map to variables in the learning intelligence system. LMS independence ensures more stability over time. It minimizes the extent of system integration that is required if the LMS should be upgraded or replaced by another system. Typically, an organization will feed performance, job code, certification, and other corporate data into the LMS’s reporting system. By adding a learning intelligence system between the LMS and other corporate systems, only one data connection must be updated if the LMS changes. 4 Learning Intelligence System A reporting and data management strategy that focuses on the LMS as the foundation only compounds the system integration challenges that make performance-based training evaluation unmanageable. Instead, the organization should adopt a cross-functional corporate reporting and data management strategy. The technical foundation for this strategy is not the LMS, but a learning intelligence system that acts as a broker between an LMS and other corporate systems. The characteristics of a learning intelligence system include: Independent of LMS Cross-functional system integration Alignment to individual and organizational performance Reporting and analytical tools Cross-Functional System Integration As a broker for business intelligence throughout the organization, a learning intelligence system needs to aggregate the data from multiple corporate systems. If assembling information is too cumbersome and time consuming and the data is outdated or not even correct, the system cannot enhance ROI evaluations by combining training with other business data. Cross-functional system integration allows the organization to leverage training and business data together in a context-sensitive manner. Technical or political requirements may dictate that decision-makers in different corporate domains access data through different systems. Cross-functional system integration allows the learning intelligence system to push data to the portal or reporting system used by a particular decision-maker. © Latitude Consulting Group 888-577-2797 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence One of the primary challenges when implementing cross-functional system integration is the migration of a diverse range of existing data sources. Data feeds, often in a flat file format, may be processed on different systems, including those of third-party vendors. Creating reports and correcting mistakes can be difficult, especially when it involves many people exchanging flat files. If the organization has a corporate data warehouse, the learning management system can push the learning management data into this consolidated data source. Any corporate reporting system can then access this learning data, combine it with other business data, and make more advanced ROI calculations. Different data owners maintain data integrity in the consolidated data source, which provides a unified data access point. If the organization does not have a centralized data warehouse, the learning intelligence system becomes even more critical to cross-functional reporting. With a single data warehouse, the system pulls data from two locations (the LMS and the data warehouse) and sends data to one other location (the data warehouse). If the data is contained in many different systems, the system must now send data and receive data through many connections. To manage these different data sources dynamically, the learning management system should send and receive data through a common data format understood by the different systems. Although the multiple data sources scenario requires a greater system integration effort to accomplish, it provides the organization with greater control over its learning and business data. Automating the collection of the training data and consolidating it with business data reduces sources of error and ensures accurate and up-to-date information, which can be shared more extensively with a minimal degree of administrative effort. Individual and Organizational Performance What differentiates a cross-functional learning intelligence system from most LMSs is its ability to align training with performance objectives that have been set for the entire extended enterprise, which includes individuals, the organization, and its business partners. The learning intelligence system can combine the course completion, certification, and assessment scores of the LMS with the evaluation and competency data in a performance management system. Likewise, the system can combine LMS data with business results from other corporate systems. Historical training and business data provides a good starting point for developing a statistical training model that will identify what training programs had the greatest impact on individual and organizational performance. After developing this initial model, the organization can apply the model to current data to quantify how training impacted performance. 5 Reporting and Analytical Tools Reporting flexibility and intuitive user interfaces are also key to making the data available and easily accessible. Portal applications can provide reporting tools that training professionals © Latitude Consulting Group 888-577-2797 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence can use to leverage their statistical training models to make more informed decisions when designing and implementing a training program. Some possible tools include: A training scorecard that evaluates training programs on ROI and other performance metrics. Sales, manufacturing, distribution, customer service, and other scorecards that provide performance metrics specific to that domain, including training Ad hoc reporting for sophisticated and quick information retrieval to meet pressing business requirements Predictive analytical tool that allowed organizations to allocate training resources to achieve desirable organizational performance For example, a training scorecard application can track ROI metrics, such as , and maintain performance accountability. The training scorecard becomes a much more powerful tool to manage interdependent activities and performance if it has the following features: Drills down to supporting data detail for ROI scores Provides a breakdown of performance scores for a wide range of training attributes, including curriculum, location, instructor, delivery method, and others Account for regional economic differences and other environmental factors beyond a the organization’s direct control With an easy-to-use “drill-down” capability, training professionals can identify how cost and performance results contribute to a training program’s ROI score. Providing this type of analysis, however, depends on a statistically validated model of cause-and-effect. Developing a Model for Learning Improvement Physicians study the signs and symptoms revealed by medical tests and prescribe medicines, diets, surgical procedures, or exercise programs to their patients. Taking the medicines in the prescribed dosages, following the recommended diet and practicing the exercises are the critical behaviors that help patients improve their health. If the medical tests were faulty the signs and symptoms would be wrong, the doctors would be unable to make proper recommendations. In medicine, physicians can properly read the signs and symptoms uncovered by tests and other diagnostic tools. The tests and diagnostic tools work because they embody scientific theories of human anatomy and physiology that explain biological interdependencies. To develop the tools that will help training professionals achieve higher ROI, they need a model of training for their organization similar to the diagnostic tools used by physicians. A model of training would identify the critical learning variables in all areas of interest and how those variables impact business results. A training model based on multivariate statistical analysis provides the necessary—and often missing—basis 6 © Latitude Consulting Group 888-577-2797 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence to reduce variation and improve training processes. (competency demonstrated by assessment scores or job performance evaluations) or business results (reductions in assembly line rework or increased customer service satisfaction) for ROI calculations. Additionally, training does not always produce the desired organizational results and improving outcomes depends on recognizing failed training and its causes. This mathematical method not only helps ROI measurements but can identify remedies for lower than expected ROI by identify what intervening factors that disrupt training effectiveness. Evaluating Effectiveness Tracking training and business performance results are critical to achieving an expected ROI. However, when an organization measures without an understanding of interdependent causeand-effect relationships, it does not accurately evaluate training effectiveness. Often, people make inferences about simple causal relationships, focusing on a single cause and effect. For example, good or bad sales may be the result of general economic conditions and other factors. A company may achieve better sales numbers following a sales training initiative even if the training itself was deficient. Tracking results does not necessarily evaluate how training modified sale staff behavior or ability. There can be a distressing independency between training metrics and the information that actually helps an organization leverage training to improve business results. Multivariate analyses can provide a tool for organizations to evaluate and quantify training effectiveness. Specialized techniques such as structural equations modeling (SEM) serve to merge a decision makers intuitive understanding of the world with multivariate analyses in order to propose and test broader models of the world (Kline, 2005; MacCallum & Austin, 2000). This statistical understanding can become a predictive model that identifies how training allocations and expenditures should be invested to have the greatest impact on training results Simplify Interdependencies by Reducing Complexity A complete ROI analysis depends on the causal connections between training and nontraining data. Developing a robust training model that can make these connections can require significant data collection and analysis effort. Many training and nontraining activities contribute to performance. Which activities does the organization attempt to manage and to what degree? Identifying what factors actually affect desired performance narrows the scope of activities that must be managed by the enterprise, reducing both the complexity of the relationship and the administrative effort required to influence the partner’s performance. A statistical analysis of performance and training data and other information can help map the interdependencies of an enterprise’s training programs with other factors and their impact on business performance. Using structured equation modeling, for 7 © Latitude Consulting Group 888-577-2797 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence 8 Figure 1 – Dealership Service Training. How specialized service manager and technical infrastructure training can affect customer satisfaction. example, results in a causal mapping that quantifies how the many-tomany relationships of a set of activities will impact overall organizational performance. These models can then help predict the way in which influencing training in one way or the other will impact desired behavior (such as key performance indicators). For example, suppose that an automotive manufacturer finds that different types of training yield different customer satisfaction results. One approach would be to create incentive systems and certification programs that encouraged all dealerships (reducing variability across the dealer channel) to allocate resources and effort in those areas that will positively affect customer service. For example, the manufacturer may mandate that dealership service managers complete specialized training that maximizes customer satisfaction more than other types of training. The training model provides both a justification for service manager training and identifies an area for improvement. The organization develops a greater confidence in how training will impact performance. One automotive dealership study contained a detailed analysis of an evaluation system used by an automotive manufacturer to measure dealer compliance to the manufacturer’s standards. The manufacturer’s dealer evaluation system measured over a hundred items, including cleanliness, size of signs, whether the waiting room had fresh coffee, and a wide range of other factors. Independent © Latitude Consulting Group 888-577-2797 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence of the compliance scores, the study obtained eight dealer performance measures, including unit sales, part sales, market share for two different vehicles, and customer satisfaction with the sales, parts and service departments. After applying statistical analysis to this data, only seventeen of the measured items in the dealer evaluations system showed consistent relationships with business results. Of those items, certified training was an item that had a positive impact on the dealer performance measures; however, non-certified training did not appear to affect dealership performance. In this case, certification could demonstrate a measurable ROI based on the business result improvement, quantified by the statistical analysis, and the cost of the certified training programs. The way structural modeling works is as follows: 1. Set initial model according to expert judgment. 2. Collect data for all variables. Data may already exist in company databases or may be commercially available. 3. Run the statistical software to estimate the model parameters. 4. Review and revise model according to data results. The resulting output of SEM – a path diagram (see figure 1) showing which variables cause changes in other variables – could represent a dealer channel causal network that describes what interrelated factors affect customer service. Not only do training professionals get an idea of what variables directly or indirectly affect customer service, they also get coefficient values that quantify how the variables affect one another. For example, an initial model for automotive customer satisfaction may include a wide range of possible relevant data values. Large amounts of data already can be collected from third-party dealer management systems and evaluation systems used by an automotive manufacturer’s field organization to score dealerships on a wide range of metrics. Using statistical software the initial model is tested. Some variables will have an impact on the customer satisfaction outcome and others will not. Variables that 9 Structural Equation Modeling Structural equations modeling (SEM) provides a statistical method to develop a causal network of exceptional service and quantify how each relevant variable affects service. It is an advanced statistical technique to study the simultaneous impact of several independent variables on a specific outcome variable. Each outcome variable of significant interest would have its own model. For example, if the goal of training is improving customer service, a model might be created for the customer satisfaction index outcome variable. © Latitude Consulting Group 888-577-2797 www.latitudecg.com LATITUDE WHITE PAPER Evaluating Training ROI with Learning Intelligence appear relevant are retained and the model parameters are run again. The emerging body of knowledge on transfer of training (Salas and Cannon-Bowers, 2001) suggests a number of factors that can affect training effectiveness. For example, (a) the transfer “climate” can have a powerful impact on the extent to which newly acquired competencies are used back on the job; (b) delays between training and actual use on the job are directly related to skill decay; (c) social, peer, subordinate, and supervisor support all play a central role in transfer; and (d) intervention strategies can be designed to improve the probability of transfer. These factors would be a good starting point for inclusion in the initial iteration of a structural equation model for training programs. In manufacturing, final inspection was the dominant model for quality control until it was demonstrated that it made more sense to focus on the processes. It is easier to locate the true cause of a failure, or potential failure, in the process rather than in the final product. Similarly, evaluating training ROI should not use on results helps rationalize what metrics to measure. By selecting those measurements that learning and training professionals can use to make valid inferences about the effectiveness of programs, they can know where to improve and how to allocate resources and effort. In the manufacturing industry, Six Sigma and Just-in-Time (JIT) production are two well-developed concepts that improve quality (Six Sigma) and reduce costs (JIT). Six Sigma is a measurement of process quality using statistical procedures to continually improve manufacturing processes. Both Six Sigma and JIT continue to significantly impact the automotive industry and can be applied to training effectiveness. Six Sigma improves the quality of each product that rolls off the manufacturing line and reduces variability in quality from one manufactured product to the next. A Six Sigma for training effectiveness would improve each program’s impact on business results. 10 Continuous Improvement In the education and training industries, measuring ROI has been criticized as an instrument of justification, rather than performance improvement. However, by using a robust learning management platform and applying statistical methodologies, ROI calculations can be used for continuous improvement as it has in other industries. © Latitude Consulting Group 888-577-2797 www.latitudecg.com