"Development of an Approach to"
Development of an Approach to Assess, Identify and Share Career Development Services Best Practices (“Best Practices Approach” project) Final Report Prepared for: The Career Development Services (CDS) Working Group, Federal-Provincial-Territorial Forum of Labour Market Ministers (FLMM) Prepared by: Goss Gilroy Inc. Management Consultants 401 Empire Avenue St. John’s, NL Tel: 754-2065 Fax: 754-6303 February 20, 2009 TABLE OF CONTENTS EXECUTIVE SUMMARY ....................................................................................................................... I 1.0 INTRODUCTION ........................................................................................................................3 1.1 PURPOSE OF THE STUDY ........................................................................................................................................3 1.2 ORGANIZATION OF THE REPORT ..............................................................................................................................4 2.0 APPROACH AND METHODOLOGY..............................................................................................4 3.0 FINDINGS & RECOMMENDATIONS ...........................................................................................5 4.0 ADDITIONAL FINDINGS ........................................................................................................... 17 5.0 CONCLUSION .......................................................................................................................... 19 6.0 RECOMMENDATIONS ............................................................................................................. 20 APPENDIX A – AMERICAN EVALUATION ASSOCIATION “EVALTALK” LISTSERV MESSAGE .................... 21 APPENDIX B – INTERVIEW GUIDES .................................................................................................... 22 APPENDIX C – INTERVIEWEES ........................................................................................................... 25 APPENDIX D - BIBLIOGRAPHY ........................................................................................................... 28 Executive Summary Introduction One of the goals of the Career Development Services (CDS) Working Group of the Forum of Labour Market Ministers (FLMM) is to identify and promote best practices in career development services. The Working Group judged that it would be wise to first examine the best way of determining best practices, and then use that information to provide a framework/approach to identify best practices in the area of career development. Hence, the Working Group sought the services of a consulting firm to undertake preliminary research around different approaches to best practices, and this report is the result of that research. Methodology The study methodology included a start-up meeting; a short literature review; an American Evaluation Association EvalTalk listserv post; and key informant interviews. The sixteen key informants included six representatives of five organizations, as well as nine experts. Key informants represented a vast array of organizational types, such as public bodies, universities, international organizations, private business, and non-profit organizations. Interviews were conducted using semi-structured interview guides for each of the organizational and expert interviews. Findings Definitions for the term “best practice” vary widely, ranging from a simple, low-evidence approach to a more highly evidenced approach. Several experts and organizations prefer alternative terms to “best practice”, such as “promising practice”, to reflect a focus on both continuous quality improvement and the importance of context. Best practices can be distinguished from regular practice in that best practices are judged to produce superior results. The structures for and process of deciding upon what practices are best varies among different organizations. Governance structures used for deciding best practices tend to be loosely structured. In any case, it is critical to include all relevant stakeholders when discussing and deciding upon best practices. Although criteria used in determining best practices are highly variable, the five key criteria are: supported, evidence-based, sustainable, replicable, and relevant. In order to be labelled as a best practice, the practice must also: already be in use elsewhere, be considered a “quick hit”, improve efficiency and cost-effectiveness, and be supported by documentation such as performance reports. Pursuing best practices is an ongoing process that requires a significant commitment of time and effort, along with capacity-building. For certain organizations, adopting an integrated approach (e.g. making it part of regular Goss Gilroy Inc. Page i business processes like annual reporting) is most suitable. As best practices are interconnected, best practices should be pursued across categories (e.g. leadership, customer focus, process management) as concurrently as possible. Events to acknowledge best practices are rare – at least “events” in the traditional sense of the word. However, many other communication modes, such as databases, newsletters, virtual communities, and posters, are used to acknowledge best practices. Lastly, best practices are essentially human practices, and the degree of latitude afforded to staff with regard to their practices should reflect the sector, e.g. high- versus low-risk sectors. Focussing on the outcome is what really matters. Recommendations The consultants recommend the following to the CDS Working Group: Consider using an alternate term for “best practice” that reflects the focus on continuous quality improvement and the reality/importance of different contexts. Criteria used by the Working Group (and associated stakeholders) should be supported, evidence-based, sustainable, replicable, and relevant to career development services in Canada. Include all relevant stakeholders, or at least representatives of each stakeholder group expected to engage in and utilize the best practice, when discussing and deciding upon best practices. Encourage integration of a best-practice approach into government planning and reporting processes. Identify at least one champion at each level of government — federal, provincial, territorial — to create enthusiasm for pursuing best practices. Work towards instilling the belief that identifying, pursuing and sharing best practices are integral duties of staff. Such a belief is especially important for key personnel and stakeholders directly involved in career- development services. Develop best-practice training sessions, and deliver this training to key personnel and stakeholders. Explore various communication modes for acknowledging best practices in career development services. Pursue best practices in an interconnected fashion, rather than a silo approach. Goss Gilroy Inc. Page ii 1.0 Introduction The Forum of Labour Market Ministers (FLMM) was formed in 1983, to facilitate and encourage dialogue on labour market matters across Canada. The FLMM is composed of federal, provincial and territorial ministers responsible for issues related to the labour market, and is “committed to work together to create a more coherent, relevant, individualized, accessible and coordinated approach to the development and delivery of labour market information 1 at the local, provincial and national levels.” The Career Development Services (CDS) Working Group was formed in April 2007 by FLMM. The Working Group’s focus is to strengthen the cooperation and collaboration between federal/provincial/territorial jurisdictions on matters related to the development and delivery of career development services at regional and pan-Canadian 2 levels. 1.1 Purpose of the Study One of the CDS Working Group’s goals is to identify and promote best practices in career development services. The Working Group judged that it would be wise to first examine the best way of determining best practices, and then use that information to provide a framework/approach to identify best practices in the area of career development. The interest of the Working Group is to examine the criteria for best practices that are used across sectors and functional areas. Hence, the Working Group sought the services of a consulting firm to undertake that preliminary research. The study aimed to answer the following eight issues: 1. How has “best practice” been variously defined? 2. What distinguishes regular practice from best/exemplary practice? 3. What are the criteria used in determining best practice? What is the rationale for each? 4. What governance structures are used for deciding upon best practices? 5. What decision rules are used to select best practices? 6. What resources are typically required to run a best practice program? 1 Service Canada (2008). Forum of Labour Market Ministers – LMI Product Guidelines. (http://www.worksearch.gc.ca/flmm.jsp;jsessionid=98FE9D3DCBBE9B091AE473508765C058.jvm8?category_id=368&lang=e) 2 The FLMM also established a Labour Market Information Working Group, which supports jurisdictions to undertake projects that address areas of mutual interest and concern related to the development and delivery of labour market information. (Forum of Labour Market Ministers, Labour Market Information Working Group . Welcome to FLMM-LMI! [http://www.flmm-lmi.org/english/View.asp?x=1]) Goss Gilroy, Inc. 3 7. What events are typically used to acknowledge best practices? 8. What categories of best practices have been utilized? 1.2 Organization of the Report This report is organized as follows: Section 2 – Approach & Methodology Section 3 – Findings & Recommendations Section 4 – Additional Findings Section 5 - Conclusions Section 6 – Summary of Recommendations Appendices A, B, C and D list the listserv message, interview guides, interviewees, and bibliography respectively. 2.0 Approach and Methodology This section provides a description of the research approach and methodology used in the study. This study of best practices was conducted by a two-member team of GGI, and was overseen by two members of the CDS Working Group. At the project’s outset, the consultants held a start-up meeting with the two Working Group members, at which the issues and general research approach were discussed. The consultants first conducted a short literature review, locating documents in peer-reviewed journals, grey literature and Internet- based documents. The consultants also scanned the literature and conducted Internet-based research to identify 3 potential key informants representing two groups: academics in the field of best practice and organizations utilizing a best practice approach. The team also posted a message on the American Evaluation Association’s EvalTalk listserv. (See Appendix A for the text of this message.) From this research, the consultant team created a list of potential interviewees. The Working Group representatives provided feedback on both the draft literature review and the draft list of key informants. As key informants were interviewed, some informants identified other potential key informants. The interviews were conducted using semi-structured interview guides for each of the two groups (see Appendix B), commencing the first of the interviews with each group with a pre-test (which indicated the interview questions would work well, and needed no modification). All interviews were held by phone, due to the location of interviewees and consultants, as well as challenges presented by winter weather. 3 The terms “[key] informant”, “interviewee” and “respondent” are used interchangeably in this report. Goss Gilroy, Inc. 4 The consultants spoke one-on-one with sixteen individuals across the two groups of respondents (experts and organizations). Respondents represented fourteen Canadian organizations, one American organization, and one international organization. A total of six respondents were interviewed from organizations, including: two public bodies, one university, one international organization, one private business, and one non-profit organization. Within these six organizations, the consultants spoke with representatives of three Canadian organizations which had been awarded National Quality Institute’s (NQI’s) Level IV Canada Award of Excellence (Gold Medal) in 2008. As discussed below, the description of the level of achievement associated with this award is closely related to common definitions of best practices. A total of nine experts were also interviewed, including representatives of two provincial government departments, four universities, one trade and industry association, and one non-profit organization. 3.0 Findings & Recommendations Issue 1: How has “best practice” been variously defined? Finding: Definitions for the term “best practice” vary widely, ranging from a low-evidenced approach to a more highly-evidenced approach. Some experts and organizations prefer alternate terms to “best practice”, to reflect the ongoing nature of identifying best practices and/or the importance of context. Definitions for the term “best practice” vary widely. The well-known American evaluation expert Michael Quinn Patton, remarks that “the widespread and indiscriminate use of the [term ...] “best practice” has devalued [it] both conceptually and pragmatically because [it] lack[s] any common meaning, standard or definition” (2001, pp. 330- 331). This is supported by one expert respondent who expressed caution with utilizing the term “best practice” so carelessly. The respondent cautioned that if the term continues to be used carelessly, it will be absorbed into everyday language, losing its worth. Best practices have been very simply defined as measures – either specific initiatives or more general policies – that achieve certain defined goals (Naegele and Walker 2006, p. 3). Another simple definition offered by one respondent organization, which has succeeded in reaching the highest certification level with the Canadian-based National Quality Institute (NQI), is a “leading-edge approach”. NQI itself defines best practice as “those practices which have been shown to produce superior results; selected by a systematic process; and judged as exemplary, good or successfully demonstrated” (National Quality Institute 2007, non-paginated). Furthermore, the Business Dictionary (operated by WebFinance Inc.) defines best practice as “methods and techniques that have consistently shown results superior than those achieved with other means, and which are used as benchmarks to strive for” (2008, non-paginated). Similarly, best practices may be defined as “the processes, practices, and systems identified in public or private organizations that performed exceptionally well and are widely recognized as improving an organization’s performance and efficiency in specific areas” Goss Gilroy, Inc. 5 (United States General Accounting Office 1995, p. 6). Performance Management Network Inc. and KPMG developed a definition for the former’s best practices in risk management study for the Treasury Board of Canada Secretariat: “a strategy, approach, method, tool or technique that was particularly effective in helping an organization achieve its objectives” (Performance Management Network Inc. 1999, non-paginated). Duignan suggests that best practice can mean any one or more of the following things: 1. A practice which practitioners know is feasible to implement because they have implemented it 2. A practice which practitioners think probably improves outcomes (but they are not making a strong high-level outcomes/impact evaluation attributional claim for it) 3. A practice which independent evaluators (or reviewers) of some sort think probably improves outcomes (but they are not making a strong high-level outcomes/impact evaluation attributional claim for it) 4. A practice for which someone has made a strong high-level outcomes/impact attributional claim (i.e. they have claimed that they have proof that the practice improves high-level outcomes). (2008, non-paginated) He notes the term “best practice” is usually used in the sense of his first or second definition (2008, non- paginated). Measures that are “replicable” and “scientifically-proven” are also widely associated with the term “best practice” (Chapman 2004, p. 1). Many evaluators commenting on the topic of “best practices” refer to the importance of context. For example, Patton cautions that “a lot of ‘best practices’ rhetoric presumes context-free adoption” (2001, p. 331). He asks his readers to imagine that automobile engineers are asked to identify the best fuel injection system, the best transmission, the best engine cooling system, etc. More likely than not, these best subsystems would come from different car models. If one tried to assemble all the “best” systems from all the best cars, they would not constitute a working car. Each best part (subsystem) would have been designed to go together with other specifically designed parts for a specific model of car. They are not interchangeable” (2001, p. 331). Patton believes that “ ‘best practices’ that are highly prescriptive and specific [...] represents bad practice of best practices”, but that “ ‘best practices’ that are principles to guide practice can be helpful” (2001, p. 331). Richard (2005) indicates that translating best practices from one context to another requires: 1) an understanding of the contextual factors in the setting in which the service will be applied; 2) adaptation require a collaborative effort to be successful; and 3) developing realistic expectations about the process of theory adaptation (pp.198). Some scholars have suggested what they believe to be alternate/more appropriate terms to “best practice”. Patton prefers the terms “better practices” or “effective practices” because these tend less to overgeneralization (2001, p. 331). “Calling something ‘best’ is typically more a political assertion than an empirical conclusion” (Patton Goss Gilroy, Inc. 6 2001, p. 331). UNICEF prefers to use the term “good practices”, which “distil innovated and validated approaches” and “attempt to better understand what works (and what does not work!), how, why, and [under] what conditions” (UNICEF 2008, non-paginated). UNICEF avoids the term “best practice” because this international organization is operating in over 150 countries around the world, in vastly different contexts. Another term frequently utilized as an alternative for best practice is “current thinking”. The term “best practice” conjures a sense of finality and authority, whereas such terms as “better practices”, “effective practices”, or “current thinking” promotes an approach of continuous quality improvement, or flexibility. In the area of health/medicine, researchers have begun to replace the term “best practices” with terms such as “evidence-based”, “research- tested interventions” and “promising practices” (Laufman 2008, non-paginated). One of the organization representatives and two of the experts also prefer the term “promising practices”. Recommendation: Consider using an alternate term for “best practice” that reflects the focus on continuous quality improvement and the reality/importance of different contexts. Given that the Working Group serves a wide variety of geographic areas – federal, provincial and territorial – and a wide spectrum in terms of economic development, types of organizations, etc., using a term that reflects the realities of different contexts would be useful, such as “promising practices”. Issue 2: What distinguishes regular practice from best/exemplary practice? Finding: Best practice can be distinguished from regular practice in that best practices are judged to produce superior results. There was little in the literature regarding what distinguishes best practice from regular practice. Certainly, any distinction depends, at least in part, upon the definition of "best practice" that is used. For instance, borrowing from the National Quality Institute's definition, best practice may be distinguished from regular practice in that the former represent a set of practices that are judged, through systematic process, to produce superior results. Furthermore, best practice may be distinguished by how it "[embraces] fairness to all […], is implemented conscientiously and shows noteworthy results" (The U.S. Equal Employment Opportunity Commission 1997, non- paginated). Interviewees suggested that regular practice might be thought of as standard, run-of-the-mill practice, which has a lower probability of achieving desired results. In this case, little or no reflection occurs as to the effectiveness or efficiency of the practice in attaining desired results. In contrast, best practices have a greater probability or track record of achieving results, in part due to the reflective process that has been undertaken to determine that the Goss Gilroy, Inc. 7 chosen approach has a higher likelihood of achieving desired results and/or is actually reaching targets; and also in part because the practice tends to be consistent, focussed and sustained over time. By way of example, a regular practice in the selling profession might be for sales persons of store R ("Regular") to highlight features of a product to potential customers, but this practice does not result in high sales. The sales staff of store B ("Best"), however, might highlight benefits and appeal to their potential clients' emotions, thereby resulting in higher volume or value of sales, i.e. attaining better results. The approach/experience of store B's sales staff's, supported by research findings – either academic or in grey literature – in the area of sales, may be described as "best practice" in selling technique. Issue 3: What are the criteria used in determining best practice? What is the rationale for each? Finding: Specific criteria utilized in determining best practices are somewhat shaky and are highly variable depending on the sector; however, five key criteria emerged from the interviews, including: supported, evidence-based, sustainable, replicable, and relevant. The typical selection process for best practices is recognized as being “somewhat shaky”. A substantial amount of best practices are presented without clearly stated criteria of their relative merit (Green 2002, pp. 4). No explicit criteria are used in most cases, or there is a lack of consistent criteria used in the selection of best practices (Green 2002, pp. 7). Key informant responses closely follow the findings from the literature, given that most respondents did not stipulate “explicit criteria” used in determining best practices. One respondent specifically noted that they do not have set criteria from the start, as the process of determining best practices is a “constantly evolving” process. Another respondent suggested that it is difficult to generate a list of generic criteria to determine best practices; criteria need to be “customized for the sector.” Responses were varied across sectors. Health-sector respondents defined best practices in a prescribed, scientific manner; whereas social-sector respondents defined best practices in a generative manner. It can be suggested that the criteria identified by respondents can be considered to exist along a continuum; wherein one end is considered to inform standards and protocols (prescriptive), and the other is considered to inform the sharing of innovative practices or ideas (generative), or as one respondent suggested criteria at this end of the continuum would feed into the perspective of “good things to try.” For the purposes of this project, the term "criteria" is defined as a standard of judgment or criticism; a [set of] rules or principles for evaluating or testing something (Dictionary.com, 2009, non-paginated). Criteria can therefore be inferred from the key informant’s stated definitions of best practices, as it can be argued that definitions are Goss Gilroy, Inc. 8 composed of a set of definiens (criteria) that constitute the definiendum (best practice). Consequently, key informants identified the following five criteria in determining best practices. The majority of respondents (7) identified support as a key criterion in determining best practices. Support should not be confused with the criterion evidence-based (described below); in this case support refers to internal support (i.e. the availability and perceived level of support within an institution). Most organizations (4) and key informant experts (3) identified the need for stakeholder support as a key element in determining a best practice. The majority of respondents (7) also identified evidence-based as a key criterion in determining best practices. Evidence-based was also referred to as “validated/validation”, “documented successful achievement” and “acknowledged by a second or third party as being exceptionally good”. “Evidence-based” includes both qualitative and quantitative data. Respondents were divided as to which form of evidence is considered the gold standard. It can be suggested, however, that this variance is attributable to the sector in which the respondent works from (see above, health vs. social). Most respondents argued that quantitative data/support for a best practice is considered more “rigorous”; however, a few respondents also noted that qualitative data should not be overlooked, as the “human side” should also be considered. Respondents (4) also identified replicable as a criterion in determining best practices (also referred to as “reproducible” and “repeatable”). Replicable refers to the ability of the best practice to be duplicated across time and location. This is an important criterion to consider, as it is often the intent of a best practice is to be utilizable across contexts (discussed below). A few respondents (3) also suggested that that sustainable is another important criterion (also referred to as “well-entrenched” or “constant”). As one respondent explained, a best practice that is considered as being the “best” today is often surpassed tomorrow. Best practices require depth to withstand the onslaught of change. It should be noted, however, that a large majority of respondents indicated that best practices are in constant evolution. It can therefore be suggested that the criterion of sustainability in determining best practices need not remain stagnant to ensure its longevity; rather sustainability refers to the ability of the best practice to retain its relevance. Similarly, respondents also identified the criterion of relevance as a key factor in determining best practices. Relevance highlights the reality of different needs across various sectors, organizations, institutions, etc. The best practice must be relevant in order for it to be determined as a best practice for a specific group. Recommendation: Criteria used by the Working Group (and associated stakeholders) should be supported, evidence-based, sustainable, replicable, and relevant to career development services in Canada. Goss Gilroy, Inc. 9 Issue 4: What governance structures are used for deciding upon best practices? Finding: Governance structures used for deciding upon best practices within an organization tend to be loosely defined. The choice of governance structure used for deciding upon best practices is related to a given organization's conceptualization of best practice. For example, one large organization with many disbursed offices sees best practices primarily as fodder for new ideas for its various departments/offices, and so the vetting process (i.e. deciding what constitutes a given best practice) is viewed as secondary to sharing. As such, the governance structure for this organization (with respect to best practices) is not a rigid one. Interviewees did not tend to identify a particular individual or group of individuals within the organization as having the "last say" with respect to what constitutes a best practice, but, generally speaking, it was a group of leaders or managers in the relevant department (or organization as a whole). Some interviewees pointed to external validation as being significant, reflecting a conceptualization of best practice as that which is acknowledged by a second or third party as being exceptionally good at meeting or exceeding standards set for a certain practice. For instance, a staff person might take note of an experience with another department in the organization that significantly exceeds the norm. There are also external-to-organization validations. For example, for an application submitted to its signature Progressive Excellence Program®, NQI makes the final decision as to whether the particular level of certification has been attained by a candidate organization. Even given this external validation process, a representative from one of the organizations that had achieved gold-level certification spoke of the requirement that at least 60% of the organization's internal team agree on how the organization is performing on various dimensions. That representative did point out how team members tended not to be quick to deem that all was acceptableand so the team's standards were assessed, at least by the respondent, to be toward the high end of the spectrum. External- to-organization validation may also happen less formally. For example, an organization posts a tool/document on its website for use by its clients, and another similar organization asks if it could use the tool with its staff. In many organizations, representatives from all levels of the organization have input into the determination of best practices. The vast majority of interviewees maintained that it is critical to involve all stakeholders in the process of identifying which areas to scrutinize so as to generate best practices, what might constitute a best practice, etc. Correspondingly, in the NQI certification process, consultations are held at all levels of an applicant organization. Ensuring that the decision-making process is as broad and inclusive as possible has several benefits — increasing engagement of partners and staff, creating buy-in, increasing sense of ownership, valuing individuals' ideas, and so Goss Gilroy, Inc. 10 forth. This often makes for an extensive process, but the benefits of consultation are seen by interviewees as outweighing the time and effort required. Recommendation: Include all relevant stakeholders, or at least representatives of each stakeholder group expected to engage in and utilize the best practice, when discussing and deciding upon best practices. Virtually all respondents in this study stated the importance of involving various stakeholder groups when discussing best practices, e.g. staff (from a variety of positions and responsibility levels), partners, and customers/clients. In the case of the CDS Working Group, this might include Working Group members (i.e. the directors, managers, senior analysts and coordinators at the central provincial/territorial agencies, plus the federal human resources/labour skills department represented on the Working Group), ensuring representation from all provinces and territories as well as the federal level; a mix of government, non-profit and private sector representatives; academics in the field of career development; representation from historically disadvantaged groups such as Aboriginal persons, women, persons with disabilities and racialized persons minorities; young and older workers; etc. Although it may not be feasible to include representation from all of these in the final decision- making group, there should at least be a process of consultation with each of these groups. Issue 5: What decision rules are used to select best practices? Finding: Decision rules used to select best practices were not explicitly identified by respondents; however, organizational guiding principles and business practices may help to identify implicit decision rules employed by organizations that would help to inform the selection of future best practices. Although most respondents were not explicitly asked to identify or explain the decision rules they use to select best practices, it can be assumed from the interview data that respondents are generally not cognisant of the specific decision rules they employ to select best practices. For example, one respondent who explicitly referred to decision rules as “guiding principles”, explained that the guiding principles used by the organization are “not on the tip of everyone's tongue and [do not] formally guide” the organization; however, the respondent explained that upon reviewing the best practice process, a set of implicit guiding principles were practised and also underlie the organization’s everyday business practices. Similarly, a number of respondents commented on “selection criteria” they use to select best practices. The idea of decision rules is best explained as the process/approach an individual or organization employs to select best practices for an identified issue/area. For example, if an organization identifies their goal as “informed decision making”, the decision rules could be considered the “set of conditions” an organization would use to Goss Gilroy, Inc. 11 select best practices (among many choices of best practices). Decision rules can also be considered as a hierarchical sequence of (contingent) decisions (“If…then” statements). The idea of decision rules stems from game theory, wherein decision rules are “a mapping from the expressed preferences of each of a group of agents 4 to a group decision” (About.com , 2009, non-paginated). Respondents representing organizations with best practices were in the best position to specifically identify decision rules used to select specific best practices within the context of their sector and/or organization. The highlighted decision rules were each identified by one respondent; commonalities across organizations were not observed. Nevertheless, respondents were in agreement concerning one key decision rule: the process of vetting best practices with relevant stakeholders, as discussed in the section above. This element of the decision-making process, however, was often employed once a group of best practices had already been identified. The decision rules identified below concern the initial stage of the selection process, in which decisions must be made to select best practices among many choices of best practices. One respondent explained that in order to select a best practice the practice must be already in use somewhere. The respondent explained that it is extremely important to build on existing practices currently utilized within the real world, rather than in theory. Often best practices are identified on paper as being “the best”; however, in actuality, the practice does not work under realistic conditions. It is therefore paramount that best practices exist in the context of where it is expected to be utilized. Another respondent explained that in order to select a best practice the practice must be considered a “quick hit”. A “quick hit” refers to the combination of effort and value. Best practices are analyzed based on a decision quadrant of low to high effort and low to high value. Best practices identified as being low effort and high value are prioritized for selection as a best practice. Best practices therefore are assessed and prioritized based on where they fall along on the quadrant. The respondent explained that best practices with low value and high effort are never selected; as one respondent explained, for best practices to be selected they must truly “make a difference”. Respondents also highlighted the need for best practices to improve efficiency and cost-effectiveness, i.e., the practice must produce a good return on investment. As noted, one respondent specifically referred to organizational guiding principles, best described as a combination of good business practices and exemplifying those business practices through action. Guiding principles employed by the organization include openness and innovation. Openness refers to the spirit of continuous improvement, in which ideas, measures and processes are exchanged to support open dialogue. Along similar lines, innovation refers to the practice of remaining curious and active around finding a better way, in the 4 About.com is part of the New York Times Company. Goss Gilroy, Inc. 12 short and longer term. This further supports the observation that the decision rules used to select a best practice are generally implicit, best identified as an organization’s guiding principles or business practices. Blurring the lines between criteria in determining best practices and the decision rules used to select best practices, another respondent specifically indicated that in order for a best practice to be selected, “the practice must be supported by performance reports, meta-evaluations, satisfaction surveys, needs assessments, informal benchmarking, and external auditing”. It can be suggested that the decision rules employed by this organization closely follow the evidence-based criterion in determining best practices. This specific point highlights the circular nature of criteria and decision rules. For example, the criteria used to determine a best practice may be employed both at the beginning and end of the process of determining a best practice. Criteria assist in identifying potential best practices among many practices; they assist the individual or group to sift through relevant information. Once areas of best practices are identified (based on a set of criteria), decision rules are utilized to further narrow the selection of best practices. For example, a best practice may be considered to meet the criteria of evidence-based, supported, and sustainable, however, if the best practice is not employed within a real-world setting and is not considered a “quick hit” (decision rules), the best practice may not be selected. Issue 6: What resources are typically required to run a best practice program? Finding: Pursuing best practices is an ongoing process that requires a significant commitment of time and effort, along with capacity-building. For certain organizations, adopting an integrated approach is most suitable. Pursuing best practices requires a high level of organizational commitment and at least one (internal) champion who can continuously inspire the organization in this quest. Interviewees emphasized how focussing upon best practices takes time. They likewise made clear that there exists a learning curve, especially for organizations in the early stages of pursuing NQI first-level certifications. For example, some organizations that acquired NQI's Gold status in 2008 began their pursuit of excellence in the mid- to-late 1990s. Even when an organization has identified its best practices, many of these are anything but stagnant. Rather, due to ongoing changes in the business/global environment (e.g. advances in information technology, new entrants into the marketplace, etc.), best practices need to be ever-evolving. Pursuing best practices takes significant effort. Some winners of the NQI Gold certification spoke of a level of exhaustion at the end of this multi-year endeavour. Nonetheless, they recognized that identification and pursuit of excellence must still continue. Goss Gilroy, Inc. 13 Attention should certainly be focussed on sufficient capacity-building within an organization – and among its stakeholders – toward the support of best practices. Education of staff is essential for maximizing engagement, fostering a sense of inclusion, and creating an increased feeling of ownership of the process and results. The preferred approach for one organization has been to integrate the identification and sharing of best practices into regular business processes (e.g. annual reporting), thereby embedding best practice into program and intervention planning. As one respondent explained, the process of identifying best practices and integrating them into everyday practice has created a cultural shift within their organization, one in which employees have come to expect that best practices are part of their job descriptions. This approach works for this organization as staff say they would otherwise be too busy to undertake what they see as the additional task(s) of identifying, discussing and sharing best practices. [See Duignan (2008) for further information on how to visually represent best practice and how to communicate in modalities that embed best practice directly into program planning.] Note, however, for some organizations finding and sharing best practices is not viewed as an "extra" by their staff. This tends to apply to private organizations that aim to provide, and indeed are providing, a high level of professional services to their clients. In essence, their clients require such a focus of them. The human resources required to pursue a best-practice approach vary from one organization to another. For instance, one large organization draws upon three HQ staff, many decentralized staff, IT resources, and the services of an editor and graphic designer (as required). Another organization maintains a department of four staff who spearhead the NQI process, while the whole organization is involved intermittently. Yet another organization employs one Full-Time Equivalent for one year. And yet another employs one person for one day a week, plus the involvement of an internal group. Recommendations: Encourage integration of a best-practice approach into government planning and reporting processes. Identify at least one champion at each level of government — federal, provincial, territorial — to create enthusiasm for pursuing best practices. Work towards instilling the belief that identifying, pursuing and sharing best practices are integral duties of staff. Such a belief is especially important for key personnel and stakeholders directly involved in career-development services. Develop best-practice training sessions, and deliver this training to key personnel and stakeholders. Goss Gilroy, Inc. 14 Issue 7: What events are typically used to acknowledge best practices? Finding: Events to acknowledge best practices are rare – at least “events” in the traditional sense of the word. However, many other communication modes are used to acknowledge best practices. The sole event informants cited to acknowledge best practices was the yearly NQI big event where award winners are presented with their trophies, which they hold for three years (and then apply for re-certification). The effect on the organization’s staff for this event is a sense of pride and recognition. At the same time, there are a number of other types of communication modes that are used by many of the organizations profiled to acknowledge best practices. That is to say, while each of these are non-events in the sense that each is not something that occurs in a certain place during a particular interval of time, each represents some set of behaviour, activities or modes of productions that serve to accomplish the acknowledgement of best practices. For instance, some organizations had databases which house best practices. The practices can be ranked by users, similar to what is done on Amazon.com where readers rate the books that they have read. This type of database serves as a source of ideas for other decentralized offices on how to approach a particular problem. Once a number of offices had implemented the best practice, with or without modifications, over time the practice would become guidance. At least one organization identified its policy and procedures database as a vehicle for documenting and sharing best practices. One large organization (and another large organization which declined an interview but has worked with NQI) produces best practices newsletter regularly and shares it throughout the organization, including on the organization’s intranet. One large organization produces annual compendiums of its best practices. A couple of organizations have virtual (web-based or intranet) communities where experts in their field/area in the organization share leading-edge knowledge with their colleagues. Related to this is the use of wikis by at least one organization. A wiki is a page or collection of web pages designed to enable anyone who accesses it to contribute or modify content, using simplified mark-up language. One organization uses posters to share best practices. This raises staff interest and engagement, pride and reinforces positive principles that the organization is trying to instil. The messages being communicated by these Goss Gilroy, Inc. 15 best practice posters are further reinforced by the leader of this best practices project making positive comments on behaviours that reflect a best practice principle, e.g. “That’s an excellent example of [principle]”. Organizations identified various meetings and training events as venues for (discussion and) sharing of best practices. Recommendation: Explore various communication modes for acknowledging best practices in career development services. Several communication modes would appear to be useful for the Working Group in acknowledging best practices. In particular, given the federal, provincial and territorial nature of the Working Group (and FLMM), a database to which members could have access and rate would likely be highly useful. Whatever communication modes are used/decided upon, it is critical to have online versions of these, given the pervasiveness and growth of Internet- based use. Issue 8: What categories of best practices have been utilized? Finding: The categories of best practices vary widely, depending on the organization’s profile. The categories of best practices varied widely, depending on whether the organization is public or private; provincial/territorial, national or international etc. Examples of categories for the organizations surveyed are listed below: business excellence (NQI), which includes the following seven subareas, or “sections”: leadership, planning, customer focus, people focus, process management, supplier/partner focus and overall business performance (into which the first six sections feed) healthy workplace (NQI), which includes the following six subareas, or “sections”: principles, leadership, planning and programs, people engagement, process management and risk assessment, and results a whole range of consulting areas – e.g. communications, the process of consulting programming – e.g. different sub-sectors within the sector served by the organization, e.g. for UNICEF, program areas such as child protection, health, and education operations – e.g. emergency operations, human resources public works – e.g. sewage treatment, pavement, infrastructure Goss Gilroy, Inc. 16 gaming – The Responsible Gaming Council’s Centre for the Advancement of Best Practices developed a Responsible Gaming Index, composed of eight areas/elements that determine the way gaming providers manage gambling. The eight areas/elements within the Index include: policies; self-exclusion; advertising and promotion; informed decision making; reaction plans and available assistance; access to money; game and venue features; and employee training. career development – The literature review revealed some examples of areas in which best practices could be identified. Specifically, Richard (2005) highlights the potential for a “framework to guide the establishment of a best practices paradigm in career planning and guidance”, which includes three areas of program implementation, namely: 1) content; 2) process; and 3) infrastructure. Content refers to the elements that are important career development competencies, which include the knowledge, skills and abilities (KSAs), required to make effective career decisions. Process specifically refers to what occurs during career counselling; that is individual interpretations of career-planning activities and decision making strategies, modelling and exposing individuals to successful career-decision making individuals, and the establishment of a support network. Process also includes the acquisition of experiential learning through career education. Lastly, infrastructure refers to the tangible, coordinated network of support within an individual’s institution, which includes leadership, staffing, facilities, and other institutional resources that support the implementation of exemplary career planning services (pp. 194-198). Closely associated with categories of best practices are levels of achievement/award levels for best practices. NQI awards four levels of achievement in its Progressive Excellence Program. Achievement of Level I indicates the organization’s readiness to make change. Level II indicates that planning is in place and that a strategy has been initiated. Level III Canada Award of Excellence (Silver Medal) indicates that implementation is widespread. Finally, Level IV Canada Award of Excellence (Gold Medal) indicates that the organization is achieving sustained results and that continuous improvement is a “way of life” in all areas (City of Kamloops, 2008, non-paginated). The descriptors/criteria for Level IV resemble most closely the definition of best practices, with its emphasis on sustained results. As discussed above, the consultants were fortunate to speak with representatives of three such organizations in the course of this study. 4.0 Additional Findings In the course of this study, the consultants found additional pieces of information that are worthy of note and likely to be of assistance to the Career Development Services (CDS) Working Group. Several informants spoke of the interconnectedness of best practices, that is, how working on a best practice in one department of an organization affects practices in another department. Consequently, separate departments would be advised to work on best practices in tandem so as to maximize efficiencies and effectiveness. This is contrasted with a silo-ed approach, where departments within the same organization communicate inadequately Goss Gilroy, Inc. 17 — or not at all — and pursue best practices in isolation from one another, resulting in an overall approach that is disjointed, inefficient and ineffective. This notion of interconnectedness must be recognized as also applying to interactions with groups and individuals outside the organization, e.g. customers, partners, etc. Engaging such “outsiders” in the best-practices approach is critical, as discussed earlier in this report. There are many benefits in adopting a best-approach focus. Positive impacts of pursuing and implementing best practices include: enthusiasm for excellence, pride in achievement, a more disciplined approach, financial and company growth, employee engagement/buy-in, better communication (between units, with stakeholders), greater understanding of the interconnectedness of various departments of an organization, superior decision- making, and a greater focus on the organization's culture. For instance, one organization experienced a shift in culture, in which now exists an expectation that best practices are part of everyone's duties. It is important to remember that best practices are essentially human practices. For sectors deemed lower risk (vs. life-or-death realms such as medicine or OH&S), it is helpful to realize that many people prefer to have some amount of leeway or choice within their work, rather than be prescribed exactly how to undertake a given activity. Employees should similarly have some amount of freedom to make mistakes (again, exempting life-or-death contexts). Such freedoms will frequently lead to innovation, which several informants pointed to as being part- and-parcel of their (or their organization's) approach to best practices. Recognizing that best practices are built on human behaviour, some informants stressed that what really matters for an organization, and by consequence, individuals within that organization, is the outcome or results that are achieved. Given the various personalities and life situations of their staff, they, as supervisors, advocate that the best approach is to decide jointly with a staff person on what the outcome(s) should be, then allow that person to determine the best way to achieve that outcome. Empowered by approaches that are less prescriptive, staff will find various/novel ways of approaching a task, which will lead to the desired outcome via the most efficient and effective manner for them. There are many sources for generating ideas as to what might make for a best practice, and many venues for further honing in on best practices. Organizations and experts pointed to: business planning exercises, NQI's networking meetings, professional development events (e.g. conferences, training sessions), internal innovation, suggestion boxes, and literature reviews/searches. Other organizations are also a key source of best practices. As Rosabeth Moss Kanter says "[s]ome of the best innovators I know are borrowers" (2003, p.1). All great borrowers – perhaps all visionary leaders – know the best practice secret. Stretching to learn from the best in any sector can make a big vision more likely to succeed. Of course, reaching for excellence requires learning what excellence is. High performers open their minds to Goss Gilroy, Inc. 18 new ideas, challenge their assumptions, and scrutinize their performance practice by practice. (Kanter, 2003, p. 1) Recommendation: Pursue best practices in an interconnected fashion, rather than a silo approach. 5.0 Conclusion Definitions for the term “best practice” vary widely, ranging from a simple, low-evidence approach to a more highly evidenced approach. Several experts and organizations prefer alternative terms to “best practice”, such as “promising practice”, to reflect a focus on both continuous quality improvement and the importance of context. The consultants recommend that an alternative term be used by the CDS Working Group. The structures for and process of deciding upon what practices are best varies among different organizations. Governance structures used for deciding best practices tend to be loosely structured. Given the variations in delivery structures/mechanisms for career development services in the federal, provincial and territorial jurisdictions, the consultants recommend that a more generative rather than prescriptive approach be used. In any case, it is critical to include all relevant stakeholders when discussing and deciding upon best practices. Although criteria used in determining best practices are highly variable, the five key criteria are: supported, evidence-based, sustainable, replicable, and relevant. In order to be labelled as a best practice, the practice must also: already be in use elsewhere, be considered a “quick hit”, improve efficiency and cost-effectiveness, and be supported by documentation such as performance reports. These criteria and decision rules represent a good starting point for the Working Group. Pursuing best practices is an ongoing process that requires a significant commitment of time and effort, along with capacity-building. For certain organizations, adopting an integrated approach (e.g. making it part of regular business processes like annual reporting) is most suitable, and the consultants suggest that the Working Group members take this approach. As best practices are interconnected, best practices should be pursued across categories (e.g. leadership, customer focus, process management) as concurrently as possible. In a career development services context, this could include the pursuit of best practice identification and sharing as it relates to content, process and infrastructure. Events to acknowledge best practices are rare – at least “events” in the traditional sense of the word. However, many other communication modes, such as databases, newsletters, virtual communities, and posters, are used to acknowledge best practices. The consultants recommend that these and other innovative approaches be explored. Goss Gilroy, Inc. 19 Online databases and virtual communities appear to have promising applications for a cross-Canada initiative, given the increasing use of and access to the internet and a widely disbursed population base. Lastly, best practices are essentially human practices, and the degree of latitude afforded to staff with regard to their practices should reflect the sector, e.g. high- versus low-risk sectors. As career development services is more of a low-risk sector, the consultants suggest a less prescriptive approach be used, with a focus on outcomes. 6.0 Recommendations The consultants recommend the following to the CDS Working Group: Consider using an alternate term for “best practice” that reflects the focus on continuous quality improvement and the reality/importance of different contexts. Criteria used by the Working Group (and associated stakeholders) should be supported, evidence-based, sustainable, replicable, and relevant to career development services in Canada. Include all relevant stakeholders, or at least representatives of each stakeholder group expected to engage in and utilize the best practice, when discussing and deciding upon best practices. Encourage integration of a best-practice approach into government planning and reporting processes. Identify at least one champion at each level of government — federal, provincial, territorial — to create enthusiasm for pursuing best practices. Work towards instilling the belief that identifying, pursuing and sharing best practices are integral duties of staff. Such a belief is especially important for key personnel and stakeholders directly involved in career- development services. Develop best-practice training sessions, and deliver this training to key personnel and stakeholders. Explore various communication modes for acknowledging best practices in career development services. Pursue best practices in an interconnected fashion, rather than a silo approach. Goss Gilroy, Inc. 20 Appendix A – American Evaluation Association “Evaltalk” Listserv message Hello, Our management consulting firm is looking for 3-4 potential interviewees who might provide information about approaches to determining best practices, either from an academic perspective or from practical implementation perspective within an organization. Goss Gilroy, the Canadian management consulting company with which I work, has been contracted by a Canadian federal-provincial-territorial government working group to examine approaches to best practices, e.g., definitions of "best practice" (or similar terms), criteria used, decision rules, resources required to run a best practices program, etc. If you have such expertise and are willing to participate in a telephone interview of approximately 30 minutes, I would be most grateful. If you can suggest any other interviewees, I would also be appreciative of contact details. You can send me an offline message (to firstname.lastname@example.org) if you wish, unless you wish to share resources that might be useful to the larger group. (I am already aware of Michael Quinn Patton's 2001 excellent article on this topic, and have also found useful discussions on related 1998 EvalTalk threads.) Thank you, Goss Gilroy, Inc. 21 Appendix B – Interview Guides BEST PRACTICES APPROACH PROJECT FEDERAL-PROVINCIAL-TERRITORIAL FORUM OF LABOUR MARKET MINISTERS (FLMM) CAREER DEVELOPMENT SERVICES (CDS) WORKING GROUP INTERVIEW GUIDE – BEST PRACTICES EXPERTS Introduction Our firm, Goss Gilroy Inc. (GGI), is assisting the Career Development Services (CDS) Working Group of the Federal- Provincial-Territorial Forum of Labour Market Ministers (FLMM) with identifying a approach for determining best practices. As part of our research, we are speaking with academics and other experts in the area of best practices, in order to, among other things, understand what distinctions can be made between good and best/exemplary practices. We wish to speak with you for approximately 30 minutes, in order to gain some knowledge from you in this area. If you wish to have any information that you share not attributed to you or your organization, you can indicate this during the interview. I will also give you opportunity at the end to indicate any information you wish to be kept anonymous. If you have any questions or require clarification with respect to the interview process or the evaluation overall, please do not hesitate to contact the GGI project manager, Sarah Mills, at (709) 754-2065. Your input will contribute to the success of this research project and we thank you for your participation. Questions 1. How might “best practices” be defined, i.e. a good working definition? a) Is “best practices” the best term? If not, what might be a better term? 2. What distinguishes best practices from good practices? 3. How are best practices typically determined? (e.g. criteria used) a) How effective do you think the typical approaches to best practices are best practices? b) Can you recommend better ways of determining best practices? 4. What things might need to be considered in applying best practices of one organization to other organizations? 5. Do you have any other comments to share regarding determining best practices? 6. Are there any organizations that have an effective best practices approach? a) Is there any other person or organization with whom we should speak as we conduct our research on approaches for determining best practices? Thank you for your input. Goss Gilroy, Inc. 22 BEST PRACTICES APPROACH PROJECT FEDERAL-PROVINCIAL-TERRITORIAL FORUM OF LABOUR MARKET MINISTERS (FLMM) CAREER DEVELOPMENT SERVICES (CDS) WORKING GROUP INTERVIEW GUIDE – ORGANIZATIONS Introduction Our firm, Goss Gilroy Inc. (GGI), is assisting the Career Development Services (CDS) Working Group of the Federal- Provincial-Territorial Forum of Labour Market Ministers (FLMM) with identifying an approach for determining best practices to use in the area of career development. As part of our research, we are speaking with representative of various organizations which have used a best practices approach to their programs or services. We wish to speak with you for approximately 30 minutes, in order to gain some knowledge from you in this area. We will be profiling different approaches to best practices in our final report. If you wish to have any information that you share not attributed to you or your organization, you can indicate this during the interview. I will also give you opportunity at the end to indicate any information you wish to be kept anonymous. If you have any questions or require clarification with respect to the interview process or the evaluation overall, please do not hesitate to contact the GGI project manager, Sarah Mills, at (709) 754-2065. Your input will contribute to the success of this research project and we thank you for your participation. Questions 1. Does your organization have a definition for “best practices” (or similar term used)? 2. In what areas have you identified best practices? 3. Is it a structured program? b) What is the title/name of your best practice program? c) How long has the program been running? d) Was it a one-time event or is it an ongoing process? 4. Who is involved in identifying/deciding upon what makes a best practice? a) Do you have specific criteria? What selection criteria do you use? How did you decide on these criteria? b) How do you evaluate practices to determine if they are “best practice”, i.e. evidence? c) What ranking system do you use? Do you use a weighting system? d) How do you involve the leaders of the organization in your best practices approach? 5. How do you use your best practices? a) What resources (human and financial) are required for implementing your best practices approach? b) How do you communicate your selection of best practices? To whom (e.g. internal, organization’s members)? c) What difference has the approach resulted in? 6. What has gone particularly well with your approach? If you were to change anything about your approach, what would it be? 7. Do you have any other comments to share about an approach to best practices? Goss Gilroy, Inc. 23 8. Are there any other organizations of which you are aware that have a best practice approach? 9. Is there any information that you have shared that you prefer not to have attributed to your organization in the report that we will be writing about approaches to best practices? Thank you for your input. Goss Gilroy, Inc. 24 Appendix C – Interviewees Organizations City of Kamloops City Hall 7 Victoria Street West Kamloops, BC V2C 1A2 www.kamloops.ca (250) 828-3396 Hill & Knowlton 160 Bloor Street East, Suite 700 Toronto, Ontario M4W 3P7 www.hillandknowlton.ca (416) 413-4622 Industrial Accident Prevention Association (IAPA) Centre for Health & Safety Innovation 5110 Creekbank Road Suite 300 Mississauga, ON L4W 0A1 www.iapa.on.ca (905) 614-4272 ext. 2336 (905) 614-4272 ext. 2287 Responsible Gaming Council's Centre for the Advancement of Best Practices 411 Richmond Street East, Suite 205 Toronto, Ontario Canada M5A 3S5 www.rgco.org (416) 499-9800 University of Waterloo Cooperative Education & Career Services 200 University Ave. W. Waterloo, ON, Canada N2L 3G1 (519) 888-4026 UNICEF (Headquarters) 3 United Nations Plaza New York, NY 10017, United States www.unicef.org (212) 824-6275 Goss Gilroy, Inc. 25 Experts Linda Atkinson Executive Director NS Youth Secretariat Nelson Place 5675 Spring Garden Road Halifax, Nova Scotia B3J 1H1 http://youth.ednet.ns.ca/participation/adult.htm (902) 424-4304 (Communications) Katherine A H Graham Dean Carlton University Faculty of Public Affairs D391 Loeb Bldg 1125 Colonel By Drive Ottawa, ON K1S 5B6 www.carleton.ca/fpa email@example.com (613) 520-3741 David Haire, VP of Best Practices and Lean Management Canadian Manufacturers and Exporters (CME), Atlantic Region 1st Floor, Parsons Building 90 O'Leary Avenue St. John's NL A1B 2C7 www.cme-mec.ca firstname.lastname@example.org (709) 772-3682, ext 7 Patricia Inman Ed.D. Adjunct Lecturer School of Professional Studies Aurora University 350 Constance Blvd. PO Box 210 Williams Bay, WI 53191-0210 www.aurora.edu email@example.com [home number only] John Perry National Quality Institute (NQI) 2275 Lake Shore Blvd. West, Suite 307 Toronto, Ontario M8V 3Y3 www.nqi.com firstname.lastname@example.org (416) 251-7600 1 (800) 263-9648 Goss Gilroy, Inc. 26 Lena Salach MA; Jessica Rogers Director of Knowledge Support Services; Director, Evidence and Guidelines Centre for Effective Practice Dept of Family and Community Medicine University of Toronto 263 McCaul Street, 3rd Fl Toronto ON M5T 1W7 www.effectivepractice.org email@example.com (416) 978-4781 Malcolm Shookner Manager Govt of NS Community Accounts Provincial Bldg, 6th floor 1723 Hollis Street PO Box 187 Halifax NS BJ3 2N3 www.gov.ns.ca/finance/community_accounts firstname.lastname@example.org (902) 424-3247 Dan Sinai Director University of Western Ontario Research Development & Services Room 5150, Support Services Bldg. London, ON N6A 3K7 www.uwo.ca/research email@example.com (519) 661-3406, ext 83406 Goss Gilroy, Inc. 27 Appendix D - Bibliography About.com. 2008. Definition of “game theory”. Retrieved January 29, 2009. http://economics.about.com/library/glossary/bldef-decision-rule.htm) Business Dictionary. 2008. Definition of “best practice”. Retrieved November 28, 2008. (http://www.businessdictionary.com/definition/best-practice.html) Chapman, L. S. 2004. Expert opinions on “best practices” in worksite health promotion. American Journal of Health Promotion, 18(6), 1-6. City of Kamloops. 2008. Our gold medal journey with the National Quality Institute. Retrieved November 27, 2008. (www.kamloops.ca) Dictionary.com. 2009. Defintion of “criteria”. Retrieved February 3, 2009. (http://dictionary.reference.com/browse/criteria) Duignan, Paul. 2008. Using visual outcomes models to communicate best practice and get it implemented on the ground: A topic article within the outcomes theory knowledge base. Retrieved November 27, 2008. (http://knol.google.com/k/paul-duignan-phd/using-visual-outcomes-models-to/2m7zd68aaz774/34#) Green, Peter. 2002. Evaluating the uses and abuses of good examples in environmental politics. Paper presented at the 2002 EES Conference: Three movements in contemporary evaluation: Learning, theory and evidence. Kanter, Rosabeth Moss. 2003. Introduction: Getting the best from best practices. From “Best practice: Ideas and insight from the world’s foremost business thinkers”. Cambridge, MA: Perseus Publishing. Laufman, Larry. (2008). Personal electronic communication dated November 27, 2008 (as a response to the consultant’s posting to the American Evaluation Association’s EvalTalk listserve) Naegele, G., and A. Walker. (2006). A guide to good practice in age management. Retrieved November 17, 2008. (http://www.eurofound.europa.eu/pubdocs/2005/137/en/1/ef05137en.pdf) National Quality Institute. 2007. BPIR core concepts. Retrieved September 4, 2008. (http://www.nqi.bpir.com/benchmarking-what-is-best-practice-bpir.com.html) Patton, Michael Quinn. 2001. Evaluation, knowledge management, best practices, and high quality lessons learned. American Journal of Evaluation, Vol. 22, No. 3. Performance Management Network Inc. 1999. Review of Canadian best practices in risk management. Retrieved November 27, 2008. (http://www.tbs-sct.gc.ca/pubs_pol/dcgpubs/RiskManagement/rm-rcbp-PR- eng.asp?printable=True) Richard, G.V. (2005). International best practices in career development: Review of the literature. International Journal for education and Vocational Guidance, Vol. 5, pp. 189-201. The U.S. Equal Employment Opportunity Commission. 1997. Best practices of private sector employers. Retrieved October 26, 2008. (http://www.eeoc.gov/abouteeoc/task_reports/practice.html) United States General Accounting Office. 1995. Best practices methodology: A new approach for improving government operations. Retrieved November 27, 2008. (http://www.gao.gov/archive/1995/ns95154.pdf) Goss Gilroy, Inc. 28