COMMUNITY INCLUSION
REPORT ON EVALUATION SUPPORT PROVIDED TO NEWFOUNDLAND & LABRADOR ASSOCIATION FOR COMMUNITY LIVING
October, 2007
DRAFT
SUBMITTED BY
Community Inclusion Provincial-Territorial Evaluation Update Organization: Newfoundland & Labrador Association for Community Living Sector Focus: Inclusive Education Introduction In 2006, Provincial Associations for Community Living and People First of Canada submitted proposals for continued federal funding from the Office of Disability Issues (ODI). This funding has been designated for the development and implementation of systemslevel interventions designed to enhance the inclusion of people with intellectual disabilities and their families in all areas of social and economic life in Canada. Using a common national framework for community inclusion, the Centre for Community Based Research1 (CCBR) has been working with the provincial/territorial associations to develop and document their approaches to systems change with the following goals in mind: To support the collection information that will help P/Ts describe and improve how their interventions are designed to impact systems; this includes facilitating the creation of program logic models, and helping PTs to identify and gather a variety of data and information about their work and its impact. To be able to consolidate CI evaluation information into a set of learnings about enhancing community inclusion in a systemic way, within and across priority sectors. To establish a balance between local flexibility in designing interventions and the need to make summary statements about effective systems-level interventions across the projects. To enhance accountability and learning of CI initiatives by promoting effective and reflexive use and mobilization of new knowledge gained from projects learnings.
The Purpose of our Evaluation Meetings
1
Formerly the Centre for Research and Education in Human Services.
1
In the summer of 2007, CCBR traveled to all the provinces and territories to meet with staff of Associations for Community Living. The purpose of these meetings was to work with staff to develop a “systems-level logic model” in a selected sector of community inclusion work. This involved creating a diagram of what project activities were implemented or planned, what information was or could be collected to track these activities, and the impacts expected to follow. This included a discussion of evaluation needs, hopes, concerns, and expectations. With a logic model beginning to form, we began to examine the “theory of change” of the initiative – a technical term that really describes the commonsense (but often overlooked) notion that there needs to be reasonable connections between “what is done” and the desired impact. This required consistently asking the question, “why do we think A will lead to B?” while also figuring what the evaluation priorities should be. We ended each day with some ideas on assessing and measuring the key activities and outcomes. Explanation of this document The purpose of this document is to summarize the work completed at these meetings. It contains a brief description of the work of NLACL in the area of inclusive education, a description of the theories of change that underlie this work, and commentary on lessons learned in advancing systems change locally and, by extension, nationally. The appendix contains the logic model which describes in detail the project activities and how they build on one another in a way that leads to desired outcomes. The logic model contains useful types of data to collect in order to assess project implementation. This document also contains recommended measurement priorities for tracking and telling the “story of system change” in the NLACL context. We conclude with some discussion of future opportunities for system change that extend beyond the current project. This document also serves the purpose of helping to summarize the community inclusion initiative nationally, by comparing, contrasting and synthesizing the work from other jurisdictions in inclusive education and community inclusion more generally. This summary task will be presented in an additional national document and/or presentation to follow separately.
2
Synopsis of NLACL’s work in Inclusive education. NLACL has been engaged in two major components within their inclusive education initiative: 1) a comprehensive policy development phase; and 2) a subsequent policy engagement phase. These two phases each contain a sequence of specific activities that build on one another towards outcomes that are supportive of inclusive education best practices and policy. This information is displayed in the logic model in Appendix A 1) Developing policy and best practices on inclusive education in NL In this development phase, NLACL created a cross-stakeholder “Inclusive Education Working Group” to advise on the development of regional focus groups to be conducted with parents and educators. In parallel, regional partnership committees (composed of educational itinerants, family networks, ACLs, etc.) were created to facilitate community engagement with the issues and to ensure focus group participation. The focus groups were conducted in order to gather local perspectives on IE policy and practice (current practice, experiences, gaps, strategies, etc.). Based on the focus group information, a policy position paper (a “vision of IE”) was prepared and circulated to NLACL’s networks. An additional document, a “national review” of IE policy, complemented the provincial policy document, and was similarly distributed to NLACL’s networks. Two other activities were included in the policy development phase. First, Community Living Ontario’s “Inclusive School Culture” program was delivered to one school board. Second, inclusive education itinerants prepared reports on best practices in IE in their jurisdictions Key outcomes: Most of the outcomes – actual benefits experienced by people and communities – follow from the policy engagement phase (see #2 below). Outcomes in this development phase include increasing the knowledge of focus group participants regarding IE principles, policies and practices and to increase participation and expansion of the Inclusive School Cultures Initiative introduced by Community Living Ontario. 2) Policy Engagement – Community & government forums on IE
3
This engagement phase was made possible by the groundwork developed in development phase. NLACL held a cross-stakeholder forum that included parents, educators, the Department of Education and other government offices, the teachers union, university faculty of education, student supports specialists, education itinerants, and others. This forum centred on the policy papers and reports prepared in the development phase. The purpose of the forum was to bring recommended approaches to IE to individuals and organizations that control policy and practice, and to gain consensus of next steps in improving the system. Alongside the forum, NLACL held meetings and presented their findings to Ministry of Education representatives. More broadly, NLACL is electronically distributing special education and policy news and hosting an “electronic learning community” on IE topics and issues. Key outcomes: To increase the knowledge & awareness of IE principles and practices among forum participants, in turn leading to greater engagement and information sharing beyond the forum to a broad range of stakeholders. In reference to specific policy-makers, greater engagement & action by decision-makers on special education related policy and following this, creation & revision of policies that support IE. Broad dissemination of policy news and issues leads to Increased general awareness of IE issues and increased participation & sharing in the electronic learning community
4
Context of the Work, Theories of Change, and Lessons Learned
The community inclusion initiatives, while varying locally in terms of focus and complexity, are all linked to a national framework of community inclusion. This framework was developed previously by CCBR and details the six priority sector areas, common mechanisms for making systems change, and the types of changes that can be expected to follow in the shorter-term (e.g., new resources, policies, partnerships, etc.) and the longer-term (e.g., greater knowledge, awareness, skills, participation, changes to practice and behaviours), all of which contribute to more inclusive environments. Discussion of context and theory of change will draw on this framework, especially in relation to the common mechanisms for systems change in the framework. Common Mechanisms for Systems Change: • Public Awareness & Dialogue (e.g. community forums, plain language) Community Capacity Building (e.g. developing partnerships, leadership, training, strengthening PF chapters) Research & Information (e.g., research studies) Knowledge Networking (e.g., conferences) Policy Development (e.g. policy position papers)
•
• • •
Policy engagement and change is a crucial element of system change, and yet impacting policy is particularly challenging because it there are potentially many factors that are simply out of the direct control of those doing the work. That said, there are a number of different components of policy development and engagement that, when combined together, have a better chance at influencing the policy and practice. NLACL has promoted and pursued a model of policy engagement that is based on a promising theory of change that lays out mutually reinforcing steps designed to lead to impact. Experiences of many social service and advocacy organizations demonstrate that smaller, circumscribed policy activities will often fail to influence the broader policy arena. For example, influence is diminished when: 1. Policy research and/or reviews are conducted without proper representation of local context. 2. Policy development proceeds without the input or endorsement of diverse stakeholders. 3. Policy recommendations are linked to single (special interest) groups and/or organizations. 4. Forums for discussion of implications of policy to inform practice do not follow.
5
5. Policy documents or positions are produced but do not reach policy makers or other key stakeholders that hold positions of power. The next section will outline the how the approach – or theory of change – of NLACL addresses these issues. Policy Development: From the bottom up NLACL wished to conduct regional focus groups to inform policy recommendations regarding a vision of IE in the province. This constitutes a research and information mechanism for systems change as a way to contribute to policy development. Gathering locally grounded information enhances the credibility of resulting policy recommendations by increasing their relevance to the NL context. This is an improvement on policy documents that may only generalize findings and ideas from the broader policy literature and whose recommendations come from an organization as opposed to the community. NLACL also recognized that the weight accorded to the focus group findings depended heavily on a credible research process itself. Two processes were put in place: to ensure effective coverage of the policy/practice issues, a working group composed of experts and practitioners was assembled to guide the research content and design. to ensure meaningful community participation and representation, regional partnership committees were assembled to help set the focus group agenda and to promote community engagement.
These two processes are straightforward (albeit time consuming) safeguards to a credible research process. Without the working group, key issues could be missed and/or the research dismissed as not credible. Without the regional partnerships, awareness and access may have suffered, leading to low participation. All this work was in service of producing a comprehensive, locally relevant, and credible policy document. It should be emphasized that a research process of this type – one that includes a range of relevant stakeholders – is also a form of “knowledge mobilization” that functions to engage key people in the issues at hand. Active participation and awareness the policy research helps
6
secure interest and participation in the engagement phase. This helps to give the policy document and the recommendations therein a central function in future discussions, rather than the document (as is often the case) being an end in itself. Policy Engagement: Mobilizing the research results Having laid the groundwork with the creation of policy document and vision of Inclusion Education, NLACL hosted a forum on Inclusive Education. Forums represent an opportunity for public awareness and dialogue and, if actions are taken as result of dialogue, community capacity building. Forums are quite common but vary in their effectiveness, if effectiveness is taken to mean that the participants and their organizations use the information presented and discussed at the forum strategically to improve policy and practice. They will tend to be more effective if they: are well-attended. are part of a larger, recognized initiative or process. provide information in advance and/or content and issues to react to. represent the diversity of stakeholder groups. promote active participation of attendees. provide information that is highly relevant to current policy and practice issues. focus on consensus building and collaborative solutions.
It appears that NLACL produced a forum that satisfied most, if not all, these criteria. However, a range of evaluation information (see sections following) is necessary to better understand how the forum met these criteria. This can be assessed in relation to the expected outcomes of the forum, which centre on raising knowledge and awareness, and increasing engagement and information sharing among key stakeholders. NLACL was also able to gain audience with government policy makers. Best practices in gaining such an audience are difficult to specify. Across similar organizations, much of the success in this regard is based on a history of long-term relationship building between organization and government. It is also worth mentioning that these relationships are easier to develop in smaller communities, like those in NL, where there is much more regular interaction with government members within service sectors.
7
Political relationships aside, the ability to promote a policy and best practice message is enhanced when it is built and endorsed by diverse and locally relevant stakeholder group, when the messenger is viewed as a leader and expert, and when there is visible community mobilization around the issue. NLACL seems to have established this position over the course of its policy development and engagement work. The desired outcomes, of course, are related to meaningful policy and practice changes that are consistent with recommendations. NLACL will need to collect evaluation information to understand how the policy information is used and where it leads, with the caveat that after all this work, the direction policy takes may also be influenced by a range of other factors outside NLACL’s control. RECOMMENDATIONS FOR EVALUATION DATA COLLECTION The logic model in Appendix A shows the flow of activities of the NLACL projects and the expected outcomes that presume to follow. The diagram represents a sequence of fairly distinct steps, and each step contains associated information that has, can, or will be collected to document the project over time. Documenting this overall process (“implementation steps”) is an important component of evaluation since it helps to understand if the intended theory of change played out in actual practice. Understanding outcomes is greatly improved by understanding how project components were implemented. The amount and types of information that system-focused projects could collect is potentially enormous. It is also the case that collecting large amounts of data is not recommended; data is useful only insofar that it is used. It is recognized that certain accountability demands often require certain reporting independent of actual use. Each major activity in the logic model contains a number of data options that may be useful for these purposes. Presently, however, we are providing a more limited set of data collection options that we recommend as strategic ways to assess your theory of change. There are two main types of relevant data and information: data that measures activities (“what was done”) and outcomes (“what was the impact”).
8
Tracking the activities and measuring outcomes for inclusive education “Policy and Best Practices – Development Phase” Lots of data collection ideas are listed in the logic model and can contribute to a better understanding of implementation. However, all data collection (if not already completed) will be retrospective and therefore places limitations on what could and should be collected. In order to provide data that “helps tell the story”, we recommend the collecting or compiling the following priority information, if it is accessible: # meetings and members of IE working group # of focus groups, # of attendees for each focus group reflections of regional partnership committee members on barriers to getting participation and “what they would do differently” participant feedback on focus group process & content (this can also help evaluate outcome #1, “increased knowledge of focus group participants…” – see logic model) # of copies of the policy document & national review distributed (sent out, downloaded, requested, etc.).
Community Living Ontario’s “Inclusive Schools Cultures” is being evaluated in relation to their own logic model. However, it may be helpful to collect local data on:
2. Greater participation of other boards in ISC orientation 3. Implementation of ISC; expansion of Labrador
# of boards expressing interest in or participating in ISCs orientation. # of boards/schools implementing the ISCs program
9
Tracking the activities and measuring outcomes for inclusive education “Policy and Best Practices – Development Phase”
Provincial forum on Inclusive Education Who: NLACL, parents, educators, boards, DofE, HRLE ( & Min), Health Comm serivces, NLTA, Fac of Ed., Educ. students, student support specialists, Itinerants, CLO (tot. 38), When: Mar. 28-29, 2007. Actions: Present history of & current policy, strategies for IE, presentations. Data: # & description of attendees, indicators of future engagement & capacity (e.g., request), resources distributed, forum feedback
NLACL’s planning leading up to forum represents some promising practices in attempting to influence inclusive education policy and practice. Implementation data should be utilized in order to ensure that the forum proceeded as planned. This may include the following: # of attendees & organizational affiliation (who was not there that should have been?) satisfaction data and feedback from forum participants resources distributed during forum proceedings report of the forum (basic description of itinerary, discussions, etc.)
1. Increased knowledge / awareness of IE principles & practice among participants
The outcomes of the forum (#1-3, at left) can be assessed in a follow up to forum participants. This requires a contact list and/or the use of the listserv. If a contact list is available a brief follow up survey could be administered by phone, email, or via the web covering the following:
2. Greater engagement in IE issues & practices
reports of new knowledge & information gained at forum. participants’ use of information after the forum. connections made with others by attending the forum sharing or intent share information they acquired. reports of changes to practice, individually and/or organizationally.
3. Sharing of information from forum to others
10
Another useful outcome indicator is the number and type of requests for more information, resources, support, and/or partnerships with NLACL.
Engage w/ Dept of Education and Boards re: policy development & implementation
Who: NLACL, Min. of Ed., ADM When: Upon policy paper completion & after IE forum. When: Ongoing. Actions: presentation dialogue w/ minister, ADM. Link issue to FPT group. Data: record of meeting discussion, policy opportunities, barriers, next steps.
After the forum, NLACL planned to engage directly with policy-makers, such as the Department of Education and school boards, through presentations of the policy development work and the forum proceedings. Records of this engagement should be tracked in order better understand impact (i.e., outcomes 4 and 5 at left). This may include: # of meetings, and who was present. presentation (or synopsis of presentation) that was made to policy makers in meeting(s). record of meeting discussions: list of options & opportunities, barriers, next steps and/or commitments coming out of meeting. new linkages made via meetings (e.g., FPT group on IE).
4. Greater engagement & action by decisionmakers on special education related policy
The outcomes themselves can be assessed by examining actual actions of government or school boards. This could include: Tracking & recording policy changes & initiatives; assessing changes against recommendations and/or next steps recorded in forum or meetings.
5. Creation & revision of policies that support IE
This could be accomplished in a number of ways. However, a policy briefing that responds to and assesses the fidelity of any new policies in reference to the recommendations serves to assess these outcomes while also creating a vehicle for awareness-raising. The final piece in NLACL’s Inclusive Education project is the public dissemination of policy news and initiatives and the establishment of an “electronic learning community”. NLACL can document basic outputs, such as: # of newsletters distributed
11
# of website hits and downloads of online material (if technically feasible) listserv membership and activity (topics discussed, # of postings) Measuring general awareness as a function of newsletters and other dissemination strategies should be modest and fairly simple. General awareness is often difficult to measure rigorously, simply because the “exposure” to the material is difficult to control (e.g., many people cannot take the time to read all they receive in this medium). Adding a few questions about the “usefulness” of a newsletter or an open-ended request for comments and feedback may be sufficient.
6. Increased general awareness of IE issues
7. Increased participation & sharing in electronic learning community
The listserv on the other hand is potentially a much more active medium for people interested in gaining policy and practice information on inclusive education. In large part, the “participation” outcome (#7) can be assessed by examining the listserv activity as mentioned above. In addition, since NLACL can moderate the content, the listserv can become an excellent vehicle for posing questions to the community. For example, new policies can be posted to the listserv for feedback and input from diverse stakeholders, which can aid NLACL in developing responses, briefings, etc. Future Directions for NLACL and Inclusive Education As with most community inclusion projects, NLACL is pursuing sustainable, long-term practices to enhance inclusion, as they have done for many years already. Given the theory of change represented in the logic model, there a number of places NLACL will be focusing on in the future. These include potential partnerships and training initiatives that are expected to form as a function of the policy development and engagement activities in the current funding period. They are represented in the logic model as longer-term enhancements to NLACL’s capacity in the inclusive education sector. Future initiatives of these types are reflective of system-change principles. In other words, systems-change requires ongoing capacity building over a significant period of time, which is built upon by the successful short-term gains in new resources, collaborations and knowledge sharing, and policy advocacy.
12