A Multi-level School Counseling Outcome Evaluation Model
by the National Leadership Cadre (NLC) October 2007
This paper presents a multi-level evaluation model that state departments of education and school districts can use to assist schools in evaluating the impact of their school counseling programs on educational outcomes. The National Leadership Cadre, an organization of nine states supporting school counseling reform with a focus on career development, calls for rigorous multi-level evaluations of school counseling programs as an accountability measure to improve programs and better support students’ educational achievement, post-secondary transition and overall life-career development. Although significant progress has been made at the state and local levels in measuring the outcomes of classroom-based instruction, little attention has been paid in measuring the impact of student support services, in particular school counseling programs. Since considerable resources are expended to provide such support services, it is time to develop a state-level outcome evaluation system to document results that can guide the development of educational policies, guidelines and regulations regarding school counseling programs. The good news is that there is a growing body of evidence and data to support this direction. Recent evaluation results from a statewide study in Missouri demonstrated the impact of secondary level school counseling programs on academic achievement and post-secondary retention. Researchers from the University of Missouri found that 47% of students graduating from high schools with more fully implemented comprehensive school counseling programs (including schools with high minority enrollment) were still enrolled in a four-year college one year after high school graduation as compared to 28% of students graduating from schools with similar enrollment patterns that were not implementing a comprehensive program (Lapan, Gysbers, Kayson, 2006).
A Conceptual Framework Based upon the evaluation plan used in Missouri, the NLC has developed the following outcome evaluation model for states to use to improve their school counseling programs. Figure 1 below depicts a multi-level evaluation model that departments of education and districts can use to determine if and how school counseling programs are impacting educational outcomes. This model uses state level data to assess whether school counseling practices result in enhanced student outcomes. Because most state departments of education already collect student, school, and district-level data, state educational leaders need only to organize and analyze existing data in relation to school counseling program implementation. This multi-level model provides a framework for effectively selecting and organizing state and district-level outcome data. The outcome evaluation model requires selection and organization of four data elements: (a) school descriptive data, (b) school counseling program descriptive data, (c) proximal student outcome data, and (d) distal program outcome data. School descriptive data capture the features of the school that are likely to influence school counseling program outcomes (e.g., percentage of low-income students, per pupil expenditures, school setting). School counseling program descriptive data include features of the school counseling program that influence outcomes. These include program processes, structures, and tools (e.g., mission statements, advisory councils, action plans); program characteristics (e.g., staffing ratios, staffing plans, counselor expertise); and program interventions, (e.g., career/educational planning, guidance curriculum, and responsive services). Proximal student outcome data reflect changes in students’ skills, knowledge, attitudes, and behavior that result from interventions and which are eventually reflected (ideally) in improved school-level data. Distal program outcome data are long range school-level data that result from the interventions implemented by an effective school counseling program.
Figure 1 – A Multi-Level Evaluation Model
Sample School Descriptive Data
percentage of low-income students per pupil expenditures school setting (urban, suburban, rural)
Sample School Counseling Program Descriptive Data
Processes, Structures, Tools
staffing ratios staffing plans counselor expertise
career plans guidance curriculum: e.g., Student Success Skills structured groups
mission statement advisory councils action plans
Sample Proximal Student Outcome Data
(Career, Academic, Personal/Social)
test-taking skills college search skills career planning ability understanding the world of work stronger decision-making skills engagement in school self-efficacy skills
Sample Distal Program Outcome Data
GPA (grade point averages) student achievement test scores number of students passing AP courses graduation rates (secondary and post-secondary) college placement data reduction of remediation rates disciplinary suspensions improved school climate substance abuse rates
Explicit in this multi-level evaluation model are the concepts of evidence-based practice and proximal and distal outcomes, terminologies that are relatively new to the school counseling profession. Evidence-based practice can be defined as the intentional use of the best available evidence in planning and implementing school counseling interventions and programs (Dimmitt, Carey, & Hatch, 2007). These programs and interventions must be evaluated within a framework that connects them to both proximal and distal outcomes. Proximal outcomes are the short range outcomes that school counselors seek to change immediately. These might reflect the consequences of students internalizing and/or applying what they have learned and will be noted in desirable changes in school behavior (e.g., being more engaged in school or possessing career identity). Distal outcomes are the long-range outcomes that school counselors seek to improve over time (e.g., GPAs, SAT scores, college graduation rates). Ideally, interventions should always be linked to both proximal and distal outcomes. For example, if a school counseling department wants to increase their students’ acceptance rates into toptier colleges, their chosen intervention might be a guidance curriculum lesson that highlights the importance of rigorous course-taking as an admissions requirement of such colleges. The proximal outcome might be to assess the increase in the number of students enrolling in Advanced Placement courses; while the distal outcomes would be the number of students who complete the courses, their exam scores, and how many apply and are admitted to top-tier colleges. As a general guideline, school counselors should seek to evaluate the relationship of a specific intervention to proximal outcomes that are likely to improve a target distal outcome (Carey, 2007). School descriptive data reflect aspects of the school that affect school counseling program outcomes. Some data elements may pertain only to the school counseling program (e.g., level of implementation of key school counseling program components) while other data elements (e.g., school size) reflect school characteristics that may affect many other programs as well. It is important to 4
statistically control for factors other than aspects of the school counseling program that affect student outcomes so that the impact of the school counseling program itself can be ascertained. Data elements that measure aspects of student diversity (e.g., the percentage of students on free and reduced lunch) must be analyzed to determine whether the beneficial effects of school counseling programs are larger or smaller in schools serving different student populations. These data enable the type of disaggregation necessary to document which groups of students are profiting most and least by school counseling interventions and activities. Student mobility (e.g., years in that school) is an important data element for assessing the impact of multi-year initiatives (e.g., preventative and developmental interventions). Sink and Stroh (2003) found that the impact of the comprehensive developmental school counseling approach was most pronounced in students who had spent at least three years in their present school. School counseling program descriptive data, while central to this outcome evaluation model, are not likely to exist at the district or state level. To date, state departments of education and school districts have not routinely collected such data. Statewide evaluation of student support services, in particular school counseling programs, requires systematic collection of data that accurately describe key characteristics of such programs and the associated interventions. To this end, Missouri (Lapan et al. 2006) used an online survey to collect data from high school counselors and administrators regarding program implementation in their school building. (It should be noted that while they were able to gather data in this way and use it to promote important changes, the collection process was not integrated into the state’s processes for collecting data from schools.) In contrast, Rhode Island developed and integrated several items describing the characteristics of school counseling programs into their state’s annual data collection instruments and process. The length of the existing process limited the number of items school counseling departments could include, however the school counseling program data deemed most important are now collected annually. While neither Missouri’s nor Rhode Island’s evaluation model is yet the ideal, states 5
should seek to collect descriptive data on all key characteristics of school counseling programs as part of their annual data collection system and process. Proximal student outcome data reflect the intended changes in student’s knowledge, attitudes and school-related behavior that result from evidence-based school counseling interventions. These changes (e.g. the level of student engagement) ultimately result in improvements in distal program outcome data elements (e.g. improved achievement test scores) which is why they are so important. School counselors use Proximal Student Outcome Data to gauge the effectiveness of their interventions and to decide whether to continue, modify, or discontinue specific interventions. However, because these data are not routinely collected or reviewed at the district and state level, it is important that school administrators encourage and support counselors’ efforts in continuously evaluating their interventions for proximal outcomes and in documenting and sharing their results. A number of state departments of education (e.g., Florida, Missouri, Utah) are promoting the use of evaluation in tracking the proximal outcomes of their school counseling programs. Distal program outcome data elements measure the potential effects of a school counseling programs on school-level outcome data. For example, schools with more elaborated career planning components may have higher attendance rates, higher rates of college attendance, and higher average achievement test scores. Much of these data are already collected and maintained by state departments of education while some (e.g., student attendance data) are collected and maintained by state departments of education at the aggregate level but not at the individual student level. Other data (e.g., college success as reflected by freshman year college GPA) are rarely collected and maintained by state departments of education, although a few states (e.g., Florida, Illinois, Massachusetts) are moving in that direction. Since data collection and warehousing is an expensive process, each state must first decide which outcome measures would be most useful to collect and maintain at the individual student level in order to gauge the effectiveness of school counseling programs. At present, the data 6
elements that reflect long-term outcomes for students (e.g., college retention and remediation) may provide the most useful information but they are the least likely data to be collected and analyzed. While much can be learned from evaluations based on data that are already routinely collected, an ideal state outcome evaluation system for school counseling programs must capture data on important long-term outcomes as well. Collecting Data All four types of data discussed above can be collected with modifications to existing processes over time. As stated earlier, most of the school descriptive data are already collected by state departments of education. School counseling program descriptive data, which are not generally collected, can be gathered if states develop a process for systemic collection of relevant school counseling data. This is best accomplished by expanding the state’s existing systems that are already used to collect descriptive data from schools (i.e., Rhode Island’s approach); however, periodic collection of such data by surveys (i.e., Missouri’s approach) can also yield the necessary data. State and district administrators can best accomplish the collection of proximal student outcome data by both promoting and supporting the use of evidence-based practice in their school counseling programs. Finally, many districts and states already have distal outcome data (e.g., middle school transition data, ninth grade post-secondary aspirations, PSAT participation) and some state departments of education are collecting post-secondary outcome data to measure student success after high school (e.g., first-year college performance). Analyzing Results After these four types of data are successfully collected, they must be combined and analyzed to determine the impact of school counseling programs on a variety of pre-identified measures. This process will require data analyses that include school counseling program descriptive data as independent variables, school descriptive data measures as covariates, and distal program outcome data as dependent variables. (Proximal outcome measures are typically used locally by 7
schools to monitor the impact of school counseling interventions and are generally not included in state-level evaluations.) This approach was used in Missouri (Lapan et al. 2006) in which the impact of the level of implementation of a comprehensive developmental school counseling program on students 10th grade achievement test scores was studied. After equating the high schools statistically for a range of factors known to affect achievement test scores (e.g., differences in student funding, teacher-to-student ratios, and number of students receiving free and reduced lunch), the level of implementation of a comprehensive developmental school counseling program was found to contribute significantly to students’ 10th grade mathematic scores. By collecting, combining, and analyzing school descriptive data, school counseling program descriptive data, and distal program outcome data, state departments of education can (a) evaluate the impact of specific aspects of the school counseling program, (b) evaluate the impact of mandated or recommended school counseling activities, (c) create recommendations for state educational policymakers, (d) generate useful information about the value of school counseling programs and activities for its school boards, superintendents and principals, and (e) identify areas needing change and improvement. If school counselors are also collecting proximal student outcome data to evaluate the impact of individual school counseling interventions, they will be able to analyze the results of state and district evaluations to identify the impact of these interventions on the distal outcomes measured. Summary and Recommendations The National Leadership Cadre strongly recommends that every state and school district develop and adopt a statewide system to evaluate the impact of school counseling programs and to generate information needed for program improvement. Much of the data needed for this evaluation are already routinely collected. Other data can easily be obtained when states develop a process for the regular collection of information regarding critical features of each school counseling program. Rhode Island’s method of building such data 8
collection into the regular yearly process is an exemplary practice. School counselors can also contribute to this data-gathering process when they evaluate their interventions and programs. While much can be learned from data that currently exist in state databases, the systematic collection of student-level outcome data that relate to overall life-career development and measures post-high school performance and achievement (e.g., work performance, college performance) would greatly enhance the evaluation of the long-term impact of school counseling programs. Similarly, the ability to routinely access key student-level data that are typically maintained in districts rather than in state databases (e.g., individual student’s college application data) would greatly enhance the evaluation of the long-term impacts of school counseling programs. Given the costs associated with data collection and warehousing, each state should decide on a finite number of data elements that are necessary for measuring the impact of school counseling programs. Sound state educational policy should be based on solid, reliable information about the relationships between school counseling program characteristics and practices and student outcomes. This information is needed by state policy makers to inform their decision-making process and by practitioners to improve their practice. With modest additional investments, state departments of education can implement statewide outcome evaluation systems that can point the direction towards sound policy and practice. The NLC strongly encourages all states to move in this direction. For additional information please see http://www.umass.edu/schoolcounseling/NLC/.