Year 1 External Evaluation Report

GUILFORD COUNTY SCHOOLS MISSION POSSIBLE PROGRAM Year 1 (2006-07) External Evaluation Report Prepared by: SERVE Center at UNCG Gateway University Research Park - Dixon Building 5900 Summit Avenue Browns Summit, NC 27214 (800) 755-3277 Contact: Holli Gottschall Bayonas, Ph.D. Project Manager hgottsch@serve.org 336-315-7438 Submitted to: Guilford County Schools Greensboro, NC April 2, 2008 TABLE OF CONTENTS Executive Summary ......................................................................................................................... i Introduction..................................................................................................................................... 1 Background Information about the Mission Possible Program ...................................................... 1 Background Information about SERVE Center.............................................................................. 2 Evaluation Plan ............................................................................................................................... 2 Key Personnel ................................................................................................................................. 3 Results............................................................................................................................................. 3 Principal Turnover ...................................................................................................................... 3 Teacher Turnover........................................................................................................................ 4 Percentage of Teachers with Initial or Continuing Licenses ...................................................... 7 Percentage of Classes Taught by Highly Qualified Teachers..................................................... 8 Annual Yearly Progress (AYP) Goals Obtained ........................................................................ 9 Percentage of Students who Passed the ABCs ......................................................................... 10 ABC Growth Met...................................................................................................................... 11 Performance Composite............................................................................................................ 12 Student Short Term Suspensions .............................................................................................. 13 Student Attendance ................................................................................................................... 14 Cohort Graduation Rate ............................................................................................................ 15 Climate Surveys ........................................................................................................................ 15 Teacher Working Conditions Survey........................................................................................ 17 Professional Development Workshops ..................................................................................... 18 Performance Incentives and Teacher Value-Added Averages ................................................. 19 Comparison of Mission Possible Schools with Non Mission Possible Schools....................... 20 Conclusion and Recommendations............................................................................................... 25 Appendix A - Mission Possible Evaluation Plan.......................................................................... 27 Appendix B - Letter to Parents Regarding Climate Survey.......................................................... 30 Appendix C - Letter to Parents Regarding Climate Survey (Spanish version)............................. 31 Appendix D - Parent/Student Consent Form ................................................................................ 32 Appendix E - Parent/Student Consent Form (Spanish version).................................................... 34 Appendix F - Parent Climate Survey ............................................................................................ 36 Appendix G - Parent Climate Survey (Spanish Version) ............................................................. 38 Appendix H - Student Climate Survey ......................................................................................... 40 Appendix I - Student Assent Form ............................................................................................... 42 Appendix J - Directions for Distributing Parent Surveys and Suggested Sampling Plan ............ 43 Appendix K - List of Coordinators for Fall 2007 Climate Survey Distribution........................... 45 Appendix L - Staff Climate Summary Results - Fall 2007........................................................... 46 Appendix M - Parent Climate Summary Results - Fall 2007 ....................................................... 49 Appendix N - Student Climate Summary Results - Fall 2007...................................................... 55 Appendix O - Student Achievement-Analytical Approach .......................................................... 58 GUILFORD COUNTY SCHOOLS MISSION POSSIBLE PROGRAM YEAR 1 (2006-07) EXTERNAL EVALUATION REPORT Executive Summary Introduction In February 2007, Guilford County Schools (GCS) contracted with SERVE Center at the University of North Carolina at Greensboro to conduct an external evaluation of the Mission Possible (MP) program. The program was in the first year of implementation when a Federal Teacher Incentive Fund (TIF) grant was awarded. The new monies from the TIF grant were allocated to pay for an additional eight schools to participate in Mission Possible and to pay for the external evaluation. The original cohort that began in 2006-07 consisted of 22 schools (20 Mission Possible schools and two Cumulative Effects High Schools 1 . The second cohort of eight schools would begin the program in the 2007-08 school year. This report documents activities that occurred during Year 1 (2006-07) of the MP Program. The majority of data reported in the Year 1 report will serve as a baseline from which to compare in subsequent years. Consequently, with the seven TIF-funded schools beginning in the 2007-08 school year, the Year 1 (2006-07) data will serve as a baseline for 22 schools and Year 2 data will serve as a baseline for the 8 additional schools. With the future addition of Year 2 data, it will be possible to look for trends among the 22 original cohort schools. In addition, with Year 3 data, trends in all 30 schools (2 years of data from the MP schools will be available) can be examined. Background Information about the Mission Possible Program Mission Possible is a comprehensive teacher incentive program. The program provides recruiting and performance incentives to help recruit and retain highly qualified teachers for schools with critical needs. Recruiting incentives range from $2,500 to $10,000 and performance incentives range from $2,500 to $5,000. Receiving the individual performance incentive hinges on the teacher completing 100% of the yearly prescribed professional development activities and earning Value-Added scores 2 that are at least one standard error above the mean. Related, if MP teachers obtain Value-Added scores of at least two standard errors below the mean for two consecutive years, they are transferred out of the MP school. This system of accountability ensures that MP schools are employing the most effective teachers for any given MP school. Teachers in their first year of participation complete workshops on Cooperative Learning and Undoing Racism. Teachers in their second year of participation complete workshops on Differentiated Instruction and TESA. The program theory is such that recruiting and retaining highly qualified teachers will lead to increased student achievement. Cumulative Effects Program is a program to recruit and retain highly qualified math teachers in Guilford County Schools. The program is a collaborative project involving Action Greensboro, the University of North Carolina system, the University of North Carolina at Greensboro, and North Carolina Agriculture and Technical State University. 2 Value-Added scores are the most unbiased way to measure gains in achievement, as opposed to just examining the percentage of students who pass the North Carolina End of Course or End of Grade tests. GCS contracts with SAS® EVAAS® to provide the Value-Added scores. 1 i Background Information about SERVE Center SERVE Center at the University of North Carolina at Greensboro (UNCG) is a university-based research, development, dissemination, evaluation, and technical assistance center. Its mission is to support and promote teaching and learning excellence in the K-12 education community. SERVE Center provides contracted evaluation services to state and local education agencies in and around the southeast region. The Program Evaluation Standards, Second Edition (The Joint Committee on Standards for Educational Evaluation, 1994) and the Guiding Principles for Evaluators (American Evaluation Association, 1995) guide the evaluation work performed at SERVE. Evaluation Plan On November 27, 2007 Eric Becoats and Emily Scott from GCS met with Pamela Finney, Kathleen Mooney, and Melissa Williams from SERVE Center to discuss the external evaluation of MP. Based on those discussions, SERVE created an evaluation plan, timeline, and budget, which was sent to GCS on December 15, 2007. In April 2007, the technical service agreement for the evaluation was signed by GCS and SERVE. The evaluation agreement covers the length of the TIF grant (until September 30, 2011). The evaluation of MP is guided by the main question stated in the original RFP: Are the goals and objectives of the project being met as outlined by the proposal? The goals as outlined in the RFP were: Goal one: Attracting and retaining the most qualified teachers and administrators at the most highly impacted schools. Goal two: Increasing student achievement on EOC and EOG tests. Goal three: Rewarding teachers and administrators who work in participating schools that meet their value-added goals or adequate yearly progress and state ABC goals. Given these goals, SERVE developed an evaluation plan using the following evaluation questions as a guide: • To what extent are the most highly qualified teachers and administrators being recruited and retained at MP Schools? • To what extent were the MP teachers and administrators trained as proposed? • To what extent did the MP training and incentives impact teacher and student outcomes? Data used to answer these questions consist primarily of student discipline referrals, student attendance and promotion rates, comparison of teacher data between MP and non-MP schools, comparison of AYP and ABC data between MP and non-MP schools, documentation of teacher/administrator participation in professional development activities, survey of teacher/administrator satisfaction with professional development activities, compilation of Teacher Working Conditions (TWC) Survey, interviews/focus groups with teachers and administrators regarding MP project activities, Parent/Student/Staff Climate Surveys, and documentation of incentives paid to teachers. The majority of data come from North Carolina Department of Public Instruction (NCDPI), through the website www.ncreportcard.org. The ii complete evaluation plan is available in Appendix A. The evaluation methodology was approved by UNCG’s Institutional Review Board (October 22, 2007, protocol #067328), which ensures the protection of human subjects in evaluation and research activities. Conclusion and Recommendations The NCDPI data for 2004-05 through 2006-07 shows improvements in teacher, principal, and student outcomes. Teacher turnover has decreased by an average of 2% in Cohort 1 schools. Seven schools had turnover that was lower than the district average. The percentage of students passing the ABCs also improved by 2%. In 2006-07, 11 schools showed an increase in AYP goals obtained, with 9 schools making AYP. A summary table of results, which shows the increase or decrease in each data category is shown in Table 1. The data from 2006-07, where comparisons were available from the 2005-06 data, show improvement. Table 1. Summary of Results for Cohort 1-Changes from 2005-06 to 2006-07 Category Principal Turnover Teacher Turnover Incentive Paid Teacher Turnover Percentage of AYP Goals Percentage of Students who Passed ABCs ABC Growth Performance Composite Short Term Suspensions Student Attendance Cohort Graduation Rate Teachers with Initial or Continuing License Classes Taught by Highly Qualified Teachers School Climate Teacher Working Conditions Teacher Value Added Scores Amount of Performance Incentives Increase Decrease Unknown-1st year of Data Average Amount22 schools combined -4.8% -2.0% N/A +6.2% +2.0% +3 schools +1.8% -84 +0.4% +10.4% +4.1% +3% N/A N/A N/A N/A iii The Climate Surveys administered in Fall 2007 show that, while parents and students perceive the majority of school components positively, there is a concern for school violence, gangs, and drugs. School staff also perceived the majority of school components positively, but perceived there to be a lack of parent involvement, student deficiencies in basic skills, low morale among staff, and racist comments from students. Both incentive-paid and non-incentivepaid teachers expressed that all teachers should be compensated in some way. However, the incentive structure is unable to be changed and the budget does not allow for incentives to be extended to all school personnel. Principals should pursue strategies to increase teacher morale within the current incentive structure. The low response rate for Parent and Student Climate Surveys is indicative of low parental involvement. Low parental involvement was also perceived to be low by teachers on the Staff survey. Principals should work on strategies to increase parental involvement. Related, in an effort to increase response rates through the surveying method, SERVE will send both Parent and Student Climate Surveys together and ask that parents return completed surveys directly to SERVE Center with a pre-addressed, postage-paid envelope. iv GUILFORD COUNTY SCHOOLS MISSION POSSIBLE PROGRAM YEAR 1 (2006-07) EXTERNAL EVALUATION REPORT Introduction In February 2007, Guilford County Schools (GCS) contracted with SERVE Center at the University of North Carolina at Greensboro to conduct an external evaluation of the Mission Possible (MP) program. The program was in the first year of implementation when a Federal Teacher Incentive Fund (TIF) grant was awarded. The new monies from the TIF grant were allocated to pay for an additional eight schools to participate in Mission Possible and to pay for the external evaluation. The original cohort that began in 2006-07 consisted of 22 schools (20 Mission Possible schools and two Cumulative Effects High Schools 3 . The second cohort of eight schools would begin the program in the 2007-08 school year. This report documents activities that occurred during Year 1 (2006-07) of the MP Program. The majority of data reported in the Year 1 report will serve as a baseline from which to compare in subsequent years. Consequently, with the eight TIF-funded schools beginning in the 2007-08 school year, the Year 1 (2006-07) data will serve as a baseline for 22 schools and Year 2 data will serve as a baseline for the 8 additional schools. With the future addition of Year 2 data, it will be possible to look for trends among the 22 original cohort schools. In addition, with Year 3 data, trends in all 30 schools (2 years of data from the MP schools will be available) can be examined. Background Information about the Mission Possible Program Mission Possible is a comprehensive teacher incentive program. The program provides recruiting and performance incentives to help recruit and retain highly qualified teachers for schools with critical needs. Recruiting incentives range from $2,500 to $10,000 and performance incentives range from $2,500 to $5,000. Receiving the individual performance incentive hinges on the teacher completing 100% of the yearly prescribed professional development activities and earning Value-Added scores 4 that are at least one standard error above the mean. Related, if MP teachers obtain Value-Added scores of at least two standard errors below the mean for two consecutive years, they are transferred out of the MP school. This system of accountability ensures that MP schools are employing the most effective teachers for any given MP school. Teachers in their first year of participation complete workshops on Cooperative Learning and Undoing Racism. Teachers in their second year of participation complete workshops on Differentiated Instruction and TESA. The program theory is such that recruiting and retaining highly qualified teachers will lead to increased student achievement. Cumulative Effects is a program intended to recruit and retain highly qualified math teachers in Guilford County Schools. The program is a collaborative project involving Action Greensboro, the University of North Carolina system, the University of North Carolina at Greensboro, and North Carolina Agriculture and Technical State University. 4 Value-Added scores are the most unbiased way to measure gains in achievement, as opposed to just examining the percentage of students who pass the North Carolina End of Course or End of Grade tests. GCS contracts with SAS® EVAAS® to provide the Value-Added scores. 3 1 Background Information about SERVE Center SERVE Center at the University of North Carolina at Greensboro (UNCG) is a university-based research, development, dissemination, evaluation, and technical assistance center. Its mission is to support and promote teaching and learning excellence in the K-12 education community. SERVE Center provides contracted evaluation services to state and local education agencies in and around the southeast region. The Program Evaluation Standards, Second Edition (The Joint Committee on Standards for Educational Evaluation, 1994) and the Guiding Principles for Evaluators (American Evaluation Association, 1995) guide the evaluation work performed at SERVE. Evaluation Plan On November 27, 2007 Eric Becoats and Emily Scott from GCS met with Pamela Finney, Kathleen Mooney, and Melissa Williams from SERVE Center to discuss the external evaluation of MP. Based on those discussions, SERVE created an evaluation plan, timeline, and budget, which was sent to GCS on December 15, 2007. In April 2007, the technical service agreement for the evaluation was signed by GCS and SERVE. The evaluation agreement covers the length of the TIF grant (until September 30, 2011). The evaluation of MP is guided by the main question stated in the original RFP: Are the goals and objectives of the project being met as outlined by the proposal? The goals as outlined in the RFP were: Goal one: Attracting and retaining the most qualified teachers and administrators at the most highly impacted schools. Goal two: Increasing student achievement on EOC and EOG tests. Goal three: Rewarding teachers and administrators who work in participating schools that meet their value-added goals or adequate yearly progress and state ABC goals. Given these goals, SERVE developed an evaluation plan using the following evaluation questions as a guide: • To what extent are the most highly qualified teachers and administrators being recruited and retained at MP Schools? • To what extent were the MP teachers and administrators trained as proposed? • To what extent did the MP training and incentives impact teacher and student outcomes? Data used to answer these questions consist primarily of student discipline referrals, student attendance and promotion rates, comparison of teacher data between MP and non-MP schools, comparison of AYP and ABC data between MP and non-MP schools, documentation of teacher/administrator participation in professional development activities, survey of teacher/administrator satisfaction with professional development activities, compilation of Teacher Working Conditions (TWC) Survey, interviews/focus groups with teachers and administrators regarding MP project activities, Parent/Student/Staff Climate Surveys, and documentation of incentives paid to teachers. The majority of data come from North Carolina Department of Public Instruction (NCDPI), through the website www.ncreportcard.org. The 2 complete evaluation plan is available in Appendix A. The evaluation methodology was approved by UNCG’s Institutional Review Board (October 22, 2007, protocol #067328), which ensures the protection of human subjects in evaluation and research activities. Key Personnel The MP program is currently directed by Amy Holcombe at GCS. In Year 1, SERVE Center staff met with the director bi-monthly to discuss the ongoing evaluation activities associated with MP. In addition, numerous correspondences were sent via email on a weekly basis. While there is an established evaluation plan, SERVE engages in participatory evaluation, and thus seeks the input of GCS for all evaluation activities. The additional key personnel for the project are listed in Table 2. Table 2. Key Personnel in 2006-07 Person Amy Holcombe Tim Moose Position Director of Mission Possible Schools (Effective August 2007) Coordinator, Mission Possible Schools (Hired in January 2007 and resignation was effective July 30, 2007) Database Manager (replaced former administrative assistant in Fall 2008) TIF Grant Monitor External Evaluator-Project Director External Evaluator-Project Manager External Evaluator External Evaluator Organization GCS GCS Wynde Brady Jeff Strohl Pamela Finney Holli Bayonas Eric Howard Fain Blackburn GCS Center for Educator Compensation Reform SERVE Center SERVE Center SERVE Center SERVE Center Results The following results are organized by type of data gathered. In most data tables, the results are separated into Cohort 1 and Cohort 2 5 . The results document progress of the goals as they pertain to the MP schools. The results do not include possible unexpected effects in other school in Guilford County, nor does the evaluation attempt to document events qualitatively for each individual MP School. The results focus solely on teacher/principal recruitment and retention, teacher professional development, and student outcomes. Principal Turnover The percentage of principal turnover is calculated by examining the principal on record for each school during each data year (taken from www.ncreportcard.org). If the principal on record changes from one year to the next, it is counted as turnover and is subsequently counted toward the Cohorts overall turnover average. The percent turnover from 2005-06 to 2006-07 in Cohort 1 schools was 19% and will serve as the benchmark for Cohort 1 schools. Turnover for each school is shown in Table 3. 5 Data for Cohort 2 are provided, even though Cohort 2 did not start until the 2007-08 school year. Providing the data in the Year 1 report will allow the reporting format to stay consistent throughout the length of the project. In addition, Academy at Smith did not begin until Fall 2006 and Academy at Central did not begin until January of 2008 so data on those two schools are often not available. 3 Table 3. Principal Turnover School 2004-05 to 2005-06 2005-06 to 2006-07 Cohort 1 Andrews High Yes No Dudley High No No Eastern Guilford High Yes No Fairview Elem No No Ferndale Middle Yes No Foust Elem No Yes Gillespie Elem No No Hairston Middle Yes No Hampton Elem No Yes High Point Central High No No Jackson Middle No No Kirkman Park Elem No No Middle College at Bennett No No Middle College at NC A&T Yes No Oak Hill Elem No No Parkview Elem No Yes Smith High No No Academy at Smith N/A N/A Southern Guilford High No No Washington Elem No No Welborn Middle No No Wiley Elem No Yes Total of Cohort 1 5 (23.8%) 4 (19%) Cohort 2 Academy at Central* N/A N/A Allen Middle Yes No Aycock Middle No Yes Bessemer Elem Yes No Cone Elem No No Falkener Elem No Yes Penn Griffin Middle No No Union Hill Elem No No Total of Cohort 2 2 (28.6%) 2 (28.6%) Note. Data taken from www.ncreportcard.org. Yes=New principal from previous year to current year. No=the principal stayed the same from one year to the next. *Academy at Central did not open until January of 2008. Data will not be available until the next academic year. Teacher Turnover Teacher turnover is a measure of how many teachers resigned and did not return to the school in the following year. Teacher turnover is important because it has been found to disrupt student learning, especially when a teacher leaves in the middle of an academic year. One of the main outcomes sought by providing teachers with MP incentives is the reduction of teacher turnover. Table 4 reflects individual schools’ percentage of teacher turnover. This includes all teachers, not just incentive-paid teachers. The numbers in parentheses represent the increase or decrease in teacher turnover from the previous school year. In 2006-07, 10 of 22 Cohort 1 schools experienced a decrease in teacher turnover compared to turnover in 2005-06. The average teacher turnover rate for Cohort 1 schools in 2006-07 was 35.9%, down 2% from the 4 previous year. The largest decreases (by more than 15%) in teacher turnover were at Fairview Elementary, Washington Elementary, and Wiley Elementary. Seven Cohort 1 schools fell under the 2006-07 district turnover average of 28%. Table 4. Percentage of Teacher Turnover School 2004-05 2005-06 2006-07 Cohort 1 Andrews High 37 40 (+3) 26 (-14) Dudley High 28 33 (+5) 35 (+2) Eastern Guilford High 19 27 (+8) 16 (-11) Fairview Elem 30 29 (-1) 14 (-15) Ferndale Middle 35 40 (+5) 38 (-2) Foust Elem 44 28 (-16) 39 (+11) Gillespie Elem 18 28 (+10) 21 (-7) Hairston Middle 36 35 (-1) 40 (+5) Hampton Elem 47 45 (-2) 47 (+2) High Point Central High 28 32 (+4) 29 (-3) Jackson Middle 24 40 (+16) 49 (+9) Kirkman Park Elem 32 55 (-23) 54 (-1) Middle College at Bennett 38 50 (+12) 56 (+6) Middle College at NC A&T 67 63 (-4) 70 (+7) Oak Hill Elem 25 22 (-3) 29 (+7) Parkview Elem 13 16 (+3) 16 (nc) Smith High 29 24 (-5) 39 (+15) Academy at Smith N/A N/A N/A Southern Guilford High 21 26 (+5) 18 (-8) Washington Elem 57 46 (-11) 25 (-21) Welborn Middle 34 34 (nc) 46 (+12) Wiley Elem 44 82 (+38) 46 (-36) Avg. of Cohort 1 33.6 37.9 35.9 Cohort 2 Academy at Central N/A N/A N/A Allen Middle 31 31 (nc) 27 (-4) Aycock Middle 28 46 (+18) 37 (-9) Bessemer Elem 34 17 (-17) 25 (+8) Cone Elem 33 24 (-9) 28 (+4) Falkener Elem 32 30 (-2) 39 (+9) Penn Griffin Middle 33 36 (+3) 28 (-8) Union Hill Elem 21 15 (-6) 15 (nc) Avg. of Cohort 2 30.3 28.4 28.4 Note. Data taken from www.ncreportcard.org. Academy at Smith opened in 2006-07 and Academy at Central opened in January of 2008 so no previous comparison data were available. In addition to examining overall teacher turnover for an individual school, records were kept by the GCS MP office that show turnover rates specific to the MP incentive-paid teachers. Table 5 shows that the total number of incentive-paid teachers who left their positions in 200607 was 35. Of those, four had low Value-Added scores for two consecutive years. The most common reasons for leaving in 2006-07 were “value-added,” “resigned,” and “retired.” Nine schools had no turnover of incentive-paid teachers. This data will serve as the baseline for Cohort 1. 5 Table 5. Mission Possible Incentive Paid Teacher Turnover School Value Added Cohort 1 Andrews High Dudley High Eastern Guilford High Fairview Elem Falkener Elem Ferndale Middle Foust Elem Gillespie Elem Hairston Middle Hampton Elem High Point Central High Jackson Middle Kirkman Park Elem Middle College at Bennett Middle College at NC A&T Oak Hill Elem Parkview Elem Smith High Academy at Smith Southern Guilford High Washington Elem Welborn Middle Wiley Elem 1 Documented Reason for Leaving in 2006-07 School Year Intervention Resigned Teacher Retired Family Surplus Team Opt Out related 1 1 1 2 Other Total 2 4 0 0 1 1 1 1 1 1 1 2 3 5 1 0 0 0 1 2 1 1 2 3 4 1 5 0 1 1 1 1 1 1 1 3 1 1 0 0 1 1 1 3 0 1 1 6 School Total Cohort 1 Cohort 2 Academy at Central Allen Middle Aycock Middle Bessemer Elem Cone Elem Penn Griffin Middle Union Hill Elem Total Cohort 2 Value Added 4 N/A N/A N/A N/A N/A N/A N/A N/A Documented Reason for Leaving in 2006-07 School Year Intervention Resigned Teacher Retired Family Surplus Team Opt Out related 2 6 1 4 3 4 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A Other 11 N/A N/A N/A N/A N/A N/A N/A N/A Total 35 N/A N/A N/A N/A N/A N/A N/A N/A Percentage of Teachers with Initial or Continuing Licenses Table 6 shows the percentage of teachers in each MP school that have obtained their initial or a continued license presented by school year. In 2006-07, 17 (77.3%) of the 22 Cohort 1 schools reported an increase in the percentage of teachers who held a license since the previous year. The total number of Cohort 1 schools that reported at least 80% of their teachers had licenses during the 2004-05 school year was 11 (52.4%); in 2005-06, 10 schools (47.6%); and in 2006-07, 16 schools (72.7%). The average percentage of teachers in Cohort 1 with an initial or continuing license was 86.3% in 2006-07, up from 82.2% in the previous year Table 6. Percentage of Teachers with Initial or Continuing Licenses School Cohort 1 Andrews High Dudley High Eastern Guilford High Fairview Elem Ferndale Middle Foust Elem Gillespie Elem Hairston Middle Hampton Elem High Point Central High Jackson Middle Kirkman Park Elem Middle College at Bennett Middle College at NC A&T Oak Hill Elem Parkview Elem Smith High Academy at Smith 2004-05 65 72 77 95 82 91 88 72 90 73 74 90 80 88 97 95 71 N/A 62 68 75 91 72 100 93 71 91 76 69 100 78 90 100 95 71 N/A 2005-06 79 73 85 93 76 97 100 86 94 84 77 83 92 90 97 98 76 41 2006-07 7 School 2004-05 2005-06 2006-07 Southern Guilford High 79 85 93 Washington Elem 63 79 96 Welborn Middle 76 79 89 Wiley Elem 86 81 100 Average Cohort 1 81.1 82.2 86.3 Cohort 2 Academy at Central N/A N/A N/A Allen Middle 77 76 83 Aycock Middle 70 76 88 Bessemer Elem 100 92 97 Cone Elem 95 93 93 Falkener Elem 94 98 100 Penn Griffin Middle 72 83 87 Union Hill Elem 91 91 97 Average Cohort 2 85.6 87 92.1 Note. Data taken from www.ncreportcard.org. Academy at Smith opened in 2006-07 and Academy at Central opened in January of 2008 so no previous comparison data were available. Percentage of Classes Taught by Highly Qualified Teachers Table 7 shows the percentage of classes in each MP school that were taught by highly qualified teachers. In 2006-07, 18 (81.8%) Cohort 1 schools reported either an increase or no change in the percentage of classes taught by highly qualified teachers since the previous year. The total number of Cohort 1 schools where 100% of classes were taught by highly qualified teachers was 7 (33.3%) during the 2004-05 school year; 8 (38.1%) in the 2005-06 school year; and 10 (45.5%) in 2006-07, demonstrating a steady increase since 2004-05. Table 7. Percentage of Classes Taught by Highly Qualified Teachers School Cohort 1 Andrews High Dudley High Eastern Guilford High Fairview Elem Ferndale Middle Foust Elem Gillespie Elem Hairston Middle Hampton Elem High Point Central High Jackson Middle Kirkman Park Elem Middle College at Bennett Middle College at NC A&T Oak Hill Elem Parkview Elem Smith High Academy at Smith Southern Guilford High Washington Elem Welborn Middle Wiley Elem 2004-05 86 90 100 100 79 100 85 76 84 93 91 100 86 100 100 83 99 N/A 100 87 88 85 92 91 99 97 90 100 100 97 100 96 93 100 71 100 100 100 97 N/A 100 87 96 88 2005-06 99 98 100 100 97 95 100 97 91 97 100 100 92 100 96 100 99 100 98 100 96 100 2006-07 8 Average Cohort 1 91 95 98 Cohort 2 Academy at Central N/A N/A N/A Allen Middle 92 99 100 Aycock Middle 86 89 97 Bessemer Elem 100 100 100 Cone Elem 100 100 100 Falkener Elem 97 100 100 Penn Griffin Middle 84 91 97 Union Hill Elem 91 96 100 Average Cohort 2 92.9 96.4 99.1% Note. Data taken from www.ncreportcard.org. Academy at Smith opened in 2006-07 and Academy at Central opened in January of 2008 so no previous comparison data were available. Annual Yearly Progress (AYP) Goals Obtained Table 8 shows that the average percentage of Annual Yearly Progress (AYP) goals obtained by each school has increased over the last three years. In 2005-06, 13 (61.9%) Cohort 1 schools showed an increase in the percentage of goals obtained compared to the previous year. In 2006-07, 11 (52.3%) schools showed an increase of goals obtained compared to the previous year. The total number of Cohort 1 schools that met AYP during the 2004-05 school year was 4 (19%); 5 (23.8%) in the 2005-06 school year; and 9 (40.9%) in the 2006-07 school year. Table 8. Percentage of AYP Goals Obtained School Cohort 1 Andrews High Dudley High Eastern Guilford High Fairview Elem Ferndale Middle Foust Elem Gillespie Elem Hairston Middle Hampton Elem High Point Central High Jackson Middle Kirkman Park Elem Middle College at Bennett Middle College at NC A&T Oak Hill Elem Parkview Elem Smith High Academy at Smith Southern Guilford High Washington Elem Welborn Middle Wiley Elem Average Cohort 1 Cohort 2 Academy at Central Allen Middle Aycock Middle 2004-05 47.1 58.8 82.3 100 77.4 100 80.0 60.9 60 47.4 72.0 53.8 100 33.3 71.4 100 35.3 N/A 76.5 53.8 71.4 76.5 69.4 N/A 93.5 80 2005-06 50.0 (+2.9) 70.6 (-11.8) 100 (+17.7) 80.0 (-20) 63.6 (-13.8) 100 (nc) 100 (+20) 100 (+39.1) 76.9 (+16.9) 71.4 (+24) 72.0 (nc) 92.3 (+38.5) 50.0 (-50) 50.0 (+16.7) 76.0 (+4.6) 100 (nc) 79.3 (+44) N/A 94.1 (+17.6) 76.9 (+23.1) 90.5 (+19.1) 69.2 (-7.3) 79.2 N/A 96.6 (+3.1) 85.7 (+5.7) 2006-07 84.6 (+34.6) 88.2 (+17.6) 100 (nc) 52.9 (-27.1) 90.9 (+27.3) 66.7 (-33.3) 61.5 (-38.5) 65.2 (-34.8) 76.9 (nc) 100 (+28.6) 60 (-12) 53.8 (-38.5) 100 (+50) 100 (+50) 100 (+24) 53.8 (-46.2) 61.9 (-17.4) 100 100 (+5.9) 100 (+23.1) 100 (+9.5) 76.9 (+7.7) 85.4 N/A 72.4 (-24.2) 100 (+14.3) 9 School 2004-05 2005-06 2006-07 Bessemer Elem 94.1 80.0 (-14.1) 52.9 (-27.1) Cone Elem 60.0 100 (+40) 64.7 (-35.3) Falkener Elem 100 100 (nc) 60.0 (-40) Penn Griffin Middle 69.6 95.2 (+25.6) 90.5 (-4.7) Union Hill Elem 69.2 76.9 (+7.7) 69.2 (-7.7) Average Cohort 2 80.9 90.6 72.8 Note. nc=No change. Academy at Smith opened in 2006-07 and Academy at Central opened in January of 2008 so no previous comparison data were available. Data taken from www.ncreportcard.org Percentage of Students who Passed the ABCs Table 9 shows the percentage of students in each MP school that passed the ABCs (presented by school). In 2005-06, 6 of the Cohort 1 schools showed an increase in the number of students who passed the ABCs compared to the previous year. In 2006-07, 11 of the Cohort 1 schools reported an increase in the number of students who passed the ABCs since the previous year. The average pass rate for Cohort 1 schools in 2006-07 was 41.9%, a 2% increase from the previous year. Table 9. Percentage of Students Passing ABCs School Cohort 1 Andrews High Dudley High Eastern Guilford High Fairview Elem Ferndale Middle Foust Elem Gillespie Elem Hairston Middle Hampton Elem High Point Central High Jackson Middle Kirkman Park Elem Middle College at Bennett Middle College at NC A&T Oak Hill Elem Parkview Elem Smith High Academy at Smith Southern Guilford High Washington Elem Welborn Middle Wiley Elem Average % of Cohort 1 Cohort 2 Academy at Central Allen Middle Aycock Middle Bessemer Elem Cone Elem Falkener Elem Penn Griffin Middle 2004-05 31.8 50.5 56.8 65.5 59.8 65.4 60.9 61.2 56.1 51.9 60.6 44.1 20.8 24.8 53.8 67.9 46.1 N/A 57.1 40.7 65.7 70.1 52.9 N/A 70.3 73.1 64.0 58.4 78.6 64.7 40.9 52.5 62.0 44.8 30.1 45.5 40.8 42.7 27.5 61.5 33.5 30.0 17.4 15.9 42.7 40.0 47.2 N/A 61.5 26.7 44.8 29.7 39.9 N/A 44.2 49.7 34.8 30.5 55.8 46.8 2005-06 45.9 42.6 55.4 43.2 38.4 45.5 40.5 42.1 32.5 62.6 31.7 33.8 38.2 32.5 43.1 38.6 35.4 30.4 56.5 44.4 56.8 31.9 41.9 N/A 39.9 64.5 35.7 38.1 42.3 56.4 2006-07 10 School 2004-05 2005-06 2006-07 Union Hill Elem 53.9 35.9 37.9 Average % of Cohort 2 66.1 42.5 45 Note. Data taken from www.ncreportcard.org. Academy at Smith opened in 2006-07 and Academy at Central opened in January of 2008 so no previous comparison data were available. ABC Growth Met Growth within the North Carolina ABC Accountability Model is determined by predicting EOG and EOC test scores for a particular school year based on test scores of the same students from the previous year, and comparing the actual scores to the predicted scores. Students meet growth when their actual score is at or above their predicted score. It is possible for students to pass the EOG or EOC yet not have growth. School growth is determined by averaging student scores together. Table 10 shows whether individual schools met the ABC Growth. In 2004-05, 8 (38.1%) Cohort 1 schools met the ABC growth; in 2005-06, 9 (42.9%) Cohort 1 schools met the ABC growth; and in 2006-07, 12 (54.5%) schools met the ABC growth. The percentage of Cohort 1 schools meeting ABC growth has increased steadily since 2004-05. Table 10. ABC Growth Met at Each School School Cohort 1 Andrews High Dudley High Eastern Guilford High Fairview Elem Ferndale Middle Foust Elem Gillespie Elem Hairston Middle Hampton Elem High Point Central High Jackson Middle Kirkman Park Elem Middle College at Bennett Middle College at NC A&T Oak Hill Elem Parkview Elem Smith High Academy at Smith Southern Guilford High Washington Elem Welborn Middle Wiley Elem Total Met Growth in Cohort 1 Cohort 2 Academy at Central Allen Middle Aycock Middle Bessemer Elem Cone Elem Falkener Elem 2004-05 No Yes Yes Yes No No No No No Yes No No Yes No No Yes Yes N/A Yes No No No 8 (38.1%) N/A No Yes Yes No Yes 2005-06 No Yes Yes Yes No No No Yes No Yes No No Yes No Yes No Yes N/A Yes No No No 9 (42.9%) N/A No No No No No 2006-07 Yes No Yes No Yes No No Yes Yes Yes No No Yes Yes No No No Yes Yes Yes Yes No 12 (54.5%) N/A No Yes No No No 11 School 2004-05 2005-06 2006-07 Penn Griffin Middle No No Yes Union Hill Elem Yes No No Total Met Growth in Cohort 2 4 (57.1%) 0 2 (28.6%) Note. Data taken from www.ncreportcard.org. Academy at Smith opened in 2006-07 and Academy at Central opened in January of 2008 so no previous comparison data were available. Performance Composite The Performance Composite is defined by NCDPI as ‘the percentage of the test scores in the school at or above Achievement Level III (also referred to as at “grade level” or “proficient”)’. Table 11 shows the performance composite for each MP school. In 2005-06, 6 (28.6%) of the Cohort 1 schools showed an increase in performance since the previous year. In 2006-07, 11 (50%) schools reported an increase in performance since the previous year. The average performance composite for Cohort 1 MP schools in 2006-07 was 51%, up 1.8% from 2005-06. Table 11. Performance Composites for Mission Possible Schools School Cohort 1 Andrews High Dudley High Eastern Guilford High Fairview Elem Ferndale Middle Foust Elem Gillespie Elem Hairston Middle Hampton Elem High Point Central High Jackson Middle Kirkman Park Elem Middle College at Bennett Middle College at NC A&T Oak Hill Elem Parkview Elem Smith High Academy at Smith Southern Guilford High Washington Elem Welborn Middle Wiley Elem Average Cohort 1 Cohort 2 Academy at Central Allen Middle Aycock Middle Bessemer Elem Cone Elem Falkener Elem Penn Griffin Middle Union Hill Elem Total Average 2004-05 35.2 50.9 57.5 76.4 69.8 77.7 71.5 73.0 66.5 52.4 68.8 58.9 20.6 24.6 66.9 77.7 45.5 N/A 58.3 54.5 74.9 76.1 59.9 N/A 77.4 82.3 72.7 70.2 87.0 74.3 69.2 76.2 2005-06 42.1 (+6.9) 51.7 (+.8) 62.3 (+4.8) 56.2 (-20.2) 46.5 (-23.3) 59.7 (-18) 56.9 (-14.6) 58.1 (-14.9) 45.2 (-21.3) 61.1 (+8.7) 50.0 (-18.8) 44.2 (-14.7) 18.7 (-1.9) 17.6 (-7) 53.3 (-13.6) 54.8 (-22.9) 47.5 (+2) N/A 61.4 (+3.1) 43.6 (-10.9) 59.4 (-15.5) 43.1 (-33) 49.2 N/A 60.8 (-16.6) 65.7 (-16.6) 50.0 (-22.7) 50.2 (-20) 66.5 (-20.5) 59.4 (-14.9) 51.3 (-17.9) 57.7 2006-07 47.4 (+5.3) 44.5 (-7.2) 54.9 (-7.4) 54.2 (-2) 56.2 (+9.7) 57.4 (-2.3) 53.6 (-3.3) 56.5 (-1.6) 47.3 (+2.1) 62.1 (+1) 50.0 (nc) 50.0 (+5.8) 40.7 (+22) 34.5 (+16.9) 58.4 (+5.1) 51.2 (-3.6) 35.5 (-12) 34.4 56.8 (-4.6) 57.4 (+13.8) 67.9 (+8.5) 50.0 (+6.9) 51.0 N/A 60.3 (-0.8) 73.5 (+7.8) 52.0 (+2) 50.5 (+.3) 53.6 (-12.9) 70.8 (+11.4) 53.4 (+2.1) 59.2 12 Note. Data taken from www.ncreportcard.org. Academy at Smith opened in 2006-07 and Academy at Central opened in January of 2008 so no previous comparison data were available. Student Short Term Suspensions The number of short term suspensions (suspensions lasting 10 days or less) in Cohort 1 decreased by 84, to 580 total suspensions in 2006-07 (Table 12). The largest decreases were in Ferndale Middle, Hairston Middle, and Jackson Middle schools. These three schools had decreases greater than 20 suspensions. The number of short term suspensions showed a steady decline since 2004-05. Table 12. Number of Student Short Term Suspensions 2004-05 2005-06 2006-07 School Cohort 1 45 50 (+5) 53 (+3) Andrews High 29 30 (+1) 24 (-6) Dudley High 36 28 (-8) 37 (+9) Eastern Guilford High 6 3 (-3) 2 (-1) Fairview Elem 84 104 (+20) 58 (-46) Ferndale Middle 3 4 (+1) 3 (-1) Foust Elem 9 9 (nc) 3 (-6) Gillespie Elem 41 62 (+21) 41 (-21) Hairston Middle 4 4 (nc) 10 (+6) Hampton Elem 52 54 (+2) 38 (-16) High Point Central High 108 107 (-1) 72 (-35) Jackson Middle 19 2 (-17) 14 (+12) Kirkman Park Elem 32 2 (-30) 6 (+4) Middle College at Bennett 67 21 (-46) 8 (-13) Middle College at NC A&T 5 4 (-1) 1 (-3) Oak Hill Elem 9 16 (+7) 9 (-7) Parkview Elem 32 45 (+13) 61 (+16) Smith High N/A N/A 3 Academy at Smith 20 34 (+14) 54 (+20) Southern Guilford High 2 9 (+7) 12 (+3) Washington Elem 85 60(-15) 62 (+2) Welborn Middle 10 16 (+6) 9 (-7) Wiley Elem Total-Cohort 1 698 664 (-34) 580 (-84) Cohort 2 Academy at Central N/A N/A N/A Allen Middle 54 47 (-7) 41 (-6) Aycock Middle 52 38 (-14) 30 (-8) Bessemer Elem 11 17 (+6) 14 (-3) Cone Elem 12 8 (-4) 11 (+3) Falkener Elem 4 5 (+1) 3 (-2) Penn Griffin Middle 48 49 (+1) 48 (-1) Union Hill Elem 10 17 (+7) 11 (-6) 191 181 (-10) 158 (-23) Total-Cohort 2 Note. Data taken from www.ncreportcard.org. Academy at Smith opened in 2006-07 and Academy at Central opened in January of 2008 so no previous comparison data were available. 13 Student Attendance Student attendance is reported as the average percentage of students who attend school daily. The average student attendance rate increased in 2006-07 for Cohort 1, from 92.9% to 93.3%. The biggest increase was at Middle College at Bennett, which rose from 81% in 2005-06 to 90% in 2006-07. Attendance rates for all MP schools are shown in Table 13. Table 13. Student Attendance Rates in Percentages School 2004-05 2005-06 2006-07 Cohort 1 Andrews High 89 92 93 Dudley High 94 92 91 Eastern Guilford High 94 94 94 Fairview Elem 94 94 94 Ferndale Middle 93 91 92 Foust Elem 95 95 95 Gillespie Elem 95 95 94 Hairston Middle 95 93 93 Hampton Elem 95 96 95 High Point Central High 92 93 93 Jackson Middle 91 91 92 Kirkman Park Elem 95 94 95 Middle College at Bennett 85 81 90 Middle College at NC A&T 93 93 90 Oak Hill Elem 94 95 95 Parkview Elem 95 95 95 Smith High N/A N/A 94 Academy at Smith 92 92 91 Southern Guilford High 95 95 94 Washington Elem 95 93 95 Welborn Middle 93 93 94 Wiley Elem 94 93 93 Average-Cohort 1 93.2 92.9 93.3 Cohort 2 Academy at Central N/A N/A N/A Allen Middle 95 95 95 Aycock Middle 95 95 94 Bessemer Elem 95 95 94 Cone Elem 94 95 94 Falkener Elem 96 96 95 Penn Griffin Middle 94 94 95 Union Hill Elem 95 95 95 Average-Cohort 2 94.9 95.0 94.6 Note. Data taken from www.ncreportcard.org. Academy at Smith opened in 2006-07 and Academy at Central opened in January of 2008 so no previous comparison data were available. 14 Cohort Graduation Rate The graduation rates for Cohort 1 increased in 2006-07. The average graduation rate of a 4-year cohort of students was 80.1% (Table 14). The largest increase was seen at Middle College at NC A&T, which increased by 23.5%. Table 14. Graduation Rate in Percentage 2005-06 2006-07 School Cohort 1 N/A N/A Academy at Central 78.8 80.5 Andrews High 74.3 83.7 Dudley High 64.1 73.3 Eastern Guilford High 62.1 77.6 High Point Central High 75.8 83.3 Middle College at Bennett 68.8 92.3 Middle College at NC A&T 56.8 63.9 Smith High N/A N/A Academy at Smith 76.6 86.4 Southern Guilford High Average-Cohort 1 69.7 80.1 Note. Data taken from www.ncreportcard.org. Academy at Smith opened in 2006-07 and Academy at Central opened in January of 2008 so no previous comparison data were available. Climate Surveys SERVE Center developed a Climate Survey for staff, parents, and students at MP schools. Climate Surveys are intended to provide a snapshot of the school climate at one point in time, and are generally used as a catalyst for change within a school. The surveys consisted of demographic questions and 5-point Likert scale items. In addition, the staff survey contained four open-ended questions about positive and negative aspects of the MP program, professional development, and other comments. The internal consistency of the items was measured using Cronbach’s alpha. This value is dependent upon how much the items intercorrelate. The alpha of the parent survey was .96, student was .91, and staff was .93. Instruments with alphas above .80 are considered to have good reliability. Parent Climate Surveys. Survey packets containing an introductory letter, IRB consent forms, and a parent survey (both English and Spanish versions of all documents-see Appendix BG for a sample of the parent materials) were sent out to parents via students at each school. In elementary schools, students in 5th grade classrooms were sent home with surveys, distributed by their homeroom teachers (Number of students surveyed varied depending on school size; see Appendix J for sampling plan). In middle and high schools, 200 students were sent home with surveys to deliver to their parents. The first dissemination of surveys was on September 24, 2007. After reviewing response rates, an email was sent from the MP Director to 13 school coordinators where less than 10 parent surveys were returned. Coordinators at each school (see Appendix K for list of coordinators) were informed of the low response rate and asked to either request additional surveys or to work at recovering uncollected surveys. Seven of 13 coordinators had responded and all requested more surveys. Elementary schools were sent the amount of surveys that the coordinator requested, or the same amount they were initially sent. Coordinators from middle and high schools where the number of responses from parents was less 15 than 10 were sent 100 additional surveys, even if they did not respond to the email. The second mailing of surveys took place on October 26, 2007. A total of 4,324 surveys were sent out to parents. Parents completed 580 surveys, a response rate of 13.4%. Of the parents that responded to the Parent Climate Survey for elementary schools, the majority had positive perceptions of all aspects of the schools. Among all Middle and High Schools, the parents perceived gangs, violence, and substance abuse to be a problem, with some variation across schools. In other areas, they perceived the climate to be generally positive. Student Climate Surveys. Of the parents who received survey packets, 432 parents granted permission 6 for their student to receive a Student Climate Survey. Student Climate Surveys were sent on November 30, 2007 via U.S. mail to each school coordinator, who in turn disseminated envelopes to the homerooms where students with parental permission were located. Individual teachers’ envelopes contained student surveys and assent forms (see Appendix H and I for a sample). Teachers were instructed via a cover letter to disseminate assent forms and have students sign them before disseminating the student survey. Of those 432 students who were sent a Climate Survey to their respective school, 243 Student Surveys were returned to SERVE by January, 2008 (Table 15). Table 15. Number of Surveys Sent and Received School Number of Parent surveys given to schools N/A 300 300 200 60 75 300 200 100 100 200 60 80 200 80 200 200 80 88 100 80 120 Number of Parent Surveys Completed N/A 47 13 19 11 7 8 0 28 16 46 21 8 32 16 60 58 0 8 8 15 7 Number of Student Consents Received from Parents N/A 30 8 14 8 6 5 0 21 11 41 15 4 23 12 58 40 0 8 6 6 6 Number of Student Surveys Completed N/A 22 0 12 0 2 0 0 21 6 18 15 4 21 4 18 35 0 5 0 6 0 Academy at Central* Allen Middle Andrews High Aycock Middle Bessemer Elem Cone Elem Dudley High Eastern Guilford High Fairview Elem Falkener Elem Ferndale Middle Foust Elem Gillespie Elem Hairston Middle Hampton Elem High Point Central High Jackson Middle Kirkman Park Elem Middle College at Bennett Middle College at NC A&T Oak Hill Elem Parkview Elem 6 The Institutional Review Board does not permit the collection of evaluation data from minors without first obtaining parental consent. 16 Number of Number of Student Student Surveys Consents Completed Received from Parents Penn Griffin Middle 200 34 23 18 Smith High 300 4 14 3 Academy at Smith 161 53 22 4 Southern Guilford High 200 5 7 6 Union Hill Elem 60 18 12 12 Washington Elem 40 13 11 3 Welborn Middle 200 22 17 8 Wiley Elem 80 3 3 0 Total 4324 580 432 243 Response rates Parent 13.4% Student 56.3% Note. *Academy at Central did not participate because the school did not open until January of 2008. School Number of Parent surveys given to schools Number of Parent Surveys Completed Students in elementary, middle, and high schools echoed parents’ concerns about gangs, violence, and substance abuse. In addition, many students disagreed with the statement that students usually follow rules. Many high school students did not agree that school lunches taste good. Staff Climate Surveys. Staff at the 29 7 MP schools were sent a web-based Climate Survey. An email invitation with a link to the web-based survey was sent to teachers through various school lists and school principals. A total of 1,086 staff surveys were submitted electronically. Calculating a response rate was not possible because it is not known how many staff and teachers the survey ultimately reached. On February 11, 2008, results of the Climate Surveys were delivered to GCS. The full results of the Staff, Parent, and Student Climate Surveys, when broken down by school, are too extensive to include in this report. The aggregate results, however, are included in Appendix LN. The results from Fall 2007 will serve as the baseline for the program period. One theme seen across all schools in the open-ended comments was that teachers, both incentive-paid and nonincentive-paid, would like to see all teachers receive an incentive. Teachers reported that paying only some teachers is decreasing morale in the school and reducing collaboration within schools. They see the positive impact of MP being that positions that used to be difficult to fill or retain are now filled. Teacher Working Conditions Survey In 2002, North Carolina Governor Mike Easley began the Teacher Working Condition (TWC) survey. According to the survey website: The survey was designed to assess whether or not state working conditions standards developed by the North Carolina Professional Teaching Standards Commission were being met. The survey was redesigned and administered online across the state again in 2004 (http://ncteachingconditions.org/research/twcnc2006.pdf). 7 Staff at Academy at Central were not included because the school did not open until January of 2008. 17 Table 16 contains results from the 2006 survey, administered in March 2006. The TWC survey will be administered again in March 2008. The 2006 results from the five domains will serve as the baseline for this project. There were three schools with no data available due to low response rates. Academy at Smith did not exist when the 2006 survey was administered, thus data are not available. Table 16. 2006 Domain Results from Teacher Working Conditions Survey 2006 Domains Schools Time Facilities And Resources Empowerment Leadership Professional Development Cohort 1 Andrews High 2.67 3.11 2.99 3.28 3.01 Dudley High 2.75 3.46 2.85 2.94 2.91 Eastern Guilford High 2.78 3.39 2.70 3.08 3.12 Fairview Elem 3.11 4.00 4.11 4.07 3.75 Ferndale Middle 2.53 3.45 2.89 2.90 2.96 Foust Elem 3.26 3.61 3.18 3.33 3.65 Gillespie Elem 3.49 4.06 4.05 4.21 3.30 Hairston Middle 3.50 3.77 3.37 3.67 3.39 Hampton Elem N/A N/A N/A N/A N/A High Point Central 2.88 3.10 2.91 3.03 3.01 Jackson Middle 2.51 3.23 1.97 1.82 2.22 Kirkman Park Elem 2.64 2.64 2.45 2.44 2.80 Middle College at Bennett 3.93 3.35 3.64 3.95 3.40 Middle College at NC A&T 3.83 2.75 2.57 2.97 2.93 Oak Hill Elem 2.93 3.52 3.52 3.53 3.61 Parkview Elem 3.20 3.43 3.44 3.66 3.76 Smith High 2.55 3.42 2.62 3.08 2.91 Academy at Smith N/A N/A N/A N/A N/A Southern High N/A N/A N/A N/A N/A Washington Elem 3.46 3.75 4.01 4.28 3.73 Welborn Middle N/A N/A N/A N/A N/A Wiley Elem 3.14 3.36 2.84 2.79 3.00 Cohort 1 Average 3.06 3.41 3.12 3.28 3.19 Cohort 2 Academy at Central N/A N/A N/A N/A N/A Allen Middle 2.57 3.49 3.05 3.02 3.36 Aycock Middle 2.68 3.71 3.04 3.12 3.09 Bessemer Elem 2.62 3.49 3.00 3.00 3.48 Cone Elem 2.42 3.42 2.54 2.77 2.95 Falkener Elem 3.16 3.96 3.31 3.34 3.45 Penn Griffin Middle 3.47 4.09 3.59 3.62 3.95 Union Hill Elem 2.78 3.53 3.28 3.50 3.33 Cohort 2 Average 2.81 3.67 3.12 3.2 3.37 Note. Academy at Smith opened in 2006-07 and Academy at Central opened in January of 2008 so data were available. In other schools where no data were available, the response rates were too low to include the data. Professional Development Workshops When the MP program initially started, there was not a systematic way of keeping track of incentive-paid teachers’ professional development attendance. As a result, during the first year of the program, incentives were not contingent on professional development and thus, teacher 18 satisfaction with professional development and attendance at professional development is not available for Year 1. Steps have been taken so that professional development records can be kept for Year 2. Performance Incentives and Teacher Value-Added Averages The total amount of money spent by GCS on performance incentives is an indicator of performance. When teacher Value-Added scores increase, the amount of performance incentivepaid will also increase. The amount spent in November 2007 on Cohort 1 teachers (2006-07 school year) serves as a baseline for Cohort 1. The TIF-funded teachers did not begin the program until 2007-08 school year, thus they will not receive their first performance incentive until November 2008. In this reporting year, it cannot be determined if teacher performance increased. The overall aggregate baseline incentives for Cohort 1 will be $268,240 for 76 total faculty, as shown in Table 17. Table 17. Total Performance Incentives Received by School School Cohort 1 Andrews High Dudley High Eastern Guilford High Fairview Elem Ferndale Middle Foust Elem Gillespie Elem Hairston Middle Hampton Elem High Point Central High Jackson Middle Kirkman Park Elem Middle College at Bennett Middle College at NC A&T Oak Hill Elem Parkview Elem Smith High Academy at Smith Southern Guilford High Washington Elem Welborn Middle Wiley Elem Totals-Cohort 1 Cohort 2 Academy at Central Allen Middle Aycock Middle Bessemer Elem Cone Elem Falkener Elem Penn Griffin Middle Union Hill Elem Totals-Cohort 2 $12,000 $24,000 $13,000 $10,500 $2,500 $4,000 $0 $33,000 $7,500 $44,500 $2,500 $0 $7,500 $7,500 $7,500 $2,500 $16,000 $11,500 $21,000 $17,500 $23,750 $0 $268,240 N/A N/A N/A N/A N/A N/A N/A N/A N/A Total $ *2006-07 (22 Schools) Total Faculty 3 6 4 3 1 1 0 9 3 12 1 0 2 2 2 1 4 3 6 6 7 0 76 N/A N/A N/A N/A N/A N/A N/A N/A N/A 19 Note. Only the 22 schools in Cohort I received incentives in 2006-07 (paid in November 2007). Year 2 report will reflect all 30 schools. Table 18 shows district totals broken down by position category. Twenty-three teachers had Value-Added Scores above 1 Standard Error, and 41 teachers had scores that were above 1.5 Standard Error. Five curriculum facilitators and seven principals earned performance incentives. Table 18. Total District Performance Incentives Paid Position Category Teachers with >1.0 SE Teachers with >1.5 SE Curriculum Facilitators Principals 23 41 5 7 2006-07 (22 Schools) Faculty $57,500 $164,000 $11,750 $35,000 Amount Total 76 $268,250 Note. Budget reflects the 22 schools in Cohort I that received incentives in 2006-07 (paid in November 2007). Year 2 report will reflect all 30 schools. The total incentives are also broken down by course categories, showing progress in the target areas (Math and Language Arts). Table 19 shows the Cohort 1 baseline for each course area. Table 19. Incentives Paid by Course Totals 2006-07 (22 schools) Total Dollars Total Faculty 3rd Grade $12,500 5 EOG Composite $73,000 22 Algebra I $58,500 15 Algebra II $19,500 6 Geometry $32,000 8 AP Calculus $5,000 2 AP Statistics $2,500 1 IB Math Studies $4,000 1 English I $39,500 11 CF AYP $11,750 5 Principal AYP $35,000 7 Totals $268,250 *76 Note. Budget reflects the 22 schools in Cohort I that received incentives in 2006-07 (paid in November 2007). Year 2 report will reflect all 30 schools. * Total incentives paid out were 83, paid to 76 unique faculty. Course-Cohort 1 Comparison of Mission Possible Schools with Non Mission Possible Schools Included in this Year 1 report are the matches of non MP schools to MP schools using a propensity scoring model. As described in the full student achievement report (Appendix O) there is currently one comparison school matched to a MP school. The matches are shown in 20 Tables 20-22 on the following page. Two additional features of the analysis will be pursued: (1) a one-to-many match using propensity scores and (2) stratifying the data into quintiles by propensity score. The second option also helps to reduce selection bias in the estimates and mitigate the impact of the unobserved factors not included in the model (Hahs-Vaughan & Onwuegbuzie, 2006; Luellen et al., 2005; and Joffe & Rosenbaum, 1999). The current matches are a good first approximation but by expanding the methodology to include a one-to-many match and by stratifying the sample into quintiles, greater accuracy can be achieved. The next phase in the student achievement analysis will be to expand the propensity score matches as outlined above and to pursue analysis at the student level within the treatment schools, possibly showing potential programmatic impacts. This will be completed within the year, depending on the acquisition of scale scores from NCDPI. 21 Table 20. Propensity Scores for One-to-One Matched Treatment Elementary Schools Elementary Schools (N = 26) One Emergency/ Teacher Prior Year School Free Lateral No. of School Total w/0-3 Yrs Achievement Grade Propensity Teacher Entry Locale Lunch Classroom Experience Rate Number Enrollment Range Score Turnover (NCES) Rate Teacher Teachers Rate (2004-2006) Rate Rate 0 670-332 422 23.3 2 55.5 1.4 35 37.1 82.9 K-5 0.00753876 1 410-385 267 28.1 2 99.3 3.6 28 7.1 66.3 PK-5 0.00760494 0 360-520 303 28.9 2 100.0 9.1 44 22.7 68.8 PK-5 0.03528028 1 410-364 551 28.5 2 100.0 4.7 43 27.9 69.3 PK-5 0.03631913 0 340-462 381 18.1 2 100.0 3.8 40 12.5 71.5 PK-5 0.03776691 1 410-511 421 15.7 2 92.9 2.6 38 23.7 71.7 PK-5 0.03789915 0 640-331 153 36.3 2 100.0 3.6 14 28.6 76.6 PK-5 0.05727090 1 410-580 406 15.1 2 91.9 3.0 33 27.3 64.2 PK-5 0.05768113 0 600-527 433 20.9 1 96.5 4.9 41 39.0 66.8 K-5 0.08021636 1 410-328 394 16.6 2 98.2 2.8 36 22.2 66.4 K-5 0.07993569 0 340-368 479 22.5 2 100.0 2.2 46 26.1 73.3 PK-5 0.10800233 1 410-349 475 24.3 2 100.0 3.5 43 27.9 67.3 PK-5 0.10943861 0 340-400 414 15.0 2 91.3 1.9 53 26.4 74.2 PK-5 0.16453770 1 410-366 571 30.1 2 94.6 0 51 37.3 80.7 PK-5 0.16836308 0 410-532 645 29.3 2 90.4 0.9 57 31.6 67.4 PK-5 0.30768022 1 410-373 311 28.1 2 94.9 0 31 32.3 73.0 PK-5 0.30924058 0 260-324 172 44.4 2 100.0 5.0 20 45.0 65.2 PK-5 0.38590229 1 410-499 355 22.2 2 100.0 0 35 20.0 64.5 PK-5 0.41501522 0 340-430 243 37.9 2 100.0 1.6 30 30.0 67.7 PK-5 0.55963081 1 410-586 203 45.8 2 100.0 6.3 24 41.7 56.1 PK-5 0.54555267 0 600-335 380 27.2 1 91.8 3.5 43 46.5 54.5 K-5 0.71064103 1 410-403 292 44.8 2 100.0 3.2 32 46.9 63.4 PK-5 0.72078979 0 340-376 482 27.2 2 100.0 1.9 52 13.5 57.9 PK-5 0.56998080 1 410-439 236 55.1 2 95.8 0 24 50.0 58.1 PK-5 0.96513462 0 600-374 360 33.3 1 95.0 0 32 34.4 66.7 K-5 0.51679736 1 410-598 186 82.1 2 100.0 1.9 26 61.5 71.5 PK-5 0.97320092 Notes. Comparison School=0, Mission Possible School=1. Baseline data presented above is from the 2005-2006 academic year, one year prior to Mission Possible implementation. School Enrollment is first month enrollment. School Locale are NCES (old version) locale codes where 1 = large city, and 2 = midsize city. Prior Achievement is the weighted average of students scoring at Level 3 or higher (percent proficient) on the mathematics and reading (summed together) end-of-grade exams across grades 3-5 for the 2003-2004, 2004-2005 and 2005-2006 academic years. Grade Range was not included as a covariate in the logistic model but was used to restrict the pool of schools to those with PK-5 and K-5 grade configurations when estimating the model. Propensity scores Mission Possible/ Treatment 22 calculated using STATA/SE version 7.0 and all propensity score matches generated in SPSS version 15.0. Source: North Carolina Department of Public Instruction, NC School Report Card Data Disc 2001-2007, version 10-3-07. Table 21. Propensity Scores for One-to-One Matched Treatment Middle Schools Middle Schools (N = 14) One Year Prior School Free No. of Teacher Achievement Grade Propensity Locale Lunch Classroom Turnover Rate Range Score (NCES) Rate Teachers Rate (2004-2006) 0 110-306 640 38.4 4 49.1 50 85.1 6-8 0.03254677 1 410-316 783 31.2 2 75.7 55 74.6 6-8 0.03272171 0 510-397 820 33.8 8 63.4 61 78.2 6-8 0.05811119 1 410-391 551 35.8 2 67.9 47 72.8 6-8 0.05827692 0 900-314 776 20.9 3 78.2 65 65.9 6-8 0.09735436 1 410-592 697 33.9 2 73.3 56 71.6 6-8 0.09333859 0 310-396 225 16.6 6 100.0 35 66.0 6-8 0.13206095 1 410-325 606 46.2 2 67.7 49 76.3 6-8 0.13182034 0 430-347 1,134 20.9 7 58.0 89 81.5 6-8 0.22639379 1 410-415 552 40.0 2 90.4 49 68.6 6-8 0.30542508 0 600-381 826 33.7 1 88.5 69 65.0 6-8 0.45008010 1 410-367 577 40.0 2 90.5 53 63.2 6-8 0.38394669 0 260-428 538 47.8 4 89.8 44 64.3 6-8 0.19950813 1 410-402 632 35.0 2 94.3 58 68.8 5-8 0.53535336 Notes. Comparison School=0, Mission Possible School=1. Baseline data presented above is from the 2005-2006 academic year, one year prior to Mission Possible implementation. School Enrollment is first month enrollment. School Locale are NCES (old version) locale codes where 1 = large city, 2 = mid-size city, 3 = fringe of large city, 4 = fringe of mid-size city, 5 = large town, 6 = small town, 7 = rural, outside CBSA/MSA, and 8 = rural, inside CBSA/MSA; locale was not included as a covariate in estimating the model. Prior Achievement is the weighted average of students scoring at Level 3 or higher (percent proficient) on the mathematics and reading (summed together) end-of-grade exams across grades 5-8 for the 2003-2004, 2004-2005 and 2005-2006 academic years. Grade Range was not included as a covariate in the logistic model but was used to restrict the pool of schools to those with 5-8, 6-8 and 6-9 grade configurations when estimating the model. Propensity scores calculated using STATA/SE version 7.0 and all propensity score matches generated in SPSS version 15.0. Source: North Carolina Department of Public Instruction, NC School Report Card Data Disc 2001-2007, version 10-3-07. Mission Possible/ Treatment School Number Total Enrollment 23 Table 22. Propensity Scores for One-to-One Matched Treatment High Schools High Schools (N = 16) One Year Teacher Prior School Free Teacher w/0-3 Yrs Achievement Graduation Grade Propensity Locale Lunch Turnover Experience Rate Rate Range Score (NCES) Rate Rate Rate (2004-2006) 0 291-336 731 22.7 6 64.4 18.8 53.5 94.8 9-12 0.04188414 1 410-556 927 25.7 8 41.5 27.3 65.1 99.3 9-12 0.04077351 0 010-360 887 34.4 2 63.2 41.9 60.9 96.7 9-12 0.08043345 1 410-406 1,278 32.0 2 55.8 37.4 59.8 96.6 9-12 0.08134238 0 422-324 315 36.0 7 78.1 24.1 57.6 100.0 9-12 0.15652275 1 410-544 1,527 23.8 2 65.6 31.8 48.3 95.6 9-12 0.15301910 0 600-592 2,223 17.8 1 51.7 31.6 48.9 98.0 9-12 0.17934483 1 410-358 930 27.2 8 51.3 41.8 55.0 99.2 9-12 0.16205180 0 340-330 1,061 28.5 2 53.1 25.8 51.2 97.7 9-12 0.31407568 1 410-355 1,583 33.3 2 65.8 39.2 48.3 96.7 9-12 0.31718808 0 600-496 1.053 34.8 1 64.0 38.0 46.6 100.0 9-12 0.67999256 1 410-483 101 62.5 2 44.6 30.0 22.7 78.3 9-12 0.77918673 0 600-576 1,651 24.5 1 79.2 30.3 38.0 99.4 9-12 0.65904289 1 410-319 1,037 39.7 2 63.0 36.5 38.8 98.0 9-12 0.87062204 0 600-376 1,137 29.0 1 64.0 36.0 44.0 96.4 9-12 0.44685721 1 410-326 83 50.0 2 51.9 22.2 24.4 95.5 9-12 0.99697083 Notes. Comparison School=0, Mission Possible School=1. Baseline data presented above is from the 2005-2006 academic year, one year prior to Mission Possible implementation. School Enrollment is first month enrollment. School Locale are NCES (old version) locale codes where 1 = large city, 2 = mid-size city, 3 = fringe of large city, 4 = fringe of mid-size city, 5 = large town, 6 = small town, 7 = rural, outside CBSA/MSA, and 8 = rural, inside CBSA/MSA; locale was not included as a covariate in estimating the model. Prior Achievement is the weighted average of students scoring at Level 3 or higher (percent proficient) on Algebra 1, Algebra 2, English 1 and geometry (summed together) end-of-course exams across grades 9-12 for the 2003-2004, 2004-2005 and 2005-2006 academic years. Grade Range was not included as a covariate in the logistic model but was used to restrict the pool of schools to those with 8-12 and 9-12 grade configurations when estimating the model. Propensity scores calculated using STATA/SE version 7.0 and all propensity score matches generated in SPSS version 15.0. Source: North Carolina Department of Public Instruction, NC School Report Card Data Disc 2001-2007, version 10-3-07. Mission Possible high school omitted from matching: Academy at Smith (410-545) (Ben L. Smith High School), since its first year was 2006-2007 and did not have baseline data. Academy at Central was also omitted because the school did not open until January 2008. Mission Possible/ Treatment School Number Total Enrollment 24 Conclusion and Recommendations The NCDPI data for 2004-05 through 2006-07 shows improvements in teacher, principal, and student outcomes. Teacher turnover has decreased by an average of 2% in Cohort 1 schools. Seven schools had turnover that was lower than the district average. The percentage of students passing the ABCs also improved by 2%. In 2006-07, 11 schools showed an increase in AYP goals obtained, with nine schools making AYP. A summary table of results, which shows the increase or decrease in each data category is shown in Table 23. The data from 2006-07, where comparisons were available from the 2005-06 data, show improvement. Table 23. Summary of Results for Cohort 1-Changes from 2005-06 to 2006-07 Category Principal Turnover Teacher Turnover Incentive Paid Teacher turnover Percentage of AYP Goals Percentage of Students who Passed ABCs ABC Growth Performance Composite Short Term Suspensions Student Attendance Cohort Graduation Rate Teachers with Initial or Continuing License Classes Taught by Highly Qualified Teachers School Climate Teacher Working Conditions Teacher Value Added Scores Amount of Performance Incentives Increase Decrease Unknown-1st Year of Data Average Amount22 schools combined -4.8% -2.0% N/A +6.2% +2.0% +3 schools +1.8% -84 +0.4% +10.4% +4.1% +3% N/A N/A N/A N/A The Climate Surveys administered in Fall 2007 show that, while parents and students perceive the majority of school components positively, there is a concern for school violence, gangs, and drugs. School staff also perceived the majority of school components positively, but perceived there to be a lack of parent involvement, student deficiencies in basic skills, low 25 morale among staff, and racist comments from students. Both incentive-paid and non-incentivepaid teachers expressed that all teachers should be compensated in some way. However, the incentive structure is unable to be changed, and the budget does not allow for incentives to be extended to all school personnel. Principals should pursue strategies to increase teacher morale within the current incentive structure. The low response rate for Parent and Student Climate Surveys is indicative of low parental involvement. Low parental involvement was also perceived to be low by teachers on the Staff Survey. Principals should work on strategies to increase parental involvement. Related, in an effort to increase response rates through the surveying method, SERVE will send both Parent and Student Climate Surveys together and ask that parents return completed surveys directly to SERVE Center with a pre-addressed, postage-paid envelope. 26 Appendix A - Mission Possible Evaluation Plan SERVE Center at the University of North Carolina at Greensboro Data Sources Benchmarks Data Gathering Points Analytical Methods Party Responsible 1. To what extent are the most highly qualified teachers and administrators being recruited and retained at Mission Possible (MP) Schools? Personnel data, e.g., vacancy ratings, turnover, numbers of unlicensed, lateral entry, and high qualified teachers. Includes: GCS application process information including Haberman Assessment Value added scores AYP status and ABC status of MP schools vs. non-MP schools) 8 System budget review to determine incentives teachers and principals receive compared to system salary budget overall Number of vacancies/ rate of turnover at MP schools Number of non-highly qualified, unlicensed and lateral entry teachers at MP schools ABC and AYP status of MP schools vs. non-MP Percentage of personnel budget that goes to pay teachers’ and principals’ incentives Years 1 – 5 Pre/post analysis of personnel data at MP schools (beginning of school year and end of school year) GCS Provide/facilitate SERVE access to the necessary data SERVE Determine matched comparison group using propensity scoring Data collection Data analysis GCS Data collection Data analysis To be included in annual evaluation report as an addendum (data to be provided by GCS) Years 1 – 5 Years 1 – 5 Comparison of AYP and ABC data between MP and non-MP schools System budget data MP schools 2. To what extent were the MP teachers and administrators trained as proposed? Participation rate of teachers/administrators Documentation of participation in professional development activities Years 1 – 5 Number/percentage of teachers participating in professional development from MP schools GCS Data collection (participant sign in sheets, rosters, etc.) SERVE Data analysis 3. To what extent did the MP training and incentives impact teacher and student outcomes? Survey of teacher and administrator satisfaction with school climate in MP schools School Climate Inventory (pre/post) Years 1 – 5 Frequencies and percentages (mean and standard deviations, if applicable) of satisfaction survey results (beginning of school year and end of school year). GCS Identify School Climate Inventory participants Administer School Climate Inventory – pre/post 8 “Non-MP” schools will be selected, by SERVE, via a matched comparison group process whereby these schools will be matched with MP schools based upon predetermined demographic variable. 27 Data Sources Benchmarks Data Gathering Points Year 2 and Year 4 Analytical Methods Party Responsible SERVE Data analysis GCS Identify focus group/interview participants SERVE Develop, collaboratively with GSC, interview/focus group protocol for teachers and administrators Conduct focus group/interviews Data analysis GCS Identify parent participants Administer survey to parents SERVE Develop, collaboratively with GSC, parent survey Data analysis GCS Identify student participants Administer student School Climate Inventory – pre/post SERVE Develop student School Climate Inventory Data analysis GCS Provide/ facilitate SERVE access to the necessary data SERVE Data collection Data analysis GCS Provide/ facilitate SERVE access to the necessary data SERVE Data collection Data analysis GCS/SAS Calculate teachers’ value-added scores at Time 1 (pre) and Time 2 (post) Teacher observation data/360 degree feedback SERVE Incorporate value-added and teacher Interviews/focus groups of teachers/ administrators to determine satisfaction with professional development activities and Mission Possible project overall Interview/focus group data Content analysis of interviews/focus group transcripts Survey of parent satisfaction with MP schools Parent Survey data Years 1 – 5 Frequencies and percentages of satisfaction survey results Survey of student satisfaction with school climate in MP schools School Climate Inventory (pre/post) – student focused Years 1 – 5 Frequencies, and percentages (mean and standard deviations, if applicable) of satisfaction survey results MP students’ discipline referrals, attendance, promotion rates School level data Years 1 – 5 MP teachers’ students’ EOG/EOC scores as compared to non-MP teachers’ students’ EOG/EOC scores EOG/EOC scores of students of MP and non-MP teachers Years 1 – 5 Comparison of EOG/EOC data between MP and non-MP schools Pre and Post MP teachers’ value added scores Teacher Observations – Incorporate results from teachers observations and evaluations Current value-added scores Years 1 – 5 Observation data/360 degree feedback 28 Data Sources Benchmarks Data Gathering Points Analytical Methods Party Responsible observation results in to evaluation reporting Attending monthly/quarterly meetings Report Writing Years 1 – 5 Years 1 – 5 SERVE SERVE 29 Appendix B - Letter to Parents Regarding Climate Survey September 24, 2007 Dear Parent or Guardian, The school that your child attends is currently participating in the Mission Possible Program. As you may know, this program is a teacher incentive program designed to attract and retain teachers for select schools and to reward teachers for outstanding results. This is a support program for teachers in selected schools focusing on high quality staff development and smaller class sizes. The SERVE Center based at the University of North Carolina at Greensboro has been asked to evaluate the program. As part of the evaluation, the parents and students are being asked to participate by answering some questions related to the school environment. You can help Guilford County improve the program by completing a survey and also by giving your consent for your child to complete a survey. There are two documents attached; a consent form and a survey. The consent form is a formal document that evaluators and researchers are required to provide to participants and parents of students under 18 years of age. This document outlines the risk of participating and other information about your rights. This document also outlines the rights of your child, if you decide to allow him or her to complete a survey at school. If you consent, your child will also have to agree by signing a separate form that will be handed out with the survey. The consent forms provide contact information should you have any questions about the surveys. Please sign the enclosed consent forms and then indicate if you consent to participate. Also indicate if you allow your child to participate. Place one copy of the consent form in the envelope along with the survey and keep the other copy of the consent form for your records. Have your child return the sealed envelope to his or her teacher. The envelopes will be delivered to the evaluators at the SERVE Center. There, your consent form will be reviewed. If you completed the survey, your survey will be separated from the consent form so that your survey is truly anonymous. If you consent for your child to take the survey, the teacher will be notified that your child should receive a student survey. If your child will not be completing a survey, they will participate in an age-appropriate activity while other students complete the survey. Please return the consent form indicating your level of participation. Thank you in advance for your participation. Sincerely, Amy Holcombe Mission Possible Project Director Guilford County Schools 712 Eugene St. Greensboro, NC 336-370-8091 holcoma@gcsnc.com 30 Appendix C - Letter to Parents Regarding Climate Survey (Spanish version) September 24, 2007 Estimado Padre o Guardián Legal, La escuela a la que su hijo asiste está participando actualmente en el Programa Misión Posible. Misión Posible es un programa de incentivo para maestros. El programa fue diseñado para atraer y retener a maestros en ciertas escuelas y para premiar a maestros que consiguen resultados sobresalientes con sus estudiantes. El Programa Misión Posible es un programa de apoyo para maestros en ciertas escuelas que ponen énfasis en la calidad alta de personal y en tener clases con un número reducido de estudiantes. El Centro SERVE, que forma parte de la Universidad de la Carolina del Norte en Greensboro, fue invitado a evaluar el programa. Como parte de la evaluación, el Centro SERVE está pidiendo a los padres y estudiantes que contesten algunas preguntas sobre la escuela. Usted puede ayudar al Condado Guilford a mejorar el programa al responder a algunas preguntas y al dar su permiso a sus hijos a responder a las preguntas también. Hay dos documentos adjuntos a esta carta; un formulario de consentimiento y un cuestionario. El formulario de consentimiento es un documento formal que los evaluadores e investigadores tienen que proveerles a los participantes y a los padres de estudiantes menores de 18 años. Este documento explica el posible riesgo de participar en esta evaluación y otra información sobres sus derechos. Este documento también explica los derechos de su hijo, si usted decide permitir a su hijo a completar un cuestionario en la escuela. Si usted da su consentimiento, su hijo también tendrá que decir que quiere participar al firmar un formulario diferente que será distribuido con el cuestionario. Si usted tiene cualquier pregunta sobre el cuestionario, el formulario de consentimiento tiene información de contacto que usted puede usar. Favor de firmar los formularios de consentimiento adjuntos e indicar si usted quiere participar y si usted permite que su hijo participe. Favor de poner una copia del formulario de consentimiento y el cuestionario completado dentro del sobre y quedarse la otra copia del formulario de consentimiento. Su hijo debe de devolverle el sobre sellado a su maestro. Los maestros les entregarán los sobres sellados a los evaluadores del Centro SERVE. Los evaluadores revisarán su formulario de consentimiento. Si usted llenó el cuestionario, los evaluadores separarán su formulario de consentimiento de su cuestionario, para que sus respuestas en el cuestionario sean verdaderamente anónimas. Si usted dio permiso para que su hijo completara el cuestionario, los evaluadores le notificarán al maestro de su hijo que su hijo debe recibir un cuestionario estudiantil para llenar. Si usted no dio permiso para que su hijo completara el cuestionario, su hijo participará en otra actividad apropiada para su edad mientras que los otros estudiantes rellenan los cuestionarios. Por favor, devuelva el formulario de consentimiento, indicando si usted y su hijo quieren participar. Agradecemos su participación. Sinceramente: Amy Holcombe Director, Programa Misión Posible Escuelas del Condado de Guilford 712 Eugene St., Greensboro, NC 336-370-8091 holcoma@gcsnc.com 31 Appendix D - Parent/Student Consent Form UNIVERSITY OF NORTH CAROLINA AT GREENSBORO CONSENT FORM PARENT Consent to Act as a Human Participant in Mission Possible Evaluation Project Title: Mission Possible Evaluation Project Director: Amy Holcombe DESCRIPTION AND EXPLANATION OF PROCEDURES: The purpose of the enclosed survey is to get parent perceptions of the school environment. Classrooms of students and their parents were randomly chosen to be invited to complete a survey. The results of this survey will be analyzed by SERVE Center and the report will be given to Guilford County Schools in an effort to help them improve the school environment and make it a better place for students to learn and for teachers to work. The student version of this survey is similar but the questions are worded slightly different. The student survey will be administered during class time. With your permission, your child will be given an assent form to sign and a survey. If you agree to participate, we ask that you complete the enclosed survey and send it back to the school with your child along with a copy of this consent form (signed). The answers provided by you and your child are confidential and will not be associated with any name. Once your envelope is received by the evaluators, your answers are guaranteed to be anonymous. We will keep all surveys in our locked file cabinet at the SERVE offices until the year 2015 and then they will be destroyed by a paper shredder. The survey should take approximately 20 minutes to complete. If you also agree that your child can participate, please indicate below by checking the appropriate line. The survey that your child completes will be collected by the teacher along with several other surveys and placed in an envelope. The surveys will not have your child’s name on them. Your child may stop filling out the survey at any time without penalty. RISKS AND DISCOMFORTS: The evaluators have determined that there is minimal risk to you and your child. Though students are asked not to identify themselves, there is a chance that a student may put their name on the survey and then their answers could be linked to them. The rate of occurrence of such a risk is thought to be minimal. Parents will have to return their survey along with a consent form for their child and themselves. These consent forms will be in the same envelope as the completed survey, so there is a chance that someone could link their answers on the survey to them. We do not anticipate that teachers will open the envelopes but instead, will give them to SERVE. POTENTIAL BENEFITS: Your responses and your child’s responses will help Guilford County improve their Mission Possible program which is anticipated to improve your child’s education experience. The benefit to society is the anticipated improvement of a school in the local community. CONSENT: By signing this consent form, you agree that you understand the procedures and any risks and benefits involved in this evaluation. You are free to refuse to participate or to withdraw your consent to participate in this evaluation at any time without penalty or prejudice; your participation is entirely voluntary. You and your child’s privacy will be protected because you will not be identified by name as a participant in this project. Your child may also refuse to participate without penalty and withdraw at any time. The University of North Carolina at Greensboro Institutional Review Board, which ensures that research and evaluation involving people follows federal regulations, has approved the evaluation and this consent form. Questions regarding your rights as a participant in this evaluation can be answered by calling Mr. Eric Allen at (336) 256-1482. If you have questions at any time about the Mission Possible evaluation or the procedures used to collect information, you may contact Dr. Holli Gottschall at PO Box 5367, Greensboro, NC 27435, 336-315-7400 or email hgottsch@serve.org Any new information that develops during the project will be provided to you if the information might affect your willingness to continue participation in the project. 32 Please sign the form and indicate your level of participation below. ____________________________________ Parent/Guardian’s Name-Printed ____________________________________ Parent/Guardian's Signature ______________ Date Parent/Guardian Participation –Place a check next to one statement: ___As a parent or guardian, I wish to participate and have completed the survey. ___As a parent or guardian, I do not wish to participate and have not completed the survey. I understand there is no penalty for this decision. Child Participation-Place a check next to one statement: ___As a parent or guardian, I allow my child to complete a survey at school. ___As a parent or guardian, I request that you do not give my child a survey. I understand there is no penalty for this decision. ____________________________________ Name of Child ____________________________________ Child’s School Name ____________________________________ Homeroom Teacher’s Name 33 Appendix E - Parent/Student Consent Form (Spanish version) UNIVERSIDAD DE LA CAROLINA DEL NORTE EN GREENSBORO FORMULARIO DE CONSENTIMIENTO PADRE consentimiento para participar en la evaluación del programa misión posible Título del Proyecto: Evaluación Misión Posible Director del Proyecto: Amy Holcombe DESCRIPCIÓN Y EXPLICACIÓN DE PROCEDIMIENTOS: El propósito del cuestionario adjunto es aprender más sobre las opiniones de los padres sobre la escuela de su hijo. Las clases de estudiantes y sus padres fueron escogidos arbitrariamente para ser invitados a completar un cuestionario. Los resultados de este cuestionario serán analizados por el Centro SERVE. Después, el Centro SERVE entregará un informe sobre los resultados del análisis de los cuestionarios a las Escuelas del Condado de Guilford. El Condado de Guilford usará este informe para mejorar sus escuelas para que sean un mejor lugar de aprendizaje para los estudiantes y un mejor lugar para trabajar para los maestros. La versión estudiantil de este cuestionario es similar a la versión para los padres, pero usa palabras diferentes para expresar ideas similares. Los estudiantes que participen en esta evaluación completarán los cuestionarios en la clase durante las horas escolares. Con su permiso, su hijo recibirá un formulario de consentimiento para firmar y un cuestionario. Si usted decide participar, le rogamos que firme el formulario de consentimiento y rellene el cuestionario adjunto. Después, devuélvalos a la escuela. Las respuestas que usted y su hijo provean son confidenciales y no serán asociadas son su nombre. Una vez que los evaluadores del Centro SERVE reciban sus respuestas, ellos garantizan que sus respuestas serán anónimas. Nosotros, los evaluadores, guardaremos todos los cuestionarios en un armario cerrado bajo llave en la oficina de SERVE hasta el año 2015; en 2015, estos cuestionarios serán destruidos. Rellenar este cuestionario toma aproximadamente 20 minutos. Si usted también quiere que su hijo participe, favor de indicareo abajo. Si su hijo llena un cuestionario, su maestro lo recogerá, lo juntará con los cuestionarios de los otros estudiantes, y los pondrá en un sobre. El cuestionario no tendrá el nombre del estudiante que lo llenó. Su hijo puede decidir no llenar el cuestionario en cualquier momento sin ningún problema. RIESGOS E INCOMODIDADES: Los evaluadores han decidido que hay un riesgo mínimo si usted y su hijo deciden participar en esta evaluación. Aunque los estudiantes no necesitan dar sus nombres, cabe la posibilidad de que un estudiante escriba su nombre en el cuestionario por accidente; en este caso, el nombre del estudiante puede ser asociado con su cuestionario. Los evaluadores creen que la probabilidad de este riesgo es mínima. Los padres tendrán que devolver su cuestionario con un formulario de consentimiento para ellos. Estos formularios de consentimiento estarán en el mismo sobre que el cuestionario completado; por eso, cabe la posibilidad de que alguien asocie el cuestionario con el nombre de la persona que lo llenó. Los maestros probablemente no abrirán los sobres; en vez de esto, es probable que los darán a los evaluadores de SERVE. BENEFICIOS POSIBLES: Sus respuestas y las respuestas de su hijo ayudarán al Condado Guilford a mejorar su programa Misión Posible, lo cual ayudará a mejorar la educación de su hijo. El beneficio para la sociedad en general es el mejoramiento de la escuela en la comunidad local que esperamos que resulte de esta evaluación CONSENTIMIENTO: Al firmar este formulario de consentimiento, usted indica que entiende los procedimientos y cualquier riesgo asociado con esta evaluación. Usted puede decidir no participar o dejar de participar en esta evaluación en cualquier momento sin problema; su participación es completamente voluntaria. Su privacidad y la de su hijo serán protegidas porque ustedes no se identificarán con nombre como participantes en este proyecto. Su hijo también puede decidir no participar o dejar de participar en esta evaluación en cualquier momento. El Comité Institucional de Revisión de la Universidad de la Carolina del Norte en Greensboro asegura que las investigaciones que incluyen a personas son conformes a las regulaciones federales. Este comité ha aprobado esta 34 evaluación y este formulario de consentimiento. Si usted tiene preguntas sobre su participación en esta evaluación, favor de llamar al Sr. Eric Allen a (336) 256-1482. Si, en cualquier momento, usted tiene preguntas sobre la evaluación del proyecto Misión Posible o los procedimientos usados para recoger información, usted puede comunicarse con la Dra. Holli Gottschall en PO Box 5367, Greensboro, NC, 27435, 336-315-7400, o por correo electrónico en hgottsch@serve.org. Si cualquier información nueva se desarrolla durante el proyecto, usted será informado si esta información podría afectar su deseo de continuar su participación en el proyecto. Por favor firme el formulario, e indigue si desea o no participar. ________________________________________ Nombre del Padre/Guardián – Escrito ________________________________________ Firma del Padre/Guardián ____________________ Fecha Participación del Padre/Guardián – Favor de marcar una de las siguientes opciones: _____ Como un padre o guardián, quiero participar en esta evaluación y he llenado el cuestionario. _____ Como un padre o guardián, no quiero participar y no he llenado el cuestionario. Entiendo que no hay ninguna penalidad al esta decisión. Participación de Hijo – Favor de marcar una de las siguientes opciones: _____ Como un padre o guardián, doy permiso a mi hijo para rellenar un cuestionario en la escuela. _____ Como un padre o guardián, no doy permiso a mi hijo para rellenar un cuestionario en la escuela. Entiendo que no hay ninguna penalidad al tomar esta decisión. ________________________________________ Nombre del Hijo ________________________________________ Nombre de la Escuela del Hijo ________________________________________ Nombre del Maestro Principal del Hijo 35 Appendix F - Parent Climate Survey Parent Climate Survey As a parent or guardian of a student in a Mission Possible school, we would like for you to complete this survey so that we can learn more about your experiences with the school. Your responses will remain confidential. The results of the survey will be summarized in terms of group response. Due to the anonymity of the survey, there are no risks involved. The survey should take no more than 20 minutes to complete. Thank you, in advance, for your feedback! Please fill circle completely that represents your answer. Be sure to fill in each circle completely. Background Information A. School Name Elementary Schools Bessemer Cone Fairview Falkener Foust Gillespie Park Hampton Kirkman Park Oak Hill Parkview Union Hill Wiley Washington Middle Schools Allen Aycock Ferndale Hairston Jackson Penn Griffin Welborn High Schools O O O O O O O O O O O O O O O O O O O O A&T Middle College Academy at Smith Andrews Middle College at Bennett Dudley High Point Central Smith O O O O O O O B. Race/Ethnicity: African American American Indian/Native Alaskan Asian/Pacific Islander Hispanic White Other C. Gender O O O O O O Male Female O O Please answer the following questions to the best of your ability by agreeing or disagreeing with the statements. Fill in the circle that best represents your answer. 5-Strongly Agree (SA), 4-Agree (A), 3- Unsure (U), 2-Disagree (D), 1-Strongly Disagree (SD) SA A U D SD 1. 2. 3. 4. The school is safe. The school is kept clean. The school is overcrowded to the degree that it affects learning. The school maintains high academic standards. O O O O O O O O O O O O O O O O O O O O 36 SA 5. 6. 7. 8. 9. The school uses adequate disciplinary measures dealing with disruptive students. The school serves lunches that are nutritious. The school serves lunches that taste good. Teachers are friendly and easy to talk to. Teachers make learning interesting. A U D SD O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O 10. Teachers make learning relevant. 11. Teachers motivate students to learn. 12. Teachers take an interest in students’ educational future. 13. Teachers are knowledgeable and understand their subject matter. 14. Teachers assign meaningful homework that helps students learn. 15. Teachers do their best to include me in matters directly affecting my child’s progress in school. 16. The school teaches students the basic academic skills in reading. 17. The school teaches students the basic academic skills in math. 18. The school teaches students how to speak correctly in English. 19. The school teaches students how to write correctly in English. 20. The school teaches how to solve problems in science. 21. The school teaches students to think critically. 22. The school teaches students to develop good study and work habits. 23. The school teaches students to get along with different kinds of people. 24. The school is free of violence. 25. The school is free of gang activity. 26. The school is free of substance abuse and drugs. 27. The principal does an effective job running my child’s school. 28. The principal is available. 29. The principal is easy to talk to. 30. The assistant principals are effective administrators. 31. School staff try to help students with personal problems. 32. School staff try to help students with academic problems. 33. School staff respond to my concerns. 34. My child is getting a good education at this school. 35. The overall climate or atmosphere at my child’s school is positive and helps my child learn. What overall grade would you give to your child’s school? A A- B B- C C- D D- F O O O O O O O O O 37 Appendix G - Parent Climate Survey (Spanish Version) Cuestionario para padre/madre o tutor legal Siendo usted un padre o guardián de un estudiante en una escuela que participa en el programa Misión Posible, favor de llenar este cuestionario. Nos gustaría aprender más sobre sus experiencias con la escuela. Sus respuestas son confidenciales. Los resultados del cuestionario serán al analizados como grupo. Debido al anonimato del cuestionario, usted no corre ningún riesgo al participar. Llenar el cuestionario no debe tomar más de 20 minutos. ¡Gracias por sus comentarios! Favor de llenar completamente el círculo que representa su respuesta. Información básica A. Nombre de la escuela Elementary Schools Bessemer Cone Fairview Falkener Foust Gillespie Park Hampton Kirkman Park Oak Hill Parkview Union Hill Wiley Washington B. Raza/Etnicidad Negro (no Hispano) Indígena Norteamericano/Indígena de Alaska Asiático/Isleño Pacífico Hispano Blanco Otro Middle Schools High Schools O O O O O O O O O O O O O Allen Aycock Ferndale Hairston Jackson Penn Griffin Welborn O O O O O O O A&T Middle College Academy at Smith Andrews Middle College at Bennett Dudley High Point Central Smith O O O O O O O C. Género O O O O O O Varón Hembra O O Favor de responder a las preguntas siguientes de la mejor forma posible indicando si usted está de acuerdo o no con cada frase. Favor de rellenar el círculo que representa su respuesta en la mejor forma. 5-Estoy totalmente de acuerdo, 4-Estoy de acuerdo, 3-No estoy seguro, 2-No estoy de acuerdo, 1-No estoy nada en absoluto 5 1. 2. 3. La escuela es un lugar seguro. La escuela es un lugar limpio. La escuela tiene demasiados estudiantes y esto afecta el aprendizaje de los estudiantes. 4 3 2 1 O O O O O O O O O O O O O O O 38 5 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. La escuela tiene altas normas académicas. La escuela sabe disciplinar a los estudiantes que causan interrupciones. La escuela sirve almuerzos nutritivos. La escuela sirve comida rica. Los maestros son amistosos. Los maestros hacen que el aprendizaje sea interesante. Los maestros hacen que el aprendizaje sea relevante. Los maestros motivan a sus estudiantes a aprender. Los maestros se interesan por el futuro educativo de sus estudiantes. Los maestros son inteligentes y tienen el conocimiento necesario para enseñar sus clases. Las tareas que los maestras dan son útiles y ayudan a los estudiantes a aprender. Los maestros me invitan a participar en asuntos que afectan directamente al progreso escolar de mi hijo. La escuela les enseña a sus estudiantes las habilidades básicas de la lectura. La escuela les enseña a sus estudiantes las habilidades básicas de las matemáticas. La escuela les enseña a sus estudiantes a hablar correctamente en inglés. La escuela les enseña a sus estudiantes a escribir correctamente en inglés. La escuela les enseña a sus estudiantes a solucionar problemas científicos. La escuela les enseña a sus estudiantes a pensar analíticamente. La escuela les enseña a sus estudiantes a tener buenos hábitos de estudiar y trabajar. La escuela les enseña a sus estudiantes a tener buenas relaciones con personas que son diferentes de ellos. No hay violencia en la escuela. No hay bandas (“gangs”) en la escuela. No hay drogas ni abuso de ninguna otra sustancia en la escuela. El director de la escuela (“the principal”) la dirige bien. El director está disponible cuando hay necesidad de comunicarse con él/ella. El director es amigable. Los subdirectores (“the assistant principals”) dirigen la escuela bien. El personal de la escuela trata de ayudar a los estudiantes con sus problemas personales. El personal de la escuela trata de ayudar a los estudiantes con sus problemas académicos. El personal de la escuela presta atención a mis preocupaciones. Mi hijo recibe una buena educación en esta escuela. La atmósfera general de la escuela es positiva y ayuda a mi hijo a aprender. 4 3 2 1 O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O ¿En términos generales, qué nota le daría usted a la escuela de su hijo? (A=la nota mejor; F=la nota peor) A A- B B- C C- D D- F O O O O O O O O O 39 Appendix H - Student Climate Survey Student Climate Survey As a student in a Mission Possible school, we would like for you to complete this survey so that we can learn more about your experiences with the school. Your responses will remain confidential. The results of the survey will be summarized in terms of group response. Due to the anonymity of the survey, there are no risks involved. The survey should take no more than 20 minutes to complete. Thank you, in advance, for your feedback! Please fill circle completely that represents your answer. Be sure to fill in each circle completely. Background Information A. School Name Elementary Schools Bessemer Cone Fairview Falkener Foust Gillespie Park Hampton Kirkman Park Oak Hill Parkview Union Hill Wiley Washington O O O O O O O O O O O O O Middle Schools Allen Aycock Ferndale Hairston Jackson Penn Griffin Welborn O O O O O O O High Schools A&T Middle College Academy at Smith Andrews Middle College at Bennett Dudley High Point Central Smith O O O O O O O B. Race/Ethnicity: African American American Indian/Native Alaskan Asian/Pacific Islander Hispanic White Other O O O O O O C. Gender Male Female O O Please answer the following questions to the best of your ability by agreeing or disagreeing with the statements. Fill in the circle that best represents your answer. 5-Strongly Agree (SA), 4-Agree (A), 3-Unsure (U), 2-Disagree (D), 1-Strongly Disagree (SD) SA A O O U O O D O O SD O O 1. 2. I feel safe at my school. The school building is kept clean. O O 40 SA 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. The students in my school usually follow school rules. My teachers require that I work very hard for the grades I get. My school has enough books and equipment to help me learn. The school serves lunches that are nutritious. The school serves lunches that taste good. Most of my teachers are friendly and easy to talk to. Most of my teachers make learning interesting. Most of my teachers make learning relevant. Most of my teachers make me to want to learn. My teachers know a lot about the subjects they teach. My teachers assign meaningful homework that helps me learn. My teachers are interested in how I do in the future. My teachers let me know how I am doing on my school work. Violence is a problem at my school. Gangs are a problem at my school. Many students are being distracted by alcohol. Many students are being distracted by drugs. The assistant principal is easy to talk to. The principal is easy to talk to. People at my school help me with personal problems. People at my school help me with learning problems. The adults at my school care about me. My teachers teach me how to get along with different kinds of people. I like coming to school. I am learning a lot at my school. A O O O O O O O O O O O O O O O O O O O O O O O O O U O O O O O O O O O O O O O O O O O O O O O O O O O D O O O O O O O O O O O O O O O O O O O O O O O O O SD O O O O O O O O O O O O O O O O O O O O O O O O O D- F O O O O O O O O O O O O O O O O O O O O O O O O O O O A O A- B O O What overall grade would you give your school? B- C O O C- D O O 41 Appendix I - Student Assent Form Student Assent Form We are doing a survey to try to see how you feel about your school. We are interested in what you think because you attend this school. We will ask students the same questions next year to see if students think any changes have been made to the school environment. If you agree to be part of our evaluation, we are going to have you complete a survey that relates to the school environment. The survey should take 20 minutes to fill out. We want you to answer honestly. You can ask questions that you might have about this survey at any time. Also, if you decide at any time not to finish, you may stop whenever you want. There are no right or wrong answers because this is not a test. Signing this paper means that you have read this or had it read to you and that you want to be part of this evaluation. If you don’t want to be in the evaluation and complete the survey, don’t sign the paper. Completing the survey is up to you, and no one will be mad if you don’t sign this paper or even if you change your mind later. Signature of Participant ____________________ Date _____________ Signature of Investigator ____________________ Date ____________ 42 Appendix J - Directions for Distributing Parent Surveys and Suggested Sampling Plan Logistics for Climate Surveys-Handed out During Meeting with Coordinators Packets that contain a letter to parents, a Parent Climate Survey, parent consent form, and sealable return envelope were prepared by SERVE to be administered in each Mission Possible School. The packets need to be handed out at each school on Monday, September 24. The attached sampling plan should be used as a guide when handing out the packets. Packet contains: Introductory letter (English on one side and Spanish on the other side) with instructions to parents 2 IRB consent forms in Spanish for Spanish speakers (both should be signed by the parent and the parent should keep one) 2 IRB consent forms in English for English speakers (both should be signed by the parent and the parent should keep one) Parent Climate Survey in Spanish Parent Climate Survey in English Envelope addressed to Holli Gottschall at SERVE If parent notifies the teacher that they do not speak English or Spanish, SERVE should be notified so that the survey and consent form can be translated into the parent’s language. Coordinators, please inform teachers to inform students to return this packet by Friday, September 28. Please send a large envelope or package containing all sealed envelopes to Tonia Purvis at Central Office via interoffice mail. Indicate which school they are coming from. Someone from SERVE will pick up the envelopes from Tonia. All envelopes from parents should remain sealed. SERVE will open the envelopes to determine which parents gave permission for their child to be surveyed. SERVE will assemble student surveys and assent forms and give back to the Coordinator via interoffice mail. Each assent form will contain name of student and name of homeroom teacher so that it will be easier to hand out surveys. Coordinator will coordinate with teacher in order to hand out student assent form and survey to appropriate children. Only students who have permission from their parents to participate and sign an assent form should be given a survey. If a student does not sign the assent form, they should not fill out a survey. Look for student surveys to be delivered to you in mid November. For future questions or assistance, contact Holli Gottschall Bayonas at SERVE hgottsch@serve.org or 315-7438 43 Suggested Sampling Plan Name of School MiddColl NCAT MiddColl Benn Washington El Gillespie El Kirkman Park El Wiley El Foust El Hampton El Oak Hill El Union Hill El Bessemer El Parkview El Cone Elem Fairview El Falkener El Ferndale Mid Jackson Mid Aycock Mid Penn Griffin Mid Hairston Mid Welborn Mid Allen Mid Andrews High HP Central High Dudley High Smith High Academy at Smith Southern Eastern Approximate # of Students 84 88 245 290 291 306 330 334 354 387 402 415 461 463 561 608 631 649 710 753 768 848 1115 1307 1344 1572 161 # of students and parents sampled* 100 88 20 40 20 40 60 40 40 60 60 60 75 100 100 200 200 200 200 200 200 200 200 200 200 200 161 200 200 Suggested Sample All students up to 100 All students up to 88 1 class 5th 2 classes 5th 1 class 5th 2 classes 5th 3 classes 5th 2 classes 5th 2 classes 5th 3 classes 5th 3 classes 5th 3 classes 5th 3 classes 5th 4 classes 5th 5 classes 5th 5 7th, 5 8th 5 7th, 5 8th 5 7th, 5 8th 5 7th, 5 8th 5 7th, 5 8th 5 7th, 5 8th 4 7th, 4 8th 4 11th, 4 12th 4 11th, 4 12th 4 11th, 4 12th 4 11th, 4 12th All students up to 161 5 7th, 5 8th 5 7th, 5 8th Coordinator Carolyn Daniel Keotia Smitherman Patricia Taylor Lana McCallum Annemarie Fiore Nikeita Constantine Tynica Lewis Jill Huddle Kristen Baker Don Flowers Jed O’Donnell Joanna Soza Johnita Readus Melissa Clowe Lauren Waterman Katina Little Pam Ford-Brown Janie Copple Gordon Thompson Adelaide Walker Tiffany Dewar MichelleDrew-Hurling NaTisha Peacock Sylvia Cates Marcus Gause Donna Matthews Patricia Via Kashanda Ray Charlie Abourjilie *Number of classrooms is based on a class size of 20 or 25. Class size data taken from 2005-06 average class size data at www.ncreportcard.org If class sizes are larger than the average, please adjust accordingly when handing out packets. For example, if class size is larger than the average reported in 05-06, there may only be enough surveys for 7 classes instead of 8, or 9 instead of 10. 44 Appendix K - List of Coordinators for Fall 2007 Climate Survey Distribution Meeting, Wed., September 19, 2007 ELEMENTARY Representative Name Jed O’Donnell Johnita Readus Melissa Clowe Lauren Waterman Tynica Lewis Lana McCallum Jill Huddle Annemarie Fiore Kristen Baker Joanna Soza Patricia Taylor Nikeita Constantine Don Flowers MIDDLE Allen Michelle Drew-Hurling Aycock Janie Copple Ferndale Katina Little Jackson Pam Ford-Brown Hairston Adelaide Walker Penn-Griffin Gordon Thompson Welborn Tiffany Dewar HIGH A&T Middle College Carolyn Daniel Andrews NaTisha Peacock Bennett Middle College Keotia Smitherman Dudley Marcus Gause Eastern Charlie Abourjilie High Point Central Sylvia Cates Southern Kashanda Ray Smith Melanie Trisch The Academy at Smith Patricia Via SCHOOL Bessemer Cone Fairview Falkener Foust Gillespie Park Hampton Academy Kirkman Park Oak Hill Parkview Washington Wiley Union Hill # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Role Curr Fac Curr Fac Counselor Counselor Curr Fac Counselor Curr Fac Teacher Counselor Curr Fac Counselor Counselor Counselor Counselor Curr Fac Counselor Curr Fac Counselor Counselor Counselor Plato Teacher Counselor Counselor Asst Prin Curr Fac Curr Fac Counselor Counselor Counselor 45 Appendix L - Staff Climate Summary Results - Fall 2007 All 29 Schools Combined (N = 1,086) Introduction The summary results contain a shortened version of the full results, with notable items highlighted for your viewing convenience. Included in the summary are results to climate questions 1-46 and the open-ended comments 9 for questions 47-50. Question H, related to attrition, was deemed an important question to also include on the summary. Results for questions A through I can be found in the Individual School Report (keysurvey.com). To control for response agreement, negatively worded items were included in the survey. These items are listed below the positively worded items. Viewing Notes Means are calculated based on a 5-point Likert scale. The scale goes from 5=Strongly Agree to 1=Strongly Disagree. When interpreting the results for positively worded questions, a Mean score of less than 3.0 (midpoint of scale) indicates that this item may be notable. When interpreting the results for negatively worded questions, a Mean score greater than 3.0 may be notable. These notable items have been highlighted for your convenience. Please review all items, not just those that are highlighted. Results for a particular question will not always sum to 100% due to nonrespondents. Question H Is it your intent to return to this school next year? Yes 67.7% No 8.1% Unsure 24.3% Questions with Mean Scores Lower than 3.0 are highlighted SA=5, A=4, U=3, D=2, SD=1 Positively Worded Questions 1 2 3 4 5 6 8 9 I feel safe at my school. The school building is kept clean. The school personnel work together as a team. Administrators solve problems effectively. My students performed at grade level or above last year. Adequate disciplinary measures are used to deal with disruptive behavior. My principal represents the school in a positive manner. I devote time to remediation. Mean 3.94 3.99 3.64 3.51 3.17 3.10 4.27 4.13 3.83 3.90 4.22 4.21 2.35 3.49 St.Dev. .89 .96 1.01 1.11 1.03 1.24 .87 .73 1.04 1.03 .93 .77 1.04 1.04 SA 24% 29.8% 15.7% 16.8% 8.9% 12% 47.1% 29.4% 27.2% 29.7% 45.4% 35.3% 1.8% 12.8% A 58.3% 54.5% 55.1% 44.8% 34% 35.4% 40.8% 58.6% 47.5% 46.8% 42.1% 56.9% 17.7% 47.8% U 7.2% 3.6% 11.1% 17.1% 26.4% 15.2% 6.9% 8.1% 10.6% 10.4% 4.9% 2.3% 15% 20.7% D 8.8% 10.3% 15.1% 16% 26.8% 26.2% 3.7% 3.4% 11.8% 10.2% 5.4% 4.5% 45.8% 13.4% SD 1.7% 1.8% 3% 5.3% 3.9% 11.2% 1.5% 0.4% 3% 2.9% 2.1% 0.9% 19.6% 5.3% 10 My grade levels/content areas plan together. 11 My principal responds to my concerns. 12 My principal treats me with respect. 13 My colleagues and I share resources. 24 Students generally come to my class at the beginning of the term prepared for the grade level or courses I teach. 25 I feel satisfied concerning how my career is progressing at this 9 Note that all open-ended responses appear as written by respondents. 46 Questions with Mean Scores Lower than 3.0 are highlighted SA=5, A=4, U=3, D=2, SD=1 Positively Worded Questions school. 26 I have a feeling of job security in my present position. 27 I like working at my school. 28 Staff morale is high at my school. 30 Teacher evaluations are fair and reasonable. 31 Teacher evaluations are used to improve teacher performance. 32 In-service programs keep me informed of the latest educational strategies. 3.64 3.96 2.91 3.82 3.68 3.66 2.92 3.91 1.00 1.00 1.17 .82 .87 .92 1.06 .88 .83 1.02 .86 .77 .93 1.00 1.08 .98 16.7% 32.3% 7.7% 17.7% 15.4% 12.5% 6% 22.7% 24.3% 14.9% 38% 31.2% 0.7% 0.5% 1% 0.9% 50.8% 44.7% 29.4% 54.2% 49.8% 58.2% 27.9% 56% 57.9% 54.6% 47.1% 54.5% 9.6% 25.2% 31.8% 27.4% 18.3% 13.9% 22.5% 22.3% 24.8% 15.4% 26.8% 12% 10.3% 12.9% 8% 9.8% 14.1% 25.1% 14.1% 23.3% 10.5% 6% 27.7% 4.7% 9.1% 11.4% 31.8% 7.8% 6.6% 13.7% 6.1% 3.9% 47% 36.9% 40.2% 40.5% 3.6% 3.2% 12.8% 1.1% 0.9% 2.6% 7.6% 1.4% 0.9% 3.9% 0.8% 0.6% 28.7% 12.3% 12.9% 8% Mean St.Dev. SA A U D SD 34 This school is free of racism. 35 I am adequately trained to differentiate instruction. 36 37 I am prepared to have students with different learning levels in my 3.97 classroom. The overall climate or atmosphere at my school is positive for learning. I am adequately prepared to use cooperative learning in the classroom. 3.62 4.16 4.12 2.06 2.64 2.68 2.73 D=Disagree, U=Unsure, 38 We are all actively trying to make this a better school. 39 40 Parental participation is high. 43 Students in my school complete homework assignments. 44 My students come to class prepared. 45 Parents respond to material sent home to them. *SA=Strongly Agree, A=Agree, SD=Strongly Disagree Questions with Mean Scores Greater than 3.0 are highlighted SA=5, A=4, U=3, D=2, SD=1 Negatively Worded Questions 7 My principal is not an effective administrator. 14 My principal is not supportive of teachers. 15 I am limited by too many students in each class. 16 I am limited by student deficiencies in basic academic skills. 17 I am limited by lack of concern/support from parents. 18 I am limited by lack of concern/support from the principal. 19 I am limited by school violence. 20 21 I am limited by insufficient resources (e.g., funds, books, equipment, supplies, etc.). I am limited by lack of concern/support from the district administration. Mean 2.21 2.07 2.74 3.60 3.58 2.07 2.12 3.04 3.07 3.00 2.76 3.55 2.04 St.Dev. 1.15 1.06 1.25 1.15 1.12 .98 .97 1.30 1.10 1.19 1.16 1.21 .88 SA 5.6% 3.1% 11.7% 23.6% 21.5% 2.4% 2.2% 14.9% 11.5% 10.2% 7.9% 24.3% 1.1% A 10.6% 8.9% 20.2% 39.6% 40.3% 8.7% 10.4% 31.3% 24.9% 29.1% 20.6% 38.5% 6.3% U 15.8% 14.5% 11.3% 12% 12.7% 10.9% 9% 9% 28.8% 23.5% 27.2% 8.5% 14.5% D 36.2% 37.7% 43.6% 21.5% 23.4% 48.9% 54.6% 34% 29.1% 25.6% 30.2% 24.2% 50.9% SD 31.9% 35.8% 13.1% 3.3% 2.1% 29% 23.9% 10.8% 5.7% 11.7% 14.2% 4.5% 27.2% 22 Student gang activity is a problem at my school. 23 Drugs are a problem at my school. I frequently feel overloaded or overwhelmed while working at my 29 school. 33 Children are not receiving a good education here. 47 Questions with Mean Scores Greater than 3.0 are highlighted SA=5, A=4, U=3, D=2, SD=1 Negatively Worded Questions 41 I hear racist comments from staff. 42 I hear racist comments from students. *SA=Strongly Agree, A=Agree, Question 46 What overall grade would you give to this school? U=Unsure, Mean 2.02 3.12 D=Disagree, A 7.4% A12.3% St.Dev. 1.00 1.18 SA 1.2% 9.3% A 11.2% 41.1% U 8.3% 9.4% D 46% 32.9% SD 33.3% 7.3% SD=Strongly Disagree B 24.3% B14.8% C 20% C11.5% D 6.4% D2.2% 1% F 48 Appendix M - Parent Climate Summary Results - Fall 2007 Parent Survey-All Elementary Schools Combined (N = 163) All Elementary Schools - N=163 Question 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 The school is safe. The school is kept clean. The school is overcrowded to the degree that it affects learning. The school maintains high academic standards. The school uses adequate disciplinary measures dealing with disruptive students. The school serves lunches that are nutritious. The school serves lunches that taste good. Teachers are friendly and easy to talk to. Teachers make learning interesting. Teachers make learning relevant. Teachers motivate students to learn. Teachers take an interest in students’ educational future. Teachers are knowledgeable and understand their subject matter. Teachers assign meaningful homework that helps students learn. Teachers do their best to include me in matters directly affecting my child’s progress in school. The school teaches students the basic academic skills in reading. The school teaches students the basic academic skills in math. The school teaches students how to speak correctly in English. The school teaches students how to write correctly in English. The school teaches how to solve problems in science. The school teaches students to think critically. The school teaches students to develop good study and work habits. The school teaches students to get along with different kinds of people. The school is free of violence. The school is free of gang activity. The school is free of substance abuse and drugs. The principal does an effective job running my child’s school. The principal is available. Surveys sent = 975 Survey response rate = 16.7% SA 63(38.7%) 75(46.0%) 19(11.7%) 32(19.6%) 40(24.5%) 52(31.9%) 37(22.7%) 69(42.3%) 56(34.4%) 45(27.6%) 62(38.0%) 69(42.3%) 53(32.5%) 60(36.8%) 64(39.3%) 61(37.4%) 64(39.3%) 62(38.0%) 64(39.3%) 48(29.4%) 46(28.2%) 61(37.4%) 62(38.0%) 41(25.2%) 60(36.8%) 66(40.5%) 71(43.6%) 63(38.7%) A 80(49.1%) 73(44.8%) 18(11.0%) 69(42.3%) 71(43.6%) 79(48.5%) 69(42.3%) 82(50.3%) 78(47.9%) 94(57.7%) 75(46.0%) 73(44.8%) 87(53.4%) 81(49.7%) 82(50.3%) 90(55.2%) 91(55.8%) 81(49.7%) 80(49.1%) 80(49.1%) 81(49.7%) 77(47.2%) 78(47.9%) 77(47.2%) 62(38.0%) 67(41.1%) 66(40.5%) 77(47.2%) U 13(8.0%) 10(6.1%) 38(23.3%) 43(26.4%) 30(18.4%) 24(14.7%) 37(22.7%) 10(6.1%) 26(16.0%) 17(10.4%) 21(12.9%) 19(11.7%) 21(12.9%) 8(4.9%) 10(6.1%) 7(4.3%) 6(3.7%) 18(11.0%) 13(8.0%) 32(19.6%) 29(17.8%) 20(12.3%) 19(11.7%) 26(16.0%) 34(20.9%) 24(14.7%) 18(11.0%) 20(12.3%) D 2(1.2%) 5(3.1%) 55(33.7%) 16(9.8%) 13(8.0%) 5(3.1%) 13(8.0%) 2(1.2%) 1(0.6%) 2(1.2%) 3(1.8%) 1(0.6%) 0 9(5.5%) 5(3.1%) 3(1.8%) 2(1.2%) 1(0.6%) 3(1.8%) 3(1.8%) 5(3.1%) 3(1.8%) 0 12(7.4%) 2(1.2%) 0 2(1.2%) 2(1.2%) 0 31(18.0%) 1(0.6%) 6(3.7%) 1(0.6%) 5(3.1%) 0 1(0.6%) 2(1.2%) 0 0 0 3(1.8%) 0 0 0 1(0.6%) 1(0.6%) 0 1(0.6%) 1(0.6%) 2(1.2%) 6(3.7%) 5(3.1%) 6(3.7%) 3(1.8%) 1(0.6%) SD 2(1.2%) Blank 3(1.8%) 0 2(1.2%) 2(1.2%) 3(1.8%) 2(1.2%) 2(1.2%) 0 1(0.6%) 3(1.8%) 2(1.2%) 1(0.6%) 2(1.2%) 2(1.2%) 2(1.2%) 2(1.2%) 0 0 2(1.2%) 0 1(0.6%) 1(0.6%) 2(1.2%) 1(0.6%) 0 0 3(1.8%) 0 49 All Elementary Schools - N=163 Question 29 30 31 32 33 34 35 36 The principal is easy to talk to. Surveys sent = 975 Survey response rate = 16.7% SA 76(46.6%) 40(24.5%) 47(28.8%) 58(35.6%) 52(31.9%) 69(42.3%) 68(41.7%) A 58(35.6%) 66(40.5%) 74(45.4%) 72(44.2%) 86(52.8%) 77(47.2%) 79(48.5%) U 25(15.3%) 50(30.7%) 36(22.1%) 28(17.2%) 17(10.4%) 11(6.7%) 8(4.9%) D 1(0.6%) 2(1.2%) 4(2.5%) 3(1.8%) 7(4.3%) 5(3.1%) 7(4.3%) 0 1(0.6%) 1(0.6%) 0 0 0 SD 1(0.6%) Blank 2(1.2%) 5(3.1%) 1(0.6%) 1(0.6%) 1(0.6%) 1(0.6%) 1(0.6%) The assistant principals are effective administrators. School staff try to help students with personal problems. School staff try to help students with academic problems. School staff respond to my concerns. My child is getting a good education at this school. The overall climate or atmosphere at my child’s school is positive and helps my child learn. What overall grade would you give to your child’s school? 62 - A, 33 - A-, 31 - B, 11 - B-, 6 - C, 7 - C-, 1 - D-, 12 - Blank *SA=Strongly Agree, A= Agree, U=Unsure, D=Disagree, SD=Strongly Disagree 50 Parent Survey-All Middle Schools Combined (N = 260) All Middle Schools - N=260 Question 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 The school is safe. The school is kept clean. The school is overcrowded to the degree that it affects learning. The school maintains high academic standards. The school uses adequate disciplinary measures dealing with disruptive students. The school serves lunches that are nutritious. The school serves lunches that taste good. Teachers are friendly and easy to talk to. Teachers make learning interesting. Teachers make learning relevant. Teachers motivate students to learn. Teachers take an interest in students’ educational future. Teachers are knowledgeable and understand their subject matter. Teachers assign meaningful homework that helps students learn. Teachers do their best to include me in matters directly affecting my child’s progress in school. The school teaches students the basic academic skills in reading. The school teaches students the basic academic skills in math. The school teaches students how to speak correctly in English. The school teaches students how to write correctly in English. The school teaches how to solve problems in science. The school teaches students to think critically. The school teaches students to develop good study and work habits. The school teaches students to get along with different kinds of people. The school is free of violence. The school is free of gang activity. The school is free of substance abuse and drugs. The principal does an effective job running my child’s school. The principal is available. Surveys sent = 1,500 SA 39(15.0%) 52(20.0%) 15(5.8%) 37(14.2%) 34(13.1%) 44(16.9%) 23(8.8%) 65(25.0%) 39(15.0%) 33(12.7%) 51(19.6%) 48(18.5%) 50(19.2%) 44(16.9%) 61(23.5%) 55(21.2%) 60(23.1%) 53(20.4%) 53(20.4%) 35(13.5%) 45(17.3%) 55(21.2%) 47(18.1%) 12(4.6%) 15(5.8%) 24(9.2%) 69(26.5%) 70(26.9%) Survey response rate = 17.3% A 144(55.4%) 162(62.3%) 34(13.1%) 103(39.6%) 123(47.3%) 126(48.5%) 81(31.2%) 146(56.2%) 156(60.0%) 162(62.3%) 152(58.5%) 139(53.5%) 163(62.7%) 161(61.9%) 129(49.6%) 167(64.2%) 169(65.0%) 141(54.2%) 151(58.1%) 146(56.2%) 139(53.5%) 136(52.3%) 137(52.7%) 63(24.2%) 55(21.2%) 66(25.4%) 127(48.8%) 114(43.8%) U 62(23.8%) 34(13.1%) 77(29.6%) 77(29.6%) 67(25.8%) 57(21.9% 91(35.0%) 33(12.7%) 50(19.2%) 54(20.8%) 41(15.8%) 53(20.4%) 42(16.2%) 30(11.5%) 31(11.9%) 32(12.3%) 25(9.6%) 43(16.5%) 37(14.2%) 69(26.5%) 55(21.2%) 47(18.1%) 55(21.2%) 85(32.7%) 108(41.5%) 117(45.0%) 48(18.5%) 53(20.4%) D 12(4.6%) 7(2.7%) 102(39.2%) 31(11.9%) 22(8.5%) 23(8.8%) 48(18.5%) 12(4.6%) 11(4.2%) 6(2.3%) 12(4.6%) 13(5.0%) 5(1.9%) 17(6.5%) 26(10.0%) 3(1.2%) 3(1.2%) 15(5.8%) 12(4.6%) 7(2.7%) 16(6.2%) 10(3.8%) 11(4.2%) 67(25.8%) 45(17.3%) 30(11.5%) 11(4.2%) 17(6.5%) 5(1.9%) 12(4.6%) 1(0.4%) 2(0.8%) 4(1.5%) 4(1.5%) 2(0.8%) 3(1.2%) 8(3.1%) 7(2.7%) 28(10.8%) 34(13.1%) 19(7.3%) 3(1.2%) 3(1.2%) 3(1.2%) 1(0.4%) 2(0.8%) 1(0.4%) 4(1.5%) 3(1.2%) 1(0.4%) 2(0.8%) 4(1.5%) 3(1.2%) 5(1.9%) 3(1.2%) 4(1.5%) 2(0.8%) 3(1.2%) SD 1(0.4%) 4(1.5%) 30(11.5%) 10(3.8%) 10(3.8%) 9(3.5%) 17(6.5%) 3(1.2%) 2(0.8%) 2(0.8%) 3(1.2%) 4(1.5%) 1(0.4%) 2(0.8%) 3(1.2%) 1(0.4%) 3(1.2%) Blank 2(0.8%) 1(0.4%) 2(0.8%) 2(0.8%) 4(1.5%) 1(0.4%) 51 All Middle Schools - N=260 Question 29 30 31 32 33 34 35 36 Surveys sent = 1,500 SA Survey response rate = 17.3% A 109(41.9%) 128(49.2%) 110(42.3%) 142(54.6%) 141(54.2%) 139(53.5%) 137(52.7%) U 64(24.6%) 67(25.8%) 90(34.6%) 61(23.5%) 46(17.7%) 49(18.8%) 54(20.8%) D 7(2.7%) 11(4.2%) 12(4.6%) 9(3.5%) 15(5.8%) 11(4.2%) 14(5.4%) SD 3(1.2%) 5(1.9%) 4(1.5%) 1(0.4%) 8(3.1%) 4(1.5%) 5(1.9%) Blank 2(0.8%) 3(1.2%) 1(0.4%) 1(0.4%) 1(0.4%) 3(1.2%) 2(0.8%) The principal is easy to talk to. 75(28.8%) The assistant principals are effective 46(17.7%) administrators. School staff try to help students with 43(16.5%) personal problems. School staff try to help students with 46(17.7%) academic problems. School staff respond to my concerns. 49(18.8%) My child is getting a good education at this 54(20.8%) school. The overall climate or atmosphere at my 48(18.5%) child’s school is positive and helps my child learn. What overall grade would you give to your child’s school? 47 - A, 51 - A-, 64 - B, 29 - B-, 26 - C, 19 - C-, 9 - D, 2 - D-, 3 - F, 10 - Blank *SA=Strongly Agree, A= Agree, U=Unsure, D=Disagree, SD=Strongly Disagree 52 Parent Survey-All High Schools Combined (N = 159) All High Schools - N=159 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 The school is safe. The school is kept clean. The school is overcrowded to the degree that it affects learning. The school maintains high academic standards. The school uses adequate disciplinary measures dealing with disruptive students. The school serves lunches that are nutritious. The school serves lunches that taste good. Teachers are friendly and easy to talk to. Teachers make learning interesting. Teachers make learning relevant. Teachers motivate students to learn. Teachers take an interest in students’ educational future. Teachers are knowledgeable and understand their subject matter. Teachers assign meaningful homework that helps students learn. Teachers do their best to include me in matters directly affecting my child’s progress in school. The school teaches students the basic academic skills in reading. The school teaches students the basic academic skills in math. The school teaches students how to speak correctly in English. The school teaches students how to write correctly in English. The school teaches how to solve problems in science. The school teaches students to think critically. The school teaches students to develop good study and work habits. The school teaches students to get along with different kinds of people. The school is free of violence. The school is free of gang activity. The school is free of substance abuse and drugs. The principal does an effective job running my child’s school. The principal is available. The principal is easy to talk to. The assistant principals are effective administrators. Surveys sent = 1,849 Survey response rate = 8.6% SA 17(10.7%) 26(16.4%) 18(11.3%) 34(21.4%) 21(13.2%) 12(7.5%) 10(6.3%) 36(22.6%) 16(10.1%) 19(11.9%) 25(15.7%) 30(18.9%) 27(17.0%) 21(13.2%) 31(19.5%) 24(15.1%) 27(17.0%) 28(17.6%) 27(17.0%) 26(16.4%) 20(12.6%) 20(12.6%) 24(15.1%) 12(7.5%) 12(7.5%) 15(9.4%) 48(30.2%) 50(31.4%) 47(29.6%) 23(14.5%) A 82(51.6%) 86(54.1%) 39(24.5%) 73(45.9%) 77(48.4%) 47(29.6%) 30(18.9%) 88(55.3%) 82(51.6%) 83(52.2%) 81(50.9%) 82(51.6%) 96(60.4%) 88(55.3%) 81(50.9%) 107(67.3%) 108(67.9%) 82(51.6%) 91(57.2%) 87(54.7%) 86(54.1%) 84(52.8%) 72(45.3%) 21(13.2%) 15(9.4%) 20(12.6%) 70(44.0%) 61(38.4%) 63(39.6%) 72(45.3%) U 38(23.9%) 25(15.7%) 26(16.4%) 33(20.8%) 27(17.0%) 56(35.2%) 64(40.3%) 20(12.6%) 43(27.0%) 46(28.9%) 38(23.9%) 38(23.9%) 31(19.5%) 28(17.6%) 23(14.5%) 20(12.6%) 17(10.7%) 30(18.9%) 25(15.7%) 37(23.3%) 36(22.6%) 28(17.6%) 40(25.2%) 27(17.0%) 35(22.0%) 31(19.5%) 21(13.2%) 26(16.4%) 29(18.2%) 44(27.7%) D 20(12.6%) 20(12.6%) 56(35.2%) 13(8.2%) 17(10.7%) 29(18.2%) 27(17.0%) 10(6.3%) 13(8.2%) 8(5.0%) 11(6.9%) 8(5.0%) 3(1.9%) 19(11.9%) 15(9.4%) 4(2.5%) 1(0.6%) 11(6.9%) 9(5.7%) 6(3.8%) 12(7.5%) 22(13.8%) 15(9.4%) 63(39.6%) 56(35.2%) 57(35.8%) 11(6.9%) 15(9.4%) 12(7.5%) 12(7.5%) SD 1(0.6%) 2(1.3%) 20(12.6%) 3(1.9%) 16(10.1%) 14(8.8%) 26(16.4%) 3(1.9%) 3(1.9%) 1(0.6%) 2(1.3%) 0 0 1(0.6%) 8(5.0%) 2(1.3%) 4(2.5%) 6(3.8%) 4(2.5%) 0 2(1.3%) 2(1.3%) 4(2.5%) 33(20.8%) 37(23.3%) 34(21.4%) 8(5.0%) 6(3.8%) 7(4.4%) 4(2.5%) Blank 1(0.6%) 0 0 3(1.9%) 1(0.6%) 1(0.6%) 2(1.3%) 2(1.3%) 2(1.3%) 2(1.3%) 2(1.3%) 1(0.6%) 2(1.3%) 2(1.3%) 1(0.6%) 2(1.3%) 2(1.3%) 2(1.3%) 3(1.9%) 3(1.9%) 3(1.9%) 3(1.9%) 4(2.5%) 3(1.9%) 4(2.5%) 2(1.3%) 1(0.6%) 1(0.6%) 1(0.6%) 4(2.5%) Question 53 All High Schools - N=159 31 32 33 34 35 36 Surveys sent = 1,849 Survey response rate = 8.6% SA 21(13.2%) 27(17.0%) 27(17.0%) 29(18.2%) 23(14.5%) A 67(42.1%) 99(62.3%) 93(58.5%) 97(61.0%) 83(52.2%) U 52(32.7%) 23(14.5%) 20(12.6%) 25(15.7%) 38(23.9%) D 12(7.5%) 8(5.0%) 12(7.5%) 6(3.8%) 9(5.7%) SD 4(2.5%) 1(0.6%) 5(3.1%) 0 4(2.5%) Blank 3(1.9%) 1(0.6%) 2(1.3%) 2(1.3%) 2(1.3%) Question School staff try to help students with personal problems. School staff try to help students with academic problems. School staff respond to my concerns. My child is getting a good education at this school. The overall climate or atmosphere at my child’s school is positive and helps my child learn. What overall grade would you give to your child’s school? 22 - A, 20 - A-, 45 - B, 25 - B-, 25 - C, 6 - C-, 5 - D, 4 - D-, 7 - Blank SA=Strongly Agree, A= Agree, U=Unsure, D=Disagree, SD=Strongly Disagree 54 Appendix N - Student Climate Summary Results - Fall 2007 Student Survey-All Elementary Schools Combined (N = 73) All Elementary Schools - N=73 Question 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 I feel safe at my school. The school building is kept clean. The students in my school usually follow school rules. My teachers require that I work very hard for the grades I get. My school has enough books and equipment to help me learn. The school serves lunches that are nutritious. The school serves lunches that taste good. Most of my teachers are friendly and easy to talk to. Most of my teachers make learning interesting. Most of my teachers make learning relevant. Most of my teachers make me to want to learn. My teachers know a lot about the subjects they teach. My teachers assign meaningful homework that helps me learn. My teachers are interested in how I do in the future. My teachers let me know how I am doing on my school work. Violence is a problem at my school. Gangs are a problem at my school. Many students are being distracted by alcohol. Many students are being distracted by drugs. The assistant principal is easy to talk to. The principal is easy to talk to. People at my school help me with personal problems. People at my school help me with learning problems. The adults at my school care about me. My teachers teach me how to get along with different kinds of people. I like coming to school. I am learning a lot at my school. What overall grade would you give to your child’s school? 23 - A, 18 - A-, 15 - B, 6 - B-, 3 - C, 2 - C-, 2 - D, 1 - F, 3 - Blank Surveys sent = 115 Survey response rate = 63.5% SA 38(52.1%) 21(28.8%) 10(13.7%) 54(74.0%) 43(58.9%) 29(39.7%) 24(32.9%) 38(52.1%) 33(45.2%) 47(64.4%) 47(64.4%) 52(71.2%) 39(53.4%) 41(56.2%) 45(61.6%) 11(15.1%) 6(8.2%) 4(5.5%) 6(8.2%) 29(39.7%)\ 37(50.7%) 28(38.4%) 33(45.2%) 40(54.8%) 43(58.9%) 33(45.2%) 49(67.1%) A 25(34.2%) 29(39.7%)\ 22(30.1%) 15(20.5%) 26(35.6%) 27(37.0%) 25(34.2%) 26(35.6%) 32(43.8%) 20(27.4%) 22(30.1%) 20(27.4%) 28(38.4%) 24(32.9%) 18(24.7%) 13(17.8%) 21(28.8%) 2(2.7%) 2(2.7%) 16(21.9%) 15(20.5%) 23(31.5%) 25(34.2%) 23(31.5%) 20(27.4%) 29(39.7%)\ 17(23.3%) U 7(9.6%) 16(21.9%) 19(26.0%) 3(4.1%) 3(4.1%) 11(15.1%) 12(16.4%) 5(6.8%) 6(8.2%) 4(5.5%) 2(2.7%) 1(1.4%) 4(5.5%) 6(8.2%) 7(9.6%) 18(24.7%) 11(15.1%) 10(13.7%) 8(11.0%) 22(30.1%) 13(17.8%) 13(17.8%) 9(12.3%) 7(9.6%) 6(8.2%) 5(6.8%) 5(6.8%) D 1(1.4%) 4(5.5%) 15(20.5%) 1(1.4%) 1(1.4%) 5(6.8%) 5(6.8%) 2(2.7%) 1(1.4%) 1(1.4%) 2(2.7%) 0 1(1.4%) 2(2.7%) 1(1.4%) 14(19.2%) 10(13.7%) 8(11.0%) 9(12.3%) 3(4.1%) 2(2.7%) 6(8.2%) 2(2.7%) 1(1.4%) 3(4.1%) 3(4.1%) 1(1.4%) SD 2(2.7%) 3(4.1%) 7(9.6%) 0 0 1(1.4%) 7(9.6%) 2(2.7%) 1(1.4%) 0 0 0 0 0 1(1.4%) 16(21.9%) 25(34.2%) 48(65.8%) 46(63.0%) 0 5(6.8%) 3(4.1%) 2(2.7%) 1(1.4%) 1(1.4%) 3(4.1%) 0 Blank 0 0 0 0 0 0 0 0 0 1(1.4%) 0 0 1(1.4%) 0 1(1.4%) 1(1.4%) 0 1(1.4%) 2(2.7%) 3(4.1%) 1(1.4%) 0 2(2.7%) 1(1.4%) 0 0 1(1.4%) *SA=Strongly Agree, A= Agree, U=Unsure, D=Disagree, SD=Strongly Disagree 55 Student Survey-All Middle Schools Combined (N = 134) All Middle Schools - N=134 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 I feel safe at my school. The school building is kept clean. The students in my school usually follow school rules. My teachers require that I work very hard for the grades I get. My school has enough books and equipment to help me learn. The school serves lunches that are nutritious. The school serves lunches that taste good. Most of my teachers are friendly and easy to talk to. Most of my teachers make learning interesting. Most of my teachers make learning relevant. Most of my teachers make me to want to learn. My teachers know a lot about the subjects they teach. My teachers assign meaningful homework that helps me learn. My teachers are interested in how I do in the future. My teachers let me know how I am doing on my school work. Violence is a problem at my school. Gangs are a problem at my school. Many students are being distracted by alcohol. Many students are being distracted by drugs. The assistant principal is easy to talk to. The principal is easy to talk to. People at my school help me with personal problems. People at my school help me with learning problems. The adults at my school care about me. My teachers teach me how to get along with different kinds of people. I like coming to school. I am learning a lot at my school. What overall grade would you give to your child’s school? 15 - A, 25 - A-, 32 - B, 21 - B-, 20 - C, 9 - C-, 3 - D, 3 - D-, 6 - Blank Surveys sent = 188 Survey response rate = 71.3% SA 20(14.9%) 8(6.0%) 4(3.0%) 69(51.5%) 43(32.1%) 20(14.9%) 11(8.2%) 33(24.6%) 23(17.2%) 16(11.9%) 27(20.1%) 62(46.3%) 30(22.4%) 38(28.4%) 39(29.1%) 15(11.2%) 16(11.9%) 4(3.0%) 4(3.0%) 25(18.7%) 26(19.4%) 21(15.7%) 28(20.9%) 23(17.2%) 17(12.7%) 26(19.4%) 44(32.8%) A 71(53.0%) 51(38.1%) 20(14.9%) 52(38.8%) 52(38.8%) 47(35.1%) 44(32.8%) 62(46.3%) 70(52.2%) 71(53.0%) 53(39.6%) 55(41.0%) 67(50.0%) 50(37.3%) 65(48.5%) 34(25.4%) 30(22.4%) 8(6.0%) 13(9.7%) 37(27.6%) 46(34.3%) 44(32.8%) 61(45.5%) 54(40.3%) 51(38.1%) 55(41.0%) 64(47.8%) U 34(25.4%) 39(29.1%) 44(32.8%) 8(6.0%) 23(17.2%) 39(29.1%) 31(23.1%) 23(17.2%) 21(15.7%) 39(29.1%) 37(27.6%) 14(10.4%) 18(13.4%) 38(28.4%) 18(13.4%) 51(38.1%) 41(30.6%) 54(40.3%) 53(39.6%) 45(33.6%) 41(30.6%) 37(27.6%) 31(23.1%) 47(35.1%) 39(29.1%) 24(17.9%) 19(14.2%) D 5(3.7%) 27(20.1%) 44(32.8%) 2(1.5%) 9(6.7%) 15(11.2%) 27(20.1%) 10(7.5%) 15(11.2%) 6(4.5%) 11(8.2%) 1(0.7%) 12(9.0%) 5(3.7%) 7(5.2%) 22(16.4%) 26(19.4%) 28(20.9%) 27(20.1%) 18(13.4%) 14(10.4%) 21(15.7%) 8(6.0%) 6(4.5%) 18(13.4%) 14(10.4%) 4(3.0%) SD 4(3.0%) 8(6.0%) 22(16.4%) 2(1.5%) 5(3.7%) 12(9.0%) 21(15.7%) 5(3.7%) 4(3.0%) 0 5(3.7%) 1(0.7%) 6(4.5%) 2(1.5%) 3(2.2%) 11(8.2%) 19(14.2%) 38(28.4%) 36(26.9%) 8(6.0%) 5(3.7%) 9(6.7%) 4(3.0%) 3(2.2%) 7(5.2%) 13(9.7%) 2(1.5%) Blank 0 1(0.7%) 0 1(0.7%) 2(1.5%) 1(0.7%) 0 1(0.7%) 1(0.7%) 2(1.5%) 1(0.7%) 1(0.7%) 1(0.7%) 1(0.7%) 2(1.5%) 1(0.7%) 2(1.5%) 2(1.5%) 1(0.7%) 1(0.7%) 2(1.5%) 2(1.5%) 2(1.5%) 1(0.7%) 2(1.5%) 2(1.5%) 1(0.7%) Question *SA=Strongly Agree, A= Agree, U=Unsure, D=Disagree, SD=Strongly Disagree 56 Student Survey-All High Schools Combined (N = 36) All High Schools - N=36 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 I feel safe at my school. The school building is kept clean. The students in my school usually follow school rules. My teachers require that I work very hard for the grades I get. My school has enough books and equipment to help me learn. The school serves lunches that are nutritious. The school serves lunches that taste good. Most of my teachers are friendly and easy to talk to. Most of my teachers make learning interesting. Most of my teachers make learning relevant. Most of my teachers make me to want to learn. My teachers know a lot about the subjects they teach. My teachers assign meaningful homework that helps me learn. My teachers are interested in how I do in the future. My teachers let me know how I am doing on my school work. Violence is a problem at my school. Gangs are a problem at my school. Many students are being distracted by alcohol. Many students are being distracted by drugs. The assistant principal is easy to talk to. The principal is easy to talk to. People at my school help me with personal problems. People at my school help me with learning problems. The adults at my school care about me. My teachers teach me how to get along with different kinds of people. I like coming to school. I am learning a lot at my school. What overall grade would you give to your child’s school? 3 - A, 9 - A-, 12 - B, 4 - B-, 4 - C, 3 - C-, 1-D Surveys sent = 41 Question Survey response rate = 87.8% SA 6(16.7%) 4(11.1%) 0 16(44.4%) 7(19.4%) 2(5.6%) 1(2.8%) 14(38.9%) 7(19.4%) 6(16.7%) 7(19.4%) 21(58.3%) 10(27.8%) 10(27.8%) 12(33.3%) 6(16.7%) 5(13.9%) 1(2.8%) 3(8.3%) 6(16.7%) 13(36.1%) 7(19.4%) 4(11.1%) 6(16.7%) 5(13.9%) 11(30.6%) 14(38.9%) A 22(61.1%) 17(47.2%) 6(16.7%) 16(44.4%) 19(52.8%) 9(25.0%) 6(16.7%) 18(50.0%) 19(52.8%) 24(66.7%) 22(61.1%) 15(41.7%) 15(41.7%) 20(55.6%) 19(52.8%) 9(25.0%) 8(22.2%) 10(27.8%) 10(27.8%) 12(33.3%) 12(33.3%) 14(38.9%) 19(52.8%) 17(47.2%) 14(38.9%) 17(47.2%) 19(52.8%) U 4(11.1%) 9(25.0%) 14(38.9%) 2(5.6%) 6(16.7%) 15(41.7%) 13(36.1%) 3(8.3%) 8(22.2%) 5(13.9%) 5(13.9%) 0 9(25.0%) 5(13.9%) 3(8.3%) 13(36.1%) 13(36.1%) 12(33.3%) 15(41.7%) 15(41.7%) 6(16.7%) 10(27.8%) 9(25.0%) 12(33.3%) 13(36.1%) 5(13.9%) 2(5.6%) D 3(8.3%) 6(16.7%) 11(30.6%) 1(2.8%) 2(5.6%) 5(13.9%) 10(27.8%) 1(2.8%) 1(2.8%) 1(2.8%) 2(5.6%) 0 1(2.8%) 1(2.8%) 2(5.6%) 7(19.4%) 8(22.2%) 8(22.2%) 4(11.1%) 1(2.8%) 2(5.6%) 3(8.3%) 3(8.3%) 0 4(11.1%) 3(8.3%) 1(2.8%) 0 5(13.9%) 1(2.8%) 2(5.6%) 5(13.9%) 6(16.7%) 0 1(2.8%) 0 0 0 1(2.8%) 0 0 1(2.8%) 2(5.6%) 5(13.9%) 2(5.6%) 1(2.8%) 3(8.3%) 2(5.6%) 1(2.8%) 1(2.8%) 0 0 0 SD 1(2.8%) Blank 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2(5.6%) 1(2.8%) 0 0 0 0 0 0 0 *SA=Strongly Agree, A= Agree, U=Unsure, D=Disagree, SD=Strongly Disagree 57 Appendix O - Student Achievement-Analytical Approach Mission Possible: Guilford County Schools Teacher Incentive Program – Student Achievement-Analytical Approach Eric S. Howard, M.A. SERVE Center at UNCG Gateway University Research Park - Dixon Building 5900 Summit Avenue Browns Summit, NC 27214 (800) 755-3277 58 Guilford County Schools (GCS) have contracted with the SERVE Center at the University of North Carolina at Greensboro to conduct an independent evaluation of their Mission Possible (MP) teacher incentive program. This proposal outlines the request for individual student level data to complete the evaluation of Mission Possible impacts on student outcomes. Section 1 provides an overview of the teacher incentive programs generally and the MP program specifically. Section 2 outlines the evaluation questions for the evaluation plan of MP program. Section 3 describes the analytical approach for the quantitative evaluation of impact of teacher incentive program on student achievement and the need for student level data. And section 4 addresses the data security and storage issues. 1. Overview Since the passage of the No Child Left Behind (NCLB) Act of 2001 educational reform has had a renewed concentration on improving teacher quality. One of NCLB’s stipulations is to promote local educational agencies and schools to develop programs that: …recruit, train, and hire highly qualified teachers to reduce class size, specially in the early grades, and professional development activities carried out in accordance with Title II, that give teachers, principals, and administrators the knowledge and skills to provide students with the opportunity to meet challenging State or local academic content standards and student academic achievement standards (NCLB 2001, 115 STAT. 1781). As a result of the renewed emphasis on teacher quality, reformers have also returned to interventions designed to address teacher incentives in the form of merit or performance pay schemes. Teacher incentive systems slowly declined throughout most of the 20th century, with approximately 48% of U.S. public school districts utilizing them in 1918, to 20% in 1939 and just 4% in 1953 (Murnane & Cohen, 1986). However, by 1993 the trend had reversed with approximately 12% of U.S. public school districts utilizing some form of teacher incentive program (Ballou, 2001, 54 table 1). Currently, the U.S. Department of Education has begun making several awards through the Teacher Incentive Fund (TIF) to support school and district efforts to pilot teacher incentive pay programs. Since fall 2006 34 awards have been made nationally totaling $99 million, with GCS being one of the recipients (Podgursky & Springer, 2007, 921-927). The Mission Possible program at GCS is a five year financial incentive program that includes 30 schools (13 elementary, 7 middle and 10 high schools) from two cohorts (22 schools in cohort 1 and 8 in cohort 2) and began with the 2006-2007 academic year (Year 1). Two content areas are being targeted for incentives: English/language arts and mathematics. Those eligible for the incentives include kindergarten through second-grade teachers, third-through eighth-grade teachers of math, language arts or reading, high school math and English teachers, curriculum facilitators and principals. 10 10 See Mission Possible Overview at: http://www.guilford.k12.nc.us/depts/mission_possible/background.htm; retrieved 01/15/2008. 59 Eligible teachers are expected to participate in two structured years of professional development to qualify for the incentives. The professional development session topics include sessions on cooperative learning and “Undoing Racism” in year 1 and differentiated learning and Teacher Expectations and Student Achievement (TESA) 11 in year 2. Professional development sessions customized to the educator’s needs are undertaken in years 3-5. Educators are allowed 13 months to complete the required sessions during the year they anticipate qualifying for the incentives. 12 Financial/pay incentives are divided into two categories: recruitment/retention incentives and performance incentives. The recruitment incentives are paid annually for eligible teachers and principals, with larger incentives paid for math at middle and high schools. Performance incentives are paid annually as well and are based on the teacher’s prior year performance (expected growth) as measured by their students’ end-of-grade (EOG) or end-of-course (EOC) value-added test scores. 13 The value-added data is from the Education Value Added Assessment System (EVAAS) from the North Carolina Department of Public Instruction (NCDPI). 14 While there are concerns over the statistical reliability and robustness of these value added estimates among researchers, they remain the primary metric for evaluating most teacher incentive programs (Podgursky & Springer, 2007, 927). Participation in Mission Possible was determined by three main criteria: (1) the school’s free and/or reduced price lunch rate, (2) their Annual Yearly Progress (AYP) and ABC status, and (3) their teacher retention history. Mission Possible has three central goals: 15 • The number of Mission Possible schools that meet Adequately Yearly Progress (AYP) will increase from 19 percent to 100 percent by 2011. • The percentage of students in Mission Possible schools who pass the ABC’s end of course tests will increase from 56.2 percent (2004-05 data) to at least the state average of 74.8 percent (2004-05 data) by 2011. • The teacher turnover rate for Mission Possible schools will decrease from: 33.1 percent (2004-05 data) in elementary schools to at least the North Carolina state average of 20 percent (2004-05 data) by 2011. o 31.6 percent (2004-05 data) in middle schools to at least the North Carolina state average of 22 percent (2004-05 data) by 2011. o 37.8 percent (2004-05 data) in high schools to at least the North Carolina state average of 19 percent (2004-05 data) by 2011. The SERVE Center at the University of North Carolina at Greensboro, which also has the contract for the Regional Educational Lab for the Southeast (REL-SE) from the U.S. Department of Education’s Institute for Education Sciences (IES), will conduct an independent evaluation of Mission Possible. The evaluation plan involves several components which are described in the following two sections. 11 12 See TESA information at: http://streamer3.lacoe.edu/tesa/. See Professional Development at: http://www.guilford.k12.nc.us/depts/mission_possible/prof_dev.htm; retrieved 01/15/2008. 13 See Pay Incentives at: http://www.guilford.k12.nc.us/depts/mission_possible/pay.htm; retrieved 01/15/2008. 14 See information on EVAAS at: http://www.ncpublicschools.org/evaas/. 15 See Frequently Asked Questions, General Information at: http://www.guilford.k12.nc.us/depts/mission_possible/FAQ.htm; retrieved 01/15/2008. 60 2. Evaluation/Research Questions GCS and the SERVE Center have outlined three overarching evaluation questions as part of the MP evaluation plan: 1. To what extent are the most highly qualified teachers and administrators being recruited and retained at Mission Possible (MP) Schools? 2. To what extent were the MP teachers and administrators trained as proposed? 3. To what extent did the MP training and incentives impact teacher and student outcomes? This proposal is concerned only with data required for Question 3; that is evaluating the impact of MP on student outcomes. Specifically, the SERVE evaluation seeks to evaluate the estimated effect of MP on student learning through analysis of MP teachers’ students’ EOG/EOC scores as compared to non-MP teachers’ students’ EOG/EOC scores in a quasi-experimental design (see Section 3) of matched comparison/control schools from outside GCS. 3. Analytical Approach and Comparison School Selection This section describes the data requirements and proposed analytical approach of the MP evaluation. Subsection 3.1 explains the evaluation design in more detail and subsection 3.2 addresses the need for matched student level data. 3.1. Analytical Approach The proposed analytical approach for the evaluation of the MP teacher incentive programs utilizes a quasi-experimental design. This is appropriate since GCS would like to compare MP school/student outcomes with non-MP school/student outcomes and due to the fact that randomization was not utilized as part of the program implementation. Quasi-experimental or non-experimental designs have limitations when attempting to make generalizations and inferences, as has been well documented (See: Fraker & Maynard, 1987; Friedlander & Robbins, 1995; Glazerman, Levy & Myers, 2003; LaLonde, 1986; Shadish, Cook & Campbell, 2002; and Winship & Morgan, 1999). The goal of the evaluation is to provide a reliable estimate of how the MP teacher incentive plan impacts student learning as measured by the EOG/EOC exams. Ideally, we would like to have outcomes for the same student while receiving the treatment (in this case receiving instruction from a MP teacher that has completed program requirements) and while in the control state (no treatment) over the same period. One of the problems posed by observational studies is that we do not have both realizations (treatment and control) of the outcomes of interest for the same student, only one of the two possible states. The purpose of non-experimental or quasi-experimental designs is to try and estimate the impact of a program on subjects where we do not have both conditions satisfied for the population under consideration. This description of empirical causality has often been labeled the counterfactual account of causality (See: Shadish, Cook & Campbell, 2002; and Winship & Morgan, 1999). One approach that is commonly used in education and medicine for non-experimental and quasiexperimental designs is propensity score matching (PSM). This model uses a matched comparison 61 group from the non-treated population to estimate the impact of the treatment on the target population. Specifically, comparison/control matches are constructed by generating a propensity score, which is the probability of being in the treatment group based on selected variables. Comparison/control subjects are then matched to treatment subjects based on their propensity score with the closest score (or another algorithm like nearest neighbor, etc.). This modeling approach produces accurate statistical matches based on the observed information provided and therefore “controls” for those variables included in the model (See: Hahs-Vaughn & Onwuegbuzie, 2006; Joffe & Rosenbaum, 1999; Luellen, Shadish & Clark, 2005; Rosenbaum & Rubin, 1983 & 1985; and Winship & Morgan, 1999). Using a PSM methodology has several advantages including: (1) it allows the researcher to identify non-comparable covariates early in the analysis; (2) it is less sensitive to model misspecification; and (3) the research is allowed to generate a non-parsimonious model (Hahs-Vaughn & Onwuegbuzie, 2006, 34). There are several disadvantages as well including the model only considers and controls for observed data. Differences between treatment and control groups based on their unobservable characteristics are not controlled for and this must be emphasized when using PSM methodologies. Other disadvantages include the need for larger data sets to get optimal performance from PSM and inclusion of irrelevant covariates will reduce model efficiency (HahsVaughn & Onwuegbuzie, 2006, 35). 3.2. Baseline Comparisons Podgursky and Springer (2007) note that due to the limited number of teacher incentive program evaluations and complicated nature of attempting to quantitatively disentangle teacher effects and impacts on student learning, it is critical that teacher incentive “programs be introduced in a manner amenable to effective evaluation” (943). Figlio and Kenny (2007) contend: The major reason why we know so little about the relation between teacher performance and incentives in the United States and student performance is that the large micro education data sets have gathered very little information about schools’ personnel practices (902). The evaluation of MP is an opportunity to overcome the limitations noted within the evaluation and research literature on teacher incentives. Due to the nature of effects, teacher impacts on their students within specific classrooms, the evaluation requires student level data matched to teachers. Examining grade and school level data will not suffice in this instance for two reasons: (1) MP program participation is targeted at two content areas (English/language arts and math) so only select teachers within a participating school are receiving “treatment” (in this case professional development and the performance incentives); and (2) analysis of grade/school level outcomes will not have sufficient statistical power to detect impacts if they are indeed generated by the program, whether positive or negative. Assignment to treatment, however, was at the school level. Comparison school matches using PSM methodology were therefore assigned also at the school level. Even though entire schools are not “treated”, teachers within MP schools may elect not to participate, assignment was determined at 62 the school level. In all, 30 GCS elementary, middle and high schools 16 are participating in MP and began in two cohorts. In year one of MP (2006-2007) 20 schools (9 elementary, 4 middle, and 7 high schools) began implementing the program while an additional 10 schools (4 elementary, 3 middle and 3 high schools) began in year two (2007-2008). Table 1 presents the summary statistical comparisons before PSM matching. One way analysis of variance (ANOVA) was used for determining statistical difference between treatment and control groups (p-value provided in the final column). Across most categories the two groups are significantly different from one another. 3.3. Computing Propensity Scores and Matching Schools Utilizing PSM methodology involves three steps: (1) selecting the matching variables and deciding on a model to estimate the propensity scores, the probability of being in the treated group; (2) calculating the propensity scores through estimating of the model using logistic regression; and (3) matching the schools based on a specific algorithm or program. Steps 1 and 2 are the most challenging and require the most time. Determining the estimation model is based on at least two criteria: the empirical literature and theory to indicate what should be included in the model and the constraints of the data (availability, missing data, sample size, etc.). While the data available for this evaluation is extensive in terms of the number of covariates available the unit of analysis (school level) and small number of treated schools (30) restrict the number of predictors that can be used in estimation. This is a common limitation in PSM modeling (Hahs-Vaughn & Onwuegbuzie, 2006, 35). For the MP evaluation three sets of propensity scores were calculated based on school type (elementary, middle and high school). Each model was estimated with a slightly different set of predictors due to smaller samples, especially for the middle and high schools. Covariates that theory and empirical research dictate are essential to include (prior achievement, school size, SES measure) were maintained throughout as well as the one year average teacher turnover rate, which was considered essential based on the intervention. Other covariates were included based on relevance to the intervention, such as teacher control variables that could possibly determine school participation (number of classroom teachers, proportion of teachers with 0-3 years experience, proportion of emergency and lateral entry teachers, etc.). See Appendix A for complete model details and specifications. 16 An additional high school, the Academy at Central (High Point Central High School, 410-406) began MP program participation in the 2007-2008 academic year, yielding a total of 30 schools. The initial analysis and matching does not reflect this school’s participation in MP. 63 Table 1: Descriptive Comparisons Prior to Matching, 2005-2006 Mission Possible Schools N = 28 35.7% (10) 21.4% (6) 21.4% (6) 21.4% (6) 590 (394.25) 79.1% (0.24) 35.5% (0.15) 34.0% (0.11) 7.0% (0.05) 49 (26.11) 94.9% (0.07) Control Group Schools N = 1,909 47.8% (913) 19.6% (374) 16.8% (320) 15.8% (302) 2,209 (378.67) 52.8% (0.26) 21.4% (0.11) 23.8% (0.11) 4.4% (0.04) 44 (23.35) 96.7% (0.07) Significant Difference b/w Groups (p-value) 0.8248 0.3398 0.0841 0.0346 0.7875 0.0000 0.0000 0.0000 0.0018 0.2527 0.4610 School Characteristic School Type (frequency ) Elementary (K-5 or PK-5) Middle (5-8, 6-8, or 6-9) High (8-12 or 9-12) Other (alternative, magnet, etc.) Enrollment Free & Reduced Price Lunch Rate One Year Teacher Turnover Rate Teachers w/0-3 Years Experience Rate Emergency/Lateral Entry Teacher Rate Number of Classroom Tachers High School Graduation Rate Prior Achievement Rate, 2004-2006 End-of Grade Exams (grades 3-8) End-of-Course Exams (grades 9-12) 68.4% (0.06) 51.1% (0.16) 80.2% (0.11) 84.1% (0.18) 0.0000 0.0000 Notes : Standard deviations in parentheses except for frequencies. Enrollment is first month school enrollment. Prior Achievement is a weighted average of math and reading EOG exams for grades 3-8 or higher (percent proficient) for years 2003-2004, 2004-2005, and 2005-2006; EOC scores are computed similarly except only for Algebra I & II, English I and geometry. Source: NCDPI, NC School Report Card Data Disc, 2001-2007, version 10-3-07. Estimating the models by school type (elementary, middle and high) as well as restricting the data to those schools with grade configurations that reflected those found in MP schools, further helped to ensure appropriate estimation. School type was not included as a covariate. A variety of other covariates including number of acts of crime on campus, proportion of high quality teachers, number of NBPTS teachers, suspension rates, proportion of teachers with advanced degrees, and whether the school had met AYP or was in improvement status were also examined. Several demonstrated strong correlation with the outcome (probability of being a MP school) but were excluded due to the reasons listed above. Step 3, determining the school matches, was conducted utilizing a one-to-one nearest neighbor matches with a macro developed by University of North Carolina Professor John Painter for SPSS. 17 These matches and the results are presented in three tables by school type in Appendix A. Table 2 presents the summary statistics for the key variables after the one-to-one matching. Nearly 17 The entire package and coding can be found at: http://sswnt5.sowo.unc.edu/VRC/Lectures/index.htm 64 all the covariates have been balanced between the MP control group schools. There are two exceptions: one year average teacher turnover and prior achievement for end-of-grade exams are both still statistically different from one another at a significant confidence level (α = 0.05), with pvalues of 0.0403 and 0.0211 respectively. However, when each of these variables is examined through one way ANOVA by school type (elementary, middle and high school) there are no statistically significant differences. 18 This raises an interesting issue in terms of methodology. Since sample sizes are small for the treatment (30), especially when disaggregating the analysis by school type (elementary, middle and high schools), a simple one-to-one match may be insufficient to provide adequate statistical matches in this case. Two additional features of the analysis will be pursued: (1) a one-to-many match using propensity scores and (2) stratifying the data into quintiles by propensity score. The second option also helps to reduce selection bias in the estimates and mitigate the impact of the unobserved factors not included in the model (Hahs-Vaughan & Onwuegbuzie, 2006; Luellen et al, 2005; and Joffe & Rosenbaum, 1999). As mentioned at the outset of the study, PSM methodology has a strong assumption in that is stipulates that all of the differences between groups are working through the observed predictors, the so called strongly ignorable assumption. By stratifying the schools by quintiles and incorporating additional comparison matches for each MP school, greater accuracy in the estimates can be obtained. The current matches are a good first approximation but by expanding the methodology to include a one-to-many matches and by stratifying the sample into quintiles, greater accuracy can be purchased at minimal cost. The next phase in the program evaluation will be to expand the propensity score matches as outlined above and to pursue analysis at the student level within the treatment schools. One way ANOVA for one year teacher turnover rate: elementary schools F2,28 = 6.00, p-value = 0.3454; middle schools F2,14 = 2.35, p-value = 0.1508; and high schools F2,16 = 2.48, p-value = 0.1373. When eon-of-course prior achievement is examined at he high school level only again the differences between MP and control schools is not statistically different: F2,16 = 0.61, p-value = 0.4490. It should also be noted that EOC exams were not included as a covariate in estimating either the elementary of middle school models. 18 65 Table 2: Comparisons After One-to-One Matching, 2005-2006 Mission Possible Schools N = 28 35.7% (10) 21.4% (6) 21.4% (6) 21.4% (6) 590 (394.25) 79.1% (0.24) 35.5% (0.15) 34.0% (0.11) 7.0% (0.05) 49 (26.11) 94.9% (0.07) Control Group Schools N = 28 42.9% (12) 21.4% (6) 17.9% (5) 17.9% (5) 674 (467.84) 80.9% (0.18) 28.7% (0.08) 31.2% (0.09) 6.7% (0.05) 67 (30.63) 97.9% (0.02) Significant Difference b/w Groups (p-value) 0.9847 0.9320 0.9167 0.8137 0.4881 0.1089 0.0403* 0.2848 0.8396 0.2589 0.2567 School Characteristic School Type (frequency ) Elementary (K-5 or PK-5) Middle (5-8, 6-8, or 6-9) High (8-12 or 9-12) Other (alternative, magnet, etc.) Enrollment Free & Reduced Price Lunch Rate One Year Teacher Turnover Rate Teachers w/0-3 Years Experience Rate Emergency/Lateral Entry Teacher Rate Number of Classroom Tachers High School Graduation Rate Prior Achievement Rate, 2004-2006 End-of Grade Exams (grades 3-8) End-of-Course Exams (grades 9-12) 68.4% (0.06) 51.1% (0.16) 70.0% (0.08) 69.8% (0.25) 0.4877 0.0211** Notes : Standard deviations in parentheses except for frequencies. Enrollment is first month school enrollment. Prior Achievement is a weighted average of math and reading EOG exams for grades 3-8 or higher (percent proficient) for years 2003-2004, 2004-2005, and 2005-2006; EOC scores are computed similarly except only for Algebra I & II, English I and geometry. Source: NCDPI, NC School Report Card Data Disc, 2001-2007, version 10-3-07. * Not statistically significant when tested by school type (elementary, middle or high; see text). ** Not statistically significant when tested at the high school level. 66 References Ballou, D. (2001). Pay for performance in public and private schools. Economics of Education Review, 20(1), 51-61. Figlio, D. N., & Kenny, L. W. (2007). Individual teacher incentives and student performance. Journal of Public Economics, 91(5-6), 901-914. Fraker, T., & Maynard, R. (1987). The adequacy of comparison group designs for evaluations of employment-related programs. Journal of Human Resources, 22(2), 194-227. Friedlander, D., & Robins, P. K. (1995). Evaluating program evaluations: New evidence on commonly used nonexperimental methods. American Economic Review, 85(4), 923-937. Glazerman, S., Levy, D. M., & Myers, D. (2003). Nonexperimental versus experimental estimates of earnings impacts. The Annuals of the American Academy of Political and Social Sciences, 589(Sept.), 63-93. Hahs-Vaughn, D. L., & Onwuegbuzie, A. J. (2006). Estimating and using propensity score analysis with complex samples. Journal of Experimental Education, 75(1), 31-65. Joffe, M. M., & Rosenbaum, P. R. (1999). Propensity scores. American Journal of Epidemiology, 150(4), 327-333. LaLonde, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review, 76(4), 604-620. Luellen, J. K., Shadish, W. R., & Clark, M. H. (2005). Propensity scores: An introduction and experimental test. Evaluation Review, 29(6), 530-558. Murnane, R. J., & Cohen, D. K. (1986). Merit pay and the evaluation problem: Why most merit pay plans fail and few survive. Harvard Education Review, 56(1), 1-17. No Child Left Behind Act of 2001, Pub. L. No. 107-110 (2001). Podgursky, M. J., & Springer, M. G. (2007). Teacher performance pay: A review. Journal of Policy Analysis and Management, 26(4), 909-949. Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41-55. Rosenbaum, P. R., & Rubin, D. B. (1985). Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. American Statistician, 39(1), 33-38. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs. Boston: Houghton Mifflin Company. Winship, C., & Morgan, S. L. (1999). The estimation of causal effects from observational data. Annual Review of Sociology, 25(1), 659-706. 67 Appendix A – Propensity Scores for One-to-One Matched Treatment Schools ELEMENTARY SCHOOLS (N = 26) Mission Possible/ Treatment 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 School Number 670-332 410-385 360-520 410-364 340-462 410-511 640-331 410-580 600-527 410-328 340-368 410-349 340-400 410-366 410-532 410-373 260-324 410-499 340-430 410-586 600-335 410-403 340-376 410-439 600-374 410-598 Total Enrollment 422 267 303 551 381 421 153 406 433 394 479 475 414 571 645 311 172 355 243 203 380 292 482 236 360 186 One Year Teacher Turnover Rate 23.3 28.1 28.9 28.5 18.1 15.7 36.3 15.1 20.9 16.6 22.5 24.3 15.0 30.1 29.3 28.1 44.4 22.2 37.9 45.8 27.2 44.8 27.2 55.1 33.3 82.1 School Locale (NCES) 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 1 2 Free Lunch Rate 55.5 99.3 100.0 100.0 100.0 92.9 100.0 91.9 96.5 98.2 100.0 100.0 91.3 94.6 90.4 94.9 100.0 100.0 100.0 100.0 91.8 100.0 100.0 95.8 95.0 100.0 Emergency/ Lateral Entry Teacher Rate 1.4 3.6 9.1 4.7 3.8 2.6 3.6 3.0 4.9 2.8 2.2 3.5 1.9 0 0.9 0 5.0 0 1.6 6.3 3.5 3.2 1.9 0 0 1.9 No. of Classroom Teachers 35 28 44 43 40 38 14 33 41 36 46 43 53 51 57 31 20 35 30 24 43 32 52 24 32 26 Teacher w/0-3 Yrs Experience Rate 37.1 7.1 22.7 27.9 12.5 23.7 28.6 27.3 39.0 22.2 26.1 27.9 26.4 37.3 31.6 32.3 45.0 20.0 30.0 41.7 46.5 46.9 13.5 50.0 34.4 61.5 Prior Achievement Rate (2004-2006) 82.9 66.3 68.8 69.3 71.5 71.7 76.6 64.2 66.8 66.4 73.3 67.3 74.2 80.7 67.4 73.0 65.2 64.5 67.7 56.1 54.5 63.4 57.9 58.1 66.7 71.5 Grade Range K-5 PK-5 PK-5 PK-5 PK-5 PK-5 PK-5 PK-5 K-5 K-5 PK-5 PK-5 PK-5 PK-5 PK-5 PK-5 PK-5 PK-5 PK-5 PK-5 K-5 PK-5 PK-5 PK-5 K-5 PK-5 Propensity Score 0.00753876 0.00760494 0.03528028 0.03631913 0.03776691 0.03789915 0.05727090 0.05768113 0.08021636 0.07993569 0.10800233 0.10943861 0.16453770 0.16836308 0.30768022 0.30924058 0.38590229 0.41501522 0.55963081 0.54555267 0.71064103 0.72078979 0.56998080 0.96513462 0.51679736 0.97320092 Notes: Baseline data presented above is from the 2005-2006 academic year, one year prior to Mission Possible implementation. School Enrollment is first month enrollment. School Locale are NCES (old version) locale codes where 1 = large city, and 2 = mid-size city. Prior Achievement is the weighted average of students scoring at Level 3 or higher (percent proficient) on the mathematics and reading (summed together) end-of-grade exams across grades 3-5 for the 2003-2004, 2004-2005 and 2005-2006 academic years. Grade Range was not included as a covariate in the logistic model but was used to restrict the pool of schools to those with PK-5 and K-5 grade configurations when estimating the model. Propensity scores calculated using STATA/SE version 7.0 and all propensity score matches generated in SPSS version 15.0. Source: North Carolina Department of Public Instruction, NC School Report Card Data Disc 2001-2007, version 10-3-07. 68 MIDDLE SCHOOLS (N = 14) Mission Possible/ Treatment 0 1 0 1 0 1 0 1 0 1 0 1 0 1 School Number 110-306 410-316 510-397 410-391 900-314 410-592 310-396 410-325 430-347 410-415 600-381 410-367 260-428 410-402 Total Enrollment 640 783 820 551 776 697 225 606 1,134 552 826 577 538 632 One Year Teacher Turnover Rate 38.4 31.2 33.8 35.8 20.9 33.9 16.6 46.2 20.9 40.0 33.7 40.0 47.8 35.0 School Locale (NCES) 4 2 8 2 3 2 6 2 7 2 1 2 4 2 Free Lunch Rate 49.1 75.7 63.4 67.9 78.2 73.3 100.0 67.7 58.0 90.4 88.5 90.5 89.8 94.3 No. of Classroom Teachers 50 55 61 47 65 56 35 49 89 49 69 53 44 58 Prior Achievement Rate (2004-2006) 85.1 74.6 78.2 72.8 65.9 71.6 66.0 76.3 81.5 68.6 65.0 63.2 64.3 68.8 Grade Range 6-8 6-8 6-8 6-8 6-8 6-8 6-8 6-8 6-8 6-8 6-8 6-8 6-8 5-8 Propensity Score 0.03254677 0.03272171 0.05811119 0.05827692 0.09735436 0.09333859 0.13206095 0.13182034 0.22639379 0.30542508 0.45008010 0.38394669 0.19950813 0.53535336 Notes: Baseline data presented above is from the 2005-2006 academic year, one year prior to Mission Possible implementation. School Enrollment is first month enrollment. School Locale are NCES (old version) locale codes where 1 = large city, 2 = mid-size city, 3 = fringe of large city, 4 = fringe of mid-size city, 5 = large town, 6 = small town, 7 = rural, outside CBSA/MSA, and 8 = rural, inside CBSA/MSA; locale was not included as a covariate in estimating the model. Prior Achievement is the weighted average of students scoring at Level 3 or higher (percent proficient) on the mathematics and reading (summed together) end-of-grade exams across grades 5-8 for the 2003-2004, 2004-2005 and 2005-2006 academic years. Grade Range was not included as a covariate in the logistic model but was used to restrict the pool of schools to those with 5-8, 6-8 and 6-9 grade configurations when estimating the model. Propensity scores calculated using STATA/SE version 7.0 and all propensity score matches generated in SPSS version 15.0. Source: North Carolina Department of Public Instruction, NC School Report Card Data Disc 2001-2007, version 10-3-07. 69 HIGH SCHOOLS (N = 16) Mission Possible/ Treatment 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 School Number 291-336 410-556 010-360 410-406 422-324 410-544 600-592 410-358 340-330 410-355 600-496 410-483 600-576 410-319 600-376 410-326 Total Enrollment 731 927 887 1,278 315 1,527 2,223 930 1,061 1,583 1.053 101 1,651 1,037 1,137 83 One Year Teacher Turnover Rate 22.7 25.7 34.4 32.0 36.0 23.8 17.8 27.2 28.5 33.3 34.8 62.5 24.5 39.7 29.0 50.0 School Locale (NCES) 6 8 2 2 7 2 1 8 2 2 1 2 1 2 1 2 Free Lunch Rate 64.4 41.5 63.2 55.8 78.1 65.6 51.7 51.3 53.1 65.8 64.0 44.6 79.2 63.0 64.0 51.9 Teacher w/0-3 Yrs Experience Rate 18.8 27.3 41.9 37.4 24.1 31.8 31.6 41.8 25.8 39.2 38.0 30.0 30.3 36.5 36.0 22.2 Prior Achievement Rate (2004-2006) 53.5 65.1 60.9 59.8 57.6 48.3 48.9 55.0 51.2 48.3 46.6 22.7 38.0 38.8 44.0 24.4 Graduation Rate 94.8 99.3 96.7 96.6 100.0 95.6 98.0 99.2 97.7 96.7 100.0 78.3 99.4 98.0 96.4 95.5 Grade Range 9-12 9-12 9-12 9-12 9-12 9-12 9-12 9-12 9-12 9-12 9-12 9-12 9-12 9-12 9-12 9-12 Propensity Score 0.04188414 0.04077351 0.08043345 0.08134238 0.15652275 0.15301910 0.17934483 0.16205180 0.31407568 0.31718808 0.67999256 0.77918673 0.65904289 0.87062204 0.44685721 0.99697083 Notes: Baseline data presented above is from the 2005-2006 academic year, one year prior to Mission Possible implementation. School Enrollment is first month enrollment. School Locale are NCES (old version) locale codes where 1 = large city, 2 = mid-size city, 3 = fringe of large city, 4 = fringe of mid-size city, 5 = large town, 6 = small town, 7 = rural, outside CBSA/MSA, and 8 = rural, inside CBSA/MSA; locale was not included as a covariate in estimating the model. Prior Achievement is the weighted average of students scoring at Level 3 or higher (percent proficient) on Algebra 1, Algebra 2, English 1 and geometry (summed together) end-of-course exams across grades 9-12 for the 2003-2004, 2004-2005 and 2005-2006 academic years. Grade Range was not included as a covariate in the logistic model but was used to restrict the pool of schools to those with 8-12 and 9-12 grade configurations when estimating the model. Propensity scores calculated using STATA/SE version 7.0 and all propensity score matches generated in SPSS version 15.0. Source: North Carolina Department of Public Instruction, NC School Report Card Data Disc 2001-2007, version 10-3-07. Mission Possible high school omitted from matching: Academy at Smith (410-545) (Ben L. Smith High School), since its first year was 2006-2007 and did not have baseline data. 70