VIEWS: 6 PAGES: 47 POSTED ON: 11/26/2011
University of Minnesota Evaluation of the Robert Noyce Teacher Scholarship Program, Final Report Section Four: The Influence of Scholar and Program Level Variables on Scholar Perceptions of the Effect of the Noyce Funding Pey-Yan Liou Frances Lawrenz, PI Noyce Program Evaluation Team Post Doctoral Members Marjorie Bullitt Bequette Michelle Fleming Deena Wassenberg Graduate Students Jim Appleton Anica Bowe Maureen Braam Chris Desjardins Karen Hofstad-Parkhill Allison Kirchhoff Kei Lee Christina Madsen Ann Ooms Mary Sande August 2009 TABLE OF CONTENTS Page Number Introduction 6 Methodology 8 Instruments 8 Data 9 Sample 10 Measures 11 Outcome variables 11 Scholar-level predictors 13 Program-level predictors 15 Analysis 15 Hierarchical Linear Models (HLMs) 15 Hierarchical Generalized Linear Models (HGLMs) 16 Results 17 Variables influencing scholars’ perceptions of the influence of the Noyce 17 funding on their commitment to become teachers Outcome variable 1: Scholars’ perception of the influence of Noyce 17 funding on becoming teachers Outcome variable 3: Would you have become a teacher if you had 20 not received the Noyce scholarship? Variables influencing scholars’ perception of the influence of the Noyce 22 funding on their commitment to teach in high need schools Outcome variable 2: Scholars’ perception of the influence of Noyce 22 funding on becoming teachers in high need schools Outcome variable 4: Would you have decided to teach in high need 24 school if you had not participated in the Noyce scholarship program? Conclusions 27 Scholars’ individual characteristics and perceptions 29 Race 29 Preparation for high need schools 30 Path to teaching 30 District/school high need environment 30 Personal beliefs towards teaching 31 Other scholars’ characteristics and perceptions 31 Programs’ characteristics 31 Noyce Evaluation Report, Section Four: HLM 3 Noyce funding 32 Preparation for high need school 32 Mentoring experience 33 Limitations 33 References 35 APPENDICES Page Number Appendix A: Items, factor loadings and sample sizes for factors from scholar 36 survey Appendix B: Groups of scholars within the eight factors 40 Appendix C: All institutions 44 Appendix D: U of MN Noyce evaluation team comprehensive evaluation report 47 sections Noyce Evaluation Report, Section Four: HLM 4 List of Tables and Figures Tables Page Number Table 1 The eight factors for the scholar survey 9 Table 2 Frequency of outcome variable 3: Would you have become a 11 teacher if you had not received the Noyce scholarship? (Three response categories) Table 3 Frequency of outcome variable 3: Would you have become a 12 teacher if you had not received the Noyce scholarship? (Two response categories) Table 4 Descriptive statistics of outcome variable 4: Would you have 12 decided to teach in a high need school if you had not participated in the Noyce scholarship program? (Three response categories) Table 5 Four outcome variables of scholars’ perception of the Noyce 13 program’s influence on their becoming a teacher and teaching in a high need school Table 6 Scholar-level predictors for all scholar group and current STEM 14 teacher group Table 7 Descriptive statistics and correlation matrix for scholar-level 14 variables for all scholar group Table 8 Descriptive statistics and correlation matrix for scholar-level 14 variables for current STEM teacher group Table 9 Program-level predictors 15 Table 10 Descriptive statistics and correlation matrix for program-level 15 variables Table 11 Effects of predictors on scholars’ perception of the influence of 20 the scholarship on becoming teachers for “all scholar” group and “current STEM teacher” group (outcome variable 1) Table 12 Effects of predictors on scholars’ perception of the influence of 22 the scholarship on becoming teachers for “all scholar” group and “current STEM teacher” group (outcome variable 3) Table 13 Effects of predictors on scholars’ perception of the influence of 24 the scholarship on becoming teachers in high need schools for “all scholar” group and “current STEM teacher” group (outcome variable 2) Table 14 Effects of predictors on scholars’ perception of the influence of 26 the scholarship on becoming teachers for “all scholar” group and “current STEM teacher” group (outcome variable 4) Noyce Evaluation Report, Section Four: HLM 5 Table 15 Summary of effects of predictors on scholars’ perception of the 28 influence of the scholarship on becoming teachers for “all scholar” group and “current STEM teacher” group Table 16 Summary of effects of predictors on scholars’ perception of the 28 influence of the scholarship on becoming teachers in high need schools for “all scholar” group and “current STEM teacher” group Noyce Evaluation Report, Section Four: HLM 6 Introduction The National Science Foundation’s (NSF) Robert Noyce Teacher Scholarship Program was funded to increase the number of highly qualified teachers working in high need schools. This program has individual projects situated nationally at various university sites. The overall purpose of these projects is to recruit and support individuals with strong academic background in science, technology, engineering, and mathematics (STEM) content areas into becoming teachers and working in high need schools. Research shows persistent correlations between student performance and teacher quality in science and mathematics (Goldhaber & Brewer, 1996; Jordan, Mendro, & Weerasinghe, 1997; National Research Council, 2000; Sanders and Rivers, 1996). Other studies (Ingersoll, 1999, 2002) show that 56% of secondary students in physical science are being taught by teachers without a major or minor in physical science, and that students in high-poverty schools are 77% more likely to be taught by an out-of-field teacher. To evaluate the effectiveness of the Robert Noyce Teacher Scholarship Program (hereafter referred to as the Noyce program) on various dimensions, the University of Minnesota (U of MN) Noyce evaluation team was founded and funded since 2005.The team has been composed of a number of graduate students and postdoctoral students with Frances Lawrenz as the Principal Investigator (PI). As suggested in the original evaluation project proposal, the University of Minnesota Noyce evaluation team (hereafter referred to as U of MN Noyce evaluation team) has four major components: preparation of an extensive literature review pertaining to effects of incentive programs in recruiting and retaining STEM K-12 teachers, thematic synthesis of the existing evaluation information through content analysis of project reports, statistical querying of the existing monitoring data to produce quantitative models of the program, and development and execution of an overall program evaluation plan through collaboration with existing projects. In addition, the following are the main evaluation questions: 1. What are the characteristics of the teacher preparation/certification programs provided by the Noyce projects? 2. In the opinions of the PIs and scholars (i.e. Noyce scholarship/stipend recipients), what role did the Noyce funding play in the scholars’ decisions to teach in high need schools? 3. In the opinions of PIs, scholars, STEM faculty and districts, in what ways and to what extent are the characteristics of the Noyce program related to Recruiting individuals with strong STEM backgrounds, including those who might not otherwise consider teaching/going into teaching careers Retaining scholarship/stipend recipients in the teaching profession Developing the pedagogical content knowledge and ability of scholarship/stipend recipients to teach STEM content Noyce Evaluation Report, Section Four: HLM 7 Developing positive profiles for pre-service teachers in STEM discipline departments This report extends the initial evaluation report by providing in-depth analyses related to the third component of the evaluation listed above, statistical querying of the data and the production of quantitative models of the Noyce program. Additionally, the U of MN Noyce evaluation team sought to examine the second evaluation question to determine the role of the Noyce program in Noyce scholarship/stipend recipients’ (hereafter referred to as scholars’) decisions to become teachers and teach in high need schools. To do this, the team analyzed the relationship between the scholars’ perceptions of the influence of the Noyce program on their commitment to become teachers and commitment to teaching in high need schools as well as scholar- and teacher preparation/certification program-level characteristics using Hierarchical Linear Modeling (HLM) and Hierarchical Generalized Linear Modeling (HGLM). These two forms of multi-level analyses are more advanced than simple single level and multiple linear regression because they can differentiate variance from different levels of predictor variables. For example, HLM and HGLM can differentiate between the variance due to the teacher preparation/certification program effects and the variance due to individual scholar effects (Raudenbush & Bryk, 2002). A key assumption in single-level models such as traditional regression is that the observations are independent of one another which is often not true where a nested structure exists (i.e., scholars within programs). In these situations, units of observations within a program tend to be more similar to one another than observations from other programs. Failing to take into account this dependency can result in biased statistical results. Multi-level models take the nested structure of this data into account and therefore reduce this potential bias (Raudenbush & Bryk, 2002). Both scholar- and teacher preparation/certification program-level variables/characteristics could influence scholars’ decisions to become teachers and to teach in high need settings. Scholar-level variables such as race and background characteristics could influence scholar perceptions of the influence of the Noyce program. Additionally, teacher preparation/certification programs provide scholars with initial teaching experiences and the educational knowledge base that is necessary for a successful teaching career. Therefore, the characteristics of the teacher preparation/certification programs are an important element that may also impact scholars’ perceptions. Because scholars in the same teacher preparation/certification program tend to be more alike than scholars in other programs, and each program may have a different impact on scholars, HLM was used to differentiate the variance due to teacher preparation/certification program effects from individual scholar effects. Therefore, the purpose of this report was to investigate how the scholars’ perceptions of the influence of the Noyce program on their commitment to teach and teach in high need schools related to the scholars’ characteristics and the characteristics of the teacher preparation/certification programs in which they received training. Noyce Evaluation Report, Section Four: HLM 8 Methodology Instruments The data utilized for this phase of the Noyce evaluation came from three main sources: the Noyce scholar survey, the Noyce PI Survey, and the ORC dataset. The scholar survey was administered online during the summer of 2007. Former and current Noyce scholars were asked to respond to a variety of items regarding their perceptions of and experiences with the Noyce program. The survey consisted of a variety of rating scale, multiple choice, and open-ended items. Completion of this survey was voluntary. Additionally, different forms of the survey were administered to the scholars based on their status at the time of the survey (e.g. in a teacher certification program, not yet a full-time teacher; completed a teacher certification program, but never taught; teaching full-time or part-time etc.). Items did not differ across surveys, but some items were either included or excluded depending on the scholars’ career progress. Appendix B of the Final Evaluation Report Section One: Planning and Survey Data, includes copies of the surveys or they can be accessed online at: http://www2.cehd.umn.edu/EdPsy/NoyceSurvey/NoyceScholar/surveySample.asp. Due to the large number of items in the scholar survey (83 items), it was necessary to combine and reduce the items for ease of analysis in HLM. In Final Report Section Two: Factor Analysis of the Evaluation Questionnaire (Liou & Lawrenz, 2009), factor analysis was used to investigate the possibility of combining many items in the Noyce scholar survey into a few broad constructs (e.g., scholars’ educational background, determination to teach in high need schools, and perceptions of their programs/Noyce scholarship). Factor variables reflect a construct more accurately than dichotomous or rating scale variables because factor variables are made up of an aggregate of items that are indicators of the same construct. This then makes factor variables more appropriate for advanced statistical analyses such as HLM. Seven factors were described in the report, including “commitment to teaching in high need schools (influence of Noyce),” “preparation for high need schools,” “path to teaching,” “district/school high need environment,” “personal beliefs towards teaching,” “school teaching environment,” and “mentoring experience” (details about the items and factors loadings for each factor are included in Appendix A). Additional analyses of the six items in the “commitment to teaching in high need schools (influence of Noyce)” factor revealed that this factor could further be reduced into two separate factors (Liou, Kirchhoff, & Lawrenz, in press). The two factors were called “influence of scholarship on STEM majors becoming teachers” and “influence of scholarship on STEM majors becoming teachers in high need schools.” Due to these findings, an eight factor structure (Table 1) was used for the HLM analyses in this report. Noyce Evaluation Report, Section Four: HLM 9 Table 1 The Eight Factors for the Scholar Survey Factor No. Factor name Factor 1 Influence of scholarship on STEM majors becoming teachers Factor 2 Influence of scholarship on STEM majors becoming high need teachers Factor 3 Preparation for high need schools Factor 4 Path to teaching Factor 5 District/school high need environment Factor 6 Personal beliefs towards teaching Factor 7 School teaching environment Factor 8 Mentoring experience The PI survey was administered online during the summer of 2007 to the Noyce project PIs at local teacher preparation/certification institutions. PIs were asked to respond to a variety of items describing their teacher preparation/certification programs and the role of the Noyce funding in their programs. Appendix A of the Final Evaluation Report Section One: Planning and Survey Data, includes a copy of the PI survey, or it can be accessed online at: http://cehd.umn.edu/EdPsych/NOYCE/PI-FacultySurveys.html. The ORC dataset was from the Noyce Program monitoring data system from ORC Macro International Inc. This system collects annual data from the Noyce project PIs about the Noyce scholars including demographic and academic background information. More information about the ORC dataset can be found in Final Report Section Five: Combined Analysis of the Robert Noyce Teacher Scholarship Program using ORC Macro and UMN Evaluation Data (Bowe, Liou, & Lawrenz, 2009). Data Not all of the data from the scholar and PI surveys and the ORC dataset were used in the analyses in this report. Data included from the scholar survey were variables related to scholars’ perception about the influence of the Noyce funding on their becoming teachers and becoming teachers in high need schools, path to teaching, school high need environment, personal beliefs towards teaching, school teaching environment, and mentoring experience (i.e., the factors described previously). One variable was included from the PI survey; this variable was related to what percentage of scholars’ tuition was provided for by the Noyce funding. Data included from the ORC dataset included variables about the scholars’ race and gender. The data used in this report exists in two levels: the scholar-level (level 1) and the program-level (level 2; throughout the report, “program-level” refers to teacher preparation/certification level variables, not the Noyce program). The scholar-level (level 1) variables include scholars’ race, gender, perceptions about preparation for high need school (Factor 3), path to teaching (Factor 4), district/school high need environment (Factor 5), personal beliefs towards teaching (Factor 6), school teaching environment (Factor 7), and mentoring experience (Factor 8). The program- level (level 2) variables include information regarding the percentage of the scholars’ tuition Noyce Evaluation Report, Section Four: HLM 10 covered by the Noyce funding, and the mean factor scores of preparation for high need schools (mean Factor 3) and mentoring experience (mean Factor 8). It is worth noting that two of the Factors ( 3 and 8) were included as variables at both the scholar- and program-level as the U of MN Noyce evaluation team considered these factors to have both scholar- and program-level effects. Factor 3: Preparation for high need schools included scholars’ responses to 13 items regarding curricula and activities provided by their teacher preparation/certification program for high need schools teaching. Factor 8: Mentoring experience included scholars’ responses to six individual items focusing on mentoring experiences they might have received during and after their teacher preparation/certification program. Although Factors 3 and 8 were constructed according to scholars’ perceptions, Factors 3 and 8 also related to program characteristics. Therefore, scholars’ Factors 3 and 8 scores from each program were used to form mean Factors 3 and 8 scores for each program. These factors at the program level can also be considered a contextual model, since it examines the impact of Factor 3 and Factor 8 at the program level over and above the individual impact of Factor 3 and Factor 8 at the scholar level. Sample Originally, 555 scholars provided data from the Noyce scholar survey, and 66 PIs provided data from the PI survey. Detailed information about the sample can be found in Lawrenz, et al. (2008). Due to concerns about statistical inference, however, some scholars’ data had to be deleted because at times there were less than 5 scholars responding from any one teacher preparation/certification program. Smaller within-program scholar samples yield relatively less reliable estimates of these population parameters than do larger within program scholar samples. Therefore, data from 527 scholars in 43 programs were available to use in this analysis. Additionally, when variables at the program-level were analyzed in HLM and HGLM, data from two teacher preparation/certification programs and 33 scholars associated with those teacher preparation/certification programs were deleted because the PIs did not respond to the PI survey. Additional information about the omitted teacher preparation/certification programs and the number of scholars in each teacher preparation/certification programs included in the analysis is listed in Appendix C. Moreover, because the HLM and HGLM procedures used pairwise deletion, the actual number of scholars and programs in any specific analysis may be lower and therefore, the actual sample size is provided for each individual analysis. The sample used in each individual analysis was further reduced due to the fact that not every scholar responded to every item in the scholar survey and therefore, some of the factors could not be used as predictors for all scholars. For example, Factor 5, district/school high need environment, only applies to scholars who were already teaching and only those scholars responded to the items included in that factor. (See Appendix B for more information about items in each factor.) To account for these differences and include the largest number of variables possible, the U of MN Noyce evaluation team decided to use two groups of scholars for the analyses and conduct separate analyses on these two groups. One group is the “all scholar” group, and the other is the “current STEM teacher” group. 427 scholars in 37 teacher preparation/certification programs were included in the all scholars group, and 265 scholars in 36 teacher preparation/certification programs were in the current STEM teacher group. Noyce Evaluation Report, Section Four: HLM 11 Measures Outcome variables: Using data from the Noyce scholar survey, four variables were considered as outcomes. The first and the second outcome variables were Factor 1: The influence of scholarship on STEM majors becoming teachers and Factor 2: The influence of scholarship on STEM majors becoming teachers in high need schools. These are continuous variables, which were created as standardized factor scores by combining the three items in each factor (Liou, Kirchhoff, & Lawrenz, in press). The third and the fourth outcome variables were categorical items from the scholar survey: Would you have become a teacher if you had not received the Noyce scholarship? and Would you have decided to teach in a high need school if you had not participated in the Noyce scholarship Program? The response options of the third outcome variable were 1) Yes, 2) Possibly, and 3) No. The frequency table of the responses for outcome variable 3 is in Table 2. Few people chose “no,” so the options “no” and “possibly” were combined which made outcome variable 3 a binary variable (yes=1; possible/no=2), and provided enough differentiation for options to analyze outcome variable 3. The frequency table of outcome variable 3 with the adjusted two categories is in Table 3. The response options of the fourth outcome variable were 1) Yes, 2) Possibly, and 3) No/I have not taught in a high need school. The frequency table of the responses for outcome variable 4 is in Table 4. No recombination of responses (Yes, Possibly, No) were made for this variable because there were sufficient enough responses in each category for differentiation analyses to take place. For these two categorical outcome variables (3 and 4), it is worth paying particular attention to their order of the option coding . “No” or “no/I have not taught in a high need school” was coded as a higher value. It is assumed that when scholars responded “no” to outcome variable 3 and 4, scholars’ perceptions of the effect of the Noyce funding for them to become teachers and to teach in high need schools were higher, and visa verse. The constructs implied in the first and third outcome variables are related to the perceived influence of Noyce funding on scholars’ decisions to become teachers. The constructs implied in the second the fourth outcome variables are related to the perceived influence of the Noyce funding on scholars’ decisions to teach in high need schools. A summary of the four outcome variables is included in Table 5. Table 2 Frequency of Outcome Variable 3: Would You Have Become a Teacher if You Had Not Received the Noyce Scholarship? (Three Response Categories) Value Frequency Percentage Cumulative Percentage 1 = Yes 339 79.6 79.6 2 = Possibly 72 16.9 96.5 3 = No 15 3.5 100 Noyce Evaluation Report, Section Four: HLM 12 Table 3 Frequency of Outcome Variable 3: Would You Have Become a Teacher if You Had Not Received the Noyce Scholarship? (Two Response Categories) Value Frequency Percentage Cumulative Percentage 1 = Yes 338 79.5 79.5 2 = No/Possibly 87 20.5 100 Table 4 Descriptive Statistics of Outcome Variable 4: Would You Have Decided to Teach in a High Need School if You Had Not Participated in the Noyce Scholarship Program? (Three Response Categories) Value Frequency Percentage Cumulative Percentage 1 = Yes 154 36.2 36.2 2 = Possibly 214 50.4 86.6 3 = No/I have not taught 57 13.4 100 in a high need school Noyce Evaluation Report, Section Four: HLM 13 Table 5 Four Outcome Variables of Scholars’ Perception of the Noyce Program’s Influence on Their Becoming a Teacher and Teaching in a High Need School Variables Item content Item option Outcome variable 1: Scholars’ (a) become a teacher 1) not at all influential perception of the influence of (b) complete the certification 2) not very influential scholarship on becoming program 3) somewhat teachers (c) take a teaching job influential (Cronbach’s Alpha=0.88) 4) very influential Outcome variable 2: Scholars’ (d) teach in a high need 1) not at all influential perception of the influence of school 2) not very influential scholarship on becoming (e) remain teaching in a high 3) somewhat teachers in high need schools need school for the full term influential (Cronbach’s Alpha=0.90) of your commitment 4) very influential (f) remain teaching in a high need school beyond the full term of your commitment Outcome variable 3: As the variable 1) yes Would you have become a 2) no/possibly teacher if you had not received the Noyce scholarship? Outcome variable 4: As the variable 1) yes Would you have decided to 2) possibly teach in a high need school if 3) no/I have not you had not participated in the taught in a high need Noyce scholarship Program? school Scholar-level predictors: Race and gender, from the ORC dataset, were used as two of predictors; race was coded as 0=Non-white and 1=White, and gender was coded as 0=Female and 1=Male. Other scholar-level predictors were factors 3-8 from the Noyce scholar survey. However, as mentioned above, not all items in each factor were answered by all scholars, so not every factor can be used as a predictor in all analyses. Therefore, Factors 3, 4, and 8 were the predictors for the all scholar group, and all six factors were the predictors for the current STEM teacher group. Therefore, the all scholar group had five total predictors and the current STEM teachers group had eight total predictors at the scholar level (see Table 6). Table 7 shows the descriptive statistics and correlation matrix for the scholar-level variables for the all scholar group, and Table 8 shows the descriptive statistics and correlation matrix for the current STEM teacher group. Noyce Evaluation Report, Section Four: HLM 14 Table 6 Scholar-Level Predictors for All Scholar Group and Current STEM Teacher Group Predictor Item Description Predictors for All Predictors for Current Scholar Group STEM Teacher Group Race Non-white or White * * Gender Female or Male * * Factor3 Preparation for high need schools * * Factor4 Path to teaching * * Factor5 District/school high need N/A * environment Factor6 Personal beliefs towards teaching N/A * Factor7 School teaching environment N/A * Factor8 Mentoring experience * * Note. N/A indicated that Factors 5 to 7 were not included in the analysis for the all scholar group. Table 7 Descriptive Statistics and Correlation Matrix for Scholar-Level Variables for All Scholar Group Variables M SD 1 2 3 4 5 6 7 8 9 1. Race .67 .47 - 2. Gender .34 .48 -.10* - 3. Factor3 -.03 .98 -.15** .04 - 4. Factor4 -.01 .90 -.04 .11* <-.01 - 5. Factor8 .01 .87 -.24** .09 .36** .06 - 6. Outcome1 -.01 .94 -.20** -.05 .07 .05 .12* - 7. Outcome2 -.07 .98 -.03 -.05 .13** -.04 -.02 .58** - 8. Outcome3 1.20 .40 -.03 .03 -.08 .16** .03 .42** .18** - 9. Outcome4 1.77 .67 .22** -.11* -.14** -.05 -.13** .11* .28** .22** - Note. The number of respondent varies from 425 to 427. *p<.05. **p<.01. Table 8 Descriptive Statistics and Correlation Matrix for Scholar-Level Variables for Current STEM Teacher Group Variables M SD 1 2 3 4 5 6 7 8 9 10 11 12 1. Race .69 .46 - 2. Gender .33 .47 -.12* - 3. Factor3 -.08 .96 -.20** .04 - 4. Factor4 .04 .90 <-.01 .10 .02 - 5. Factor5 -.04 .89 -.15* .03 .10 .07 - 6. Factor6 -.08 1.11 .07 -.05 .30** .02 .04 - 7. Factor7 <-.01 .88 -.01 -.03 .16* .05 -.14* .35** - 8. Factor8 .01 .83 -.31** .04 .43** .12* .19** .29** .15* - 9. Outcome1 <-.01 .94 -.16* -.09 .04 .04 .16** -.07 -.03 .09 - 10. Outcome2 -.11 .99 -.05 -.04 .11 -.02 .07 -.05 -.07 .01 .62** - 11. Outcome3 1.25 .43 -.04 -.03 -.12* .13* .03 -.10 -.13* -.02 .45** .23** - 12. Outcome4 1.77 .68 .15* -.05 -.19** -.02 -.17** -.19** -.16** -.13* .16** .30** .29** - Note. The number of respondent is 265.*p<.05. **p<.01. Noyce Evaluation Report, Section Four: HLM 15 Program-level predictors: The percentage of scholars’ total tuition provided for by the Noyce scholarship was used as one of the program-level predictors. It is an ordinal variable (1= 0% (money not used for tuition); 2=1-24%; 3=25-49 %; 4=50-74 %; 5=75-99 %; 6=100 %). In addition, individual scholar scores on Factor 3 and Factor 8 for each teacher preparation/certification program were aggregated and averaged within teacher preparation/certification programs and used as program-level predictors. Table 9 summarizes the three predictors at the program level. Table 10 shows the descriptive statistics and correlation matrix for the program-level variables. Table 9 Program-Level Predictors Predictor Item Description Item Coding Noyce Funding What proportion of the total tuition 1) 0 % (money not used for scholars need to pay is provided by the tuition); 2) 1-24 %; 3) 25-49 %; Noyce scholarship funding? 4) 50-74 %; 5) 75-99 %; 6) 100 % Mean Factor3 Preparation for high need schools Mean Factor8 Mentoring experience Table 10 Descriptive Statistics and Correlation Matrix for Program-Level Variables Variables M SD 1 2 3 1. Noyce Funding 4.11 1.65 - 2. Mean Factor3 .04 .52 <.01 - 3. Mean Factor8 -.01 .43 -.30 .40* - Note. The number of PI is 37. *p<.05. Analysis In this report, HLMs and HGLMs were used to explore 1) what variables influence scholars’ perceptions of the influence of the Noyce funding on their decisions to become teachers and 2) the influence of the Noyce funding on their decisions to become teachers in high need schools. In other words, HLMs and HGLMs can examine the quantitative relationship between the predictors and the outcome variables. Hierarchical Linear Models (HLMs): As described in the introduction, HLMs are able to differentiate between the variance due to the teacher preparation/certification program effects and the variance due to individual scholar’s variables effects (Raudenbush & Bryk, 2002). HLMs can only be used for analyzing continuous outcome variables, such as outcome variables 1 and 2, but can not allow the analysis of Noyce Evaluation Report, Section Four: HLM 16 categorical variables such as outcome variables 3 and 4: Would you have become a teacher if you had not received the Noyce scholarship? and Would you have decided to teach in a high need school if you had not participated in the Noyce funding Program? Therefore, HGLM was used for analyzing outcome variables 3 and 4. Hierarchical Generalized Linear Models (HGLMs): HGMLs, also known as generalized linear mixed models or generalized linear models with random effects (Raudenbush & Bryk, 2002), offer a coherent modeling framework for multilevel data with nonlinear structural models and nonnormally distributed errors. There are three reasons why HGLMs are appropriate for ordinal variables. First, HGLMs restrict the outcome probability to an interval of (0, 1). This constraint gives meaning to the effect sizes defined by the model. A nonlinear transformation of the predicted value satisfies this constraint. Second, given the predicted value of outcome variables, HGLMs relax the normal distribution of level-1 random effects. Third, HGLMs do not require heterogeneous variance like HLMs. For the binary outcome variable 3: Would you have become a teacher if you had not received the Noyce scholarship? a Bernoulli distribution with a log-link function was used. For the ordinal outcome variable 4: Would you have decided to teach in a high need school if you had not participated in the Noyce scholarship Program? an ordered logit-link function was used. HLMs and HGLMs were estimated using HLM6 (Raudenbush, Bryk, & Congdon, 2005). The default estimation procedure in HLM6 employs restricted maximum likelihood where fixed effects are estimated using generalized least squares and variance-covariance components are estimated using maximum likelihood (Raudenbush et al., 2005). In this report, the unit-specific model was used. This function “defines fixed regression coefficients that can be interpreted as the expected change in the outcome associated with a one-unit increase in the relevant predictor, holding constant other predictors and all random effects in the model (Raudenbush & Bryk, 2002, p.334).” Grand-mean centering was used for adjusting predictors in equations, and maximum likelihood estimation was used to estimate parameters. All models reported are random-intercept models. The random part of the intercept was freely estimated to reflect between-program differences in the influence of Noyce funding to become teachers. Since there is no a priori hypothesis concerning between-program differences of the predictor variables, the random parts of the slopes were not relevant. In other words, only the intercept varied across programs, but other level-1 coefficients remained constant. All differences described in this report are statistically significant at α=0.05. No statistical adjustments to account for multiple comparisons were used. Noyce Evaluation Report, Section Four: HLM 17 Results This results section contains two parts: which scholar- and program-level variables are related to scholars’ perceptions of the influence of the Noyce funding on their commitment to become teachers, and which scholar- and program-level variables are related to scholars’ perceptions of the influence of the Noyce funding on their commitment to teaching in high need schools. Variables influencing scholars’ perceptions of the influence of the Noyce funding on their commitment to become teachers To investigate which scholar- and program-level variables were related to scholars’ perceptions of the influence of the Noyce funding on their commitment to become teachers, outcome variable 1: Scholars’ perception of the influence of scholarship on becoming teachers, and outcome variable 3: Would you have become a teacher if you had not received the Noyce scholarship? were used. In addition, models for each of the two groups “all scholars” and “current STEM teacher” are discussed separately. Outcome variable 1: Scholars’ perception of the influence of Noyce funding on becoming teachers For outcome variable 1, an unconditional HLM model (one-way random-effects ANOVA model) was first fit to determine whether substantial variance could be explained at the program level of the model for the all scholar group and the current STEM teacher group. There was significance in the program-level variance for outcome variable 1 in both groups. For the all scholar group, the estimate for the within-program variance was 0.81, and the overall variability among the true program means on outcome variable 1 was 0.08. This resulted in an intraclass correlation of 0.09. This result indicated that most of the variance for the all scholar group occurred at the scholar level, with 9% of the variance located at the program level. For the current STEM teacher group, the estimate for the within-program variance was 0.67, and the overall variability among the true program means on outcome variable 2 was 0.19. This resulted in an intraclass correlation of 0.22. This result indicated 22% of the variance for the current STEM teacher group was located at the program level. In order to explain more variability by adding potential predictors, conditional models were fit (Model 1-all scholar group in equation 1, Model 2-current STEM teacher group in equation 2) as follows: Model 1: HLM/HGLM for the all scholar group (1) Level 1: All scholar Yij 0 j 1 j ( Race) 2 j (Gender ) 3 j ( Factor3) 4 j ( Factor 4) 8 j ( Factor8) rij Level 2: Program 0 j 00 01 ( NoyceFunding ) 02 ( MeanFactor3) 03 ( MeanFactor8) u 0 j 1 j 10 Noyce Evaluation Report, Section Four: HLM 18 2 j 20 3 j 30 4 j 40 8 j 80 Model 2: HLM/HGLM for the current STEM teacher group (2) Level 1: Current STEM teacher Yij 0 j 1 j ( Race) 2 j (Gender ) 3 j ( Factor3) 4 j ( Factor 4) 5 j ( Factor5) 6 j ( Factor6) 7 j ( Factor7) 8 j ( Factor8) rij Level 2: Program 0 j 00 01 ( NoyceFunding ) 02 ( MeanFactor3) 03 ( MeanFactor8) u 0 j 1 j 10 2 j 20 3 j 30 4 j 40 5 j 50 6 j 60 7 j 70 8 j 80 In Model 1 (the all scholar group), two predictors were significantly related to outcome variable 1 at the program level. Noyce funding ( b 0.13 ; SE 0.03 ; p-value 0.01 ) was positively related to outcome variable 1, while Mean Factor 3: Preparation for high need schools ( b 0.38 ; SE 0.13 ; p-value 0.01 ) had a negative effect. This means that outcome variable 1 scores were substantially higher in programs where the Noyce funding was a higher percentage of total tuition. This result suggests that greater the percentage of scholars' tuition covered by the Noyce funding, the greater the Noyce program influenced scholars' decisions to become teachers. On the other hand, outcome variable 1 scores were lower in programs with higher Mean Factor 3 scores. This result suggests that greater degree of exposure to curricula and activities that prepared pre-teachers for high need schools, the less the Noyce program influenced scholars’ decisions to become teachers. At the scholar level, only Race ( b 0.41; SE 0.08 ; p- value 0.01 ) was significant for outcome variable 1. This means that White scholars tended to have lower perceptions of the influence of the Noyce funding on becoming teachers than Non- white scholars on average. Model 1 incorporated the three program-level variables and the five scholar-level variables relating to the perceived influence of Noyce funding on scholars becoming teachers. The variance at the program level was 0.04, and at the scholar level was 0.77 which represents unexplained variance after taking into account scholars’ race, gender, Factor 3, Factor 4, and Factor 8. In other words, these predictors decreased 5% of the variance at the program level as well as 4% of the variance at the scholar level. Therefore, this resulted in an intraclass correlation of 0.05. This result indicated that most of the variance occurred at the scholar level, with 5% of the variance located at the program level. The intraclass correlation Noyce Evaluation Report, Section Four: HLM 19 explained dropped to 0.05 from 0.09 after adding predictors at both levels. This demonstrates that predictors at the scholar level have more variability than the predictors at the program level. In Model 2 (the current STEM teacher group), no predictors at the program level were significant as related to outcome variable 1. Two predictors were significant at the scholar level. Factor 4: Path to teaching ( b 0.25 ; SE 0.12 ; p-value 0.05 ) was positive in relation to outcome variable 1. It showed that when Factor 4 increased one unit above the grand mean Factor 4 score, the slope increased 0.25 units after controlling for other predictors. This means that scholars that had higher Factor 4 scores tended to have higher outcome variable 1 scores. Factor 4: Path to teaching is a construct which the scholars responded to seven items regarding various aspects of courses they took and decisions about becoming teachers, including previous career status. A basic interpretation of this score is that the higher the score, the more likely the scholar had another full-time career before becoming a teacher and considered themselves to have made a career change. In addition, they were ones who were likely to take more STEM classes. Therefore, this result suggests that when scholars had higher Factor 4 scores, they were the ones who considered themselves to be more of a career changer and their perception of the influence of the Noyce funding on becoming teachers was higher. In addition, Factor 5: District/school high need environment ( b 0.19 ; SE 0.09 ; p-value 0.05 ) had an positive influence on outcome variable 1. Factor 5: District/school high need environment is a construct which the scholars responded to five items. To determine this, the scholars were asked to indicate their district/school status regarding the percentage of students receiving free or reduced lunch; the percentage of teachers lacking sufficient training in the content area they did most of their teaching in; and the percentage of teacher attrition over the last three years. A basic interpretation of these scores is that higher scores correspond to districts/schools meeting Title I requirements for being considered high need. Therefore, this result suggests that when scholars had higher Factor 5 scores, they were the ones who tended to work in districts that met title I criteria for high need and their perception of the influence of the Noyce funding on becoming teachers was higher. Race ( b 0.37 ; SE 0.19 ; p-value 0.06 ) almost had a significant effect on outcome variable 1. Again, this suggests that White scholars perceived that the Noyce funding had less influence on their decisions to becoming teachers compared to Non-white scholars. These predictors explained an additional 8% of the variance at the program level, and decreased 4 % of the variance at the scholar level. Therefore, this resulted in an intraclass correlation of 0.29. This result indicated that although most of the variance occurred at the scholar level, the program level had 29% of the variance. The results of the two conditional HLMs (for the whole scholar and the current STEM teacher group) are presented below in Table 11. Noyce Evaluation Report, Section Four: HLM 20 Table 11 Effects of Predictors on Scholars’ Perception of the Influence of the Scholarship on Becoming Teachers for “All Scholar” Group and “Current STEM Teacher” Group (Outcome Variable 1) All Scholars Current STEM Teachers b SE P-value b SE P-value Level 2 Intercept, 00 -.01 .05 .89 .14 .12 .24 Noyce Funding, 01 .13** .03 <.01 .05 .08 .56 Mean Factor3, 02 -.38** .13 .01 -.03 .25 .91 Mean Factor8, 03 .27 .17 .12 .25 .32 .45 Level 1 Race, 10 -.41** .08 <.01 -.37 .19 .06 Gender, 20 -.12 .09 .16 -.21 .21 .34 Factor3, 30 .06 .06 .27 .13 .13 .29 Factor4, 40 .07 .06 .25 .25* .12 .05 Factor5, 50 N/A .20* .09 .04 Factor6, 60 N/A -.07 .09 .47 Factor7, 70 N/A -.12 .07 .11 Factor8, 80 .09 .07 .21 .02 .15 .87 Note. There were 405 scholars in 37 programs in the “all scholars” group. There were 250 scholars in 36 programs in the “current STEM teachers” group. N/A indicated that Factors 5 to 7 were not included in the analysis for the all scholar group. *p<.05. ** p<.01. Outcome variable 3: Would you have become a teacher if you had not received the Noyce scholarship? For outcome variable 3, an unconditional HGLM model was first fit to determine whether sufficient variance could be explained at the program level of the model for the all scholar group and the current STEM teacher group. In Model 1 (the all scholar group), the test of variance component showed there was significant program-level variance for outcome variable 3 in both groups (p-value 0.05 ). No intraclass correlation can be computed via HGLM. In order to explain more variability by adding potential predictors, conditional models were fit (Model 1-all scholar group in equation 1, Model 2-current STEM teacher group in equation 2 with a log-link function). In Model 1 (the all scholar group), Noyce funding ( OR 1.38; SE 0.13 ; p-value 0.05 ) was positively related to outcome variable 3 after controlling for other predictors at the program level. This means when teacher preparation/certification programs with higher percentage of scholars’ total tuition was covered by Noyce funding, scholars tended to answer “no” or “possibly” rather than “yes” of outcome variable 3. In other words, Noyce funding at the program-level seemed to have a positive relationship with scholars’ perception on Noyce funding for them to become Noyce Evaluation Report, Section Four: HLM 21 teachers, since scholars would not become teachers if they had not received the Noyce funding. At the scholar level, Factor 4: Path to teaching ( OR 1.44 ; SE 0.16 ; p-value 0.05 ) had a positive relationship with outcome variable 3. This result suggests that when scholars had higher Factor 4 scores, they tended to choose “no” or “possibly” rather than “yes” of outcome variable 3. This indicates that when scholars had higher Factor 4 scores, their perception of the influence of the Noyce funding on becoming a teacher was higher. The variance component dropped to 0.74 from 0.96 after adding predictors at both levels. However, the test of the variance component was still significant ( 2 65 .44 ; p-value 0.01). It means that these predictors cannot explain the variance. In Model 2 (the current STEM teacher group), Mean Factor 8: Mentoring experience ( OR 6.32 ; SE 0.83 ; p-value 0.05 ) was positively related to outcome variable 3 after controlling for other predictors at the program level. This means that scholars tend to choose “no” or “possibly” rather than “yes” of outcome variable 3 when Mean Factor 8 scores at the program-level were higher. In other words, the greater degree of mentoring experience scholar had received during and after their teacher preparation/certification program, the more Noyce funding affected scholars to teach. At the scholar level, no predictor showed a significant relationship with outcome variable 3. The variance component dropped to 0.68 from 1 after adding predictors at the both levels. The test of the variance component was not significant ( 2 36 .75 ; p-value=0.26). It means that these predictors can explain the variance in the model. The results of the two conditional HGLMs (for the whole scholar and the current STEM teacher group) are presented in Table 12. Noyce Evaluation Report, Section Four: HLM 22 Table 12 Effects of Predictors on Scholars’ Perception of the Influence of the Scholarship on Becoming Teacher for “All Scholar” Group and “Current STEM teacher” Group (Outcome Variable 3) All Scholars Current STEM Teachers OR 95% CI P-value OR 95% CI P-value Level 2 Intercept, 00 .22** .15, .33 <.01 .37* .18, .77 .01 Noyce Funding, 01 1.38* 1.06, 1.79 .02 1.29 .81, 2.05 .27 Mean Factor 3, 02 .43 .17, 1.05 .06 1.24 .37, 4.19 .72 Mean Factor 8, 03 2.50 .94, 6.65 .07 6.32* 1.18,33.98 .03 Level 1 Race, 10 .81 .41, 1.58 .54 .83 .31, 2.22 .71 Gender, 20 1.12 .69, 1.82 .64 .47 .11, 1.93 .29 Factor3, 30 .82 .57, 1.18 .28 .64 .32, 1.28 .21 Factor4, 40 1.44* 1.05, 1.99 .03 1.48 .62, 3.52 .37 Factor5, 50 N/A .87 .43, 1.78 .70 Factor6, 60 N/A .94 .41, 2.17 .89 Factor7, 70 N/A .59 .29, 1.22 .15 Factor8, 80 1.15 .84, 1.57 .39 1.42 .69, 2.90 .34 Note. There were 403 scholars in 37 programs in the “all scholars” group. There were 250 scholars in 36 programs in the “current STEM teachers” group. OR=odds ratio; CI=confidence interval. N/A indicated that Factors 5 to 7 were not included in the analysis for the all scholar group. *p<.05. ** p<.01. Variables influencing scholars’ perception of the influence of the Noyce funding on their commitment to teach in high need schools To address which scholar- and program-level variables were related to scholars’ perceptions of their commitment to teaching at high need schools, outcome variable 2: Scholars’ perception of the influence of Noyce funding on becoming teachers in high need schools, and outcome variable 4: Would you have decided to teach in a high need school if you had not participated in the Noyce scholarship Program? were used. Outcome variable 2: Scholars’ perception of the influence of Noyce funding on becoming teachers in high need schools For outcome variable 2, an unconditional HLM model was first fit to determine if sufficient variance could be explained at the program level of the model for the all scholar group and the current STEM teacher group. There was significance in the program-level variance for outcome variable 2 in the all scholar group, but not in the current STEM teacher group. For the all scholar group, the estimate for the within-program variance was 0.90, and the overall variability among Noyce Evaluation Report, Section Four: HLM 23 the true program means on the outcome variable 2 was 0.05. This resulted in an intraclass correlation of 0.06. This result indicated that most of the variance occurred at the scholar level, with 6% of the variance located at the program level. For the current STEM teacher group, the estimate for the within-program variance was 0.62, and the overall variability among the true program means on outcome variable 2 was 0.02. This resulted in an intraclass correlation of 0.03. This result indicated that the variance which can be accounted for at the program level was limited since it was only 3%. Although using HLMs for outcome variable 2 for the current STEM teacher group may not be necessary, the U of MN Noyce evaluation team decided to fit HLM conditional models for outcome variable 2 for both groups as equation 1 and 2. In Model 1 (the all scholar group), Noyce funding ( b 0.09 ; SE 0.04 ; p-value 0.05 ) was positively related to outcome variable 2. This means that outcome variable 2 scores were substantially higher in teacher preparation/certification programs where a higher percentage of scholars’ tuition was provided for by the Noyce funding. Therefore, scholars’ perceptions of the influence of the Noyce funding on becoming teachers in high need schools seems to be positively related to the percentage of the tuition covered by the Noyce funding. At the scholar level, only Factor 3: Preparation for high need schools ( b 0.14 ; SE 0.05 ; p-value 0.01 ) was significant for outcome variable 2. This suggests that scholars in teacher preparation/certification programs with more preparation for high need schools tended to have higher perceptions of the influence of the Noyce funding on becoming teachers in high need schools. Model 1 incorporated the three program-level variables and the five scholar-level variables relating to the influence of Noyce funding on scholars becoming teachers in high need schools. These predictors decreased 2% of the variance at the program level and 0.1% of the variance at the scholar level. Therefore, this resulted in an intraclass correlation of 0.04. This result indicated that most of the variance occurred at the scholar level, with 4% of the variance located at the program level. In Model 2 (the current STEM teacher group), no predictor at either the program level or the scholar level were significant related to outcome variable 2. These predictors decreased 6% of the variance at the program level and 16% of variance at the scholar level. Therefore, this resulted in an intraclass correlation of 0.12. The results of the two conditional HLMs (for the whole scholar and the current STEM teacher group) are presented in Table 13 below. Noyce Evaluation Report, Section Four: HLM 24 Table 13 Effects of Predictors on Scholars’ Perception of the Influence of Scholarship on Becoming Teachers in High Need Schools for “All Scholar” Group and “Current STEM Teacher” Group (Outcome Variable 2) All Scholars Current STEM Teachers b SE P-value b SE P-value Level 2 Intercept, 00 -.04 .05 .47 .09 .09 .37 Noyce Funding, 01 .09* .04 .03 .01 .06 .90 Mean Factor3, 02 -.07 .14 .60 .07 .17 .68 Mean Factor8, 03 -.05 .18 .81 .14 .22 .53 Level 1 Race, 10 -.07 .09 .43 -.09 .17 .65 Gender, 20 -.13 .10 .17 -.12 .21 .58 Factor3, 30 .14** .05 <.01 .15 .10 .13 Factor4, 40 .02 .05 .73 .10 .09 .27 Factor5, 50 N/A .07 .08 .40 Factor6, 60 N/A .02 .08 .79 Factor7, 70 N/A -.12 .10 .23 Factor8, 80 -.03 .06 .66 -.02 .10 .86 Note. There were 405 scholars in 37 programs in the “all scholars” group. There were 250 scholars in 36 programs in the “current STEM teachers” group. N/A indicated that Factors 5 to 7 were not included in the analysis for the all scholar group. *p<.05. ** p<.01. Outcome variable 4: Would you have decided to teach in a high need school if you had not participated in the Noyce scholarship program? For outcome variable 4, an unconditional HGLM model was first fit to determine whether sufficient variance could be explained at the program level of the model for the all scholar group and the current STEM teacher group. In Model 1 (the all scholar group), the test of variance component showed there was significant program-level variance for outcome variable 4 in both groups (p-value 0.05 ). In order to explain more variability by adding potential predictors, conditional models were then fit (Model 1-all scholar group in equation 1, Model 2-current STEM teacher group in equation 2 with an ordered logit-link function). In Model 1 (the all scholar group), none of the predictors at the program level was significantly related to outcome variable 4. At the scholar level, race ( OR 0.44 ; SE 0.19 ; p-value 0.01) had a negative relationship with outcome variable 4. This suggests that non-White scholars tended to have a greater perception of the influence of Noyce funding on their decision to become teachers in high need schools. The variance component dropped to 0.14 from 0.26 after adding predictors at the both levels. However, the test of the variance component was not Noyce Evaluation Report, Section Four: HLM 25 significant ( 2 41 .79 ; p-value 0.14 ), which suggests that these predictors can explain the variance in the model. In Model 2 (the current STEM teacher group), Noyce Funding ( OR 0.80 ; SE 0.11 ; p- value 0.05 ) was negatively related to outcome variable 4 after controlling for other predictors at the program level. At the scholar level, Factor 6: Personal beliefs towards teaching ( OR 2.07 ; SE 0.30 ; p-value 0.05 ) showed a positive relationship with outcome variable 4. Factor 6: Personal beliefs towards teaching included nine items which the scholars responded on the scholar survey. Higher scores correspond to higher levels of job satisfaction, opportunities of professional growth, and higher self-efficacy towards teaching. Therefore, this result suggests that when scholars had higher Factor 6 scores, they tended to choose “No/I have not taught in a high need school” rather than “Yes” of outcome variable 4. In other words, when scholars had higher Factor 6 scores, their perception of the influence of the Noyce funding on teaching in high need schools was higher. The variance component dropped to <.01 from 0.13 after adding predictors at the both levels. The test of the variance component was not significant ( 2 28 .86 ; p-value>0.50), which suggests that these predictors can explain the variance in the model. The results of the two conditional HGLMs (for the whole scholar and the current STEM teacher group) are presented in Table 14. Noyce Evaluation Report, Section Four: HLM 26 Table 14 Effects of Predictors on Scholars’ Perception of the Influence of Scholarship on Becoming Teachers in High Need Schools for “All Scholar” Group and “Current STEM Teacher” Group (Outcome Variable 4) All Scholars Current STEM Teachers OR 95% CI P-value OR 95% CI P-value Level 2 Intercept, 00 .56** .44, .72 <.01 .50** .33, .76 <.01 Noyce Funding, 01 .90 .79, 1.03 .13 .80* .64, 1.00 .05 Mean Factor 3, 02 1.58 .92, 2.72 .10 .86 .35, 2.10 .73 Mean Factor 8, 03 .94 .51, 1.73 .84 .45 .15, 1.33 .14 Level 1 Race, 10 .44** .30, .64 <.01 .74 .28, 1.94 .53 Gender, 20 1.42 .85, 2.37 .18 1.57 .55, 4.48 .40 Factor3, 30 1.03 .82, 1.30 .77 1.18 .71, 1.96 .53 Factor4, 40 .98 .75, 1.29 .90 .54 .29, 1.00 .51 Factor5, 50 N/A .94 .54, 1.63 .81 Factor6, 60 N/A 2.07* 1.14, 3.78 .02 Factor7, 70 N/A 1.11 .67, 1.85 .68 Factor8, 80 1.07 .85, 1.36 .56 .83 .51, 1.35 .46 Threshold Difference, 2 14.09** 10.22, 19.42 <.01 39.10* 17.00, 89.89 <.01 * Level-2 variance, 00 0.14 .14 <.01 >.50 Note. There were 403 scholars in 37 programs in the “all scholars” group. There were 250 scholars in 36 programs in the “current STEM teachers” group. N/A indicated that Factors 5 to 7 were not included in the analysis for the all scholar group. *p .05. ** p<.01. Noyce Evaluation Report, Section Four: HLM 27 Conclusions This report examined the relationships among potential predictors and scholars’ perceptions of the influence of Noyce funding on becoming teachers and becoming teachers in high need schools. Two sets of data were used. One set, the all scholars group, was a survey of 405 scholars from 37 programs; the second, the current STEM teacher group, was a survey of 250 scholars from 36 programs. Results from the two-level HLM analyses revealed that most of the variance in the influence of the Noyce funding occurred at the scholar level rather than the program level. In addition, more variance at the program level occurred for scholars’ perceptions of the influence of the Noyce funding to become teachers than for explaining scholars’ perceptions of the influence of the Noyce funding on teaching in high need schools. Nine percent of the variance from the unconditional HLM model of the all scholar group, and 22% of the variance of the current STEM teacher group were located at the program level for outcome variable 1 (scholars’ perceptions of the influence of the Noyce funding to become teachers). On the other hand, 5% of the variance from the unconditional HLM model of the all scholar group, and 3% of the variance of the current STEM teacher group were located at the program level for outcome variable 2 (scholars’ perceptions of the influence of the Noyce funding to teach in high need schools). The summary of effects of predictors on scholars’ perceptions of the influence of the Noyce funding on becoming teachers and becoming teachers in high need schools for the all scholar group and the current STEM teacher group are listed in Tables 15 and 16, respectively. From the two tables, it can be inferred that only a few predictors have significant relationships with the outcome variables, and that these significant predictors do not have consistent relationships across the outcome variables and two groups. However, although the results for the four outcome variables and the two groups are not totally the same, a few conclusions may be drawn and they are presented in the following two sections. The first section summarizes the scholars’ individual characteristics and perceptions, and the second section summarizes programs’ characteristics. Noyce Evaluation Report, Section Four: HLM 28 Table 15 Summary of Effects of Predictors on Scholars’ Perception of the Influence of the Scholarship on Becoming Teachers for “All Scholar” Group and “Current STEM Teacher” Group All Scholars Current STEM Teachers Outcome Outcome Outcome Outcome variable 1 variable 3 variable 1 variable 3 Level 2 Noyce Funding ↑ ↑ Mean Factor3: Preparation for high need schools ↓ Mean Factor8: Mentoring experiences ↑ Level 1 Race: 0=Non-white; 1=White ↓ Gender: 0=Female; 1=Male Factor3: Preparation for high need schools Factor4: Path to teaching ↑ ↑ Factor5: District/school high need environment N/A N/A ↑ Factor6: Personal beliefs towards teaching N/A N/A Factor7: School teaching environment N/A N/A Factor8: Mentoring experience Note. N/A indicated that Factors 5 to 7 were not included in the analysis for the all scholar group. The direction of the arrows shows the relationship between a predictor and an outcome variable; “↑” indicates a positive relationship, and “↓” indicates a negative relationship. Table 16 Summary of Effects of Predictors on Scholars’ Perception of the Influence of the Scholarship on Becoming Teachers in High Need Schools for “All Scholar” Group and “Current STEM Teacher” Group All Scholars Current STEM Teachers Outcome Outcome Outcome Outcome variable 2 variable 4 variable 2 variable 4 Level 2 Noyce Funding ↑ ↓ Mean Factor3: Preparation for high need schools Mean Factor8: Mentoring experiences Level 1 Race: 0=Non-white; 1=White ↓ Gender: 0=Female; 1=Male Factor3: Preparation for high need schools ↑ Factor4: Path to teaching Factor5: District/school high need environment N/A N/A Factor6: Personal beliefs towards teaching N/A N/A ↑ Factor7: School teaching environment N/A N/A Factor8: Mentoring experience Note. N/A indicated that Factors 5 to 7 were not included in the analysis for the all scholar group. The direction of the arrows shows the relationship between a predictor and an outcome variable; “↑” indicates a positive relationship, and “↓” indicates a negative relationship. Noyce Evaluation Report, Section Four: HLM 29 Scholars’ individual characteristics and perceptions Because of the extra items on the surveys for the current STEM teachers, there were different numbers of predictors at the scholar level for the two different groups of scholars. There were five predictors for the all scholar group, and eight predictors for the current STEM teacher group. The five predictors used for both groups were Race, Gender, Factor 3: Preparation for high need schools, Factor 4: Path to Teaching, and Factor 8: Mentoring experience. An additional three predictors for the current STEM teacher group were Factor 5: District/school high need environment, Factor 6: Personal beliefs towards teaching, and Factor 7: School teaching environment. Race: Race (Whites vs. Non-whites) was an important predictor in the all scholar group for two of the four outcome variables. Race was a significant predictor for outcome variable 1: Influence of scholarship on STEM majors becoming teachers for the all scholar group. The results showed that the Noyce funding influences Non-whites more than Whites to become teachers. Race was also a significant predictor for outcome variable 4: Would you have decided to teach in a high need school if you had not participated in the Noyce funding Program? for the all scholar group. These results indicated that Non-whites tended to have higher perceptions of the effect of Noyce funding on teaching in high need schools than Whites. These results about race corroborates most of the results from Final Report Section Five: Combined Analysis of the Robert Noyce Teacher Scholarship Program Using ORC Macro and UMN Evaluation Data (Bowe et al., 2009). That report also showed a significant difference between Whites and Non-whites on the continuous outcome variable 1 with Non-whites having higher perceptions of the effect of the Noyce funding and no significant effects for outcome variable 2 and 3. In contrast to the present report, Final Report Section Five showed a higher percentage of Non-whites saying “yes” they would have taught in high need schools even without getting Noyce funding; suggesting that more Whites reported that they felt influenced by the Noyce funding to teach in a high need school. However, it was difficult to accurately interpret the results in Final Report Section Five because of the effect of the “possibly” category and because only visual inspection was possible—not a statistical test. The results for outcome variable 4 in the present report with more sophisticated analyses indicated that Non-whites perceived the Noyce funding as being more influential in their decision to become teachers in high need schools compared to Whites. The effect of race for the current STEM teachers on the outcome variables was also examined through cluster analysis (Liou, Desjardins, & Lawrenz, in press). In that analysis three clusters emerged but the clusters combined teaching and teaching in high need schools rather than separating these outcomes as is the case in this HLM analysis. In the cluster analyses 47% of the Non-whites were in the cluster, “Highly committed to becoming a teacher and teaching in a high needs schools,” while 27% and 26% were in the other two clusters “Highly committed to becoming a teacher but not in high needs schools” and “Not highly committed to becoming a Noyce Evaluation Report, Section Four: HLM 30 teacher or to teach in high need schools.” In the cluster analysis, it was suggested that Non- white scholars would have already been more committed to teaching in high need schools, so Noyce funding would have had less effect on Non-whites in terms of teaching in high need schools. This result appears to conflict with the results in the present report; however, the analyses are different and the group of current STEM teachers in the cluster analysis is a subset of the all scholars group which showed the significant findings in the HLM analyses reported here. In terms of analyses, the cluster analyses used the four outcome variables to produce clusters and assign the current STEM teachers into the clusters. Then the current STEM teachers’ characteristics were further investigated. However, in the present report using HLM, the relationships between current STEM teachers’ characteristics and each of the outcome variables were analyzed simultaneously. Preparation for high need schools: Factor 3: Preparation for high need schools was a significantly positive predictor for outcome variable 2 for the all scholar group. This result indicated that including curricula and activities for high need schools in Noyce programs played a role in increasing scholars’ perceptions of the effect of Noyce funding on their decision to teach in high need schools. These curricula and activities included developing specific strategies for teaching students from diverse racial and ethnic backgrounds, considering the relationship between education and social justice and/or democracy, education about how to work in high need schools specifically, etc.. However, Factor 3 did not show a significant relationship with outcome variable 4 which also related to the effect of the Noyce funding on teaching in high needs schools for either group. Additionally, Factor 3 was not significantly related to the effect of Noyce funding on scholars becoming teachers (outcome variable 1 and 3) in either group. Path to teaching: Factor 4: Path to teaching was a significantly positive predictor for outcome variable 1 for the current STEM teacher group, and correspondingly was a positive predictor for outcome variable 3 for the all scholar group. This result indicated that scholars’ path to teaching increased the effect of Noyce funding on scholars’ decisions to become teachers. The construct of path to teaching included various aspects of courses scholars took and decisions about becoming teachers, including previous career status. In other words, scholars that had another full-time career before becoming a teacher and considered themselves to have made a career change tended to perceive a greater effect of the Noyce funding on the decision to become teachers. District/school high need environment: Factor 5: District/school high need environment is a significantly positive predictor for outcome variable 1 for the current STEM teacher group. This result indicated that current STEM teachers’ perceptions of the effect of the Noyce program on their decision to become teachers had a positive relationship with their district/school high need environment. Results of items in Noyce Evaluation Report, Section Four: HLM 31 District/school high need environment included items pertaining to district or school high need status according to Title I criteria for percentage of teachers lacking sufficient training in the content area they did most of their teaching in either in; the percentage of students receiving free or reduced lunch in their districts or schools, and the percentage of teacher attrition over the last three years. It may be inferred that in short, the more current STEM teachers perceived their teaching environment as high need the more they perceived the Noyce funding as having influenced them to become teachers. Personal beliefs toward teaching: Factor 6: Personal beliefs towards teaching was a positively significant predictor for outcome variable 4 for the current STEM teacher group. This result indicated that the more current STEM teachers’ personal beliefs toward teaching increased, the higher they perceived the effect of Noyce funding on their decisions to teach in high need schools. These personal beliefs include I am satisfied with my current teaching job, I would still become a teacher if I had to do it all over again, I would still choose the same teacher preparation program into teaching if I had to do it all over again, etc.. It may also be inferred that when these STEM teachers had more positive perceptions of their levels of job satisfaction, opportunities of professional growth, and self- efficacy towards teaching, their perceptions of the effect of the Noyce funding on their decision to teach in high need schools was greater. Other scholars’ characteristics and perceptions: Three other predictors, gender, Factor 7: School teaching environment and Factor 8: Mentoring experience, at the scholar level did not show a significant relationship with the outcome variables. These results about gender confirmed the results from Final Report Section Five (Bowe, et al., 2009) and the cluster analysis (Liou, Desjardins, & Lawrenz, in press). The results in Final Report Section Five showed that gender was not significantly related to the continuous outcome variables 1 and 3 although females’ scores were higher than males’. In addition, the results from the cluster analysis (Liou, et al., in press) also supported the notion that gender is not an essential variable related to scholars’ perceptions about the effects of the Noyce funding. Factor 7 and Factor 8 did not reach significance for any outcome variable. However, according to the correlation matrices (Tables 7 and 8); there were positive relationships between school teaching environment and the outcome variables as well as mentoring experience and the outcome variables. Programs’ characteristics Three predictors at the program level were analyzed in this report. Noyce money, Mean Factor 3: Preparation for high need schools, and Mean Factor 8: Mentoring experience. All of the program-level predictors showed statistically significant relationships with an outcome variable in at least one of the two groups. Noyce Evaluation Report, Section Four: HLM 32 Noyce funding: Noyce funding was an important variable in explaining scholars’ perceptions of the effect of Noyce funding on becoming a teacher and teaching in high need schools. Noyce funding was a positively significant variable for outcome variables 1, 2, and 3 for the all scholar group, and also a positively significant variable for outcome variable 3 for the current STEM teacher group. These results indicated that the greater the percentage of scholars' tuition covered by the Noyce funding, the more the scholars perceived that the Noyce program influenced their decision to become teachers and to teach in high need schools. However, Noyce funding showed a negatively significant relationship with outcome variable 4 for the current STEM teacher group. This result suggested that that greater the percentage of scholars' tuition covered by the Noyce funding at the program level, the less the Noyce funding influenced current STEM teachers' decisions to teach in high need schools. This result is contrary to the other results. One explanation is that more current STEM teachers chose “possibly” instead of “no/I have no taught in a high need school” or “yes,” and choosing the “possibly” option caused a relatively uncertain effect when analyzing the data. If “no/I have not taught in a high need school” and “possibly” were combined into one option instead of the trichotomous data that were used, and the HLM rerun, the results from the dichotomous data showed a positive relationship indicating that the Noyce funding was more influential in the scholars’ decisions to teach in high need schools. Therefore, based on the summary results from these four outcome variables, it can be concluded that the amount of Noyce funding at the program-level had a positive relationship with the effect of the Noyce funding on becoming teachers and teaching in high need schools. Preparation for high need schools: Mean Factor 3: Preparation for high need schools was a negatively significant predictor for outcome variable 1 for the all scholar group. This indicates that as the mean score of the perceptions of the scholars in a preparation program became higher the scholars’ perceptions of the effect of the Noyce funding became lower. In other words, scholars in programs that they perceived as having opportunities for them to learn about high need schools were less likely to perceive the Noyce funding as influential in them becoming teachers. Recall that mean Factor 3 was produced by aggregating and averaging Factor 3: Preparation for high need schools scores from individual scholars within a program. One explanation for a negative relationship might be that there was more focused recruitment in programs that were specifically designed to prepare teachers for high need schools. In other words, these types of programs were more likely to give funding to scholars who already had high commitment to being teachers. Or perhaps, scholars who were more influenced by the Noyce funding were more critical of the preparation for teaching in high need schools provided by their programs. In contrast to this result, the relationship between Factor 3 and outcome variable 1 for the all scholar group was positive at the scholar level. One possible explanation is that there is a shift in the meaning of Factor 3 from the individual level to the aggregated level; that aggregation by program changed the meaning that should be ascribed to the Factor. This may imply that items which form Factor 3 at the scholar level need to be validated for use at the program level. Noyce Evaluation Report, Section Four: HLM 33 Mentoring experience: Mean Factor 8: Mentoring experience is a positively significant predictor for outcome variable 3 for the current STEM teacher group. This result indicated that the mentoring experience had a positive effect on scholars’ perceptions of the effect of Noyce funding on becoming teachers. The content of Factor 8 included mentoring experiences provided by their certification program and their district during their first and second year of teaching, a guaranteed job at a participating school district(assuming successful completion of the program), and continuing contact with participants in their teacher education program. Therefore, it can be inferred that when programs and districts provided more mentoring on these current STEM teachers, their perceptions of the influence of the Noyce funding for them to become teachers were higher. Noyce Evaluation Report, Section Four: HLM 34 Limitations There were several limitations of this report that require discussion. First, the data were from a cross-sectional evaluation survey and it did not employ an experimental or quasi-experimental design. Therefore, there is no evidence or basis to say causal relationships exist among the scholars’ demographic characteristics, their perceptions, and their perceived effect of Noyce funding for them to become teachers and teach in high need schools. The only credible statements that can be made from the data are statements regarding correlations. Because this is a cross-sectional study and not a longitudinal study, it is impossible to examine retention and continued satisfaction with teaching or Noyce programs. Second, race is an important variable explaining scholars’ perceptions about the influence of Noyce funding on becoming teachers and becoming teachers in high need schools, but only Whites and Non-whites were used in this study because of small within-group sample sizes. All scholars identified as Non-white may not share convergent beliefs about teaching and it is likely that with a finer grained lens, certain ethnic groups might perceive an effect of the scholarship on their decision to enter teaching, to teach at high need schools, or both. Third, the sample size of scholars in the current STEM teacher group is small. Statistical power becomes an issue when analyzing a small number of programs. Although 36 programs existed in the analysis, there were only 250 scholars from these 36, therefore it is possible that not enough information was obtained. Another limitation of this study was that these predictors variables (race, gender, and Factor 3-8 at the scholar level, and Noyce funding, mean factor 3 and mean factor 8 at the program level) were unable to explain the variance, since only a small increase in the amount of variance was accounted for difference between the unconditional models and conditional models for the four outcome variables for both groups. Lastly, the potential impact from non-response bias is another limitation for this report, because responding to the survey was voluntary. Therefore, it may be that the scholars who responded were different from those who did not respond in some systematic ways. If there were systematic differences between respondents and nonrespondents, the data might be biased and unrepresentative of the whole population. Therefore, the response bias is possible and could have influenced the results. Noyce Evaluation Report, Section Four: HLM 35 References Bowe, A., Liou, P.-Y., & Lawrenz, F. (2009). Final Report Section Five: Combined Analysis of the Robert Noyce Teacher Scholarship Program using ORC Macro and UMN Evaluation Data. University of Minnesota, Minneapolis. Lawrenz, P., Appleton, J., Bullitt Bequette, M., Desjardins, C., Liou, P.-Y., Madsen, C., & Ooms, A. (2008). Final Report Section One: Planning and Survey Data Noyce Program Evaluation. University of Minnesota, Minneapolis. Liou, P.-Y., & Lawrenz, P. (2008). Final Report Section Two: Factor Analysis of Robert Noyce Scholarship Program Evaluation. University of Minnesota, Minneapolis. Liou, P.-Y., Desjardins, C., & Lawrenz, F. (in press). Demographics of STEM teacher scholarship recipients and their perceptions about teaching in high-needs schools via cluster analysis. School Science and Mathematics. Liou, P.-Y., Kirchhoff, A., & Lawrenz, P. (in press). Perceived effects of scholarships on STEM majors’ commitment to teaching in high need schools. Journal of Science Teacher Education. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage. 2nd edition. Raudenbush, S. W., Bryk, A. S., & Congdon, R. (2005). HLM 6: Hierarchical linear and nonlinear modeling. Lincolnwood, IL: Scientific Software International, Inc. Noyce Evaluation Report, Section Four: HLM 36 Appendix A: Items, factor loadings and sample sizes for factors from scholar survey Table A1 Items, Loading and Sample Size for “Commitment to Teaching in High Need Schools (Influence of Noyce)” Factor (6 items) Item Factor Sample loading size IV.8.c Take a teaching job 0.82 270 IV.8.d Teach in a high need school 0.80 270 IV.8.e Remain teaching in a high need school for the full term of 0.78 269 your commitment IV.8.a Become a teacher 0.77 270 IV.8.f Remain teaching in a high need school beyond the full term of 0.75 269 your commitment IV.8.b Complete the certification program 0.68 270 Note. All items from scholar survey question IV.8: “How influential is the Noyce scholarship money in your commitment to: _____?” Table A2 Items, Factor Loadings and Sample Sizes for “Preparation for High Need Schools” Factor (13 items) Factor Sample Item loading size II.4.c Develop specific strategies for teaching students from diverse 0.66 273 a racial and ethnic backgrounds II.4.d Consider the relationship between education and social justice 0.63 273 a and/or democracy II.1.e Education about how to work in high need schools 0.52 274 b specifically II.4.a Develop specific strategies for teaching English language 0.50 273 learners (those with limited English proficiency)a II.4.b Develop specific strategies for teaching students identified 0.49 273 with learning disabilitiesa II.1.h Student teaching experience in a high need schoolb 0.48 272 II.2.d Supervised actual classroom teaching in high need schools 0.45 263 c (this may be called student teaching, internship, etc. in your state) II.1.f Opportunities to observe/work at high need schools (not 0.35 274 b student teaching) II.1.c Education about different culturesb 0.35 274 II.1.g Student teaching experienceb 0.34 274 b II.1.b Opportunities to interact with children from different cultures 0.32 274 II.2.a Education field experience working in schools (e.g. tutoring, 0.32 267 Noyce Evaluation Report, Section Four: HLM 37 teacher aide) with young people like those who attend high need schoolsc II.1.a Opportunities to interact with adults from different culturesb 0.28 274 a Items from scholar survey question II.4: “Five types of experiences are listed below. Please indicate whether each of the experiences below is a requirement of your teacher certification program. Indicate the length of any required experiences” b Items from scholar survey question II.1: “Which of these are part of your experience in your teacher certification program?” c Items from scholar survey question II.2: “In your teacher certification program, how much opportunity do you have to do the following:_____?” Table A3 Items, Factor Loadings and Sample Sizes for “Path to Teaching” Factor (7 items) Factor Sample Item loading size IV.2 What age were you when you began the teacher certification 0.81 270 program?__years V.3 Did you work full time before the teacher certification program? 0.69 231 If yes, in what field was the majority of your work? IV.1 At what point in your life did you decide to become a STEM 0.68 270 teacher? V.4 In entering the teacher certification program, do you consider 0.68 267 yourself to have made a “career change”? V.1.a How many upper level (junior/senior) or graduate level classes 0.30 171 have you taken in: STEM? V.2.a In what year did you last take a formal course for college -0.25 258 credit in: Mathematics V.2.b In what year did you last take a formal course for college -0.25 239 credit in: Science Table A4 Items, Factor Loadings and Sample Sizes for “District/school high need environment” Factor (5 items) Factor Sample Item loading size III.4.b Over 33% of teachers lack sufficient training in their 0.79 161 academic area.(district)a III.3.b Over 33% of teachers lack sufficient training in their 0.67 161 b academic area.(school) III.3.a Over 50% of students receive free or reduced lunch.(school) b 0.44 163 a III.4.a Over 50% of students receive free or reduced lunch.(district) 0.44 162 III.1 Which of the following describes your current teaching status? 0.36 158 a Items from scholar survey question III.4: “Which of the following characterize your school? For each option, indicate whether the description applies or not, or whether you are not sure” b Items from scholar survey question III.3: “Which of the following characterize your district? For each option, indicate whether the description applies or not, or whether you are not sure” Noyce Evaluation Report, Section Four: HLM 38 Table A5 Items, Factor Loadings and Sample Sizes for “Personal beliefs towards teaching” Factor (9 items) Factor Sample Item loading size a III.2.a I am satisfied with my current teaching job 0.67 157 III.2.c If I had to do it all over again, in view of my present 0.65 164 knowledge, I would become a teachera III.2.d If I had it to do all over again, I would choose the same 0.58 164 teacher preparation program and/or route into teachinga III.2.e In the next three years, I am likely to assume a leadership 0.35 158 position (e.g., lead teacher, department chair, official or unofficial mentor)a IV.3.f I feel that I have a talent for teaching STEMb 0.32 263 a III.2.b I really dislike STEM teaching -0.32 160 III.5 Within the last three years have you held any professional 0.28 157 educational leadership positions, e.g., lead mathematics teacher, science committee chair, etc. IV.3.d I like the flexibility and/or autonomy of STEM teachingb 0.26 259 b IV.3.b I like working with young people 0.23 268 a Items from scholar survey question III.2: “How much do you agree or disagree with each of the following statements about teaching?” b Items from scholar survey question IV.3: “Did any of the following help you decide to become a STEM teacher?” Table A6 Items, Factor Loadings and Sample Sizes for “School teaching environment" (4 items) Factor Sample Item loading size III.6.b Strong collaborative leadership (e.g., principals and other 0.77 116 leaders provide teachers with opportunities to do well; principals and other leaders can be trusted; principals and other leaders share your vision of successful classroom practice) III.6.a Collegial relationships (e.g., teachers consult on the quality of 0.71 115 student work and make joint decisions based on assessment, collaborate to solve classroom challenges, observe and discuss each others’ instruction) III.6.d Mentoring and/or induction support (e.g., organized, 0.66 116 supported contact with a more experienced teacher, help with issues particular to early career teaching) III.6.c Availability of supplies or material (e.g., textbooks, print 0.48 115 resources, instructional materials such as lab supplies or math manipulatives, and classroom supplies such as paper, pencils, or tape) Noyce Evaluation Report, Section Four: HLM 39 Note. All items from scholar survey question III.6: “Please rate your school environment as high, medium, or low on the following features” Table A7 Items, Factor Loadings and Sample Sizes for “Mentoring Experience” Factor (6 items) Factor Sample Item loading size II.1.l Mentoring experiences provided by your certification program 0.58 164 during your second year of teaching II.1.m Mentoring experiences provided by your district during your 0.55 264 second year of teaching II.1.j Mentoring experiences provided by your certification program 0.51 272 during your first year of teaching II.1.k Mentoring experiences provided by your district during your 0.49 271 first year of teaching II.1.i A guaranteed job (assuming successful completion of program) 0.37 271 at a participating school district II.1.n Continuing contact with participants in your teacher education 0.30 270 program Note. All items from scholar survey question II.1: “Which of these are part of your experience in your teacher certification program?” Noyce Evaluation Report, Section Four: HLM 40 Appendix B: Groups of scholars within the eight factors The scholars were grouped into 7 groups based on their current teaching status. These groups were given different forms of the scholar survey. Because of this, not all groups were included in all factors. This appendix details which groups of scholars were included in each factor. Table B1 Groups of scholars based on current educational/teaching status Group Teaching status No. 1. In a teacher certification program, not yet a full-time teacher 2. Completed a teacher certification program, but never taught 3. Did not complete a teacher certification program and will not return 4. In a teacher certification program and teaching full-time as part of that program 5. Teacher full-time or part-time 6. Taught after being certified and now working in education but not as a teacher 7. Taught after being certified and now not working in education Table B2 Groups of scholars included in each factor Factor 1: Influence of scholarship on STEM majors becoming teachers (the total number of items=3) Item Groups IV.8.a become a teacher All groups IV.8.b complete the certification program All groups IV.8.c take a teaching job All groups Factor 2: Influence of scholarship on STEM majors becoming high need teachers Item Groups IV.8.d teach in a high need school All groups IV.8.e remain teaching in a high need school for the full term of All groups your commitment IV.8.f remain teaching in a high need school beyond the full term of All groups your commitment Factor 3: Preparation for high need schools (the total number of items=13) Item Groups II.4.c Develop specific strategies for teaching students from diverse All groups racial and ethnic backgrounds II.4.d Consider the relationship between education and social justice All groups and/or democracy Noyce Evaluation Report, Section Four: HLM 41 II.1.e Education about how to work in high need schools specifically All groups II.4.a Develop specific strategies for teaching English language All groups learners (those with limited English proficiency) II.4.b Develop specific strategies for teaching students identified All groups with learning disabilities II.1.h Student teaching experience in a high need school All groups II.2.d Supervised actual classroom teaching in high need schools All groups (this may be called student teaching, internship, etc. in your state) II.1.f Opportunities to observe/work at high need schools (not All groups student teaching) II.1.c Education about different cultures All groups II.1.g Student teaching experience All groups II.1.b Opportunities to interact with children from different cultures All groups II.2.a Education field experience (e.g. tutoring, teacher aide) All groups working in schools with young people like those who attend high need schools II.1.a Opportunities to interact with adults from different cultures All groups Factor 4: Path to teaching (the total number of items=7) Item Groups IV.2 What age were you when you began the teacher certification All groups program?__years V.3 Did you work full time before becoming a teacher? If yes, in All groups what field was the majority of your work? If yes, in what field was the majority of your work? IV.1 At what point in your life did you decide to become a STEM All groups teacher? V.4 In becoming a teacher, do you consider yourself to have made a All groups “career changer”? V.1.a How many STEM classes were taken? All groups V.2.a In what year did you last take a formal course for college All groups credit in: Mathematics V.2.b In what year did you last take a formal course for college All groups credit in: Science Factor 5: District/school high need environment (the total number of items=5) Item Groups III.4.b Over 33% of teachers lack sufficient training in their 4, 5, 6, 7 academic area.(district) III.3.b Over 33% of teachers lack sufficient training in their 4, 5, 6, 7 academic area.(school) III.3.a Over 50% of students receive free or reduced lunch.(school) 4, 5, 6, 7 III.4.a Over 50% of students receive free or reduced lunch.(district) 4, 5, 6, 7 III.1 Which of the following describes your current teaching status? 4, 5 Factor 6: Personal beliefs towards teaching (the total number of items=9) Noyce Evaluation Report, Section Four: HLM 42 Item Groups III.2.a I am satisfied with my current teaching job. 4, 5, 6, 7 III.2.c If I had to do it all over again, in view of my present 4, 5, 6, 7 knowledge, I would become a teacher III.2.d If I had it to do all over again, I would choose the same 4, 5, 6, 7 teacher preparation program and/or route into teaching III.2.e In the next three years, I am likely to assume a leadership 4, 5 position (e.g., lead teacher, department chair, official or unofficial mentor) IV.3.f I feel that I have a talent for teaching STEM All groups III.2.b I dislike STEM teaching 4, 5, 6, 7 III.5 Within the last three years have you held any professional 4, 5 educational leadership positions, e.g., lead mathematics teacher, science committee chair, etc. IV.3.d I like the flexibility and/or autonomy of STEM teaching. All groups IV.3.b I like working with young people All groups Factor 7: School teaching environment (the total number of items=4) Item Groups III.6.b Strong collaborative leadership (e.g., principals and other 4, 5, 6, 7 leaders provide teachers with opportunities to do well; principals and other leaders can be trusted; principals and other leaders share your vision of successful classroom practice) III.6.a Collegial relationships (e.g., teachers consult on the quality of 4, 5, 6, 7 student work and make joint decisions based on assessment, collaborate to solve classroom challenges, observe and discuss each others III.6.d Mentoring and/or induction support (e.g., organized, 4, 5, 6, 7 supported contact with a more experienced teacher, help with issues particular to early career teaching) III.6.c Availability of supplies or material (e.g., textbooks, print 4, 5, 6, 7 resources, instructional materials such as lab supplies or math manipulatives, and classroom supplies such as paper, pencils, or tape) Factor 8: Mentoring experience (the total number of items=6) Item Groups II.1.l Mentoring experiences provided by your certification program All groups during your second year of teaching II.1.m Mentoring experiences provided by your district during your All groups second year of teaching II.1.j Mentoring experiences provided by your certification program All groups during your first year of teaching II.1.k Mentoring experiences provided by your district during your All groups first year of teaching II.1.i A guaranteed job (assuming successful completion of program) All groups Noyce Evaluation Report, Section Four: HLM 43 at a participating school district II.1.n Continuing contact with participants in your teacher education All groups program Noyce Evaluation Report, Section Four: HLM 44 Appendix C: All institutions Number of Scholars Who Number of PIs Who Institution Responded Responded Auburn Universitya 2 1 Baylor College of Medicine 12 1 BOE City of St. Louisa 1 1 Boise Statea 1 1 Brownsville ISD 6 1 California Polytechnic State No responders 1 Foundation, Inc. California Polytechnic, Pomona 8 1 California State University Fresno 28 1 California State University- No responders 1 Fullerton Foundation California State University-Long 12 1 Beach Foundation California State University-San No responders 1 Bernardino Foundation California State University San 9 1 Marcos Claremont Graduate University 7 1 Clark Atlanta University No responders 1 Cornell University 9 1 Dowling College 14 1 Drexel University 5 1 Duke University 6 1 Foundation @ NJIT, New Jersey 1 1 Institute of Technologya Georgia State University Research 8 1 Foundation, Inc. Harvard University 7 1 Howard University 6 1 Indiana State University 4 1 Indiana University 6 2 Kean University 6 1 Kentucky Science & Technology 1 6 Council Lake City Community Collegea 3 1 Louisiana State University & 1 Agricultural and Mechanical 15 College Maine Math and Science Alliance 29 1 Michigan State Universityb 22 No response Noyce Evaluation Report, Section Four: HLM 45 Our Lady of the Lake University 11 1 PA State System of Higher 1 No responders Education Portland State University No responders 1 Saint Ambrose University 7 1 San Diego State Universityb 11 No response San Jose State University 15 1 Seattle Pacific University No responders 1 SUNY at Buffalo 10 1 SUNY at Fredonia No responders 1 SUNY at Stony Brook 6 1 Texas A&M Univ, College Station 22 1 Texas A&M University, Kingsville 16 1 Texas A&M University, Main 1 No responders Campus Texas A&M University, Texarkana 14 1 Trinity University 10 1 University of Arizona 5 1 University of Arkansas, Little Rocka 4 1 University of California Los 1 17 Angeles University of Central Floridaa 4 1 University of Cincinnati Main 1 9 Campus University of Colorado at Boulder 10 1 University of Illinois at Chicago 36 1 University of Kansas No responders 1 University of Massachusetts 1 8 Amherst University of Massachusetts 1 2 Bostona University of Massachusetts Lowell 1 2 Research Foundationa University of Minnesota 25 1 University of Missouri-Columbia 11 1 University of New Mexico 5 1 University of North Caroline 1 4 Pembrokea University of North Texas 13 1 University of Southern Mississippi No responders 1 University of Texas at Austin 21 1 University of Texas at El Paso No responders 1 University of Texas at San Antonio 6 1 Washington State University No responders 1 Wayne State University 18 1 Noyce Evaluation Report, Section Four: HLM 46 The University Corporation, 1 No responders Northridge a These institutions were omitted from the analyses due to low numbers of scholars within the programs which violated assumptions of multi-level regression analyses b Scholars from these institutions were omitted from the analyses as the PIs did not provide program-level data from the PI survey Noyce Evaluation Report, Section Four: HLM 47 Appendix D: U of MN Noyce evaluation team comprehensive evaluation report sections Lawrenz, F., Appleton, J., Bullitt Bequette, M., Desjardins, C., Liou, P.-Y., Madsen, C., & Ooms, A. (2008). University of Minnesota Evaluation of the Robert Noyce Teacher Scholarship Program, Final Report Section One: Planning and survey data. Liou, P.-Y. and Lawrenz, F. (2008). University of Minnesota Evaluation of the Robert Noyce Teacher Scholarship Program, Final Report Section Two: Factor analysis of the evaluation questionnaire. Kirchhoff, A. and Lawrenz, F. (2008). University of Minnesota Evaluation of the Robert Noyce Teacher Scholarship Program, Final Report Section Three: District representative perceptions of and experiences with the Robert Noyce Teacher Scholarship Program. Liou, P.-Y. and Lawrenz, F. (2009). University of Minnesota Evaluation of the Robert Noyce Teacher Scholarship Program, Final Report Section Four: The influence of scholar and program level variables on scholar perceptions of the effect of the Noyce funding. Bowe, A., Liou, P.-Y. and Lawrenz, F. (2009). University of Minnesota Evaluation of the Robert Noyce Teacher Scholarship Program, Final Report Section Five: Combined analysis of the Robert Noyce Teacher Scholarship Program using ORC Macro and UMN evaluation data. Kirchhoff, A. and Lawrenz, F. (2009). University of Minnesota Evaluation of the Robert Noyce Teacher Scholarship Program, Final Report Section Six: A model of the pathway to retention in high need settings: Analysis of Noyce scholar interviews.