Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

HLM_ Final

VIEWS: 6 PAGES: 47

									 University of Minnesota Evaluation of the Robert Noyce
Teacher Scholarship Program, Final Report Section Four:
The Influence of Scholar and Program Level Variables on
 Scholar Perceptions of the Effect of the Noyce Funding


                         Pey-Yan Liou
                      Frances Lawrenz, PI



                 Noyce Program Evaluation Team

                    Post Doctoral Members
                    Marjorie Bullitt Bequette
                      Michelle Fleming
                      Deena Wassenberg

                       Graduate Students
                         Jim Appleton
                          Anica Bowe
                        Maureen Braam
                        Chris Desjardins
                     Karen Hofstad-Parkhill
                       Allison Kirchhoff
                            Kei Lee
                       Christina Madsen
                          Ann Ooms
                          Mary Sande

                          August 2009
                                  TABLE OF CONTENTS



                                                                                Page Number
Introduction                                                                          6
Methodology                                                                           8
        Instruments                                                                   8
        Data                                                                          9
        Sample                                                                        10
        Measures                                                                      11
               Outcome variables                                                      11
               Scholar-level predictors                                               13
               Program-level predictors                                               15
        Analysis                                                                      15
               Hierarchical Linear Models (HLMs)                                      15
               Hierarchical Generalized Linear Models (HGLMs)                         16
Results                                                                               17
        Variables influencing scholars’ perceptions of the influence of the Noyce     17
        funding on their commitment to become teachers
               Outcome variable 1: Scholars’ perception of the influence of Noyce 17
               funding on becoming teachers
               Outcome variable 3: Would you have become a teacher if you had         20
               not received the Noyce scholarship?
      Variables influencing scholars’ perception of the influence of the Noyce      22
      funding on their commitment to teach in high need schools
             Outcome variable 2: Scholars’ perception of the influence of Noyce     22
             funding on becoming teachers in high need schools
             Outcome variable 4: Would you have decided to teach in high need       24
             school if you had not participated in the Noyce scholarship program?
Conclusions                                                                         27
      Scholars’ individual characteristics and perceptions                          29
             Race                                                                   29
             Preparation for high need schools                                      30
             Path to teaching                                                       30
             District/school high need environment                                  30
             Personal beliefs towards teaching                                      31
             Other scholars’ characteristics and perceptions                        31
      Programs’ characteristics                                                     31
                                                 Noyce Evaluation Report, Section Four: HLM 3


              Noyce funding                                                          32
              Preparation for high need school                                       32
              Mentoring experience                                                   33
Limitations                                                                          33
References                                                                           35



                                          APPENDICES                           Page Number

Appendix A: Items, factor loadings and sample sizes for factors from scholar         36
survey
Appendix B: Groups of scholars within the eight factors                              40
Appendix C: All institutions                                                         44
Appendix D: U of MN Noyce evaluation team comprehensive evaluation report            47
sections
                                               Noyce Evaluation Report, Section Four: HLM 4


                                 List of Tables and Figures

Tables                                                                         Page Number
Table 1    The eight factors for the scholar survey                                  9
Table 2    Frequency of outcome variable 3: Would you have become a                 11
           teacher if you had not received the Noyce scholarship? (Three
           response categories)
Table 3    Frequency of outcome variable 3: Would you have become a                 12
           teacher if you had not received the Noyce scholarship? (Two
           response categories)
Table 4    Descriptive statistics of outcome variable 4: Would you have             12
           decided to teach in a high need school if you had not
           participated in the Noyce scholarship program? (Three response
           categories)
Table 5    Four outcome variables of scholars’ perception of the Noyce              13
           program’s influence on their becoming a teacher and teaching
           in a high need school
Table 6    Scholar-level predictors for all scholar group and current STEM          14
           teacher group
Table 7    Descriptive statistics and correlation matrix for scholar-level          14
           variables for all scholar group
Table 8    Descriptive statistics and correlation matrix for scholar-level          14
           variables for current STEM teacher group
Table 9    Program-level predictors                                                 15
Table 10   Descriptive statistics and correlation matrix for program-level           15
           variables
Table 11   Effects of predictors on scholars’ perception of the influence of        20
           the scholarship on becoming teachers for “all scholar” group
           and “current STEM teacher” group (outcome variable 1)
Table 12   Effects of predictors on scholars’ perception of the influence of         22
           the scholarship on becoming teachers for “all scholar” group
           and “current STEM teacher” group (outcome variable 3)
Table 13   Effects of predictors on scholars’ perception of the influence of         24
           the scholarship on becoming teachers in high need schools for
           “all scholar” group and “current STEM teacher” group
           (outcome variable 2)
Table 14   Effects of predictors on scholars’ perception of the influence of         26
           the scholarship on becoming teachers for “all scholar” group
           and “current STEM teacher” group (outcome variable 4)
                                             Noyce Evaluation Report, Section Four: HLM 5


Table 15   Summary of effects of predictors on scholars’ perception of the      28
           influence of the scholarship on becoming teachers for “all
           scholar” group and “current STEM teacher” group
Table 16   Summary of effects of predictors on scholars’ perception of the      28
           influence of the scholarship on becoming teachers in high need
           schools for “all scholar” group and “current STEM teacher”
           group
                                                 Noyce Evaluation Report, Section Four: HLM 6



Introduction

The National Science Foundation’s (NSF) Robert Noyce Teacher Scholarship Program was
funded to increase the number of highly qualified teachers working in high need schools. This
program has individual projects situated nationally at various university sites. The overall
purpose of these projects is to recruit and support individuals with strong academic background
in science, technology, engineering, and mathematics (STEM) content areas into becoming
teachers and working in high need schools. Research shows persistent correlations between
student performance and teacher quality in science and mathematics (Goldhaber & Brewer,
1996; Jordan, Mendro, & Weerasinghe, 1997; National Research Council, 2000; Sanders and
Rivers, 1996). Other studies (Ingersoll, 1999, 2002) show that 56% of secondary students in
physical science are being taught by teachers without a major or minor in physical science, and
that students in high-poverty schools are 77% more likely to be taught by an out-of-field teacher.

To evaluate the effectiveness of the Robert Noyce Teacher Scholarship Program (hereafter
referred to as the Noyce program) on various dimensions, the University of Minnesota (U of
MN) Noyce evaluation team was founded and funded since 2005.The team has been composed
of a number of graduate students and postdoctoral students with Frances Lawrenz as the
Principal Investigator (PI). As suggested in the original evaluation project proposal, the
University of Minnesota Noyce evaluation team (hereafter referred to as U of MN Noyce
evaluation team) has four major components:

      preparation of an extensive literature review pertaining to effects of incentive programs in
       recruiting and retaining STEM K-12 teachers,
      thematic synthesis of the existing evaluation information through content analysis of
       project reports,
      statistical querying of the existing monitoring data to produce quantitative models of the
       program, and
      development and execution of an overall program evaluation plan through collaboration
       with existing projects.

In addition, the following are the main evaluation questions:

   1. What are the characteristics of the teacher preparation/certification programs provided by
      the Noyce projects?
   2. In the opinions of the PIs and scholars (i.e. Noyce scholarship/stipend recipients), what
      role did the Noyce funding play in the scholars’ decisions to teach in high need schools?
   3. In the opinions of PIs, scholars, STEM faculty and districts, in what ways and to what
      extent are the characteristics of the Noyce program related to
           Recruiting individuals with strong STEM backgrounds, including those who
              might not otherwise consider teaching/going into teaching careers
           Retaining scholarship/stipend recipients in the teaching profession
           Developing the pedagogical content knowledge and ability of scholarship/stipend
              recipients to teach STEM content
                                                  Noyce Evaluation Report, Section Four: HLM 7


              Developing positive profiles for pre-service teachers in STEM discipline
               departments

This report extends the initial evaluation report by providing in-depth analyses related to the
third component of the evaluation listed above, statistical querying of the data and the production
of quantitative models of the Noyce program. Additionally, the U of MN Noyce evaluation team
sought to examine the second evaluation question to determine the role of the Noyce program in
Noyce scholarship/stipend recipients’ (hereafter referred to as scholars’) decisions to become
teachers and teach in high need schools. To do this, the team analyzed the relationship between
the scholars’ perceptions of the influence of the Noyce program on their commitment to become
teachers and commitment to teaching in high need schools as well as scholar- and teacher
preparation/certification program-level characteristics using Hierarchical Linear Modeling
(HLM) and Hierarchical Generalized Linear Modeling (HGLM).

These two forms of multi-level analyses are more advanced than simple single level and multiple
linear regression because they can differentiate variance from different levels of predictor
variables. For example, HLM and HGLM can differentiate between the variance due to the
teacher preparation/certification program effects and the variance due to individual scholar
effects (Raudenbush & Bryk, 2002). A key assumption in single-level models such as traditional
regression is that the observations are independent of one another which is often not true where a
nested structure exists (i.e., scholars within programs). In these situations, units of observations
within a program tend to be more similar to one another than observations from other programs.
Failing to take into account this dependency can result in biased statistical results. Multi-level
models take the nested structure of this data into account and therefore reduce this potential bias
(Raudenbush & Bryk, 2002).

Both scholar- and teacher preparation/certification program-level variables/characteristics could
influence scholars’ decisions to become teachers and to teach in high need settings. Scholar-level
variables such as race and background characteristics could influence scholar perceptions of the
influence of the Noyce program. Additionally, teacher preparation/certification programs provide
scholars with initial teaching experiences and the educational knowledge base that is necessary
for a successful teaching career. Therefore, the characteristics of the teacher
preparation/certification programs are an important element that may also impact scholars’
perceptions. Because scholars in the same teacher preparation/certification program tend to be
more alike than scholars in other programs, and each program may have a different impact on
scholars, HLM was used to differentiate the variance due to teacher preparation/certification
program effects from individual scholar effects. Therefore, the purpose of this report was to
investigate how the scholars’ perceptions of the influence of the Noyce program on their
commitment to teach and teach in high need schools related to the scholars’ characteristics and
the characteristics of the teacher preparation/certification programs in which they received
training.
                                                 Noyce Evaluation Report, Section Four: HLM 8



Methodology


Instruments

The data utilized for this phase of the Noyce evaluation came from three main sources: the
Noyce scholar survey, the Noyce PI Survey, and the ORC dataset. The scholar survey was
administered online during the summer of 2007. Former and current Noyce scholars were asked
to respond to a variety of items regarding their perceptions of and experiences with the Noyce
program. The survey consisted of a variety of rating scale, multiple choice, and open-ended
items. Completion of this survey was voluntary. Additionally, different forms of the survey were
administered to the scholars based on their status at the time of the survey (e.g. in a teacher
certification program, not yet a full-time teacher; completed a teacher certification program, but
never taught; teaching full-time or part-time etc.). Items did not differ across surveys, but some
items were either included or excluded depending on the scholars’ career progress. Appendix B
of the Final Evaluation Report Section One: Planning and Survey Data, includes copies of the
surveys or they can be accessed online at:
http://www2.cehd.umn.edu/EdPsy/NoyceSurvey/NoyceScholar/surveySample.asp.

Due to the large number of items in the scholar survey (83 items), it was necessary to combine
and reduce the items for ease of analysis in HLM. In Final Report Section Two: Factor Analysis
of the Evaluation Questionnaire (Liou & Lawrenz, 2009), factor analysis was used to investigate
the possibility of combining many items in the Noyce scholar survey into a few broad constructs
(e.g., scholars’ educational background, determination to teach in high need schools, and
perceptions of their programs/Noyce scholarship). Factor variables reflect a construct more
accurately than dichotomous or rating scale variables because factor variables are made up of an
aggregate of items that are indicators of the same construct. This then makes factor variables
more appropriate for advanced statistical analyses such as HLM. Seven factors were described in
the report, including “commitment to teaching in high need schools (influence of Noyce),”
“preparation for high need schools,” “path to teaching,” “district/school high need environment,”
“personal beliefs towards teaching,” “school teaching environment,” and “mentoring experience”
(details about the items and factors loadings for each factor are included in Appendix A).
Additional analyses of the six items in the “commitment to teaching in high need schools
(influence of Noyce)” factor revealed that this factor could further be reduced into two separate
factors (Liou, Kirchhoff, & Lawrenz, in press). The two factors were called “influence of
scholarship on STEM majors becoming teachers” and “influence of scholarship on STEM majors
becoming teachers in high need schools.” Due to these findings, an eight factor structure (Table
1) was used for the HLM analyses in this report.
                                                   Noyce Evaluation Report, Section Four: HLM 9


Table 1

The Eight Factors for the Scholar Survey
    Factor No.        Factor name
     Factor 1         Influence of scholarship on STEM majors becoming teachers
     Factor 2         Influence of scholarship on STEM majors becoming high need teachers
     Factor 3         Preparation for high need schools
     Factor 4         Path to teaching
     Factor 5         District/school high need environment
     Factor 6         Personal beliefs towards teaching
     Factor 7         School teaching environment
     Factor 8         Mentoring experience

The PI survey was administered online during the summer of 2007 to the Noyce project PIs at
local teacher preparation/certification institutions. PIs were asked to respond to a variety of items
describing their teacher preparation/certification programs and the role of the Noyce funding in
their programs. Appendix A of the Final Evaluation Report Section One: Planning and Survey
Data, includes a copy of the PI survey, or it can be accessed online at:
http://cehd.umn.edu/EdPsych/NOYCE/PI-FacultySurveys.html.

The ORC dataset was from the Noyce Program monitoring data system from ORC Macro
International Inc. This system collects annual data from the Noyce project PIs about the Noyce
scholars including demographic and academic background information. More information about
the ORC dataset can be found in Final Report Section Five: Combined Analysis of the Robert
Noyce Teacher Scholarship Program using ORC Macro and UMN Evaluation Data (Bowe, Liou,
& Lawrenz, 2009).


Data

Not all of the data from the scholar and PI surveys and the ORC dataset were used in the
analyses in this report. Data included from the scholar survey were variables related to scholars’
perception about the influence of the Noyce funding on their becoming teachers and becoming
teachers in high need schools, path to teaching, school high need environment, personal beliefs
towards teaching, school teaching environment, and mentoring experience (i.e., the factors
described previously). One variable was included from the PI survey; this variable was related to
what percentage of scholars’ tuition was provided for by the Noyce funding. Data included from
the ORC dataset included variables about the scholars’ race and gender.

The data used in this report exists in two levels: the scholar-level (level 1) and the program-level
(level 2; throughout the report, “program-level” refers to teacher preparation/certification level
variables, not the Noyce program). The scholar-level (level 1) variables include scholars’ race,
gender, perceptions about preparation for high need school (Factor 3), path to teaching (Factor
4), district/school high need environment (Factor 5), personal beliefs towards teaching (Factor
6), school teaching environment (Factor 7), and mentoring experience (Factor 8). The program-
level (level 2) variables include information regarding the percentage of the scholars’ tuition
                                                Noyce Evaluation Report, Section Four: HLM 10


covered by the Noyce funding, and the mean factor scores of preparation for high need schools
(mean Factor 3) and mentoring experience (mean Factor 8). It is worth noting that two of the
Factors ( 3 and 8) were included as variables at both the scholar- and program-level as the U of
MN Noyce evaluation team considered these factors to have both scholar- and program-level
effects. Factor 3: Preparation for high need schools included scholars’ responses to 13 items
regarding curricula and activities provided by their teacher preparation/certification program for
high need schools teaching. Factor 8: Mentoring experience included scholars’ responses to six
individual items focusing on mentoring experiences they might have received during and after
their teacher preparation/certification program. Although Factors 3 and 8 were constructed
according to scholars’ perceptions, Factors 3 and 8 also related to program characteristics.
Therefore, scholars’ Factors 3 and 8 scores from each program were used to form mean Factors 3
and 8 scores for each program. These factors at the program level can also be considered a
contextual model, since it examines the impact of Factor 3 and Factor 8 at the program level over
and above the individual impact of Factor 3 and Factor 8 at the scholar level.


Sample

Originally, 555 scholars provided data from the Noyce scholar survey, and 66 PIs provided data
from the PI survey. Detailed information about the sample can be found in Lawrenz, et al. (2008).
Due to concerns about statistical inference, however, some scholars’ data had to be deleted
because at times there were less than 5 scholars responding from any one teacher
preparation/certification program. Smaller within-program scholar samples yield relatively less
reliable estimates of these population parameters than do larger within program scholar samples.
Therefore, data from 527 scholars in 43 programs were available to use in this analysis.
Additionally, when variables at the program-level were analyzed in HLM and HGLM, data from
two teacher preparation/certification programs and 33 scholars associated with those teacher
preparation/certification programs were deleted because the PIs did not respond to the PI survey.
Additional information about the omitted teacher preparation/certification programs and the
number of scholars in each teacher preparation/certification programs included in the analysis is
listed in Appendix C. Moreover, because the HLM and HGLM procedures used pairwise
deletion, the actual number of scholars and programs in any specific analysis may be lower and
therefore, the actual sample size is provided for each individual analysis.

The sample used in each individual analysis was further reduced due to the fact that not every
scholar responded to every item in the scholar survey and therefore, some of the factors could
not be used as predictors for all scholars. For example, Factor 5, district/school high need
environment, only applies to scholars who were already teaching and only those scholars
responded to the items included in that factor. (See Appendix B for more information about
items in each factor.) To account for these differences and include the largest number of
variables possible, the U of MN Noyce evaluation team decided to use two groups of scholars for
the analyses and conduct separate analyses on these two groups. One group is the “all scholar”
group, and the other is the “current STEM teacher” group. 427 scholars in 37 teacher
preparation/certification programs were included in the all scholars group, and 265 scholars in 36
teacher preparation/certification programs were in the current STEM teacher group.
                                                 Noyce Evaluation Report, Section Four: HLM 11


Measures


Outcome variables:

Using data from the Noyce scholar survey, four variables were considered as outcomes. The first
and the second outcome variables were Factor 1: The influence of scholarship on STEM majors
becoming teachers and Factor 2: The influence of scholarship on STEM majors becoming
teachers in high need schools. These are continuous variables, which were created as
standardized factor scores by combining the three items in each factor (Liou, Kirchhoff, &
Lawrenz, in press).

The third and the fourth outcome variables were categorical items from the scholar survey:
Would you have become a teacher if you had not received the Noyce scholarship? and Would
you have decided to teach in a high need school if you had not participated in the Noyce
scholarship Program? The response options of the third outcome variable were 1) Yes, 2)
Possibly, and 3) No. The frequency table of the responses for outcome variable 3 is in Table 2.
Few people chose “no,” so the options “no” and “possibly” were combined which made outcome
variable 3 a binary variable (yes=1; possible/no=2), and provided enough differentiation for
options to analyze outcome variable 3. The frequency table of outcome variable 3 with the
adjusted two categories is in Table 3. The response options of the fourth outcome variable were
1) Yes, 2) Possibly, and 3) No/I have not taught in a high need school. The frequency table of the
responses for outcome variable 4 is in Table 4. No recombination of responses (Yes, Possibly,
No) were made for this variable because there were sufficient enough responses in each category
for differentiation analyses to take place. For these two categorical outcome variables (3 and 4),
it is worth paying particular attention to their order of the option coding . “No” or “no/I have not
taught in a high need school” was coded as a higher value. It is assumed that when scholars
responded “no” to outcome variable 3 and 4, scholars’ perceptions of the effect of the Noyce
funding for them to become teachers and to teach in high need schools were higher, and visa
verse.

The constructs implied in the first and third outcome variables are related to the perceived
influence of Noyce funding on scholars’ decisions to become teachers. The constructs implied in
the second the fourth outcome variables are related to the perceived influence of the Noyce
funding on scholars’ decisions to teach in high need schools. A summary of the four outcome
variables is included in Table 5.

Table 2

Frequency of Outcome Variable 3: Would You Have Become a Teacher if You Had Not Received
the Noyce Scholarship? (Three Response Categories)
         Value                   Frequency         Percentage        Cumulative Percentage
1 = Yes                             339               79.6                   79.6
2 = Possibly                         72               16.9                   96.5
3 = No                               15                3.5                   100
                                             Noyce Evaluation Report, Section Four: HLM 12


Table 3

Frequency of Outcome Variable 3: Would You Have Become a Teacher if You Had Not Received
the Noyce Scholarship? (Two Response Categories)
         Value                  Frequency          Percentage        Cumulative Percentage
1 = Yes                            338                79.5                   79.5
2 = No/Possibly                     87                20.5                   100

Table 4

Descriptive Statistics of Outcome Variable 4: Would You Have Decided to Teach in a High Need
School if You Had Not Participated in the Noyce Scholarship Program? (Three Response
Categories)
          Value                    Frequency           Percentage       Cumulative Percentage
1 = Yes                               154                 36.2                  36.2
2 = Possibly                          214                 50.4                  86.6
3 = No/I have not taught               57                 13.4                  100
in a high need school
                                                  Noyce Evaluation Report, Section Four: HLM 13


Table 5

Four Outcome Variables of Scholars’ Perception of the Noyce Program’s Influence on Their
Becoming a Teacher and Teaching in a High Need School
Variables                       Item content                    Item option
Outcome variable 1: Scholars’   (a) become a teacher            1) not at all influential
perception of the influence of  (b) complete the certification  2) not very influential
scholarship on becoming         program                         3) somewhat
teachers                        (c) take a teaching job         influential
(Cronbach’s Alpha=0.88)                                         4) very influential

Outcome variable 2: Scholars’       (d) teach in a high need          1) not at all influential
perception of the influence of      school                            2) not very influential
scholarship on becoming             (e) remain teaching in a high     3) somewhat
teachers in high need schools       need school for the full term     influential
(Cronbach’s Alpha=0.90)             of your commitment                4) very influential
                                    (f) remain teaching in a high
                                    need school beyond the full
                                    term of your commitment

Outcome variable 3:                 As the variable                   1) yes
Would you have become a                                               2) no/possibly
teacher if you had not received
the Noyce scholarship?

Outcome variable 4:                 As the variable                   1) yes
Would you have decided to                                             2) possibly
teach in a high need school if                                        3) no/I have not
you had not participated in the                                       taught in a high need
Noyce scholarship Program?                                            school



Scholar-level predictors:

Race and gender, from the ORC dataset, were used as two of predictors; race was coded as
0=Non-white and 1=White, and gender was coded as 0=Female and 1=Male. Other scholar-level
predictors were factors 3-8 from the Noyce scholar survey. However, as mentioned above, not all
items in each factor were answered by all scholars, so not every factor can be used as a predictor
in all analyses. Therefore, Factors 3, 4, and 8 were the predictors for the all scholar group, and all
six factors were the predictors for the current STEM teacher group. Therefore, the all scholar
group had five total predictors and the current STEM teachers group had eight total predictors at
the scholar level (see Table 6). Table 7 shows the descriptive statistics and correlation matrix for
the scholar-level variables for the all scholar group, and Table 8 shows the descriptive statistics
and correlation matrix for the current STEM teacher group.
                                                           Noyce Evaluation Report, Section Four: HLM 14


Table 6

Scholar-Level Predictors for All Scholar Group and Current STEM Teacher Group
Predictor           Item Description             Predictors for All Predictors for Current
                                                  Scholar Group     STEM Teacher Group
  Race     Non-white or White                            *                    *
 Gender Female or Male                                   *                    *
 Factor3 Preparation for high need schools               *                    *
 Factor4 Path to teaching                                *                    *
 Factor5 District/school high need                     N/A                    *
           environment
 Factor6 Personal beliefs towards teaching             N/A                    *
 Factor7 School teaching environment                   N/A                    *
 Factor8 Mentoring experience                            *                    *
Note. N/A indicated that Factors 5 to 7 were not included in the analysis for the all scholar group.

Table 7

Descriptive Statistics and Correlation Matrix for Scholar-Level Variables for All Scholar Group
Variables             M     SD          1      2        3     4        5     6               7            8           9
1. Race               .67    .47        -
2. Gender             .34    .48     -.10*     -
3. Factor3           -.03    .98    -.15**    .04       -
4. Factor4           -.01    .90      -.04   .11*    <-.01    -
5. Factor8            .01    .87    -.24**    .09    .36**  .06        -
6. Outcome1          -.01    .94    -.20**   -.05      .07  .05     .12*     -
7. Outcome2          -.07    .98      -.03   -.05    .13**  -.04     -.02 .58**               -
8. Outcome3          1.20    .40      -.03    .03     -.08 .16**      .03 .42**             .18**          -
9. Outcome4          1.77    .67     .22**  -.11*   -.14**  -.05   -.13**  .11*             .28**        .22**        -
 Note. The number of respondent varies from 425 to 427. *p<.05. **p<.01.

Table 8

Descriptive Statistics and Correlation Matrix for Scholar-Level Variables for Current STEM
Teacher Group
Variables         M      SD        1      2       3     4          5        6        7              8            9         10      11     12
1. Race           .69     .46      -
2. Gender         .33     .47   -.12*     -
3. Factor3       -.08     .96  -.20**   .04       -
4. Factor4        .04     .90   <-.01   .10      .02    -
5. Factor5       -.04     .89   -.15*   .03      .10   .07           -
6. Factor6       -.08    1.11     .07   -.05   .30**   .02          .04       -
7. Factor7      <-.01     .88    -.01   -.03    .16*   .05        -.14*    .35**       -
8. Factor8        .01     .83  -.31**   .04    .43**  .12*        .19**    .29**     .15*           -
9. Outcome1     <-.01     .94   -.16*   -.09     .04   .04        .16**     -.07     -.03          .09          -
10. Outcome2     -.11     .99    -.05   -.04     .11  -.02          .07     -.05     -.07          .01        .62**         -
11. Outcome3     1.25     .43    -.04   -.03   -.12*  .13*          .03     -.10    -.13*         -.02        .45**       .23**     -
12. Outcome4     1.77     .68    .15*   -.05  -.19**  -.02       -.17**   -.19**   -.16**        -.13*        .16**       .30**   .29**   -
 Note. The number of respondent is 265.*p<.05. **p<.01.
                                                 Noyce Evaluation Report, Section Four: HLM 15


Program-level predictors:

The percentage of scholars’ total tuition provided for by the Noyce scholarship was used as one
of the program-level predictors. It is an ordinal variable (1= 0% (money not used for tuition);
2=1-24%; 3=25-49 %; 4=50-74 %; 5=75-99 %; 6=100 %). In addition, individual scholar scores
on Factor 3 and Factor 8 for each teacher preparation/certification program were aggregated and
averaged within teacher preparation/certification programs and used as program-level predictors.
Table 9 summarizes the three predictors at the program level. Table 10 shows the descriptive
statistics and correlation matrix for the program-level variables.

Table 9

Program-Level Predictors
   Predictor                  Item Description                             Item Coding
Noyce Funding What proportion of the total tuition             1) 0 % (money not used for
                 scholars need to pay is provided by the       tuition); 2) 1-24 %; 3) 25-49 %;
                 Noyce scholarship funding?                    4) 50-74 %; 5) 75-99 %; 6) 100
                                                               %
  Mean Factor3       Preparation for high need schools
  Mean Factor8       Mentoring experience

Table 10

 Descriptive Statistics and Correlation Matrix for Program-Level Variables
Variables                      M        SD           1          2          3
1. Noyce Funding             4.11      1.65          -
2. Mean Factor3               .04       .52        <.01         -
3. Mean Factor8              -.01       .43        -.30       .40*         -
Note. The number of PI is 37. *p<.05.


Analysis

In this report, HLMs and HGLMs were used to explore 1) what variables influence scholars’
perceptions of the influence of the Noyce funding on their decisions to become teachers and 2)
the influence of the Noyce funding on their decisions to become teachers in high need schools. In
other words, HLMs and HGLMs can examine the quantitative relationship between the
predictors and the outcome variables.


Hierarchical Linear Models (HLMs):

As described in the introduction, HLMs are able to differentiate between the variance due to the
teacher preparation/certification program effects and the variance due to individual scholar’s
variables effects (Raudenbush & Bryk, 2002). HLMs can only be used for analyzing continuous
outcome variables, such as outcome variables 1 and 2, but can not allow the analysis of
                                                 Noyce Evaluation Report, Section Four: HLM 16


categorical variables such as outcome variables 3 and 4: Would you have become a teacher if
you had not received the Noyce scholarship? and Would you have decided to teach in a high
need school if you had not participated in the Noyce funding Program? Therefore, HGLM was
used for analyzing outcome variables 3 and 4.


Hierarchical Generalized Linear Models (HGLMs):

HGMLs, also known as generalized linear mixed models or generalized linear models with
random effects (Raudenbush & Bryk, 2002), offer a coherent modeling framework for multilevel
data with nonlinear structural models and nonnormally distributed errors.

There are three reasons why HGLMs are appropriate for ordinal variables. First, HGLMs restrict
the outcome probability to an interval of (0, 1). This constraint gives meaning to the effect sizes
defined by the model. A nonlinear transformation of the predicted value satisfies this constraint.
Second, given the predicted value of outcome variables, HGLMs relax the normal distribution of
level-1 random effects. Third, HGLMs do not require heterogeneous variance like HLMs.

For the binary outcome variable 3: Would you have become a teacher if you had not received the
Noyce scholarship? a Bernoulli distribution with a log-link function was used. For the ordinal
outcome variable 4: Would you have decided to teach in a high need school if you had not
participated in the Noyce scholarship Program? an ordered logit-link function was used.

HLMs and HGLMs were estimated using HLM6 (Raudenbush, Bryk, & Congdon, 2005). The
default estimation procedure in HLM6 employs restricted maximum likelihood where fixed
effects are estimated using generalized least squares and variance-covariance components are
estimated using maximum likelihood (Raudenbush et al., 2005). In this report, the unit-specific
model was used. This function “defines fixed regression coefficients that can be interpreted as
the expected change in the outcome associated with a one-unit increase in the relevant predictor,
holding constant other predictors and all random effects in the model (Raudenbush & Bryk, 2002,
p.334).” Grand-mean centering was used for adjusting predictors in equations, and maximum
likelihood estimation was used to estimate parameters. All models reported are random-intercept
models. The random part of the intercept was freely estimated to reflect between-program
differences in the influence of Noyce funding to become teachers. Since there is no a priori
hypothesis concerning between-program differences of the predictor variables, the random parts
of the slopes were not relevant. In other words, only the intercept varied across programs, but
other level-1 coefficients remained constant. All differences described in this report are
statistically significant at α=0.05. No statistical adjustments to account for multiple comparisons
were used.
                                                          Noyce Evaluation Report, Section Four: HLM 17



Results

This results section contains two parts: which scholar- and program-level variables are related to
scholars’ perceptions of the influence of the Noyce funding on their commitment to become
teachers, and which scholar- and program-level variables are related to scholars’ perceptions of
the influence of the Noyce funding on their commitment to teaching in high need schools.


Variables influencing scholars’ perceptions of the influence of the Noyce funding on their
commitment to become teachers

To investigate which scholar- and program-level variables were related to scholars’ perceptions
of the influence of the Noyce funding on their commitment to become teachers, outcome
variable 1: Scholars’ perception of the influence of scholarship on becoming teachers, and
outcome variable 3: Would you have become a teacher if you had not received the Noyce
scholarship? were used. In addition, models for each of the two groups “all scholars” and
“current STEM teacher” are discussed separately.


Outcome variable 1: Scholars’ perception of the influence of Noyce funding on becoming
teachers

For outcome variable 1, an unconditional HLM model (one-way random-effects ANOVA model)
was first fit to determine whether substantial variance could be explained at the program level of
the model for the all scholar group and the current STEM teacher group. There was significance
in the program-level variance for outcome variable 1 in both groups. For the all scholar group,
the estimate for the within-program variance was 0.81, and the overall variability among the true
program means on outcome variable 1 was 0.08. This resulted in an intraclass correlation of 0.09.
This result indicated that most of the variance for the all scholar group occurred at the scholar
level, with 9% of the variance located at the program level. For the current STEM teacher group,
the estimate for the within-program variance was 0.67, and the overall variability among the true
program means on outcome variable 2 was 0.19. This resulted in an intraclass correlation of 0.22.
This result indicated 22% of the variance for the current STEM teacher group was located at the
program level. In order to explain more variability by adding potential predictors, conditional
models were fit (Model 1-all scholar group in equation 1, Model 2-current STEM teacher group
in equation 2) as follows:

Model 1: HLM/HGLM for the all scholar group                                                                    (1)
Level 1: All scholar
Yij   0 j  1 j ( Race)   2 j (Gender )   3 j ( Factor3)   4 j ( Factor 4)   8 j ( Factor8)  rij

Level 2: Program
 0 j   00   01 ( NoyceFunding )   02 ( MeanFactor3)   03 ( MeanFactor8)  u 0 j
 1 j   10
                                                                      Noyce Evaluation Report, Section Four: HLM 18


 2 j   20
 3 j   30
 4 j   40
 8 j   80

Model 2: HLM/HGLM for the current STEM teacher group                                                                                       (2)
Level 1: Current STEM teacher
Yij   0 j  1 j ( Race)   2 j (Gender )   3 j ( Factor3)   4 j ( Factor 4)   5 j ( Factor5)   6 j ( Factor6)   7 j ( Factor7)   8 j ( Factor8)  rij

Level 2: Program
 0 j   00   01 ( NoyceFunding )   02 ( MeanFactor3)   03 ( MeanFactor8)  u 0 j
 1 j   10
 2 j   20
 3 j   30
 4 j   40
 5 j   50
 6 j   60
 7 j   70
 8 j   80

In Model 1 (the all scholar group), two predictors were significantly related to outcome variable
1 at the program level. Noyce funding ( b  0.13 ; SE  0.03 ; p-value  0.01 ) was positively
related to outcome variable 1, while Mean Factor 3: Preparation for high need schools
( b  0.38 ; SE  0.13 ; p-value  0.01 ) had a negative effect. This means that outcome variable 1
scores were substantially higher in programs where the Noyce funding was a higher percentage
of total tuition. This result suggests that greater the percentage of scholars' tuition covered by the
Noyce funding, the greater the Noyce program influenced scholars' decisions to become teachers.
On the other hand, outcome variable 1 scores were lower in programs with higher Mean Factor 3
scores. This result suggests that greater degree of exposure to curricula and activities that
prepared pre-teachers for high need schools, the less the Noyce program influenced scholars’
decisions to become teachers. At the scholar level, only Race ( b  0.41; SE  0.08 ; p-
value  0.01 ) was significant for outcome variable 1. This means that White scholars tended to
have lower perceptions of the influence of the Noyce funding on becoming teachers than Non-
white scholars on average. Model 1 incorporated the three program-level variables and the five
scholar-level variables relating to the perceived influence of Noyce funding on scholars
becoming teachers. The variance at the program level was 0.04, and at the scholar level was 0.77
which represents unexplained variance after taking into account scholars’ race, gender, Factor 3,
Factor 4, and Factor 8. In other words, these predictors decreased 5% of the variance at the
program level as well as 4% of the variance at the scholar level. Therefore, this resulted in an
intraclass correlation of 0.05. This result indicated that most of the variance occurred at the
scholar level, with 5% of the variance located at the program level. The intraclass correlation
                                                  Noyce Evaluation Report, Section Four: HLM 19


explained dropped to 0.05 from 0.09 after adding predictors at both levels. This demonstrates
that predictors at the scholar level have more variability than the predictors at the program level.

In Model 2 (the current STEM teacher group), no predictors at the program level were significant
as related to outcome variable 1. Two predictors were significant at the scholar level. Factor 4:
Path to teaching ( b  0.25 ; SE  0.12 ; p-value  0.05 ) was positive in relation to outcome
variable 1. It showed that when Factor 4 increased one unit above the grand mean Factor 4 score,
the slope increased 0.25 units after controlling for other predictors. This means that scholars that
had higher Factor 4 scores tended to have higher outcome variable 1 scores. Factor 4: Path to
teaching is a construct which the scholars responded to seven items regarding various aspects of
courses they took and decisions about becoming teachers, including previous career status. A
basic interpretation of this score is that the higher the score, the more likely the scholar had
another full-time career before becoming a teacher and considered themselves to have made a
career change. In addition, they were ones who were likely to take more STEM classes.
Therefore, this result suggests that when scholars had higher Factor 4 scores, they were the ones
who considered themselves to be more of a career changer and their perception of the influence
of the Noyce funding on becoming teachers was higher. In addition, Factor 5: District/school
high need environment ( b  0.19 ; SE  0.09 ; p-value  0.05 ) had an positive influence on
outcome variable 1. Factor 5: District/school high need environment is a construct which the
scholars responded to five items. To determine this, the scholars were asked to indicate their
district/school status regarding the percentage of students receiving free or reduced lunch; the
percentage of teachers lacking sufficient training in the content area they did most of their
teaching in; and the percentage of teacher attrition over the last three years. A basic interpretation
of these scores is that higher scores correspond to districts/schools meeting Title I requirements
for being considered high need. Therefore, this result suggests that when scholars had higher
Factor 5 scores, they were the ones who tended to work in districts that met title I criteria for
high need and their perception of the influence of the Noyce funding on becoming teachers was
higher. Race ( b  0.37 ; SE  0.19 ; p-value  0.06 ) almost had a significant effect on outcome
variable 1. Again, this suggests that White scholars perceived that the Noyce funding had less
influence on their decisions to becoming teachers compared to Non-white scholars.

These predictors explained an additional 8% of the variance at the program level, and decreased
4 % of the variance at the scholar level. Therefore, this resulted in an intraclass correlation of
0.29. This result indicated that although most of the variance occurred at the scholar level, the
program level had 29% of the variance.

The results of the two conditional HLMs (for the whole scholar and the current STEM teacher
group) are presented below in Table 11.
                                                           Noyce Evaluation Report, Section Four: HLM 20


Table 11

Effects of Predictors on Scholars’ Perception of the Influence of the Scholarship on Becoming
Teachers for “All Scholar” Group and “Current STEM Teacher” Group (Outcome Variable 1)
                                       All Scholars                   Current STEM Teachers
                                    b       SE       P-value           b        SE     P-value
Level 2
   Intercept,  00                -.01      .05         .89           .14       .12      .24
   Noyce Funding,  01           .13**      .03        <.01           .05       .08      .56
   Mean Factor3,  02           -.38**      .13         .01          -.03       .25      .91
   Mean Factor8,  03              .27      .17         .12           .25       .32      .45
Level 1
  Race, 10                           -.41**         .08         <.01             -.37         .19          .06
  Gender,  20                          -.12         .09         .16              -.21         .21          .34
  Factor3,  30                          .06         .06         .27               .13         .13          .29
  Factor4,  40                          .07         .06         .25              .25*         .12          .05
  Factor5,  50                        N/A                                        .20*         .09          .04
  Factor6,  60                        N/A                                        -.07         .09          .47
  Factor7,  70                        N/A                                        -.12         .07          .11
  Factor8,  80                          .09         .07          .21              .02         .15          .87
Note. There were 405 scholars in 37 programs in the “all scholars” group. There were 250 scholars in 36 programs
in the “current STEM teachers” group. N/A indicated that Factors 5 to 7 were not included in the analysis for the all
scholar group. *p<.05. ** p<.01.


Outcome variable 3: Would you have become a teacher if you had not received the Noyce
scholarship?

For outcome variable 3, an unconditional HGLM model was first fit to determine whether
sufficient variance could be explained at the program level of the model for the all scholar group
and the current STEM teacher group. In Model 1 (the all scholar group), the test of variance
component showed there was significant program-level variance for outcome variable 3 in both
groups (p-value  0.05 ). No intraclass correlation can be computed via HGLM. In order to
explain more variability by adding potential predictors, conditional models were fit (Model 1-all
scholar group in equation 1, Model 2-current STEM teacher group in equation 2 with a log-link
function).

In Model 1 (the all scholar group), Noyce funding ( OR  1.38; SE  0.13 ; p-value  0.05 ) was
positively related to outcome variable 3 after controlling for other predictors at the program level.
This means when teacher preparation/certification programs with higher percentage of scholars’
total tuition was covered by Noyce funding, scholars tended to answer “no” or “possibly” rather
than “yes” of outcome variable 3. In other words, Noyce funding at the program-level seemed to
have a positive relationship with scholars’ perception on Noyce funding for them to become
                                                 Noyce Evaluation Report, Section Four: HLM 21


teachers, since scholars would not become teachers if they had not received the Noyce funding.
At the scholar level, Factor 4: Path to teaching ( OR  1.44 ; SE  0.16 ; p-value  0.05 ) had a
positive relationship with outcome variable 3. This result suggests that when scholars had higher
Factor 4 scores, they tended to choose “no” or “possibly” rather than “yes” of outcome variable 3.
This indicates that when scholars had higher Factor 4 scores, their perception of the influence of
the Noyce funding on becoming a teacher was higher. The variance component dropped to 0.74
from 0.96 after adding predictors at both levels. However, the test of the variance component
was still significant (  2  65 .44 ; p-value  0.01). It means that these predictors cannot explain
the variance.

In Model 2 (the current STEM teacher group), Mean Factor 8: Mentoring experience
( OR  6.32 ; SE  0.83 ; p-value  0.05 ) was positively related to outcome variable 3 after
controlling for other predictors at the program level. This means that scholars tend to choose
“no” or “possibly” rather than “yes” of outcome variable 3 when Mean Factor 8 scores at the
program-level were higher. In other words, the greater degree of mentoring experience scholar
had received during and after their teacher preparation/certification program, the more Noyce
funding affected scholars to teach. At the scholar level, no predictor showed a significant
relationship with outcome variable 3. The variance component dropped to 0.68 from 1 after
adding predictors at the both levels. The test of the variance component was not significant
(  2  36 .75 ; p-value=0.26). It means that these predictors can explain the variance in the model.

The results of the two conditional HGLMs (for the whole scholar and the current STEM teacher
group) are presented in Table 12.
                                                        Noyce Evaluation Report, Section Four: HLM 22


Table 12

Effects of Predictors on Scholars’ Perception of the Influence of the Scholarship on Becoming
Teacher for “All Scholar” Group and “Current STEM teacher” Group (Outcome Variable 3)
                                          All Scholars                       Current STEM Teachers
                                  OR          95% CI      P-value          OR         95% CI   P-value
Level 2
   Intercept,  00               .22**        .15, .33      <.01          .37*        .18, .77   .01
   Noyce Funding,  01           1.38*      1.06, 1.79      .02           1.29       .81, 2.05   .27
   Mean Factor 3,  02             .43       .17, 1.05      .06           1.24       .37, 4.19   .72
   Mean Factor 8,  03            2.50       .94, 6.65      .07          6.32*      1.18,33.98   .03
Level 1
   Race, 10                       .81       .41, 1.58      .54            .83       .31, 2.22   .71
   Gender,     20                     1.12         .69, 1.82         .64             .47         .11, 1.93        .29
   Factor3,    30                      .82         .57, 1.18         .28             .64         .32, 1.28        .21
   Factor4,    40                    1.44*        1.05, 1.99         .03            1.48         .62, 3.52        .37
   Factor5,    50                     N/A                                            .87         .43, 1.78        .70
   Factor6,    60                     N/A                                            .94         .41, 2.17        .89
   Factor7,    70                     N/A                                            .59         .29, 1.22        .15
   Factor8,    80                     1.15         .84, 1.57         .39            1.42         .69, 2.90        .34
Note. There were 403 scholars in 37 programs in the “all scholars” group. There were 250 scholars in 36 programs
in the “current STEM teachers” group. OR=odds ratio; CI=confidence interval. N/A indicated that Factors 5 to 7
were not included in the analysis for the all scholar group. *p<.05. ** p<.01.


Variables influencing scholars’ perception of the influence of the Noyce funding on their
commitment to teach in high need schools

To address which scholar- and program-level variables were related to scholars’ perceptions of
their commitment to teaching at high need schools, outcome variable 2: Scholars’ perception of
the influence of Noyce funding on becoming teachers in high need schools, and outcome
variable 4: Would you have decided to teach in a high need school if you had not participated in
the Noyce scholarship Program? were used.


Outcome variable 2: Scholars’ perception of the influence of Noyce funding on becoming
teachers in high need schools

For outcome variable 2, an unconditional HLM model was first fit to determine if sufficient
variance could be explained at the program level of the model for the all scholar group and the
current STEM teacher group. There was significance in the program-level variance for outcome
variable 2 in the all scholar group, but not in the current STEM teacher group. For the all scholar
group, the estimate for the within-program variance was 0.90, and the overall variability among
                                                  Noyce Evaluation Report, Section Four: HLM 23


the true program means on the outcome variable 2 was 0.05. This resulted in an intraclass
correlation of 0.06. This result indicated that most of the variance occurred at the scholar level,
with 6% of the variance located at the program level. For the current STEM teacher group, the
estimate for the within-program variance was 0.62, and the overall variability among the true
program means on outcome variable 2 was 0.02. This resulted in an intraclass correlation of 0.03.
This result indicated that the variance which can be accounted for at the program level was
limited since it was only 3%. Although using HLMs for outcome variable 2 for the current
STEM teacher group may not be necessary, the U of MN Noyce evaluation team decided to fit
HLM conditional models for outcome variable 2 for both groups as equation 1 and 2.

In Model 1 (the all scholar group), Noyce funding ( b  0.09 ; SE  0.04 ; p-value  0.05 ) was
positively related to outcome variable 2. This means that outcome variable 2 scores were
substantially higher in teacher preparation/certification programs where a higher percentage of
scholars’ tuition was provided for by the Noyce funding. Therefore, scholars’ perceptions of the
influence of the Noyce funding on becoming teachers in high need schools seems to be
positively related to the percentage of the tuition covered by the Noyce funding. At the scholar
level, only Factor 3: Preparation for high need schools ( b  0.14 ; SE  0.05 ; p-value  0.01 ) was
significant for outcome variable 2. This suggests that scholars in teacher
preparation/certification programs with more preparation for high need schools tended to have
higher perceptions of the influence of the Noyce funding on becoming teachers in high need
schools. Model 1 incorporated the three program-level variables and the five scholar-level
variables relating to the influence of Noyce funding on scholars becoming teachers in high need
schools. These predictors decreased 2% of the variance at the program level and 0.1% of the
variance at the scholar level. Therefore, this resulted in an intraclass correlation of 0.04. This
result indicated that most of the variance occurred at the scholar level, with 4% of the variance
located at the program level.

In Model 2 (the current STEM teacher group), no predictor at either the program level or the
scholar level were significant related to outcome variable 2. These predictors decreased 6% of
the variance at the program level and 16% of variance at the scholar level. Therefore, this
resulted in an intraclass correlation of 0.12. The results of the two conditional HLMs (for the
whole scholar and the current STEM teacher group) are presented in Table 13 below.
                                                           Noyce Evaluation Report, Section Four: HLM 24


Table 13

Effects of Predictors on Scholars’ Perception of the Influence of Scholarship on Becoming
Teachers in High Need Schools for “All Scholar” Group and “Current STEM Teacher” Group
(Outcome Variable 2)
                                       All Scholars                  Current STEM Teachers
                                   b        SE       P-value          b        SE     P-value
Level 2
   Intercept,  00               -.04       .05         .47          .09       .09      .37
   Noyce Funding,  01           .09*       .04         .03          .01       .06      .90
   Mean Factor3,  02            -.07       .14         .60          .07       .17      .68
   Mean Factor8,  03            -.05       .18         .81          .14       .22      .53
Level 1
  Race, 10                             -.07         .09         .43              -.09         .17          .65
  Gender,  20                          -.13         .10         .17              -.12         .21          .58
  Factor3,  30                        .14**         .05         <.01              .15         .10          .13
  Factor4,  40                          .02         .05         .73               .10         .09          .27
  Factor5,  50                         N/A                                        .07         .08          .40
  Factor6,  60                         N/A                                        .02         .08          .79
  Factor7,  70                         N/A                                       -.12         .10          .23
  Factor8,  80                         -.03         .06          .66             -.02         .10          .86
Note. There were 405 scholars in 37 programs in the “all scholars” group. There were 250 scholars in 36 programs
in the “current STEM teachers” group. N/A indicated that Factors 5 to 7 were not included in the analysis for the all
scholar group. *p<.05. ** p<.01.


Outcome variable 4: Would you have decided to teach in a high need school if you had not
participated in the Noyce scholarship program?

For outcome variable 4, an unconditional HGLM model was first fit to determine whether
sufficient variance could be explained at the program level of the model for the all scholar group
and the current STEM teacher group. In Model 1 (the all scholar group), the test of variance
component showed there was significant program-level variance for outcome variable 4 in both
groups (p-value  0.05 ). In order to explain more variability by adding potential predictors,
conditional models were then fit (Model 1-all scholar group in equation 1, Model 2-current
STEM teacher group in equation 2 with an ordered logit-link function).

In Model 1 (the all scholar group), none of the predictors at the program level was significantly
related to outcome variable 4. At the scholar level, race ( OR  0.44 ; SE  0.19 ; p-value  0.01)
had a negative relationship with outcome variable 4. This suggests that non-White scholars
tended to have a greater perception of the influence of Noyce funding on their decision to
become teachers in high need schools. The variance component dropped to 0.14 from 0.26 after
adding predictors at the both levels. However, the test of the variance component was not
                                                  Noyce Evaluation Report, Section Four: HLM 25


significant (  2  41 .79 ; p-value  0.14 ), which suggests that these predictors can explain the
variance in the model.

In Model 2 (the current STEM teacher group), Noyce Funding ( OR  0.80 ; SE  0.11 ; p-
value  0.05 ) was negatively related to outcome variable 4 after controlling for other predictors
at the program level. At the scholar level, Factor 6: Personal beliefs towards teaching
( OR  2.07 ; SE  0.30 ; p-value  0.05 ) showed a positive relationship with outcome variable 4.
Factor 6: Personal beliefs towards teaching included nine items which the scholars responded on
the scholar survey. Higher scores correspond to higher levels of job satisfaction, opportunities of
professional growth, and higher self-efficacy towards teaching. Therefore, this result suggests
that when scholars had higher Factor 6 scores, they tended to choose “No/I have not taught in a
high need school” rather than “Yes” of outcome variable 4. In other words, when scholars had
higher Factor 6 scores, their perception of the influence of the Noyce funding on teaching in high
need schools was higher. The variance component dropped to <.01 from 0.13 after adding
predictors at the both levels. The test of the variance component was not significant (  2  28 .86 ;
p-value>0.50), which suggests that these predictors can explain the variance in the model.

The results of the two conditional HGLMs (for the whole scholar and the current STEM teacher
group) are presented in Table 14.
                                                          Noyce Evaluation Report, Section Four: HLM 26


Table 14

Effects of Predictors on Scholars’ Perception of the Influence of Scholarship on Becoming
Teachers in High Need Schools for “All Scholar” Group and “Current STEM Teacher” Group
(Outcome Variable 4)
                                         All Scholars                    Current STEM Teachers
                                         OR            95% CI        P-value          OR           95% CI         P-value
Level 2
  Intercept,  00                       .56**         .44, .72        <.01           .50**         .33, .76        <.01
  Noyce Funding,  01                    .90         .79, 1.03         .13            .80*        .64, 1.00         .05
  Mean Factor 3,  02                   1.58         .92, 2.72         .10             .86        .35, 2.10         .73
  Mean Factor 8,  03                    .94         .51, 1.73         .84             .45        .15, 1.33         .14
Level 1
  Race, 10                            .44**          .30, .64        <.01             .74        .28, 1.94         .53
  Gender,  20                          1.42         .85, 2.37         .18            1.57        .55, 4.48         .40
  Factor3,  30                         1.03         .82, 1.30         .77            1.18        .71, 1.96         .53
  Factor4,  40                          .98         .75, 1.29         .90             .54        .29, 1.00         .51
  Factor5,  50                         N/A                                            .94        .54, 1.63         .81
  Factor6,  60                         N/A                                          2.07*       1.14, 3.78         .02
  Factor7,  70                         N/A                                           1.11        .67, 1.85         .68
  Factor8,  80                         1.07         .85, 1.36         .56             .83        .51, 1.35         .46
   Threshold Difference,  2          14.09**      10.22, 19.42       <.01          39.10*      17.00, 89.89       <.01
                                                                                      *

Level-2 variance,  00                  0.14                           .14           <.01                          >.50
Note. There were 403 scholars in 37 programs in the “all scholars” group. There were 250 scholars in 36 programs
in the “current STEM teachers” group. N/A indicated that Factors 5 to 7 were not included in the analysis for the all
scholar group. *p  .05. ** p<.01.
                                                Noyce Evaluation Report, Section Four: HLM 27



Conclusions

This report examined the relationships among potential predictors and scholars’ perceptions of
the influence of Noyce funding on becoming teachers and becoming teachers in high need
schools. Two sets of data were used. One set, the all scholars group, was a survey of 405 scholars
from 37 programs; the second, the current STEM teacher group, was a survey of 250 scholars
from 36 programs. Results from the two-level HLM analyses revealed that most of the variance
in the influence of the Noyce funding occurred at the scholar level rather than the program level.
In addition, more variance at the program level occurred for scholars’ perceptions of the
influence of the Noyce funding to become teachers than for explaining scholars’ perceptions of
the influence of the Noyce funding on teaching in high need schools. Nine percent of the
variance from the unconditional HLM model of the all scholar group, and 22% of the variance of
the current STEM teacher group were located at the program level for outcome variable 1
(scholars’ perceptions of the influence of the Noyce funding to become teachers). On the other
hand, 5% of the variance from the unconditional HLM model of the all scholar group, and 3% of
the variance of the current STEM teacher group were located at the program level for outcome
variable 2 (scholars’ perceptions of the influence of the Noyce funding to teach in high need
schools).

The summary of effects of predictors on scholars’ perceptions of the influence of the Noyce
funding on becoming teachers and becoming teachers in high need schools for the all scholar
group and the current STEM teacher group are listed in Tables 15 and 16, respectively. From the
two tables, it can be inferred that only a few predictors have significant relationships with the
outcome variables, and that these significant predictors do not have consistent relationships
across the outcome variables and two groups. However, although the results for the four outcome
variables and the two groups are not totally the same, a few conclusions may be drawn and they
are presented in the following two sections. The first section summarizes the scholars’ individual
characteristics and perceptions, and the second section summarizes programs’ characteristics.
                                                         Noyce Evaluation Report, Section Four: HLM 28


Table 15

Summary of Effects of Predictors on Scholars’ Perception of the Influence of the Scholarship on
Becoming Teachers for “All Scholar” Group and “Current STEM Teacher” Group
                                                              All Scholars                 Current STEM Teachers
                                                        Outcome        Outcome             Outcome      Outcome
                                                        variable 1    variable 3           variable 1  variable 3
Level 2
   Noyce Funding                                              ↑               ↑
   Mean Factor3: Preparation for high need schools            ↓
   Mean Factor8: Mentoring experiences                                                                           ↑
Level 1
   Race: 0=Non-white; 1=White                                 ↓
   Gender: 0=Female; 1=Male
   Factor3: Preparation for high need schools
   Factor4: Path to teaching                                                  ↑                   ↑
   Factor5: District/school high need environment           N/A             N/A                   ↑
   Factor6: Personal beliefs towards teaching               N/A             N/A
   Factor7: School teaching environment                     N/A             N/A
   Factor8: Mentoring experience
Note. N/A indicated that Factors 5 to 7 were not included in the analysis for the all scholar group. The direction of
the arrows shows the relationship between a predictor and an outcome variable; “↑” indicates a positive relationship,
and “↓” indicates a negative relationship.

Table 16

Summary of Effects of Predictors on Scholars’ Perception of the Influence of the Scholarship on
Becoming Teachers in High Need Schools for “All Scholar” Group and “Current STEM
Teacher” Group
                                                              All Scholars                 Current STEM Teachers
                                                        Outcome        Outcome             Outcome      Outcome
                                                        variable 2    variable 4           variable 2  variable 4
Level 2
   Noyce Funding                                              ↑                                                  ↓
   Mean Factor3: Preparation for high need schools
   Mean Factor8: Mentoring experiences
Level 1
   Race: 0=Non-white; 1=White                                                 ↓
   Gender: 0=Female; 1=Male
   Factor3: Preparation for high need schools                 ↑
   Factor4: Path to teaching
   Factor5: District/school high need environment           N/A             N/A
   Factor6: Personal beliefs towards teaching               N/A             N/A                                  ↑
   Factor7: School teaching environment                     N/A             N/A
   Factor8: Mentoring experience
Note. N/A indicated that Factors 5 to 7 were not included in the analysis for the all scholar group. The direction of
the arrows shows the relationship between a predictor and an outcome variable; “↑” indicates a positive relationship,
and “↓” indicates a negative relationship.
                                                Noyce Evaluation Report, Section Four: HLM 29



Scholars’ individual characteristics and perceptions

Because of the extra items on the surveys for the current STEM teachers, there were different
numbers of predictors at the scholar level for the two different groups of scholars. There were
five predictors for the all scholar group, and eight predictors for the current STEM teacher group.
The five predictors used for both groups were Race, Gender, Factor 3: Preparation for high
need schools, Factor 4: Path to Teaching, and Factor 8: Mentoring experience. An additional
three predictors for the current STEM teacher group were Factor 5: District/school high need
environment, Factor 6: Personal beliefs towards teaching, and Factor 7: School teaching
environment.


Race:

Race (Whites vs. Non-whites) was an important predictor in the all scholar group for two of the
four outcome variables. Race was a significant predictor for outcome variable 1: Influence of
scholarship on STEM majors becoming teachers for the all scholar group. The results showed
that the Noyce funding influences Non-whites more than Whites to become teachers. Race was
also a significant predictor for outcome variable 4: Would you have decided to teach in a high
need school if you had not participated in the Noyce funding Program? for the all scholar group.
These results indicated that Non-whites tended to have higher perceptions of the effect of Noyce
funding on teaching in high need schools than Whites.

These results about race corroborates most of the results from Final Report Section Five:
Combined Analysis of the Robert Noyce Teacher Scholarship Program Using ORC Macro and
UMN Evaluation Data (Bowe et al., 2009). That report also showed a significant difference
between Whites and Non-whites on the continuous outcome variable 1 with Non-whites having
higher perceptions of the effect of the Noyce funding and no significant effects for outcome
variable 2 and 3. In contrast to the present report, Final Report Section Five showed a higher
percentage of Non-whites saying “yes” they would have taught in high need schools even
without getting Noyce funding; suggesting that more Whites reported that they felt influenced by
the Noyce funding to teach in a high need school. However, it was difficult to accurately
interpret the results in Final Report Section Five because of the effect of the “possibly” category
and because only visual inspection was possible—not a statistical test. The results for outcome
variable 4 in the present report with more sophisticated analyses indicated that Non-whites
perceived the Noyce funding as being more influential in their decision to become teachers in
high need schools compared to Whites.

The effect of race for the current STEM teachers on the outcome variables was also examined
through cluster analysis (Liou, Desjardins, & Lawrenz, in press). In that analysis three clusters
emerged but the clusters combined teaching and teaching in high need schools rather than
separating these outcomes as is the case in this HLM analysis. In the cluster analyses 47% of the
Non-whites were in the cluster, “Highly committed to becoming a teacher and teaching in a high
needs schools,” while 27% and 26% were in the other two clusters “Highly committed to
becoming a teacher but not in high needs schools” and “Not highly committed to becoming a
                                                Noyce Evaluation Report, Section Four: HLM 30


teacher or to teach in high need schools.” In the cluster analysis, it was suggested that Non-
white scholars would have already been more committed to teaching in high need schools, so
Noyce funding would have had less effect on Non-whites in terms of teaching in high need
schools. This result appears to conflict with the results in the present report; however, the
analyses are different and the group of current STEM teachers in the cluster analysis is a subset
of the all scholars group which showed the significant findings in the HLM analyses reported
here. In terms of analyses, the cluster analyses used the four outcome variables to produce
clusters and assign the current STEM teachers into the clusters. Then the current STEM teachers’
characteristics were further investigated. However, in the present report using HLM, the
relationships between current STEM teachers’ characteristics and each of the outcome variables
were analyzed simultaneously.


Preparation for high need schools:

Factor 3: Preparation for high need schools was a significantly positive predictor for outcome
variable 2 for the all scholar group. This result indicated that including curricula and activities
for high need schools in Noyce programs played a role in increasing scholars’ perceptions of the
effect of Noyce funding on their decision to teach in high need schools. These curricula and
activities included developing specific strategies for teaching students from diverse racial and
ethnic backgrounds, considering the relationship between education and social justice and/or
democracy, education about how to work in high need schools specifically, etc.. However,
Factor 3 did not show a significant relationship with outcome variable 4 which also related to the
effect of the Noyce funding on teaching in high needs schools for either group. Additionally,
Factor 3 was not significantly related to the effect of Noyce funding on scholars becoming
teachers (outcome variable 1 and 3) in either group.


Path to teaching:

Factor 4: Path to teaching was a significantly positive predictor for outcome variable 1 for the
current STEM teacher group, and correspondingly was a positive predictor for outcome variable
3 for the all scholar group. This result indicated that scholars’ path to teaching increased the
effect of Noyce funding on scholars’ decisions to become teachers. The construct of path to
teaching included various aspects of courses scholars took and decisions about becoming
teachers, including previous career status. In other words, scholars that had another full-time
career before becoming a teacher and considered themselves to have made a career change
tended to perceive a greater effect of the Noyce funding on the decision to become teachers.


District/school high need environment:

Factor 5: District/school high need environment is a significantly positive predictor for outcome
variable 1 for the current STEM teacher group. This result indicated that current STEM teachers’
perceptions of the effect of the Noyce program on their decision to become teachers had a
positive relationship with their district/school high need environment. Results of items in
                                                   Noyce Evaluation Report, Section Four: HLM 31


District/school high need environment included items pertaining to district or school high need
status according to Title I criteria for percentage of teachers lacking sufficient training in the
content area they did most of their teaching in either in; the percentage of students receiving free
or reduced lunch in their districts or schools, and the percentage of teacher attrition over the last
three years. It may be inferred that in short, the more current STEM teachers perceived their
teaching environment as high need the more they perceived the Noyce funding as having
influenced them to become teachers.


Personal beliefs toward teaching:

Factor 6: Personal beliefs towards teaching was a positively significant predictor for outcome
variable 4 for the current STEM teacher group. This result indicated that the more current STEM
teachers’ personal beliefs toward teaching increased, the higher they perceived the effect of
Noyce funding on their decisions to teach in high need schools. These personal beliefs include I
am satisfied with my current teaching job, I would still become a teacher if I had to do it all over
again, I would still choose the same teacher preparation program into teaching if I had to do it all
over again, etc.. It may also be inferred that when these STEM teachers had more positive
perceptions of their levels of job satisfaction, opportunities of professional growth, and self-
efficacy towards teaching, their perceptions of the effect of the Noyce funding on their decision
to teach in high need schools was greater.


Other scholars’ characteristics and perceptions:

Three other predictors, gender, Factor 7: School teaching environment and Factor 8: Mentoring
experience, at the scholar level did not show a significant relationship with the outcome variables.
These results about gender confirmed the results from Final Report Section Five (Bowe, et al.,
2009) and the cluster analysis (Liou, Desjardins, & Lawrenz, in press). The results in Final
Report Section Five showed that gender was not significantly related to the continuous outcome
variables 1 and 3 although females’ scores were higher than males’. In addition, the results from
the cluster analysis (Liou, et al., in press) also supported the notion that gender is not an essential
variable related to scholars’ perceptions about the effects of the Noyce funding. Factor 7 and
Factor 8 did not reach significance for any outcome variable. However, according to the
correlation matrices (Tables 7 and 8); there were positive relationships between school teaching
environment and the outcome variables as well as mentoring experience and the outcome
variables.


Programs’ characteristics

Three predictors at the program level were analyzed in this report. Noyce money, Mean Factor 3:
Preparation for high need schools, and Mean Factor 8: Mentoring experience. All of the
program-level predictors showed statistically significant relationships with an outcome variable
in at least one of the two groups.
                                                 Noyce Evaluation Report, Section Four: HLM 32


Noyce funding:

Noyce funding was an important variable in explaining scholars’ perceptions of the effect of
Noyce funding on becoming a teacher and teaching in high need schools. Noyce funding was a
positively significant variable for outcome variables 1, 2, and 3 for the all scholar group, and also
a positively significant variable for outcome variable 3 for the current STEM teacher group.
These results indicated that the greater the percentage of scholars' tuition covered by the Noyce
funding, the more the scholars perceived that the Noyce program influenced their decision to
become teachers and to teach in high need schools. However, Noyce funding showed a
negatively significant relationship with outcome variable 4 for the current STEM teacher group.
This result suggested that that greater the percentage of scholars' tuition covered by the Noyce
funding at the program level, the less the Noyce funding influenced current STEM teachers'
decisions to teach in high need schools. This result is contrary to the other results. One
explanation is that more current STEM teachers chose “possibly” instead of “no/I have no taught
in a high need school” or “yes,” and choosing the “possibly” option caused a relatively uncertain
effect when analyzing the data. If “no/I have not taught in a high need school” and “possibly”
were combined into one option instead of the trichotomous data that were used, and the HLM
rerun, the results from the dichotomous data showed a positive relationship indicating that the
Noyce funding was more influential in the scholars’ decisions to teach in high need schools.
Therefore, based on the summary results from these four outcome variables, it can be concluded
that the amount of Noyce funding at the program-level had a positive relationship with the effect
of the Noyce funding on becoming teachers and teaching in high need schools.


Preparation for high need schools:

Mean Factor 3: Preparation for high need schools was a negatively significant predictor for
outcome variable 1 for the all scholar group. This indicates that as the mean score of the
perceptions of the scholars in a preparation program became higher the scholars’ perceptions of
the effect of the Noyce funding became lower. In other words, scholars in programs that they
perceived as having opportunities for them to learn about high need schools were less likely to
perceive the Noyce funding as influential in them becoming teachers. Recall that mean Factor 3
was produced by aggregating and averaging Factor 3: Preparation for high need schools scores
from individual scholars within a program. One explanation for a negative relationship might be
that there was more focused recruitment in programs that were specifically designed to prepare
teachers for high need schools. In other words, these types of programs were more likely to give
funding to scholars who already had high commitment to being teachers. Or perhaps, scholars
who were more influenced by the Noyce funding were more critical of the preparation for
teaching in high need schools provided by their programs. In contrast to this result, the
relationship between Factor 3 and outcome variable 1 for the all scholar group was positive at the
scholar level. One possible explanation is that there is a shift in the meaning of Factor 3 from the
individual level to the aggregated level; that aggregation by program changed the meaning that
should be ascribed to the Factor. This may imply that items which form Factor 3 at the scholar
level need to be validated for use at the program level.
                                                  Noyce Evaluation Report, Section Four: HLM 33


Mentoring experience:

Mean Factor 8: Mentoring experience is a positively significant predictor for outcome variable 3
for the current STEM teacher group. This result indicated that the mentoring experience had a
positive effect on scholars’ perceptions of the effect of Noyce funding on becoming teachers.
The content of Factor 8 included mentoring experiences provided by their certification program
and their district during their first and second year of teaching, a guaranteed job at a participating
school district(assuming successful completion of the program), and continuing contact with
participants in their teacher education program. Therefore, it can be inferred that when programs
and districts provided more mentoring on these current STEM teachers, their perceptions of the
influence of the Noyce funding for them to become teachers were higher.
                                                 Noyce Evaluation Report, Section Four: HLM 34



Limitations
There were several limitations of this report that require discussion. First, the data were from a
cross-sectional evaluation survey and it did not employ an experimental or quasi-experimental
design. Therefore, there is no evidence or basis to say causal relationships exist among the
scholars’ demographic characteristics, their perceptions, and their perceived effect of Noyce
funding for them to become teachers and teach in high need schools. The only credible
statements that can be made from the data are statements regarding correlations. Because this is a
cross-sectional study and not a longitudinal study, it is impossible to examine retention and
continued satisfaction with teaching or Noyce programs.

Second, race is an important variable explaining scholars’ perceptions about the influence of
Noyce funding on becoming teachers and becoming teachers in high need schools, but only
Whites and Non-whites were used in this study because of small within-group sample sizes. All
scholars identified as Non-white may not share convergent beliefs about teaching and it is likely
that with a finer grained lens, certain ethnic groups might perceive an effect of the scholarship on
their decision to enter teaching, to teach at high need schools, or both.

Third, the sample size of scholars in the current STEM teacher group is small. Statistical power
becomes an issue when analyzing a small number of programs. Although 36 programs existed in
the analysis, there were only 250 scholars from these 36, therefore it is possible that not enough
information was obtained. Another limitation of this study was that these predictors variables
(race, gender, and Factor 3-8 at the scholar level, and Noyce funding, mean factor 3 and mean
factor 8 at the program level) were unable to explain the variance, since only a small increase in
the amount of variance was accounted for difference between the unconditional models and
conditional models for the four outcome variables for both groups.

Lastly, the potential impact from non-response bias is another limitation for this report, because
responding to the survey was voluntary. Therefore, it may be that the scholars who responded
were different from those who did not respond in some systematic ways. If there were systematic
differences between respondents and nonrespondents, the data might be biased and
unrepresentative of the whole population. Therefore, the response bias is possible and could have
influenced the results.
                                               Noyce Evaluation Report, Section Four: HLM 35



References

Bowe, A., Liou, P.-Y., & Lawrenz, F. (2009). Final Report Section Five: Combined Analysis of
      the Robert Noyce Teacher Scholarship Program using ORC Macro and UMN Evaluation
      Data. University of Minnesota, Minneapolis.

Lawrenz, P., Appleton, J., Bullitt Bequette, M., Desjardins, C., Liou, P.-Y., Madsen, C., &
      Ooms, A. (2008). Final Report Section One: Planning and Survey Data Noyce Program
      Evaluation. University of Minnesota, Minneapolis.

Liou, P.-Y., & Lawrenz, P. (2008). Final Report Section Two: Factor Analysis of Robert Noyce
       Scholarship Program Evaluation. University of Minnesota, Minneapolis.

Liou, P.-Y., Desjardins, C., & Lawrenz, F. (in press). Demographics of STEM teacher
       scholarship recipients and their perceptions about teaching in high-needs schools via
       cluster analysis. School Science and Mathematics.

Liou, P.-Y., Kirchhoff, A., & Lawrenz, P. (in press). Perceived effects of scholarships on STEM
       majors’ commitment to teaching in high need schools. Journal of Science Teacher
       Education.

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and
      data analysis methods. Thousand Oaks, CA: Sage. 2nd edition.

Raudenbush, S. W., Bryk, A. S., & Congdon, R. (2005). HLM 6: Hierarchical linear and
      nonlinear modeling. Lincolnwood, IL: Scientific Software International, Inc.
                                                        Noyce Evaluation Report, Section Four: HLM 36



Appendix A: Items, factor loadings and sample sizes for factors from scholar
survey

Table A1

Items, Loading and Sample Size for “Commitment to Teaching in High Need Schools (Influence
of Noyce)” Factor (6 items)
                                Item                                  Factor      Sample
                                                                     loading        size
IV.8.c Take a teaching job                                             0.82         270
IV.8.d Teach in a high need school                                     0.80         270
IV.8.e Remain teaching in a high need school for the full term of      0.78         269
your commitment
IV.8.a Become a teacher                                                0.77         270
IV.8.f Remain teaching in a high need school beyond the full term of   0.75         269
your commitment
IV.8.b Complete the certification program                              0.68         270
Note. All items from scholar survey question IV.8: “How influential is the Noyce scholarship money in your
commitment to: _____?”

Table A2

Items, Factor Loadings and Sample Sizes for “Preparation for High Need Schools” Factor (13
items)
                                                                           Factor Sample
                                  Item
                                                                          loading   size
II.4.c Develop specific strategies for teaching students from diverse       0.66    273
                               a
racial and ethnic backgrounds
II.4.d Consider the relationship between education and social justice       0.63    273
                   a
and/or democracy
II.1.e Education about how to work in high need schools                     0.52    274
            b
specifically
II.4.a Develop specific strategies for teaching English language            0.50    273
learners (those with limited English proficiency)a
II.4.b Develop specific strategies for teaching students identified         0.49    273
with learning disabilitiesa
II.1.h Student teaching experience in a high need schoolb                   0.48    272
II.2.d Supervised actual classroom teaching in high need schools            0.45    263
                                                                      c
(this may be called student teaching, internship, etc. in your state)
II.1.f Opportunities to observe/work at high need schools (not              0.35    274
                  b
student teaching)
II.1.c Education about different culturesb                                  0.35    274
II.1.g Student teaching experienceb                                         0.34    274
                                                                        b
II.1.b Opportunities to interact with children from different cultures      0.32    274
II.2.a Education field experience working in schools (e.g. tutoring,        0.32    267
                                                          Noyce Evaluation Report, Section Four: HLM 37


teacher aide) with young people like those who attend high need
schoolsc
II.1.a Opportunities to interact with adults from different culturesb                      0.28              274
a
   Items from scholar survey question II.4: “Five types of experiences are listed below. Please indicate whether each
of the experiences below is a requirement of your teacher certification program. Indicate the length of any required
experiences”
b
  Items from scholar survey question II.1: “Which of these are part of your experience in your teacher certification
program?”
c
  Items from scholar survey question II.2: “In your teacher certification program, how much opportunity do you have
to do the following:_____?”

Table A3

Items, Factor Loadings and Sample Sizes for “Path to Teaching” Factor (7 items)
                                                                       Factor                             Sample
                                  Item
                                                                      loading                              size
IV.2 What age were you when you began the teacher certification         0.81                               270
program?__years
V.3 Did you work full time before the teacher certification program?    0.69                                 231
If yes, in what field was the majority of your work?
IV.1 At what point in your life did you decide to become a STEM         0.68                                 270
teacher?
V.4 In entering the teacher certification program, do you consider      0.68                                 267
yourself to have made a “career change”?
V.1.a How many upper level (junior/senior) or graduate level classes    0.30                                 171
have you taken in: STEM?
V.2.a In what year did you last take a formal course for college       -0.25                                 258
credit in: Mathematics
V.2.b In what year did you last take a formal course for college       -0.25                                 239
credit in: Science

Table A4

Items, Factor Loadings and Sample Sizes for “District/school high need environment” Factor (5
items)
                                                                         Factor     Sample
                                Item
                                                                        loading       size
III.4.b Over 33% of teachers lack sufficient training in their            0.79        161
academic area.(district)a
III.3.b Over 33% of teachers lack sufficient training in their            0.67        161
                        b
academic area.(school)
III.3.a Over 50% of students receive free or reduced lunch.(school) b     0.44        163
                                                                      a
III.4.a Over 50% of students receive free or reduced lunch.(district)     0.44        162
III.1 Which of the following describes your current teaching status?      0.36        158
a
 Items from scholar survey question III.4: “Which of the following characterize your school? For each option,
indicate whether the description applies or not, or whether you are not sure”
b
  Items from scholar survey question III.3: “Which of the following characterize your district? For each option,
indicate whether the description applies or not, or whether you are not sure”
                                                       Noyce Evaluation Report, Section Four: HLM 38



Table A5

Items, Factor Loadings and Sample Sizes for “Personal beliefs towards teaching” Factor (9
items)
                                                                        Factor      Sample
                                    Item
                                                                       loading        size
                                                      a
III.2.a I am satisfied with my current teaching job                      0.67         157
III.2.c If I had to do it all over again, in view of my present          0.65         164
knowledge, I would become a teachera
III.2.d If I had it to do all over again, I would choose the same        0.58         164
teacher preparation program and/or route into teachinga
III.2.e In the next three years, I am likely to assume a leadership      0.35         158
position (e.g., lead teacher, department chair, official or unofficial
mentor)a
IV.3.f I feel that I have a talent for teaching STEMb                    0.32         263
                                           a
III.2.b I really dislike STEM teaching                                  -0.32         160
III.5 Within the last three years have you held any professional         0.28         157
educational leadership positions, e.g., lead mathematics teacher,
science committee chair, etc.
IV.3.d I like the flexibility and/or autonomy of STEM teachingb          0.26         259
                                             b
IV.3.b I like working with young people                                  0.23         268
a
 Items from scholar survey question III.2: “How much do you agree or disagree with each of the following
statements about teaching?”
b
  Items from scholar survey question IV.3: “Did any of the following help you decide to become a STEM teacher?”

Table A6

Items, Factor Loadings and Sample Sizes for “School teaching environment" (4 items)
                                                                           Factor   Sample
                                   Item
                                                                          loading    size
III.6.b Strong collaborative leadership (e.g., principals and other         0.77     116
leaders provide teachers with opportunities to do well; principals and
other leaders can be trusted; principals and other leaders share your
vision of successful classroom practice)
III.6.a Collegial relationships (e.g., teachers consult on the quality of   0.71     115
student work and make joint decisions based on assessment,
collaborate to solve classroom challenges, observe and discuss each
others’ instruction)
III.6.d Mentoring and/or induction support (e.g., organized,                0.66     116
supported contact with a more experienced teacher, help with issues
particular to early career teaching)
III.6.c Availability of supplies or material (e.g., textbooks, print        0.48     115
resources, instructional materials such as lab supplies or math
manipulatives, and classroom supplies such as paper, pencils, or
tape)
                                                         Noyce Evaluation Report, Section Four: HLM 39

Note. All items from scholar survey question III.6: “Please rate your school environment as high, medium, or low on
the following features”

Table A7

Items, Factor Loadings and Sample Sizes for “Mentoring Experience” Factor (6 items)
                                                                       Factor       Sample
                                   Item
                                                                      loading        size
II.1.l Mentoring experiences provided by your certification program     0.58         164
during your second year of teaching
II.1.m Mentoring experiences provided by your district during your      0.55         264
second year of teaching
II.1.j Mentoring experiences provided by your certification program     0.51         272
during your first year of teaching
II.1.k Mentoring experiences provided by your district during your      0.49         271
first year of teaching
II.1.i A guaranteed job (assuming successful completion of program)     0.37         271
at a participating school district
II.1.n Continuing contact with participants in your teacher education   0.30         270
program
Note. All items from scholar survey question II.1: “Which of these are part of your experience in your teacher
certification program?”
                                                 Noyce Evaluation Report, Section Four: HLM 40



Appendix B: Groups of scholars within the eight factors

The scholars were grouped into 7 groups based on their current teaching status. These groups
were given different forms of the scholar survey. Because of this, not all groups were included in
all factors. This appendix details which groups of scholars were included in each factor.

Table B1

Groups of scholars based on current educational/teaching status
 Group                                       Teaching status
  No.
   1.       In a teacher certification program, not yet a full-time teacher
   2.       Completed a teacher certification program, but never taught
   3.       Did not complete a teacher certification program and will not return
   4.       In a teacher certification program and teaching full-time as part of that
            program
   5.       Teacher full-time or part-time
   6.       Taught after being certified and now working in education but not as a
            teacher
   7.       Taught after being certified and now not working in education

Table B2

Groups of scholars included in each factor
              Factor 1: Influence of scholarship on STEM majors becoming teachers
                                   (the total number of items=3)
                                 Item                                         Groups
IV.8.a become a teacher                                                     All groups
IV.8.b complete the certification program                                   All groups
IV.8.c take a teaching job                                                  All groups

         Factor 2: Influence of scholarship on STEM majors becoming high need teachers
                                  Item                                       Groups
IV.8.d teach in a high need school                                          All groups
IV.8.e remain teaching in a high need school for the full term of           All groups
your commitment
IV.8.f remain teaching in a high need school beyond the full term of        All groups
your commitment

            Factor 3: Preparation for high need schools (the total number of items=13)
                                  Item                                            Groups
II.4.c Develop specific strategies for teaching students from diverse           All groups
racial and ethnic backgrounds
II.4.d Consider the relationship between education and social justice           All groups
and/or democracy
                                                 Noyce Evaluation Report, Section Four: HLM 41


II.1.e Education about how to work in high need schools specifically             All groups
II.4.a Develop specific strategies for teaching English language                 All groups
learners (those with limited English proficiency)
II.4.b Develop specific strategies for teaching students identified              All groups
with learning disabilities
II.1.h Student teaching experience in a high need school                         All groups
II.2.d Supervised actual classroom teaching in high need schools                 All groups
(this may be called student teaching, internship, etc. in your state)
II.1.f Opportunities to observe/work at high need schools (not                   All groups
student teaching)
II.1.c Education about different cultures                                        All groups
II.1.g Student teaching experience                                               All groups
II.1.b Opportunities to interact with children from different cultures           All groups
II.2.a Education field experience (e.g. tutoring, teacher aide)                  All groups
working in schools with young people like those who attend high
need schools
II.1.a Opportunities to interact with adults from different cultures             All groups

                     Factor 4: Path to teaching (the total number of items=7)
                                 Item                                             Groups
IV.2 What age were you when you began the teacher certification                  All groups
program?__years
V.3 Did you work full time before becoming a teacher? If yes, in                 All groups
what field was the majority of your work? If yes, in what field was
the majority of your work?
IV.1 At what point in your life did you decide to become a STEM                  All groups
teacher?
V.4 In becoming a teacher, do you consider yourself to have made a               All groups
“career changer”?
V.1.a How many STEM classes were taken?                                          All groups
V.2.a In what year did you last take a formal course for college                 All groups
credit in: Mathematics
V.2.b In what year did you last take a formal course for college                 All groups
credit in: Science

          Factor 5: District/school high need environment (the total number of items=5)
                                 Item                                             Groups
III.4.b Over 33% of teachers lack sufficient training in their                   4, 5, 6, 7
academic area.(district)
III.3.b Over 33% of teachers lack sufficient training in their                   4, 5, 6, 7
academic area.(school)
III.3.a Over 50% of students receive free or reduced lunch.(school)              4, 5, 6, 7
III.4.a Over 50% of students receive free or reduced lunch.(district)            4, 5, 6, 7
III.1 Which of the following describes your current teaching status?                4, 5

            Factor 6: Personal beliefs towards teaching (the total number of items=9)
                                                  Noyce Evaluation Report, Section Four: HLM 42


                                    Item                                         Groups
III.2.a I am satisfied with my current teaching job.                            4, 5, 6, 7
III.2.c If I had to do it all over again, in view of my present                 4, 5, 6, 7
knowledge, I would become a teacher
III.2.d If I had it to do all over again, I would choose the same               4, 5, 6, 7
teacher preparation program and/or route into teaching
III.2.e In the next three years, I am likely to assume a leadership               4, 5
position (e.g., lead teacher, department chair, official or unofficial
mentor)
IV.3.f I feel that I have a talent for teaching STEM                           All groups
III.2.b I dislike STEM teaching                                                 4, 5, 6, 7
III.5 Within the last three years have you held any professional                   4, 5
educational leadership positions, e.g., lead mathematics teacher,
science committee chair, etc.
IV.3.d I like the flexibility and/or autonomy of STEM teaching.                All groups
IV.3.b I like working with young people                                        All groups

               Factor 7: School teaching environment (the total number of items=4)
                                   Item                                         Groups
III.6.b Strong collaborative leadership (e.g., principals and other            4, 5, 6, 7
leaders provide teachers with opportunities to do well; principals and
other leaders can be trusted; principals and other leaders share your
vision of successful classroom practice)
III.6.a Collegial relationships (e.g., teachers consult on the quality of      4, 5, 6, 7
student work and make joint decisions based on assessment,
collaborate to solve classroom challenges, observe and discuss each
others
III.6.d Mentoring and/or induction support (e.g., organized,                   4, 5, 6, 7
supported contact with a more experienced teacher, help with issues
particular to early career teaching)
III.6.c Availability of supplies or material (e.g., textbooks, print           4, 5, 6, 7
resources, instructional materials such as lab supplies or math
manipulatives, and classroom supplies such as paper, pencils, or
tape)

                    Factor 8: Mentoring experience (the total number of items=6)
                                 Item                                            Groups
II.1.l Mentoring experiences provided by your certification program             All groups
during your second year of teaching
II.1.m Mentoring experiences provided by your district during your              All groups
second year of teaching
II.1.j Mentoring experiences provided by your certification program             All groups
during your first year of teaching
II.1.k Mentoring experiences provided by your district during your              All groups
first year of teaching
II.1.i A guaranteed job (assuming successful completion of program)             All groups
                                                Noyce Evaluation Report, Section Four: HLM 43


at a participating school district
II.1.n Continuing contact with participants in your teacher education       All groups
program
                                           Noyce Evaluation Report, Section Four: HLM 44



Appendix C: All institutions

                                     Number of Scholars Who       Number of PIs Who
           Institution
                                          Responded                  Responded
        Auburn Universitya                      2                        1
   Baylor College of Medicine                  12                        1
      BOE City of St. Louisa                    1                        1
            Boise Statea                        1                        1
         Brownsville ISD                        6                        1
  California Polytechnic State           No responders                   1
         Foundation, Inc.
 California Polytechnic, Pomona                 8                          1
California State University Fresno             28                          1
   California State University-           No responders                    1
       Fullerton Foundation
California State University-Long               12                          1
        Beach Foundation
 California State University-San          No responders                    1
      Bernardino Foundation
 California State University San               9                           1
              Marcos
 Claremont Graduate University                  7                          1
     Clark Atlanta University             No responders                    1
        Cornell University                      9                          1
         Dowling College                       14                          1
        Drexel University                       5                          1
         Duke University                        6                          1
Foundation @ NJIT, New Jersey                   1                          1
     Institute of Technologya
Georgia State University Research              8                           1
         Foundation, Inc.
        Harvard University                     7                           1
        Howard University                      6                           1
     Indiana State University                  4                           1
        Indiana University                     6                           2
         Kean University                       6                           1
Kentucky Science & Technology                                              1
                                               6
              Council
 Lake City Community Collegea                  3                           1
  Louisiana State University &                                             1
  Agricultural and Mechanical                  15
              College
Maine Math and Science Alliance                29                          1
   Michigan State Universityb                  22                     No response
                                        Noyce Evaluation Report, Section Four: HLM 45


 Our Lady of the Lake University            11                          1
    PA State System of Higher                                           1
                                       No responders
             Education
     Portland State University         No responders                    1
     Saint Ambrose University                7                          1
    San Diego State Universityb             11                     No response
     San Jose State University              15                          1
     Seattle Pacific University        No responders                    1
         SUNY at Buffalo                    10                          1
         SUNY at Fredonia              No responders                    1
       SUNY at Stony Brook                   6                          1
Texas A&M Univ, College Station             22                          1
Texas A&M University, Kingsville            16                          1
   Texas A&M University, Main                                           1
                                       No responders
              Campus
Texas A&M University, Texarkana             14                          1
         Trinity University                 10                          1
       University of Arizona                 5                          1
University of Arkansas, Little Rocka         4                          1
    University of California Los                                        1
                                            17
              Angeles
   University of Central Floridaa           4                           1
   University of Cincinnati Main                                        1
                                            9
              Campus
 University of Colorado at Boulder          10                          1
  University of Illinois at Chicago         36                          1
        University of Kansas           No responders                    1
    University of Massachusetts                                         1
                                            8
              Amherst
    University of Massachusetts                                         1
                                            2
              Bostona
University of Massachusetts Lowell                                      1
                                            2
       Research Foundationa
      University of Minnesota               25                          1
 University of Missouri-Columbia            11                          1
    University of New Mexico                 5                          1
   University of North Caroline                                         1
                                            4
             Pembrokea
     University of North Texas              13                          1
University of Southern Mississippi     No responders                    1
   University of Texas at Austin            21                          1
   University of Texas at El Paso      No responders                    1
University of Texas at San Antonio           6                          1
   Washington State University         No responders                    1
      Wayne State University                18                          1
                                                         Noyce Evaluation Report, Section Four: HLM 46


    The University Corporation,                                                                     1
                                                       No responders
           Northridge
a
  These institutions were omitted from the analyses due to low numbers of scholars within the programs which
violated assumptions of multi-level regression analyses
b
  Scholars from these institutions were omitted from the analyses as the PIs did not provide program-level data from
the PI survey
                                               Noyce Evaluation Report, Section Four: HLM 47



Appendix D: U of MN Noyce evaluation team comprehensive evaluation
report sections

Lawrenz, F., Appleton, J., Bullitt Bequette, M., Desjardins, C., Liou, P.-Y., Madsen, C., &
Ooms, A. (2008). University of Minnesota Evaluation of the Robert Noyce Teacher Scholarship
Program, Final Report Section One: Planning and survey data.

Liou, P.-Y. and Lawrenz, F. (2008). University of Minnesota Evaluation of the Robert Noyce
Teacher Scholarship Program, Final Report Section Two: Factor analysis of the evaluation
questionnaire.

Kirchhoff, A. and Lawrenz, F. (2008). University of Minnesota Evaluation of the Robert Noyce
Teacher Scholarship Program, Final Report Section Three: District representative perceptions of
and experiences with the Robert Noyce Teacher Scholarship Program.

Liou, P.-Y. and Lawrenz, F. (2009). University of Minnesota Evaluation of the Robert Noyce
Teacher Scholarship Program, Final Report Section Four: The influence of scholar and program
level variables on scholar perceptions of the effect of the Noyce funding.

Bowe, A., Liou, P.-Y. and Lawrenz, F. (2009). University of Minnesota Evaluation of the Robert
Noyce Teacher Scholarship Program, Final Report Section Five: Combined analysis of the
Robert Noyce Teacher Scholarship Program using ORC Macro and UMN evaluation data.

Kirchhoff, A. and Lawrenz, F. (2009). University of Minnesota Evaluation of the Robert Noyce
Teacher Scholarship Program, Final Report Section Six: A model of the pathway to retention in
high need settings: Analysis of Noyce scholar interviews.

								
To top